Upload Google Storage to BigQuery
This is an npm package that provides functionality to upload files from Google Cloud Storage to BigQuery.
Installation
You can install the package using npm:
npm install upload-google-storage-to-bigquery
Usage
import BigQueryUploader from 'upload-google-storage-to-bigquery';
const uploader = new BigQueryUploader('path/to/keyfile.json');
uploader.uploadToGCS('path/to/file.csv', 'bucket-name')
.then(() => {
console.log('File uploaded to Google Cloud Storage successfully.');
})
.catch((error) => {
console.error('Failed to upload file to Google Cloud Storage:', error);
});
uploader.loadToBigQuery('dataset-id', 'table-id', 'bucket-name', 'path/to/file.csv')
.then(() => {
console.log('File loaded to BigQuery successfully.');
})
.catch((error) => {
console.error('Failed to load file to BigQuery:', error);
});
uploader.runQuery('SELECT * FROM `myDataset.myTable`')
.then((result) => {
console.log('Query result:', result);
})
.catch((error) => {
console.error('Failed to run query:', error);
});
Replace 'path/to/keyfile.json'
with the path to your Google Cloud service account key file. Make sure you have the necessary permissions to access Google Cloud Storage and BigQuery.
API
BigQueryUploader
The BigQueryUploader
class provides the following methods:
constructor(keyFilename: string)
Creates a new instance of the BigQueryUploader
class with the specified Google Cloud service account key file.
uploadToGCS(filePath: string, bucketName: string): Promise<void>
Uploads the specified file to the specified Google Cloud Storage bucket.
-
filePath
: The path to the file to upload. -
bucketName
: The name of the Google Cloud Storage bucket.
loadToBigQuery(datasetId: string, tableId: string, bucketName: string, filePath: string): Promise<void>
Loads the specified file from Google Cloud Storage to BigQuery.
-
datasetId
: The ID of the BigQuery dataset. -
tableId
: The ID of the BigQuery table. -
bucketName
: The name of the Google Cloud Storage bucket. -
filePath
: The path to the file in Google Cloud Storage.
runQuery(sql: string): Promise<any>
Runs the specified SQL query in BigQuery.
-
sql
: The SQL query to run.
Contributing
Contributions are welcome! If you find any issues or have suggestions for improvements, please feel free to open an issue or submit a pull request.
License
This package is MIT licensed.
Feel free to customize and modify the README file according to your package's specific features and requirements.