lambda-etl
This module is used along with aws lambda / Serverless Framework to perform ETL
How Does It Work?
This module provide an extendable abstract class, thus the child class will override some of the provided methods:
getImportHandler()
Which handle result data after parsing and processing line (persisting to database for example).
getLineProcessorCallback()
Which handle the logic how to process each line after parsing (adapting result to DTO, or transform data before passing to the handler).
## Getting Started
Installation
$ npm i lambda-etl
Example
Initialization
After Setting up aws credentials
const AbstractImportService = /** * @returns */ { return 'MyServerlessMsa-devStage-handlePersistRowToDb' } /** * @param * Initialize ETL process based on ImportTrack Model or whatever */ async { /** * Initialize a new Row with file information and start time */ return thisImportModel } /** * @returns */ { if extension === 'json' // create custom http event for json files for example return { return body: parsedChunkdata pathParameters: id: parsedChunkid } return { return body: parsedChunk } } /** * @param * Initialize ETL process based on ImportTrack Model or whatever */ async { /** * Update DB ETL track row and mark as finished and log errors number * if any */ return thisImportModel }
Usage
let importService = let bucketName = "uploads"let filname = "data.json"importService