Needs Perl Modules

    flight-scrappper

    2.3.3 • Public • Published

    flight-scrappper

    A Web scraper made with nodejs and selenium-webdriver that gathers flight data and stores it in a mongodb database.

    NPM Version Codacy Badge dependencies Status devDependencies Status MIT License

    let FlightScrappper = require('flight-scrappper');
     
    FlightScrappper.run().then((flights) => {
        console.log(flights);
    }).catch(function(err) {
        console.log('Found an error! ' + err);
    });

    Requirements

    Installation

    $ npm install --save flight-scrappper

    Options

    The following options can be defined as an argument of the FlightScrappper.run() method.

    This can be done passing an object {option1:'abc',option2:'abc',...}.

    If an option is not defined, a default value will be used instead.

    These are the default values:

    const defaultDateFormat = 'DD-MM-YYYY';
    let defaultOptions = {
        periods: 1,
        interval: 48,
        currency: 'EUR',
        directFlight: false,
        dateFormat: defaultDateFormat,
        targetDate: Utils.getDefaultDateString(defaultDateFormat),
        database: 'localhost:27017/flight-scrappper',
        collection: 'flight-data',
        timeout: 80000,
        browser: 'chrome',
        chromedriverArgs: [],
        maximize: false,
        retries: 1,
        routes: [{
            from: 'LIS',
            to: 'PAR'
        }]
    };

    These queried dates are calculated with the following formula targetDate + options.interval x options.periods times.

    Example: Setting periods to 2, interval to 24 and targetDate to 5/01/2000 will generate an array such as ['5/01/2000','07/01/2000'].

    Running

    First, start your mongodb database. You can try npm run mongo-linux/win/mac to start your database in an easy way, or do it manually. For more information on what these commands are doing, just read the scripts object in the package.json file.

    If you want to scrap flights, without storing data, you can set database to 'none'.

    To start the flight-scrappper with the default values just type $ npm start.

    If you want to run with different options just add arguments as specified in Options.

    If you want to get feedback in the console please check Debugging.

    Output

    FlightScrappper.run will return a promise which will resolve into the number of inserted documents or into an error.

    The output data that will look like this:

    "search" : {
        "from" : "LIS",
        "to" : "AKL",
        "source" : "momondo",
        "queried" : ISODate("2016-10-23T12:09:21.566Z")
    },
    "data" : {
        "duration" : 2080,
        "stops" : 2,
        "flightClass" : 0,
        "airline": ["TAP","Ryanair"],
        "price" : {
            "amount" : 778,
            "currency" : "EUR"
        },
        "departure" : {
            "time" : {
                "minute" : 15,
                "hour" : 14,
                "day" : 25,
                "month" : 10,
                "year" : 2016
            },
            "airport" : "LIS"
        },
        "arrival" : {
            "time" : {
                "minute" : 55,
                "hour" : 12,
                "day" : 27,
                "month" : 10,
                "year" : 2016
            },
            "airport" : "AKL"
        }
    }

    Tests

    To run the test suite, first install the dependencies, then run npm test:

    $ npm install
    $ npm test

    Debugging or Verbose

    $ npm run debug to have console output.

    Contributing

    Contributions, requests or pull requests are welcome & appreciated!

    Send your pull requests to the developing branch please!

    Send me an email if you have questions regarding possible contributions.

    License

    MIT

    Install

    npm i flight-scrappper

    DownloadsWeekly Downloads

    24

    Version

    2.3.3

    License

    MIT

    Last publish

    Collaborators

    • bertolo1988