pop-api-scraper
Features
The pop-api-scraper project aims to provide the core modules for the
popcorn-api
scraper, but
can also be used for other purposes by using middleware.
- Strategy pattern with providers
- Cronjobs
- Scraper wrapper class
- HttpService with
got
Installation
$ npm install --save pop-api-scraper pop-api
Documentation
Usage
For the basic setup you need to create a Provider
(strategy) the
PopApiScraper
instance can use. The PopApiScraper
implements the strategy
pattern, where the providers are the strategies.
The example below makes a HTTP GET request to a web service or website. from there on you are free to implement how and what data you want to get from it.
// ./ExampleProvider.js // Extend from the internal AbstractProvider. { superPopApiScraper name configs maxWebRequests } // Override the `scrapeConfig` method to get the content from one // configuration. { // A HTTP service to send HTTP requests. thishttpService = baseUrl: configbaseUrl // HTTP GET request to: https://jsonplaceholder.typicode.com/posts?foo=bar return thishttpService }
Bundle it all up together with
pop-api
:
// ./index.js async { try // Let the PopApiScraper use the ExampleProvider o scrape data. PopApiScraper // Register the PopApiScraper middleware to the pop-api instance. PopApi // Optionally you can use the Cron middleware to scrape for content on a // regulat basis. PopApi // PopApi now has a `scraper` instance. const res = await PopApiscraper console catch err console }
License
MIT License