Aniscrape
About
Aniscrape is an experimental scraping framework which supplies a 'provider' api to assist in scraping video & anime information from online anime sites.
Install
npm install aniscrape
Usage
Foreword
Please note that aniscrape is currently under heavy development and as a result the API is constantly changing and is by no means final. I don't claim to be an expert in module design so suggestions regarding class structure are welcome.
var Aniscrape = ;var animebam = ; // Check source on GitHub for more info. var scraper = ;scraper; scraper // You can also do searchAll to search through all providers.// When you call fetchSeries & fetchVideos, aniscrape will detect the provider automatically// and use the correct methods to retrieve it.scraper;
Providers
Aniscrape uses a modular design whereby you simply provide the scraping structure of your website (urls, html class names etc.) in the form of a search provider module, and aniscrape will use that when providing search results, episode lists and more.
You can see the current structure of search providers in the provider guide.
Todo
Provider API
There are still more options I need to include for use in the provider API, such as:
- Support sites that have episode lists on a different page to their anime page (allow promise based episode returns)
Overall features that need to be included
Some must have features for the base scraper in general:
- Ability to control more aspects of the web requests (instanced needle modules per provider) so that cookies & headers can be modified.
- A promise based intialise method for sites that have things that delay immediate scraping (CloudFlare & KissAnime are one example)
- Throttling and rate limiting requests. Currently requests are sent immediately, this is less than ideal especially if you want to bulk grab video URLs.
Contributing
The project is currently in its infancy so I don't really have any contributor rules. The source is written in CoffeeScript and I would prefer it remain that way. CoffeeScript lends itself well to the key value based structure of the provider API as it exists currently.
Open to all issues and pull requests, submit away.
License
MIT. See LICENSE.