Newman's Personal Motorcade

    @elioway/spider

    1.2.5 • Public • Published

    spider

    Get your schemon! Tim Bushell

    Scrape Schema.org objects into mongoose schema files the elioWay.

    This is a requirement of bones but it can also be run as the boilerplate of a web spidering project with scheming intentions.

    Install

    npm install @elioWay/spider --save

    Usage

    // yourapp.js
    const yourAppSpider = require('@elioWay/spider');
    var today = new Date()
    // Create schemon the spider.
    let schemon = new yourAppSpider(
        version = today.getFullYear() + '.' + today.getMonth() + '.' + today.getDate(), // Do change.
        depth = 2, // The deeper you go, the more objects you get. Go crazy.
        thingsSelector = '#thing_tree', // Don't change - but there is a bigger tree on the page.
        useOjectFields = true // Instead of 1 to 1 relationships to other Things, force String type.
      )
    // Let schemon do spider things.
    schemon.spider(
      // Wrap what schemon scraped.
      data => Spider.optimize(
        data
      )
    )
    node yourapp

    Seeing is believing

    git clone https://gitlab.com/elioschemers/spider/
    cd spider
    node test_spider

    Credits

    License

    MIT Tim Bushell

    elioway

    Install

    npm i @elioway/spider

    DownloadsWeekly Downloads

    10

    Version

    1.2.5

    License

    MIT

    Unpacked Size

    76 kB

    Total Files

    48

    Last publish

    Collaborators

    • timitee