Share your code. npm Orgs help your team discover, share, and reuse code. Create a free org »

    cw-spiderpublic

    Build Status

    The best module ever.

    Getting Started

    Install the module with: npm install cw-spider

     
    {Spider} = require 'cw-spider'
     
     
    class XiaMiArtlistSpider extends Spider
      mapUrl: (id)->
        "http://www.xiami.com/artist/#{id}"
     
      parse: (id, $,result)->
        name = $('div#title>h1').contents().first().text()
        avatar = $('a#cover_lightbox>img').attr('src')
        console.log "id:#{id},name:#{name},photo:#{avatar}"
     
        {id: id, name: name}
     
      save: (data)->
        console.log data
     
    spider = new XiaMiArtlistSpider()
    spider.start [1...10000]
     

    Documentation

    运行环境 iojs 1.5.0 coffee 1.9.1

    构造函数默认参数

    options =
          co:
            limt: 2
          rate:
            rate: 60
            interval: 30
            backoffCode: 429
            backoffTime: 10
            maxWaitingTime: 300
          request:
            timeout: 5000
            maxSockets: 1
    
    co: co-limiter的配置
    rate:request-rate-limiter配置,请参考https://www.npmjs.com/package/request-rate-limiter
    request:request配置,请参考:
    
    

    Examples

    (Coming soon)

    Contributing

    In lieu of a formal styleguide, take care to maintain the existing coding style. Add unit tests for any new or changed functionality. Lint and test your code using Grunt.

    License

    Copyright (c) 2015
    Licensed under the MIT license.

    Keywords

    install

    npm i cw-spider

    Downloadsweekly downloads

    12

    version

    0.0.6

    license

    MIT

    repository

    githubgithub

    last publish

    collaborators

    • avatar