Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
published 1.1.9 4 years agoMongoDB queue implementation for simplecrawler
published 1.0.1 4 years ago
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
MongoDB queue implementation for simplecrawler