Parse data fetched by docparse scrapers
Parse scraped documents
To start the parseScraped worker, execute
node parseScrapedWorker.js --config test/config.json
This will create a new Parser object which can receive remote requests to start parsing documents. See app.js for the construction of a new Parser object. Also See the parseScrapedWorker.js file in the project root for details
var ParseScrapedWorker = require'./index'var config = require'nconf'defaultsseaport:host: 'localhost'port: 4598var parser = config
The parseScrapedWorker object has a function parseScraped. This function should be called with a scraped document as the first parameter and a callback function as the second.
var inspect = require'eyespect'inspector;var config = require'nconf'defaultsseaport:host: 'localhost'port: 4598var parser = configvar scrapedDoc =supplierCode: 'HES'payload:billDate: '2011-02-23 00:00:00 +00:00'accountNumber: 'fooAccountNumber'billNumber: 'fooBillNumber'loginID: 'fooLoginID'supplierCode: 'HES'textPages: 'foo page 1' 'bar page 2' // these are the extracted text pages from the bill pdf fileparserparseScrapedscrapedDocif errinspecterr 'error parsing scraped document'returninspectreply 'parsed scraped document correctly'
A scraped document is parsed differently for each supplier. Therefore in the DocParse system, there need to parsing servers online for each supplier. When the ParseScrapedWorker is initiated, it binds to a request Axon socket for each supported supplier. See
lib/getRemoteParseSockets.js for details. It also registers a service with seaport so that the supplier response sockets know where to connect
The system currently supports HES (Hess), NST (NStar), NGE (NGrid Electric), and NGA (NGrid Gas). For HES, the ParseScrapedWorker would bind to a socket as follows
var config = require'nconf'defaultsseaport:host: 'localhost'port: 4598var seaConfig = configget'seaport'var seaHost = seaConfighostvar seaPort = seaConfigportvar ports = seaportconnectseaHost seaPortvar role = 'pushScrapedParseJobHES' // all push roles are in the format "pushScrapedParseJob<supplier code>"var port = portsregisterrole // register with seaport so the remote HES parsing server knows where to connectvar socket = axonsocket'req';socketformat'json'socketbindport
This binding happens for each supported supplier. The ParseScrapedWorker keeps a reference to each supplier-specific push socket in its self.sockets property. The sockets property is an object keyed by supplierCode When the ParseScrapedWorker.parseScraped method is called, it gets the appropriate supplier-specific socket and sends out a request to the remote parsing server. The supplier-specific socket is an Axon req socket.
Each instatiated ParseScrapedWorker object has a
parseScraped function. This function is called with an unparsed scraped document and a callback. The parseScraped function gets the appropriate supplier-specific request socket and sends a new parsing request out the the remote supplier-specific parsing server
var self = thisvar config = selfconfigvar supplierCode = docsupplierCodevar sockets = selfsocketsvar socket = socketssupplierCodesocketsendscrapedDoc cb
In the example above, the parseScrape function actually calls
lib/parseRemote function. The functionality is the same, but parseRemote adds a some timeout logic in case the remote parsing server goes down request fails for some reason