Over my past couple years in the industry, there have been several times where I need to scrape structured information from (relatively) unstructured XHTML websites.
My approach to doing this has gradually evolved to include the following technologies:
{"headline":"Obama Tries to Turn Focus to Medicare From Jobs Figures","firstParagraph":"SEMINOLE, Fla. — President Obama on Saturday began hammering away at the Republican ticket’s plans for Medicare, using a campaign swing through Florida, with its large number of retired and elderly voters, to try to turn the page from anemic employment growth, his biggest weakness, to entitlements, a Democratic strength."}
An Optional Closure can be Provided for Processing the Value
A closure can optionally be provided as the third parameter for the set() method.
If a closure is given, the return value of the closure will be set as a key's value, rather than the text value of the selector.
{"headline":"Obama Tries to Turn Focus to Medicare From Jobs Figures","firstParagraph":"SEMINOLE, Fla. — President Obama on Saturday began hammering away at the Republican ticket’s plans for Medicare, using a campaign swing through Florida, with its large number of retired and elderly voters, to try to turn the page from anemic employment growth, his biggest weakness, to entitlements, a Democratic strength.","image":"http://graphics8.nytimes.com/images/2012/09/09/us/JP-CANDIDATE-1/JP-CANDIDATE-1-articleLarge.jpg"}
The closure will be passed the following values:
element: a jQuery element matching the CSS selector specified in set().
prev: if multiple elements on the page match the selector, the closure is will be executed once for each. prev can be used to interact with the object created by previous executions of the closure. As an example, we might want to increment a counter if the same link occurs multiple times on the same page.
this: the state is shared between multiple executions of the same closure (see examples/wikipedia.js, to get an idea of why this is useful).
Closure Return Types
strings: the last string returned by the closure will be used as the value.
numbers: the last number returned by the closure will be used as the value.
arrays: when an array is returned, it will be merged with all other arrays returned for the given key. The final merged array will be set as value.
objects: when an object is returned, the object will be merged with all other objects returned. The final object will be used as the value.
key/object-pair: this special return type allows value to be populated with an object that has dynamically generated key names.
{"paragraphs":["SEMINOLE, Fla. — President Obama on Saturday began hammering away at the Republican ticket’s...","Kicking off a two-day bus tour through...",...]}
{"links":{"#cite_note-MSW3_Lupus-1":{"title":"","href":"#cite_note-MSW3_Lupus-1","occurrences":1},"#cite_note-ADW-2":{"title":"","href":"#cite_note-ADW-2","occurrences":1},"/wiki/Gray_wolf_subspecies":{"title":"Gray wolf subspecies","href":"/wiki/Gray_wolf_subspecies","occurrences":1},"/wiki/Gray_wolf":{"title":"Gray wolf","href":"/wiki/Gray_wolf","occurrences":1},"/wiki/Canidae":{"title":"Canidae","href":"/wiki/Canidae","occurrences":1}}}
That's About It
I'm excited about jDistiller, I think it solves the scraping problem in an elegant way.
Don't be shy with your feedback, and please contribute.