bill-scraper

BillScraper.js 📝📝📝

A NPM module to scrape and analyze bills in Congress. Works for bills from the 113th, 114th, and 115th congress. THIS PROJECT IS STILL UNDER DEVELOPMENT. DO NOT DOWNLOAD YET.

Table of Contents

• docs: Holds markdown files that explain the project in more detail.
• examples: Holds example code.
• example_outputs: Holds example outputs.
• lib: Holds all Javascript files
• misc: Hold all graphics for the README.md
• test: Holds all unit tests.

Why?

The government hosts bulk data for all congressional bills at gpo.gov/fdsys/bulkdata. However, this repository is horrible to use for developers, as bills are uploaded to unpredictable URLS as XML files. Each bill file is too structured, as XML style elements like paragraphs, bold text, and quotes are preserved, making it hard to parse the actual text of a bill.

What does this project do

• Returns full Congressional bill text and metadata from Congress.gov as JSON. This library can scrape both the actual bill text (bills) and meta information such as cosponsors, amendments, and committees (bill statuses). For example, the following code will convert the XML (on the top) into the JSON (on the bottom)

var billScraper = require("bill-scraper");

billScraper.getBill("BILLS-113hr1033rs", function(res){
  billScraper.printBill(res, function(parsedResult){
    console.log(JSON.stringify(parsedResult));
  });
}, {session: 2});

XML	JSON

For more detailed examples, visit examples/get_bills.js.

• Parses bills for key points and ideas: This project generates "tags" for bills, allowing them to more easily searched. For instance, searching for all bills tagged with "battlefield".

var billScraper = require("bill-scraper");

billScraper.getBill("BILLS-113hr1033rs", function(res){
 billScraper.printBill(res, function(parsedResult){
   billScraper.tagBill(parsedResult, function(tags){
      console.log(tags);
   });
 });
}, {session: 2});

//Will return ['battlefield', 'sites','war','section','land','subsection','period','funds','appropriation','protection','revolutionary','public','sub','term','nations','civil','preservation','service','sellers','prohibition','purposes','person','entity','matter','influence','law','ratification','policy'];

For more detailed examples, visit examples/tag_bills.js.

• Generates a searchable Mongo database of bills: For example, the following code will locally cache a database (see photo below) of parsed, searchable bills.

var billScraper = require("bill-scraper");
var mongoose = require("mongoose");

mongoose.connect("mongodb://localhost/bills");
billScraper.generateDB("hr", 115, 1);

For documentation on the Bill struct stored in the database, visit BILL_TEMPLATE.md in docs, for more detailed examples, visit examples/database.js, and for information on searching bills, see the examples in examples/search_bills.js.

Documentation

• To get started, visit GET_STARTED.md in docs.
• For examples of how to use the library, visit examples.
• Documentation for all the methods is available here.

Community

This repository encourages all users to contribute. Please read the Code of Conduct and Contributing.md. For things to contribute, check out the issues or TODO.md.

Datasets used

• GPO Bill bulk data

Relevant libraries used

• chai and mocha: To run unit tests
• jsdoc: To generate function-level documentation
• istanbul and codecov: To generate code coverage reports
• mongoose and mongo: For the database
• retext and retext-keywords: To automatically tag bills by subject
• htmlparser, request, soupselect, and xml2js for web scraping.

View the package.json for the full list of dependencies.

Contributors

• Jeffrey Shen: Initial development. Currently maintains the code.
• Amanda Li
• Malcolm Essaid
• Ryan Greenberg

View the full list of contributors here.

License

This repository is licensed as CC0. See the LICENSE for more information.

---

To the extent possible under law, Jeffrey Shen has waived all copyright and related or neighboring rights to BillScraper.js. This work is published from: United States.