Nicer Package Manager

    bill-scraper

    1.2.0 • Public • Published

    BillScraper.js 📝📝📝

    Build Status codecov npm version Inline docs

    A NPM module to scrape and analyze bills in Congress. Works for bills from the 113th, 114th, and 115th congress. THIS PROJECT IS STILL UNDER DEVELOPMENT. DO NOT DOWNLOAD YET.

    Table of Contents

    • docs: Holds markdown files that explain the project in more detail.
    • examples: Holds example code.
    • example_outputs: Holds example outputs.
    • lib: Holds all Javascript files
    • misc: Hold all graphics for the README.md
    • test: Holds all unit tests.

    Why?

    The government hosts bulk data for all congressional bills at gpo.gov/fdsys/bulkdata. However, this repository is horrible to use for developers, as bills are uploaded to unpredictable URLS as XML files. Each bill file is too structured, as XML style elements like paragraphs, bold text, and quotes are preserved, making it hard to parse the actual text of a bill.

    What does this project do

    • Returns full Congressional bill text and metadata from Congress.gov as JSON. This library can scrape both the actual bill text (bills) and meta information such as cosponsors, amendments, and committees (bill statuses). For example, the following code will convert the XML (on the top) into the JSON (on the bottom)
    var billScraper = require("bill-scraper");
    
    billScraper.getBill("BILLS-113hr1033rs", function(res){
      billScraper.printBill(res, function(parsedResult){
        console.log(JSON.stringify(parsedResult));
      });
    }, {session: 2});
    
    XML JSON
    XML JSON

    For more detailed examples, visit examples/get_bills.js.

    • Parses bills for key points and ideas: This project generates "tags" for bills, allowing them to more easily searched. For instance, searching for all bills tagged with "battlefield".
    var billScraper = require("bill-scraper");
    
    billScraper.getBill("BILLS-113hr1033rs", function(res){
     billScraper.printBill(res, function(parsedResult){
       billScraper.tagBill(parsedResult, function(tags){
          console.log(tags);
       });
     });
    }, {session: 2});
    
    //Will return ['battlefield', 'sites','war','section','land','subsection','period','funds','appropriation','protection','revolutionary','public','sub','term','nations','civil','preservation','service','sellers','prohibition','purposes','person','entity','matter','influence','law','ratification','policy'];
    

    For more detailed examples, visit examples/tag_bills.js.

    • Generates a searchable Mongo database of bills: For example, the following code will locally cache a database (see photo below) of parsed, searchable bills.
    var billScraper = require("bill-scraper");
    var mongoose = require("mongoose");
    
    mongoose.connect("mongodb://localhost/bills");
    billScraper.generateDB("hr", 115, 1);
    

    Database

    For documentation on the Bill struct stored in the database, visit BILL_TEMPLATE.md in docs, for more detailed examples, visit examples/database.js, and for information on searching bills, see the examples in examples/search_bills.js.

    Documentation

    • To get started, visit GET_STARTED.md in docs.
    • For examples of how to use the library, visit examples.
    • Documentation for all the methods is available here.

    Community

    This repository encourages all users to contribute. Please read the Code of Conduct and Contributing.md. For things to contribute, check out the issues or TODO.md.

    Datasets used

    Relevant libraries used

    View the package.json for the full list of dependencies.

    Contributors

    View the full list of contributors here.

    License

    This repository is licensed as CC0. See the LICENSE for more information.


    CC0
    To the extent possible under law, Jeffrey Shen has waived all copyright and related or neighboring rights to BillScraper.js. This work is published from: United States.

    Install

    npm i bill-scraper

    DownloadsWeekly Downloads

    2

    Version

    1.2.0

    License

    SEE LICENSE IN LICENSE

    Unpacked Size

    7.56 MB

    Total Files

    49

    Last publish

    Collaborators

    • jeffreyshen19