GSpan.js
Transcribe and annotate C-Span closed captions in Google.
Why This?
Providing necessary context and checking the facts when politicians and other public figures speak is a critical part of our journalism at POLITICO. Doing them live comes with some hard technical challenges, especially when that context requires many fact-checkers and when we want to show our work alongside the original source.
This library aims to make that process easier by leveraging a platform that is already very good at solving the hard problems involved: Google docs. Using GSpan you’ll have an end-to-end system for acquiring transcripts of a live event, creating a collaborative editing environment in Google docs for fact-checkers to annotate that transcript and, finally, structuring all that data in a way it can be easily consumed by a front-end application or site.
How It Works
GSpan comes with two main features that are designed to work together: transcription and downloading/parsing.
It uses Opened Captions created by Dan Schultz as the source of the transcription. You can read more about the Opened Captions project on Source. These captions are written into a Google Doc using a special markup to signify things like speaker changes and soundbites. Then, reporters can use the native commenting functionality to annotate documents.
Once annotations are done you can convert that Google Doc (with all its annotations) into structured data using GSpan's download functionality. All the unique markup will be turned into an easy-to-use JSON format with rich text elements rendered as Markdown.
What you do with that data and how your page looks is up to you, but here's what we did.
Quick Start
Install the package globally to use the CLI
$ npm install -g gspan
Create a Google Doc. Share it with your Google service account's client email (see Authenticating). Get the Doc ID for that document. You can find it in the URL between d/ and /edit like this:
With an ID in hand you can begin transcribing and downloading with the CLI.
Transcribing from C-Span into a Google Doc
gspan transcribe <doc>
Begins transcribing CSPAN into a document.
Positionals:
doc The Google Doc ID [string] [required]
Options:
--verbose, -v Log new entries in the console [boolean]
--backfill, -b Start from a saved backup file [boolean]
--backupFile, -f A filepath to save a backup [string]
Downloading and parsing data from a Google Doc
gspan download <doc> [output]
Downloads a GSpan doc as a JSON file.
Positionals:
doc The Google Doc ID [string] [required]
output The directory to save the file to [string]
Options:
--authorAPI, -a A link to an authors API [string]
--authorNameAccessor, -n Accessor for Google display name [string]
--authorIdAccessor, -i Accessor for unique ID [string]
--defaultPublish, -p Default for annoation published state [boolean]
* Boolean arguments can be set to false by setting them equal to false or by adding the no- prefix to their name. (e.g. gspan transcribe DOC_ID --no-verbose or gspan download DOC_ID -p=false)
Authenticating
In order to use this app you'll need a Google Service Account which has write access to your Google Doc. For help with creating a Google Service Account, see Making A Google Service Account.
Once you have your credentials file, you'll need the client_email and private_key. You'll need to make sure that these are saved as environment variables in your runtime under the keys of GAPI_CLIENT_EMAIL and GAPI_PRIVATE_KEY respectively.
There's a number of ways to do this depending on your workflow. If you've never worked with environment variables in node check out this guide.
Finally, also make sure to share the Google Doc you're using with the client_email. It may look long and not like a valid email, but Google will allow it.
Demo
Our inaugural use of GSpan was for the 2019 State of the Union. Check out the page!
API Documentation
- Transcribing To A Doc: API reference for GSpan transcribe.
- Downloading A Doc: API reference for GSpan download.
- The Data Signature: API Reference for the output of GSpan download.
Guides
For Users
- Annotating Docs: A guide to making the annotations in a Google Doc.
- Advanced Annotating: A guide for handling some of the fringe cases.
For Developers
- Formatting Comments: A guide to formatting rich text annoations and the markdown output they produce.
- Making A Google Service Account: A guide to setting up a Google service account and getting your credentials.