____ ______| | | | __ __ _ | | / /___ __/ /_ ___ _____ (_)___ | | / __/ / / / __ \/ _ \/ ___/ / / __ \ | | / /_/ /_/ / /_/ / __(__ ) / / /_/ / | | \__/\__,_/_.___/\___/____(_)_/\____/ ______| | |____|
The tubes.io library for node is a collection of clients and utilities for interacting with tubes.io services. The library consists of two parts:
The CLI is used to manage scripts and deployments while the library is meant to be included in your node or coffee scripts to provide access to essential tubes.io services such as the proxy servers and other commonly used tools.
curl http://npmjs.org/install.sh | sh
[sudo] npm install -g tubesio
Command Line Interface
The tubes.io CLI simplifies the process of environment setup and subsequent management of scripts deployed on tubes.io. This is what you'll want to use if you plan on developing scripts using your own IDE and storing scripts in some kind of version control instead of the web-based interface.
Create a working directory where your scripts will live:
$ mkdir my_username && cd my_username
$ tubesio init
This command will prompt you for your tubes.io username and API key. Your API key can be found on the edit profile page once logged in over at tubes.io.
This will download both the necessary and available libraries for your script once deployed. It will also create a .tubesio file in the current working directory containing your environment configuration.
The current feature set is quite limited. Only automated deployments are supported which means you will need to copy/paste existing scripts from tubes.io to your local environment. Creating new tubes must also be done from the tubes.io website.
By convention scripts should be named
<slug>.js where slug
is the slugified version of your tube's name. For example if your
tube's name is My 1st Tube then your script name should be
my-1st-tube.js. This generally means replacing spaces and
special charachters with dashes. If you're unsure of what your
tube's slug name is you should take a look at your tube's endpoint.
The slug will be the portion after the last "/" and before the format
".json" (e.g. http://tubes.io/username/tube/my-1st-tube.json)
Once you've copied your hub down into your local work directory you can deploy by running:
$ tubesio deploy my-1st-tube
A word of warning: The change will take effect immediately. In case you need to rollback, a backup of the previous script is created with the .bak extension in your working directory. This can be deployed using the
-fswitch to specify an alternate file to use instead of the default. For example:
$ tubes.io deploy -f my-1st-tube.js.bak my-1st-tube
You can test the deployed script by running:
curl <endpoint URL>
Note: be sure to include your API key in the querystring and any other args your tube requires.
Usage of the tubesio node library is probably best demonstrated by example. Here's a simple node script that performs a HTTP request, parses the resulting HTML and extracts the page title:
// Required importvar tubesio = '<username>' '<apiKey>';// Optional importsvar cheerio = ;// Shortcutsrequest = tubesiohttprequestargs = tubesioutilsargslast = tubesioutilslastResult;;
The first line imports the tubesio lib. Notice the two arguments passed to the call to require. This is required for authentication against tubesio services such as the HTTP proxies.
Next we import cheerio which is a nice, lightweight DOM parsing and normalizing library with support for jQuery like selectors and syntax. We also create a shortcut reference to the tubesio request method.
After that we set up some additional shortcuts to the request method for brevity, the tube arguments (passed as either GET or POST parameters) and the result of the last tube run (this will be populated if you turn on "Cache Last Result" in your tube settings).
Calling the request method we pass the URL and attach a callback to the
property of the request settings object. The callback is passed the body of
the resulting HTTP response. We then load the body into cheerio and extract the
title text out.
Lastly a call to the finish completes the request. It's
important that all code paths eventually call
finish. Neglecting to do so
will cause your script to timeout as the node runtime won't know when
your script is finished. The
finish method is also important because the
data you pass to it is what gets returned from the API.
finishonly accepts objects that are JSON serializable (i.e. passing a naked number or string won't work).
Exits the current script.
err argument if defined and an instance of
Error will cause the script
to exit with a non-zero exit code and will print the error message to stderr. The
exit code defaults to 1 but may be overridden by setting the exitCode property of
the err object.
data you wish to output. The result will be passed through JSON.stringify and
printed to stdout before exiting with exit code 0.
A vastly simplified and elegant HTTP request method. Supports smart switching between HTTP/HTTPS based on URL, automatic GZIP and DEFLATE decompression, object serialization and proxy servers.
Important: It's strongly advised that you use this function over node's raw http.request or other third party libraries as it automatically proxies requests via our IP rotation service. We will not provide support for people wishing to use other methods.
location argument should be a string containing a URL or an object
location property set to the URL being requested.
settings argument can be either a callback function or an object
complete property set to the callback function to call
when the request completes.
- complete: a callback function that takes the arugments err, body.
- cookieJar: an instance of CookieJar for cookie support. Defaults to null.
- data: a string or object. Strings are sent verbatim whilst objects are serialized
according to the content-type set in the headers. Method will also be set to
POSTwhere method is not explicitly set. Currently detects
application/json. Where no content-type header exists defaults to form encoding.
- headers: an object containing key-value pairs of headers to send with request.
- method: the HTTP method to perform the request with. Defaults to
Sets the global proxy configuration object. This will impact all requests during the lifetime of the script. Under normal circumstances calling setProxy is not required. Importing the tubesio library automatically configures the correct proxy.
config argument can be a string, object or function. Strings are processed using
method. When passing a function it should produce either a string or configuration object.
Configuration objects should conform to the following structure:
host: 'hostname'port: 80proxyAuth: 'user:pass'
Note: port and proxyAuth are optional. Defaults to port 80 and no authentication.
CookieJar provides cookie support and persistance across HTTP requests. It behaves like a browser for the most part, simply accepting cookies and storing their values as well as passing cookie values with subsequent requests.
As a rule you shouldn't need to use the CookieJar class directly. Creating a CookieJar instance and passing it with your requests is all that's required.
var cookieJar = tubesiohttp;;
Note: you should create the CookieJar instance in the global scope in order to persist and reuse it across requests.
A logger that logs exclusively to stderr. Includes timestamps in log output and supports log level filtering.
Utility functions and helpful properties.
An object containing your GET or POST parameters.
An object containing the result of the last successful run of your tube. One of it's uses is for diffing against the current result set to see if anything has changed.
This property will be null unless "Cache Last Result" is turned on within the tube meta/settings.