A layer on top of the request library to further abstract and simplify web requests
Request Enhanced is a Node.js library that functions as a layer on top of the request library to further abstract and simplify web requests. No worries about handling simple errors or retries, pooling requests, dealing with the complexity of writing to a file, or even manually searching the fetched content. Just fetch and done.
In addition to the awesome features that request offers, the following bonus features are available with request-enhanced
Greater File Saving Simplicity - No more worrying about odd edge cases or strange syntax when downloading to files. Directories are also automatically created along the specified file path as necessary.
Request Pooling - All requests are pooled to prevent EMFILE errors from too many simultaneous requests.
Request Prioritizing - Requests can be given a priority so that higher priority requests can execute first.
Easy Searching of Results - With regex queries, searching the resulting textual data becomes extremely simple.
Automatic Retries - When the connection fails with one of
ECONNREFUSED or when an HTTP 5xx error occurrs, the request will automatically be re-attempted as these are often recoverable errors and will go away on retry.
npm install request-enhanced
To perform a simple GET request, the following code can be used:
var re = ;re;
Don't want the result in memory and would rather pipe it to a file? No problem!
get function actually has a bunch of optional parameters to achieve additional functionality. A detailed description of each of the parameters is below.
get ( options, [filename or regex], [priority], callback )
The options parameter behavior is nearly identical to request's options paramter. It is possible to pass a string URL of the target or any of request's parameters in an options object with the following additional options:
maxAttempts- The maximum number of attempts to retry the request, defaults to
retryDelay- The delay in milliseconds before trying again after a recoverable failure, defaults to
defaultValue- The default value to assign to regex queries that find no data, defaults to
The following options already existing in request now have default values:
timeout- Now defaults to
pool- Now defaults to
The filename is an optional string representing a location on disk where the results of the GET request should be written to. Any directories in the path not currently existing will be created automatically. This option cannot be used at the same time as
To perform regular expression searches on the returned data, an optional regex query object can be provided. Regex queries are explained in the "Regex Queries" section below. This option cannot be used at the same time as
The priority is an optional number representing the request's priority. Requests with a lesser value will be performed prior to those with a higher valued priority. Requests with the same priority will be performed in a FIFO order. If not specified, a request is given the default priority of
The callback function will be called on success or error due to unrecoverable error or reaching the maximum retries. The following parameters will be passed back to this function in the following order:
error- This will either be an error object if there was an error, or
nullif there was not. The error object may have a
codeparamter for an HTTP status code if the error was HTTP related.
data- If a filename was specified in the call, a string filename will be returned of the newly saved file on disk. If a filename was not specified in the call, the string data returned from the GET request will be returned.
results- If regex queries were specified in the call and a filename was not, the results of the queries running on the
datawill be returned in this object
setDefaults function allows for setting of the default values across all requests. A detailed description of each of the parameter it takes is below.
setDefaults ( newDefaults )
The newDefaults object contains a mapping of keys with their defaults. Only the defaults that are specified will be overridden. All others will remain as they were previously and any additional non-existant defaults specified will be ignored.
Here is a list of the defaults that can be overridden:
defaults =maxAttempts: 10 // The default maximum number of retries before a request is deemed a failurepriority: 0 // The default priority a request is given if not specifiedtimeout: 10000 // The default delay in milliseconds before a request is automatically aborted and retriedretryDelay: 5000 // The default delay in milliseconds before a request is attempted again after a recoverable failuredefaultValue: '' // The default value to give a regex query if nothing is foundpool: // request's default pool parametermaxSockets: Infinity // request's max sockets is set to Infinity to allow request-enhanced to control poolingmaxConcurrent: 100 // The maximum number of concurrent requests;
Regex queries are a much simpler way to deal with searching in the returned content. When a regex query object is present in a call to the
get function, the fetched data will be automatically parsed.
Here is a very simple regex query:
query:regex: /My query: ""/iresults: 1
What this means is that the fetched data will be searched for the regex and then the first result of the match will be assigned to the
query key of the
results object returned.
As an example, if using the regex query above and the fetched data contained
my query: "hello world", the results object would look like this:
query: "hello world"
Instead of supplying a native RegExp object, it is also possible to supply a string as the regex.
query:regex: 'My query: "(.*?)"'results: 1
By default, only the case insensitive flag
i will be set automatically, but it is possible to set or disable any of the three flags.
query:regex: 'My query: "(.*?)"'results: 1caseSensitive: true // disables the i flagmultiple: true // enables the g flagmultiline: true // enables the m flag
The values of these parameters need not be the boolean
true, but anything that equates to true.
By default, a regex query will return only the first match, but it is possible to return the results of every match by setting the global flag
g in the RegExp object or by using the
multiple: true setting. Instead of returning the value of the result directly as the value of the key as in the basic example, the value will be an array of matches.
Take for example, the following regex query:
query:regex: /My query: ""/giresults: 1
If the fetched data contained both
my query: "hello world" and
my query: "goodbye world", the results object would look like this:
query: "hello world" "goodbye world"
Note: An array will always be returned for multiple matches queries. Simply use
array.length to determine the number of matches found.
It is also possible to return multiple results from a single query. Simply supply a mapping of result indexes to keys to store the resulting data on in place of the numerical index from the above examples. The results object will assign the matches to the keys and return the new mapping as the result.
This query will return multiple results:
query:regex: /My query: "", ~~/iresults:1: 'myquery'2: 'tilde'
If the fetched data contained
my query: "hello world", ~hello!~, the results object would look like this:
query:myquery: 'hello world'tilde: 'hello!'
Multiple results can be returned from multiple matches as well. By tweaking the example query above to add a global flag
g and also finding
my query: "goodbye world", ~goodbye!~ in the fetched data, the results object would be changed to this:
query:myquery: 'hello world'tilde: 'hello!'myquery: 'goodbye world'tilde: 'goodbye!'
Finally, it is also possible to do as many queries as desired on the fetched data. Simply supply more keys mapping to regex queries.
Use this type of query:
queryOne: ...queryTwo: ...queryThree: ...
to produce this type of results object:
queryOne: ...queryTwo: ...queryThree: ...
min the RegExp object, or use
multiline: truefor a string regular expression if this functionality is desired.
String.match()function. Requesting an index of 0 will return the entire matched string (e.g.
my query: "hello world").
results:declaration (or set to null). The
String.match()results as described in #2 will all be returned in an array. An empty array will be returned in cases where nothing is found.