Nuclear Pumpkin Mayhem

    urllib
    TypeScript icon, indicating that this package has built-in type declarations

    3.3.0 • Public • Published

    urllib

    NPM version Node.js CI Test coverage Known Vulnerabilities npm download

    Request HTTP URLs in a complex world — basic and digest authentication, redirections, cookies, timeout and more.

    Install

    npm install urllib --save

    Usage

    TypeScript and ESM

    import { request } from 'urllib';
    
    const { data, res } = await request('http://cnodejs.org/');
    // result: { data: Buffer, res: Response }
    console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);

    CommonJS

    const { request } = require('urllib');
    
    const { data, res } = await request('http://cnodejs.org/');
    // result: { data: Buffer, res: Response }
    console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);

    Global response event

    You should create a urllib instance first.

    import { HttpClient } from 'urllib';
    
    const httpclient = new HttpClient();
    httpclient.on('response', (info) => {
      error: err,
      ctx: args.ctx,
      req: {
        url: url,
        options: options,
        size: requestSize,
      },
      res: res
    });
    
    const { data, res } = await httpclient.request('https://nodejs.org');
    console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);

    API Doc

    Method: async request(url[, options])

    Arguments

    • url String | Object - The URL to request, either a String or a Object that return by url.parse.
    • options Object - Optional
      • method String - Request method, defaults to GET. Could be GET, POST, DELETE or PUT. Alias 'type'.
      • data Object - Data to be sent. Will be stringify automatically.
      • content String | Buffer - Manually set the content of payload. If set, data will be ignored.
      • stream stream.Readable - Stream to be pipe to the remote. If set, data and content will be ignored.
      • writeStream stream.Writable - A writable stream to be piped by the response stream. Responding data will be write to this stream and callback will be called with data set null after finished writing.
      • files {Array<ReadStream|Buffer|String> | Object | ReadStream | Buffer | String - The files will send with multipart/form-data format, base on formstream. If method not set, will use POST method by default.
      • contentType String - Type of request data. Could be json (Notes: not use application/json here). If it's json, will auto set Content-Type: application/json header.
      • dataType String - Type of response data. Could be text or json. If it's text, the callbacked data would be a String. If it's json, the data of callback would be a parsed JSON Object and will auto set Accept: application/json header. Default callbacked data would be a Buffer.
      • fixJSONCtlChars Boolean - Fix the control characters (U+0000 through U+001F) before JSON parse response. Default is false.
      • headers Object - Request headers.
      • timeout Number | Array - Request timeout in milliseconds for connecting phase and response receiving phase. Defaults to exports.TIMEOUT, both are 5s. You can use timeout: 5000 to tell urllib use same timeout on two phase or set them seperately such as timeout: [3000, 5000], which will set connecting timeout to 3s and response 5s.
      • auth String - username:password used in HTTP Basic Authorization.
      • digestAuth String - username:password used in HTTP Digest Authorization.
      • agent http.Agent - HTTP Agent object. Set false if you does not use agent.
      • httpsAgent https.Agent - HTTPS Agent object. Set false if you does not use agent.
      • followRedirect Boolean - follow HTTP 3xx responses as redirects. defaults to false.
      • maxRedirects Number - The maximum number of redirects to follow, defaults to 10.
      • formatRedirectUrl Function - Format the redirect url by your self. Default is url.resolve(from, to).
      • beforeRequest Function - Before request hook, you can change every thing here.
      • streaming Boolean - let you get the res object when request connected, default false. alias customResponse
      • gzip Boolean - Accept gzip, br response content and auto decode it, default is false.
      • timing Boolean - Enable timing or not, default is false.

    Options: options.data

    When making a request:

    await request('https://example.com', {
      method: 'GET',
      data: {
        'a': 'hello',
        'b': 'world',
      }
    });

    For GET request, data will be stringify to query string, e.g. http://example.com/?a=hello&b=world.

    For others like POST, PATCH or PUT request, in defaults, the data will be stringify into application/x-www-form-urlencoded format if content-type header is not set.

    If content-type is application/json, the data will be JSON.stringify to JSON data format.

    Options: options.content

    options.content is useful when you wish to construct the request body by yourself, for example making a content-type: application/json request.

    Notes that if you want to send a JSON body, you should stringify it yourself:

    await request('https://example.com', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json'
      },
      content: JSON.stringify({
        a: 'hello',
        b: 'world',
      })
    });

    It would make a HTTP request like:

    POST / HTTP/1.1
    host: example.com
    content-type: application/json
    
    {
      "a": "hello",
      "b": "world"
    }

    This exmaple can use options.data with application/json content type:

    await request('https://example.com', {
      method: 'POST',
      headers: {
        'content-type': 'application/json'
      },
      data: {
        a: 'hello',
        b: 'world',
      }
    });

    Options: options.files

    Upload a file with a hello field.

    await request('https://example.com/upload', {
      method: 'POST',
      files: __filename,
      data: {
        hello: 'hello urllib',
      },
    });

    Upload multi files with a hello field.

    await request('https://example.com/upload', {
      method: 'POST',
      files: [
        __filename,
        fs.createReadStream(__filename),
        Buffer.from('mock file content'),
      ],
      data: {
        hello: 'hello urllib with multi files',
      },
    });

    Custom file field name with uploadfile.

    await request('https://example.com/upload', {
      method: 'POST',
      files: {
        uploadfile: __filename,
      },
    });

    Options: options.stream

    Uploads a file with formstream:

    import formstream from 'formstream';
    
    const form = formstream();
    form.file('file', __filename);
    form.field('hello', '你好urllib');
    
    await request('https://example.com/upload', {
      method: 'POST',
      headers: form.headers(),
      stream: form,
    });

    Response Object

    Response is normal object, it contains:

    • status or statusCode: response status code.
      • -1 meaning some network error like ENOTFOUND
      • -2 meaning ConnectionTimeoutError
    • headers: response http headers, default is {}
    • size: response size
    • aborted: response was aborted or not
    • rt: total request and response time in ms.
    • timing: timing object if timing enable.
    • remoteAddress: http server ip address
    • remotePort: http server ip port
    • socketHandledRequests: socket already handled request count
    • socketHandledResponses: socket already handled response count

    Run test with debug log

    NODE_DEBUG=urllib:* npm test

    Mocking Request

    export from undici

    import { strict as assert } from 'assert';
    import { MockAgent, setGlobalDispatcher, request } from 'urllib';
    
    const mockAgent = new MockAgent();
    setGlobalDispatcher(mockAgent);
    
    const mockPool = mockAgent.get('http://localhost:7001');
    
    mockPool.intercept({
      path: '/foo',
      method: 'POST',
    }).reply(400, {
      message: 'mock 400 bad request',
    });
    
    const response = await request('http://localhost:7001/foo', {
      method: 'POST',
      dataType: 'json',
    });
    assert.equal(response.status, 400);
    assert.deepEqual(response.data, { message: 'mock 400 bad request' });

    Request through a http proxy

    export from undici

    import { ProxyAgent, request } from 'urllib';
    
    const proxyAgent = new ProxyAgent('http://my.proxy.com:8080');
    const response = await request('https://www.npmjs.com/package/urllib', {
      dispatcher: proxyAgent,
    });
    console.log(response.status, response.headers);

    Benchmarks

    Fork undici benchmarks script

    Connections 1

    Tests Samples Result Tolerance Difference with slowest
    http - no keepalive 15 6.38 req/sec ± 2.44 % -
    http - keepalive 10 6.77 req/sec ± 2.35 % + 6.13 %
    urllib2 - request 45 40.13 req/sec ± 2.88 % + 528.66 %
    urllib3 - request 10 58.51 req/sec ± 2.52 % + 816.64 %
    undici - pipeline 5 59.12 req/sec ± 2.47 % + 826.18 %
    undici - fetch 15 60.42 req/sec ± 3.00 % + 846.60 %
    undici - dispatch 5 60.58 req/sec ± 1.39 % + 848.99 %
    undici - stream 5 61.30 req/sec ± 1.31 % + 860.39 %
    undici - request 5 61.74 req/sec ± 2.03 % + 867.20 %

    Connections 50

    Tests Samples Result Tolerance Difference with slowest
    urllib2 - request 51 1465.40 req/sec ± 14.40 % -
    undici - fetch 40 3121.10 req/sec ± 2.82 % + 112.99 %
    http - no keepalive 45 3355.42 req/sec ± 2.84 % + 128.98 %
    http - keepalive 51 5179.55 req/sec ± 36.61 % + 253.46 %
    urllib3 - request 30 7045.86 req/sec ± 2.93 % + 380.82 %
    undici - pipeline 50 8306.92 req/sec ± 2.99 % + 466.87 %
    undici - request 51 9552.59 req/sec ± 13.13 % + 551.88 %
    undici - stream 45 12523.45 req/sec ± 2.97 % + 754.61 %
    undici - dispatch 51 12970.18 req/sec ± 3.15 % + 785.10 %

    Contributors


    fengmk2


    dead-horse


    xingrz


    popomore


    JacksonTian


    ibigbug


    greenkeeperio-bot


    atian25


    paambaati


    denghongcai


    XadillaX


    alsotang


    leoner


    hyj1991


    isayme


    killagu


    cyjake


    whxaxes


    chadxz


    adapt0


    danielwpz


    danielsss


    Jeff-Tian


    jedahan


    nick-ng


    rishavsharan


    willizm


    davidkhala


    aleafs


    Amunu


    azure-pipelines[bot]


    changzhiwin


    yuzhigang33


    fishbar


    gxcsoccer


    mars-coder


    rockdai


    dickeylth


    aladdin-add

    This project follows the git-contributor spec, auto updated at Fri Sep 09 2022 21:45:49 GMT+0800.

    License

    MIT

    Install

    npm i urllib

    DownloadsWeekly Downloads

    158,271

    Version

    3.3.0

    License

    MIT

    Unpacked Size

    254 kB

    Total Files

    65

    Last publish

    Collaborators

    • dead_horse
    • fengmk2