Nest Query Parser

A query string parser to be used in applications developed with NestJS.

NOTE: This library will no longer have updates. To use the same existing features, and take advantage of new features and eventual fixes, use the nest-mongo-query-parser library. For applications that use Typeorm, I'm developing a library called nest-typeorm-query-parser. Happy Coding! :)

Summary

• Prerequisites
• Installing
• Usage
• Examples
• Explain the Resources
  • Queries with @MongoQuery() | @MongoQueryParser()
    • Pagination
    • Ordering
    • Select
    • Filters
      • Simple Filters
      • Partial Filters
      • Comparison Filters
      • Element Filters
      • AND | OR Filters
    • Populate
• Rules
• Observations
• Practical Examples
• Upcoming Features
• License
• Authors

Prerequisites

As the name of the library suggests, this library was built to work together with the NestJS framework. However, as future work, another library will be implemented that can be used as middleware of APIs that use Express or HapiJS.

Installing

Use the follow command:

npm i --save nest-query-parser

Usage

There are two ways to use the parsers available in this library: as a ParamDecorator or as a MethodDecorator.

If you want to use it as a ParamDecorator, just add the tag referring to the Parser to be used as a method parameter. Example:

import { Get } from '@nestjs/common';
import { Controller } from '@nestjs/common';
import { ResourceService } from './resource.service';
import { MongoQuery, MongoQueryModel } from 'nest-query-parser';

@Controller('resources')
export class ResourceController {
  constructor(private readonly _service: ResourceService) {}

  @Get()
  public find(@MongoQuery() query: MongoQueryModel) {
    return this._service.find(query);
  }
}

It can also be used as a MethodDecorator. Just use the tag referring to the Parser to be used as the method decorator. Example:

import { Injectable } from '@nestjs/common';
import { MongoQueryParser, MongoQueryModel } from 'nest-query-parser';

@Injectable()
export class ResourceService {
  @MongoQueryParser()
  public find(query: MongoQueryModel) {
    return [];
  }
}

NOTE: When using the library as a MethodDecorator, you can receive other arguments in the method in question, but the query has to be passed as the first argument of the function, so that the treatment is done properly.

Examples

Request: http://localhost:3000/resources

Query:

{
  "limit": 100,
  "skip": 0,
  "select": {},
  "sort": {},
  "populate": [],
  "filter": {}
}

Request: http://localhost:3000/resources?limit=10&page=2&select=\_id,name,age&sort=-created_at&age=gt:30

Query:

{
  "limit": 10,
  "skip": 10,
  "select": {
    "_id": 1,
    "name": 1,
    "age": 1
  },
  "sort": {
    "created_at": -1
  },
  "populate": [],
  "filter": {
    "age": {
      "$gt": 30
    }
  }
}

Explain the Resources

Queries with @MongoQuery() | @MongoQueryParser()

Pagination

The paging feature is very useful for clients who will consume your API. It is through this feature that applications can define the data limit in a query, as well as define which page to be displayed. Each time a page of an application is selected, it means that some resources have been displaced (data offset or skip data).

There is a mathematical rule that relates page number to resource offset. Basically:

offset = (page - 1) * limit, where page > 0.

This means that for a limit of 10 elements per page:

• To access page 1, the offset will be equal to = (1 - 1) * 10, so offset = 0
• To access page 2, the offset will be equal to = (2 - 1) * 10, so offset = 10
• To access page 3, the offset will be equal to = (3 - 1) * 10, so offset = 20

And so on.

With this library, it is possible to use pagination with the page parameter, or using the skip manually. By default, the limit value is 100 and skip value is 0.

Example:

Request: http://localhost:3000/resources?limit=10&page=3

Query:

{
  "limit": 10,
  "skip": 20
}

Request: http://localhost:3000/resources?limit=10&skip=20

Query:

{
  "limit": 10,
  "skip": 20
}

Ordering

To work with ordering, you need to specify one or more sorting parameters, and whether you want the sorting to be ascending or descending. For ascending ordering, just put the name of the ordering parameter. For descending ordering, you need to put a "-" symbol before the name of the ordering parameter. Example:

Request: http://localhost:3000/resources?sort=created_at

Query:

{
  "sort": {
    "created_at": 1
  }
}

Request: http://localhost:3000/resources?sort=-created_at

Query:

{
  "sort": {
    "created_at": -1
  }
}

Request: http://localhost:3000/resources?sort=-age,name

Query:

{
  "sort": {
    "age": -1,
    "name": 1
  }
}

In multiple-parameter ordering, the first ordering parameter has higher priority than the second, and so on. In the example above, the ordering will be given primarily by the age parameter, in descending order. If there are two or more objects with the same value in age, then those objects will be sorted by name in ascending order.

Select

With this library, you can choose which parameters should be returned by the API. However, Mongo has a peculiarity: you can also specify which parameters you don't want to be returned. The logic is similar to ordering: to specify which parameters are to be returned, simply enter the parameter name; and to specify which parameters should not be returned, just place a "-" symbol before the parameter.

Example:

Request: http://localhost:3000/resources?select=\_id,name,age

Query:

{
  "select": {
    "_id": 1,
    "name": 1,
    "age": 1
  }
}

Request: http://localhost:3000/resources?select=-\_id,-created_at,-updated_at

Query:

{
  "select": {
    "_id": 0,
    "created_at": 0,
    "updated_at": 0
  }
}

It is interesting to use one or the other in your queries, as one is complementary to the other. If you want almost all parameters except a few, use the option to ignore parameters. If you want some parameters, and ignore the others, use the option to select the ones you want.

Filters

Now let's go to the most complex part of the library: the filters. There are several ways to apply filters in this library, so I'm going to break this topic down into subtopics for every possible filter approach.

Simple Filters

Simple filters are equality filters. Basically it's set key=value. All filter parameters are defined as string, so there are some validations that are done on these values.

1. If the value is a string number, it is transformed into a number, either integer or float/double (up to 16 decimal places);

2. If the value is in yyyy-MM-dd format or yyyy-MM-ddThh:mm:ss.sZ format, it is transformed into a Date object;

3. If the value is 'true' or 'false', it is transformed into a boolean value, according to your value;

4. Otherwise, the value is considered as a string.

Example:

Request: http://localhost:3000/resources?name=John%20Doe&age=31&birth_date=1990-01-01

Query:

{
  "filter": {
    "name": "John Doe",
    "age": 31,
    "birth_date": 1990-01-01T00:00:00.000Z
  }
}

MultiLevel Filters

You can specify multilevel filters. This means that, if you have an object that has a field that is another object, you can perform a search with filters through the parameters of the internal object. Example:

Object

{
  "_id": "613532a350857c1c8d1d10d9",
  "name": "Filippo Nyles",
  "age": 28,
  "current_job": {
    "title": "Budget/Accounting Analyst III",
    "salary": 4776.8
  }
}

Request: http://localhost:3000/resources?current_job.title=Budget/Accounting%20Analyst%20III

Query:

{
  "filter": {
    "current_job.title": "Budget/Accounting Analyst III"
  }
}

Partial Filters

Partial filters are a way to search a string type value for a part of the value. There are three ways to use partial filters. Making an analogy with javascript, it would be like using the startsWith, endsWith and includes methods, where:

• startsWith: search for a string-type value that starts with a given substring. To do this, just add a "*" at the beginning of the substring.
• endsWith: search for a string-type value that ends with a given substring. To do this, just add a "*" at the end of the substring.
• includes: search for a string value that contains a specific substring. To do this, just add a "*" at the beginning and end of the substring.

Example:

Request: http://localhost:3000/resources?name=_Lu&email=gmail.com_&job=_Developer_

Query:

{
  "filter": {
    "name": {
      "$regex": "^Lu",
      "$options": "i"
    },
    "email": {
      "$regex": "gmail.com$",
      "$options": "i"
    },
    "job": {
      "$regex": "Developer",
      "$options": "i"
    }
  }
}

Comparison Filters

Comparison operators are specific filtering options to check whether a parameter has a value. It is possible to check not only equality, but other mathematical operators, such as: ">", ">=", "<", "<=", "!=". In addition, you can use comparison operators to check whether an element is in an array.

According to the mongodb documentation, the available comparison operators are:

• $eq: Matches values that are equal to a specified value.
• $gt: Matches values that are greater than a specified value.
• $gte: Matches values that are greater than or equal to a specified value.
• $in: Matches any of the values specified in an array.
• $lt: Matches values that are less than a specified value.
• $lte: Matches values that are less than or equal to a specified value.
• $ne: Matches all values that are not equal to a specified value.
• $nin: Matches none of the values specified in an array.

To use these operators, just pass the comparator tag without the "$" symbol. Example:

Request: http://localhost:3000/resources?age=gt:30

Query:

{
  "filter": {
    "age": {
      "$gt": 30
    }
  }
}

I won't put an example with all operators here, but you can test arithmetic comparison operators on parameters with values of type string or number, or test the operators of $in and $nin on parameters of type array.

Element Filters

Element filters are filters used to deal with parameters that make up the entity's schema. There are two types of element filter possibilities:

• $exists: returns elements that have or do not have a specific field

• $type: returns elements whose field has a specific type.

Example:

Request: http://localhost:3000/resources?created_at=exists:true&updated_at=exists:false&jobs=type:array

Query:

{
  "filter": {
    "created_at": {
      "$exists": true
    },
    "updated_at": {
      "$exists": false
    },
    "jobs": {
      "$type": "array"
    }
  }
}

The $exists filter only works with true or false values. If a different value is entered, the filter will be ignored.

The same goes for the $type filter, which only works with valid type values defined in the mongodb documentation ( except deprecated ones):

 {
  "validTypes": [
    "double",
    "string",
    "object",
    "array",
    "binData",
    "objectId",
    "bool",
    "date",
    "null",
    "regex",
    "javascript",
    "int",
    "timestamp",
    "long",
    "decimal",
    "minKey",
    "maxKey"
  ]
}

AND | OR filters

Finally, it is possible to use filters with AND | OR operator. The usage logic follows the arithmetic rule.

To use the AND operator, you must pass the same value twice in a query. Example:

Request: http://localhost:3000/resources?age=gt:30&age=lt:50

Query:

{
  "filter": {
    "$and": [
      {
        "age": {
          "$gt": 30
        }
      },
      {
        "age": {
          "$lt": 50
        }
      }
    ]
  }
}

To use the OR operator, you must enter the values separated by a comma. Example:

Request: http://localhost:3000/resources?age=30,50

Query:

{
  "filter": {
    "$or": [
      {
        "age":  30
      },
      {
        "age": 50
      }
    ]
  }
}

Populate

If any collection uses references to other objects, in some operations it is interesting to return this information populated in the object in a single request. For this, the library supports the populate feature.

There are three ways to add the populate parameter to the query string:

• Specifying only the field to be populated:

Request: http://localhost:3000/resources?populate=jobs

Query:

{
  "populate": {
    "path": "jobs"
  }
}

• Specifying the field to be populated and which fields should be returned:

Request: http://localhost:3000/resources?populate=jobs;title,salary

Query:

{
  "populate": {
    "path": "jobs",
    "select": {
      "title": 1,
      "salary": 1
    }
  }
}

• Specifying the field to be populated, which fields should be returned and a resource filter (useful parameter when the populated field is a list):

Request: http://localhost:3000/resources?populate=jobs;title,salary;salary=gt:3000

Query:

{
  "populate": {
    "path": "job",
    "select": {
      "title": 1,
      "salary": 1
    },
    "match": {
      "salary": {
        "$gt": 3000
      }
    }
  }
}

• Specifying more than one field to be populated:

Request: http://localhost:3000/resources?populate=jobs&populate=currentJob

Query:

{
  "populate": [
    {
      "path": "jobs"
    },
    {
      "path": "currentJob"
    }
  ]
}

There are some rules to consider in populate. The populate must be specified as follows: populate=field;select;filter. Soon:

1. If you specify only the field to be populated, all field parameters will be returned, and if it is an array, all array elements will be returned;
2. If you want to specify which parameters are to be returned from the populated field, you need to specify which fields are to be returned;
3. If you want to filter the populated parameters, you need to specify the parameters that should be returned. If you want to return all object parameters, the select parameter must be informed as all. Example: populate=jobs;all;salary=gt:3000

Rules

• For pagination, you should use limit, skip and page only;
• For ordination, you should use sort only;
• For select, you should use select only;
• For populate, you should use populateonly;
• Anything other than limit, skip, page, sort, select and populate will be considered a filter;
• Parameters never contain characters that don't fit the regex /[^A-z0-9_.]/g;
• Filter values never contain characters that don't fit the regex /[^\w\s@.-:]/g;

Observations

This library is generic. This means that it handles the query based on the query object itself. Therefore, it is not possible to control events such as filter parameters with types incompatible with the types defined in the base. Use proper queries for your API, to prevent implementation errors from being thrown into your app.

Practical Examples

Check out how the configuration of the library in an API works in practice in this project.

Upcoming Features:

• Add a query handling option for Typeorm.

License

Distributed under the Apache License 2.0. See LICENSE for more information.

Authors

• Lucas Rocha - Initial Work.