Intro video: https://www.youtube.com/watch?v=RYBSJeUnZbc
What is Backpack?
Backpack is a REST API starter kit which includes a handful of helpful tools to get you up and running quickly without locking you into anything. After running the init script, you'll start with directory which looks like this:
api/
devops/
lib/
models/
routes/
tests/
server.js
...and some other stuff
...So that's the first thing. Using Backpack means you start with a sensibly organized Express/Mongoose app. When you want to add a new CRUD collection like books
for example, instead of manually creating a new models file and a new routes file and then importing those into server.js
, you can just create them in their appropriate directories and they'll be auto-imported. Better yet, just run npm run model Book
to automatically generate them, then just edit models/Book.js
to define your schema. The goal is to start with a good boilerplate, then hack at it to make it your own.
The 3 core pieces to Backpack
- The init script
npm init backpack
which instantly pulls down the Backpack boilerplate and creates a newapi/
directory. - The Backpack middleware which simplifies things like sending response JSON or errors, and handling JSON web tokens and protected routes.
- The client-side
backpack-fetch
NPM module which streamlines and improvesfetch
requests to your API.
Quick start
- Create a new directory called
myProject
andcd
into it. - Run:
npm init backpack
. - Browse to
http://localhost:8100/api
Standardized JSON Response
When designing a REST API, it's important to impose some standards on what the response will look like and then adhere to those standards. For example, what properties can we count on our API response to always include? Using the provided Backpack middleware functions res.sendData()
or res.sendError()
, will always include at least status
, data
, and error
properties; and when applicable, it will also include token
and/or meta
(for pagination data). Either data
or error
will always be populated with an object and the other will be null
.
Example data response:
{
"status": 200,
"data": [...book1, ...book2, ...],
"error": null
}
Example error response:
{
"status": 404,
"data": null,
"error": {"message": "Not found"}
}
One of the main benefits of using Backpack is that planning this sort of standardization is already all figured out and implemented sensibly.
Core piece 1: The init script
To add a Backpack REST API to your project, just cd
into the root of your app directory and run the NPM init script:
npm init backpack
This will add a new api/
directory at the current location, install dependencies, and start the server. You now have a running Backpack server.
api/
directory
Exploring the The file server.js
is where the bulk of the action is. Take a look at that to discover that this is really just a pretty standard Express server which imports some middleware and auto-imports whatever files exist in the routes/
directory. Other than that, there's really nothing special.
The two most important directories in api/
are: models/
and routes
. In models/
, you'll find User.js
and UserAuth.js
which contain default Mongoose model schemas for managing users and JWT-based authorization. More on this later. The routes/
directory is where we define our Express routes and controller functions and starts off with a file for users.js
. All pretty standard stuff. Even if we were to stop here and didn't use any other features of Backpack, we've already got a running API server with sensible defaults and auto-loading routes in about a minute.
Core piece 2: The middleware
Backpack includes middleware to help us work more efficiently. These middleware and the tools they provide are totally optional. They provide simplified methods for sending a response, easier token management, and a couple other handy tools.
req
and res
objects.
New properties and methods added to Express When writing controller functions to handle routes in Express, we're used to dealing with the req
and res
objects. Here's a typical example querying the database with Mongoose:
// GET /people/:id
router.get('/:id', async (req, res, next) => {
const person = await Person.findById(req.params.id).catch(err => next(err));
if (person) {
res.status(200).json(status: 200, data: { person: person },
}
});
You're certainly welcome to do that in your Backpack routes. But there's an easier way because Backpack middleware adds some nifty properties and methods to both the req
and res
objects. Probably the two most useful are sendData
and sendError
.
res.sendData(obj, [meta])
Sends the JSON response along with optional meta data and auto-included token if applicable.
Example: res.sendData( { name: 'Bob', age: 35 })
.
This is almost the equivalent of res.status(200).json(status: 200, data: { name: 'Bob', age: 35}, error: null)
. The difference is that sendData
will also include pagination and token properties when appropriate (see below).
res.sendError(status, [message], [details])
Sends an error response with appropriate HTTP status
code and the appropriate message
(automatically populated according to status code if not supplied explicitly) and optional details
.
req.token
If a token was included in the request headers, the token
middleware will try to validate it. If the token is valid, it will be decoded and assigned to req.token
available for use in controller functions. Otherwise, res.token
will be undefined. See "Tokens and Authorization" section below for details.
req.pagination
If the request has a querystring with either _page
or _limit
properties, those values will be used to create an object like { page: 2, limit: 20 }
and set to req.pagination
available for use in controller functions. See "Pagination" section below for details.
req.getQuery(Model)
Converts querystring parameters to a Mongoose query. For example, if a request came in with /employees?age=21
, then req.GetQuery(Employee)
would return { age: 21 }
. This can then be passed into Mongoose find()
to filter results. Note that the model is passed in so that the key ("age" in this case) can be checked to make sure it's a valid schema key and also to handle keys which are String types differently. For example, /employees?name=bob
would be converted to { name: { $regex: 'bob', $options: 'i' } }
. This allows for matching partial strings and is case-insensitive.
res.encodedToken
Whenever a new token is created or refreshed, the newly created raw (not decoded) token is assigned here. While this is unlikely to be very useful in controller functions. It's used by the middleware for the response method sendData()
. If res.encodedToken
is set, it will be included in the response JSON.
res.setToken(obj)
See this in action in routes/users.js
. When the user has successfully authenticated with password or validation key or whatever method we decide to use, we need to issue a token. Do that with res.setToken()
and pass in an object with whatever data you want to save in the token. Using the defaults provided with Backpack user auth standards, those properties will be _id
, and roles
.
Core piece 3: The client side module
Rather than using fetch
or axios
to make requests from the client-side, we can use the companion NPM module backpack-fetch
which was designed to work perfectly with Backpack. From backpack-fetch
we import the api
module which has methods get()
, post()
, patch()
, and delete()
for our standard CRUD requests. It also has another method setUrl()
which allows us to set the base URL for the API server just once for the whole application (be sure to call setUrl()
early in the life cycle of your app).
The standard CRUD methods all accept an endpoint as the first argument and post()
, and patch()
accept a payload object as the second request.
fetch
without backpack-fetch
:
Example using standard const baseUrl = 'http://localhost:8100/api/';
async function postProduct(productObj) {
try {
const res = await fetch(baseUrl + 'product', {
method: post,
body: JSON.stringify(productObj),
headers: {
Accept: 'application/json',
'Content-Type': 'application/json'
}
});
const json = await res.json();
console.log(json);
} catch (error) {
console.error(error);
}
}
postProduct({ name: 'iPhone', description: "Duh. It's an iPhone.", price: 899 });
This will be a lot easier using the api
module from backpack-fetch
.
backpack-fetch
:
Example with // First: `npm install backpack-fetch`
import api from 'backpack-fetch';
api.setUrl('http://localhost:8100/api');
async function postProduct(productObj) {
const res = await api.post(productObj).catch(console.error(error));
res.error ? console.error(res.error) : console.log(res.data);
}
postProduct({ name: 'iPhone', description: "Duh. It's an iPhone.", price: 899 });
The postProduct()
function went from 14 lines of code to 2 and does precisely the same thing. Besides being slimmer and sexier, there are other advantages to using the api
module:
- Conversion from JSON response to object is automatic
-
Accept
andContent-Type
Headers for JSON payloads added automatically - Guaranteed Backpack standardized response object (even if the server doesn't respond)
- Handles server timeouts (default is 4 seconds)
- Configurable base URL
- Automatically includes localStorage
token
as Authorization header for every request - Response
token
automatically saved to localStorage
Bonus core piece: The code generator
There's nothing magical about the files in models/
or routes/
and it's completely fine to manually add more models and routes here directly. But there's a faster way:
npm run model <MyModel>
Creates 3 new files:
models / MyModel.js; // A Mongoose model with empty schema
routes / MyModel.js; // An Express routes file with basic CRUD routes
tests / MyModel.http; // HTTP tests for VS Code "HTTP Client" extension
It's important to emphasize that this script only generates these three files. Nothing more. Basic CRUD routes and tests are now ready to go. But the model schema still needs to be edited. So the typical workflow is to run npm run model <MyModel>
, then edit the schema for your new model. At some point, you'll likely want to edit the routes file and maybe the tests too – but immediately after running the generator and editing the schema, you've got basic CRUD routes already working. Again, these three files are meant to be hacked. They're' just a starting point.
tests/
dir?
What's in the If you use VS Code and have installed the amazing "Rest Client" extension, then this file is a simple alternative to using something like Postman to test your routes. The code generator above will create this file with a handful of standard CRUD routes which are clickable so you can test API endpoints. And of course you can add more routes or hack it up however you like. Or you can just ignore these files and use Postman or whatever.
Pagination
https://github.com/aravindnc/mongoose-paginate-v2
Backpack uses mongoose-paginate-v2
with a couple config tweaks that make it work with our standardized JSON response. Model files made using the generator script will include this plugin. Then in your controllers, when you want paginated results, just use <Model>.paginate()
instead of <Model>.find()
.
Whereas <Model>.find()
will return an array, <Model>.paginate()
will return an object like this: { data, meta }
.
find()
method:
Example using standard const data = Book.find();
res.sendData(data);
paginate()
method:
Example using const { data, meta } = Book.paginate({}, { page: 3, limit: 20 });
res.sendData(data, meta);
Notice that we send meta
as the second argument to res.sendData()
.
The meta object contains pagination information like this:
{
(totalDocs = 547),
(limit = 20),
(page = 3),
(totalPages = 28),
(hasNextPage = true),
(nextPage = 4),
(hasPrevPage = true),
(prevPage = 2),
(pagingCounter = 3);
}
As discussed above, the middleware also has a convenience helper for finding pagination parameters passed into querystrings that look like this: /books?_page=5&_limit=20
. These should be prefixed with an underscore since it's possible that page
or limit
could be names of properties you'd want to be able to use. So we reserve these two names prefixed with underscores instead.
When _page
and optionally _limit
are included in the querystring, the middleware removes these special keys from req.query
and instead sets them at req.pagination
as page
and limit
. This makes it very easy to get paginated results like this:
/books?_page=5&_limit=20
Ex: // Middleware has populated this --> req.pagination == { page: 5, limit: 20 }
const { data, meta } = Book.paginate({}, req.pagination);
sendData(data, meta);
The paginate()
method accepts a query as the first argument, and an options object as the second. We passed in the options object with properties page
and limit
since that's what req.pagination will always include.
💡 The
limit
property is optional and has a default value of 50 (this setting is inserver.js
).
Tokens and Authorization
If you're not a JSON Webtoken aficionado, the basic idea is that the server is able to validate a token by hashing it against the key it was generated with – without hitting the database. So whatever value exists for JWT_KEY
in the .env
file is used to both generate tokens and also to validate them. This key was randomly generated during the configuration phase of npm init backpack
.
The basic idea
We want to avoid doing a database lookup for every single protected request. For example, an admin user logged into a control panel we've built into the app might make a hundred are so requests in a 10 minute session. Using JSON webtokens means we can issue the token once, then just validate that token on the server side for every subsequent request – until the token has expired or is due for a refresh.
exp
and refresh
The reason for having both The problem with issuing a token which grants privileges without any database calls is that there's no way to invalidate the token from the server side without invalidating all tokens. So we want to keep the interval somewhat short in case privileges have been revoked (or downgraded or upgraded) for whatever reason. In other words, if this user's admin
privileges have been revoked, the only way her token becomes invalidated is if we check the database and discover that. This is why we "refresh" the token at a set interval (default is 15 minutes).
If the exp
timestamp is outdated, the user is actually required to get a new token (usually by logging in again). We don't want to make the user log in every 15 minutes. So behind the scenes, if the token refresh
is outdated but the exp
is not, we just check the database and then issue a new token with updated values for exp
, refresh
, and roles
.
req.token
Auto-populated If an Authorization header was included with a request AND if the token can be validated, that token will automatically be decoded and set to req.token
. Then in your Express routes function, you can easily access things like req.token.roles
(to check for admin privileges, for example) or whatever other properties you may be storing in the token.
💡 It's worth pointing out here that these "magical" new properties aren't really magical at all. And it doesn't lock you into any particular way of doing things. The middleware file
lib/token.js
simply looks for an Authorization header and if there's a valid token there, it decodes it and sets it toreq.token
. It's available for you to use or ignore. Totally up to you.
res.setToken()
Setting a new token with When a user has authenticated (by clicking an email link or whatever), this method accepts an object with properties to be stored in the token. The token must include an _id
property so that the user can be authenticated for token refresh. If you don't supply exp
or refresh
(usually you won't), those will be populated for you. The default users
routes include endpoints for getting a new token with validation key or password. Both of these prebaked routes create a new User
instance, then pass the _id
and roles
properties to res.setToken()
. Check out routes/users.js
to see this stuff in action.
Auto token refresh
By default, a token will expire after 7 days. When a token expires, the user will be required to get a new token (typically by re-authenticating). In addition to all tokens having an exp
property, tokens created with res.setToken()
will also have a refresh
property with a default of 15 minutes.
💡 Defaults can be changed at the top of
lib/token.js
So during a session, instead of checking the user auth against the database for every request, we can just validate the token for every request until the token is due for a refresh. When the current time exceeds the value of refresh
, the middleware will lookup the User
by the _id
in the token and then generate a new token with updated exp
, refresh
, and roles
values.
token
inclusion with response
Auto When using res.sendData()
, the new tokens will be included as a property of the JSON response object. Whenever a new token is generated either by calling res.setToken()
or because the token was refreshed, in either case the new token is set to res.encodedToken
. That's the new base-64 encoded token which isn't likely very useful for us to access in the routes. But the fact that it's there means that res.sendData()
will include it in the response.
So instead of:
{ "status": 200, "data": {...theData }, "error": null }
The client should instead get something like:
{ "status": 200, "data": {...theData }, "error": null, "token": "123456.123456.123456" }
Putting all that together...
So that may all seem like a lot random bits of token management. But when you put it all together and use it in combination with the api
methods provided by backpack-fetch
on the frontend, token management becomes completely automated from the point you call res.setToken()
during user authentication (which is already done for you in the default routes in routes/users.js
). Let's look at how the token is handled during a round-trip request:
-
The user hits an endpoint perhaps with a validation code so we know we can now safely authenticate this user.
-
Your controller function calls
res.setToken()
which will create a new token and this value is set tores.encodedToken
. -
When you send the response using
res.sendData()
, the token will be included in the response becauseres.encodedToken
exists. -
If the request was made using one of the
api
methods provided bybackpack-fetch
, thetoken
property of the JSON response will be set to the browser localStorage. -
On the next request the user makes, if there's a token found in localStorage, it will be included in the next request as an "Authorization" header.
-
Back on the server side, any request that includes an Authorization header will try to validate that token value. If it validates AND isn't expired AND isnt due for a refresh, the token is decoded and set to
res.token
for easy access in your routes. -
If the token is due for a refresh, the token middleware finds that user by the token
_id
and sets new values forexp
,refresh
, androles
and saves the new token tores.encodedToken
and the decoded version tores.token
. Which means it will be included inres.sendData()
, and assuming you're usingbackpack-fetch
on the client side, will automatically update the token in localStorage.
Token guarantees
Using the token
middleware with sendData
and backpack-fetch
on the client side. You can count on these things to always be true:
- On the server side, If you create a new token or a token has automatically refreshed, it will be included in the JSON response using
sendData
. - Using
backpack-fetch
on the client side, if there's atoken
included in the JSON response, it will always be saved to localStorage. - Also using
backpack-fetch
, if there's atoken
in localStorage, it will always be included in the request headers. - On the server side, whenever an Authorization header exists, the included token is validated and refreshed if due for refresh, then included with the JSON response (see Step 1).
So this effectively gives us automated rolling sessions with JSON webtokens.
Protected routes
In lib/auth.js
which can be modified like anything else, there are two middleware functions available to be used in protected routes: requireAuth
and requireAdmin
. For any routes you want restricted to a logged in user (with a valid token), you could do this in your routes file:
// Example for GET /secretThing/:id
router.get('/:id', requireAuth, async (req, res, next) => { // NOTICE THE SECOND ARG
const thing = await secretThing.findById(req.params.id).catch(err => next(err));
thing ? res.sendData(thing) : res.sendError(404);
}
That second argument requireAuth
simply checks for the presence of res.token
. Remember, for every request, the token middleware always attempts to validate any token found in the headers and then sets it res.token
only if it's present and valid. So determining whether a user has a valid token is extremely easy. If there's a res.token
, the user has a valid token.
Instead of using the middleware above, we could have just added a line like this as the first line inside the function block:
if (!res.token) return res.sendError(401);
If you open up lib/auth.js
, you'll see that's exactly what it does. In fact, let's take a look at that right now:
In /lib/auth.js
requireAuth(req, res, next) {
// Validate token.
if (!res.token) return res.sendError(401);
next();
},
Pretty dang simple. Either adding that line as shown above or inserting the middleware into your route function as shown at the top of this section, either approach has the same exact effect. The middleware way is a little more terse which is nice for a full page of protected routes functions – keeps things DRY.
The other auth function available by default is requireAdmin
. This is basically the same but takes the additional step of checking that the token.roles
array includes the value 'admin'
. Of course, it's trivial to add other auth functions here as your app may require.
Additional planned features
File uploads
Although this is not baked in yet, there's nothing stopping you from adding this functionality in your routes. The plan is to add baked in functionality using "Multer" at some point.
Logging
Currently only logging to console is supported (using Morgan). Should be pretty easy to configure Morgan in server.js
to log to files. Not to be a broken record but all this stuff is just a starting point. You aren't locked into anything. Hack at it!