node package manager


ideas about recurring tasks in Web- and Backend-Application building

Table of Contents generated with DocToc

Alpha One

About: This is a meta-project to collect scattered thoughts about and working solutions for recurring tasks in Web- and Backend-Application building. Do not expect polished, working code, but rather fragmentary, sketchy hints.

The project takes its name from the 1970s British TV series Moonbase Alpha / Space: 1999.



Rationale: We often want to restrict access to the ressources a web application publishes; for example, you want subscribers to have unlimited access, guest users to enjoy a realistic peek into your offerings without handing over the full power of your application and the data therein, and at the same time deter any automated downloads. One part of the equation that can make this happen is User Authentication; the other is identifying anonymous visitors and selectively throttle or block their access, based on automatically collected behavior details and / or manual black- and whitelisting.

  • A limiter should be completely agnostic as to the task it is limiting—it should help to generally put limits to ressource usage, not, say, be built to specifically limit HTTP requests (a drop-in middleware could do that–one that uses a generic limiter).

  • Because of its genericity, a limiter should work with arbitrary ID tokens that represent clients—these could be any piece of data (an internal user ID, an IP address, a session ID) that is fit to identify a client in the sense of the application.

  • Since a limiter should be able to base its behavior on the past behavior of a given client and on the client's ranking (say, customer vs. guest vs. rogue), it must either entertain a suitable datastructure to keep such group affiliation and usage data (or else be fed (some of) these details when called).

  • Whether or not a limiter should take care of data persistence is an open question; ideally, server restart or redirection to another server process should not impact limiter behavior (this consideration would appear to favor Redis-like data persistence plans).

  • A limiter should distinguish between and be configurable for at least three behavioral patterns: full access, deferred (throttled) access, and denial of service (the last two with timeouts and permanent).

CoffeeNode Limit is an sttempt to bring all of the above points to the backend. It currently works with a slightly patched version of node-rate-limiter.

Browser Console / ANSI Colors


Command Line Option Parsing

Using path-extra to get the canonical OS-dependent location for user-level application option files:

### ###
njs_path = require 'path-extra'
info njs_path.tempdir()
info njs_path.homedir()
info njs_path.datadir 'app-name-here'


See discussion of initialization vector (iv):

General Data

Probably use openpgpjs or sjcl:

User Authentication & Management

The specific purpose of password encryption should probably be done using bcrypt, for which there are 100% JS implementations available. It does have the advantage of adaptable algorithmic complexity. APIs typically do not offer decryption, only comparison of data.

Jed Schmidt writes in the README:

since it doesn't handle logins, persistence, sessions, or anything past authentication, it is more of a tool and less of a framework

'Easy DB'

NPM & GitHub Publishing Helpers, Module Administration

Our erstwhile solution to preparing CoffeeScript source files for both development and production, larq, is very much on the way out.

It still has one distinctive feature, though, and that is larq dev, which compiles a JavaScript file for each CoffeeScript file that does not contain the actual code, but rather dynamically loads (and compiles) the CoffeeScript source; in this way, the CoffeeScript sources may change any time, while to outside consumers, everything appears as though code was actually loaded from JavaScript files, from the locations where those JavaScript loader files are located.

Thus, every effort is made to make the existence of CoffeeScript sources transparent; the result is convenience during development and assurance that production versions of the same code (produced with larq pub or coffee --compile) are fully functional (even in the absence of the CS sources—the plan is to publish those, too, though, so potential contributors get a chance to work on either JS or CS, whatever they feel most comfotable with).

I started larq ('Language-Aware ReQuire') when I realized that the global require.extensions hook in NodeJS has huge problems—long story short, it's a global hook, so if any of your dependencies have different ideas on how to resolve a particular filename extensions, you're BFE. It would be fine if it was some very local thing, but as it stands, its not, and care must be exercised.

There are some thoughts on how to repair this, but so far no overarching facility or best-practices solution has emerged, short of the advisory to compile your 3rd-party-language sources down to the standard ones (*.js, *.css, *.html) and only ever deal with these compilation targets in production.

A possible, maybe makeshift solution is to use CoffeeNode Monitor or similar to watch all sources and compile on change; this may prove a better alternative than trying to do the same, but in more convoluted ways, using yet another tool (larq).

Persisting, Responsive Long Running Processes

Rationale: When we want to have a single HTTP serving process to run on a production server, we can accomplish that by daemonizing the server process. Having more than one process and processes that are resumed after abortion requires more work. On a development machine, we want to typically have a single HTTP server to be run explicitly (not automatically on system startup), and, when any of its dependencies have been modified, we want to re-run any build steps and then restart the serving process. In case of syntax or runtime errors, we typically would like the monitoring instance to wait for any subsequent changes to the codebase and then try and re-run the build steps. A server is a Long Running Process—as distinguished from, say, a backup script that runs once a day for a minute or so. A Responsive Process is one that responds to (rather, is managed so it can respond to) changes in source files. A Persisting Process is one that is resumed on termination.

  • coffeenode-monitor

Jizura Data Queries

  • coffeenode-mingkwai
  • coffeenode-solr
  • coffeenode-mojikura

Paper Publishing

Publication Helpers / Build Tools

Generate TOC for your with npm install --global doctoc:


Git Automation

Git Automation


-1 for not updating their examples on their homepage. -1 for the Pyramids of Doom in their examples. -1 for an API that makes the programmatical equivalent of git add --all; commit -m "foo" look like rocket science. -1 for embracing asynchronicity so damn hard you'll get a hundred callbacks when you ask for a commit history. -1 for embracing Object-Oriented Programming so damn hard that you'll never get an answer to your question but an object whose methods you must call—to asynchronously receive another object on which to call more methods.


Cake is better.

NodeJS Version Management

Rationale: With the advent of breaking changes in NodeJS unstable 0.11.x, many modules relying on gyp / waf do not compile any more. Some tools (such as browserify) compile with 0.11.7 but not with 0.11.8. Consequently we must be able to switch between NodeJS versions in a swift and painless manner; n by visionmedia is one way to do that.

After the painless install: npm install --global n, you get a new executable n that lets you install and / or use new versions with commands like n latest, n stable or n 0.11.7. Simply executing n yields a menu in the terminal that lets you choose one of the installed versions. The global node command is always re-bound to the last chosen version with n; in order to get commands for specific versions, cd into a directory on the path and do something along the lines of:

ln -s /usr/local/n/versions/0.11.7/bin/node node0117
ln -s /usr/local/n/versions/0.11.7/bin/node node-latest
ln -s /usr/local/n/versions/0.10.22/bin/node node-stable

Application Architecture & Design

Control Flow

And the Winner is: Yield!

NB CoffyScript is still (highly) experimental; its use in underlying libraries is especially strongly discouraged. CyS is currently implemented as a random patch of (an earlier version of) CoffeeScript 1.6.3 which has been shown to posses some subtle bugs (especially faulty error reporting that does not point out the source of the error). There was an attempt to upgrade to the latest bugfix editions of CoffeeScript, but the code had already suffieciently changed to make simple copy-and-paste of the changes unviable. The CS Redux project is, unfortunately, not yet advanced / adopted enough for me switch over, so i decided to just wait it out: wait for yield to become the available in a stable Node release (i.e. 0.12 or 1.x) and wait for CS Redux to land in the master branch of CoffeeScript.

With all the caveats, CoffyScript is already usable and has greatly helped to write some very readable, working code that feels magnitudes better than the Promises/A+ ridden code it replaced.

The Journey Continues

  • caolan async
Making Things Synchronous

Sometimes the solution to async woes lies in choosing an underlying functionality provider that is synchronous—for example, when doing application startup stuff like compiling sources and so on, you might choose fs.writeFileSync over fs.writeFile because (1) it lets you write linear code, (2) you can't run your app before this step has completed anyhow. Using synchronous IO is an antipattern in online serving, but its much more of a best practice for a startup sequence.

In the same vein, it may be desirable to execute shell commands in a synchronous fashion. Sadly, this piece of functionality is not included in NodeJS proper; fortunately, there are modules to do that, like execSync. It compiles with warnings under NodeJS 0.11.7 on my OSX box but appears to work, so now you can do TRM = require 'coffeenode-trm', result = TRM.execute 'ls' alongside with TRM.execute 'ls', ( error, result ) -> ....


Promises, with all the neat /A+ marketing ribbons tacked unto them, are horrible IMHO.

They come fraught with conceptual burden (eg. promises vs. deferreds), force boilerplate on you (never forget to call .done() when you're done), and they urge you to wrap all of your callback-based functions so as to make them play nice (which is not nice—callbacks were there first, and they are the conceptually simpler mechanism. You just threw out the trusty hammer, now you got a beeping flashy gadget).

And: The API of the leading promises implementation is huge. If you insist on doing promises, i'll ask you to memorize this 60+ items list:

Q.all, Q.allSettled, Q.async, Q.defer, Q.delay, Q.denodeify, Q.getUnhandledReasons, Q.isPromise, Q.longStackSupport, Q.makePromise, Q.nbind, Q.nextTick, Q.nfapply, Q.nfcall, Q.ninvoke, Q.npost, Q.onerror, Q.promise, Q.promised, Q.reject, Q.resetUnhandledRejections, Q.spawn, Q.stopUnhandledRejectionTracking, promise.all, promise.allSettled, promise.catch, promise.delay, promise.delete, promise.dispatch, promise.done, promise.fapply, promise.fbind, promise.fcall, promise.finally, promise.get, promise.inspect, promise.invoke, promise.isFulfilled, promise.isPending, promise.isRejected, promise.keys, promise.nodeify,, promise.progress, promise.set, promise.spread, promise.then, promise.thenReject, promise.thenResolve, promise.timeout, deferred.promise, deferred.resolve, deferred.reject, deferred.notify, deferred.makeNodeResolver.

You're of course not done yet—care to learn that (from the API docs) "Q.when( 5, onFulfilled ) is equivalent to Q( 5 ).then( onFulfilled )"? I never wanted to know this.

One might argue that this particular API is just the result of some particular diligent programmer going somewhat over the top, but that doesn't devalidate the argument that there is more than a fair number of promises libraries out there, some of them broken, others with uncertain merits. Finding out how the ~30 libraries listed alongside the /A+ specs interact and compare in terms of features, performance and memory consumption is not something i want to spend my time with. And interaction between promises libraries is a real concern. To cite a proponent of promises (emphases mine):

As authors of Promises/A-consuming libraries, we would like to assume this statement to be true: that something that is “thenable” [meaning you can say x.then()on it] actually behaves as a Promises/A promise, with all the power that entails.

If you can make this assumption, you can write very extensive libraries that are entirely agnostic to the implementation of the promises they accept! Whether they be from Q, when.js, or even WinJS, you can use the simple composition rules of the Promises/A spec to build on promise behavior. For example, here's a generalized retry function that works with any Promises/A implementation.

Unfortunately, libraries like jQuery break this. This necessitates ugly hacks to detect the presence of objects masquerading as promises, and who call themselves in their API documentation promises, but aren't really Promises/A promises. If the consumers of your API start trying to pass you jQuery promises, you have two choices: fail in mysterious and hard-to-decipher ways when your compositional techniques fail, or fail up-front and block them from using your library entirely. This sucks.

Interesting to hear that the jQuery team tried to implement promises and failed. Interesting, too, that (some) promises library authors (according to the above statement) appearently try to live up to the duck typing dream—i.e. they rationalize "this must be a promises object!" when they get a return value x where x.then happens to be a function. That's about as reliable as saying "this must be a cinema!" because "it has seats inside".

Frankly, as much as asynchronous calls and Pyramids of Doom tend to be a PITA, promises look and feel wrong.


Standalone Apps & Isomorphic Modules

Rationale. NodeJS makes it possible to use the same code on the server and in the browser; tools like node-webkit make it possible to run one-file downloads as no-installation / 'portable' desktop apps (with the VM (runtime) either bundled or installed globally on the target machine). This spells good-bye to all (or at least most) of the many competing GUI toolkits (if and where a 70MB minimum size for the VM is acceptable, where a given platform is supported, and where performance is sufficient).



(Web) Application (Backend) Design

To Framework or Not To Framework?

Tero Piirainen writes under the title Technology lock-in:

If you look at the history of frameworks in any programming language, it's a history of failures. Frameworks come and go.

The Stack

Ordered from the earliest, most general, down to the latest, most specific steps, the 'stack of actions' in a web application may be sketched as follows:

  • Preparation
    • Limit (blocking)
    • Routing & Query Parsing
    • Favicon
    • Static
    • Authentication
    • Limit (throttle)
  • App
    • Response Initiation
    • Session Handling
    • DB Access
    • Template (View) Building
  • Finalization
    • Response Finalization

Modern Wire Communication Protocols

Chunked HTTP

  • what it means for templating


Web Application Frontend Design

Icons &c

Web Application Deployment




Probably better than using float:

Custom HTML Elements

Rationale: getting away from <div class='foo'><div class='bar'>, moving towards <foo><bar>.

Probably need a polyfill for IE compatibility; without namespacing, custom tags may face problems with future updates to HTML5 standard.

Alpha-One Design Philosophy (Sketches)

Templates, Layouts, Views

A template is a function that accepts a request and a response argument and returns HTML as text (or as a POD—not yet implemented).

A layout is a function that accepts a request, a response, and a content argument; it returns HTML as text (or as a POD—not yet implemented).

A view is a middleware method that accepts a request, a response, and a done argument; it gets bound to a constant or parametrized HTTP route. Views are somewhat special in that they can act either in a synchronous or a asynchronous fashion.

  • Asynchronous views must return undefined or null to signal that output should not yet be sent to the client (this is expecially important to keep in mind when using CoffeeScript, as functions may be compiled with an explicit return statement, so it's good practice to end asynchronous view methods with an explicit return null).

  • **Asynchronous views must call done content when they're ready to, otherwise the client will hang on waiting indefinitely for a response.

  • Synchronous views must return a text; no other return values are allowed at this point in time, although it conceivably makes sense to return an object that represents some content and may be manipulated before beign serialized and sent to the client.

After a view has returned a text, it must not call done; this is only allowed once. Conversely, once done has been called, returning anything but undefined or null is not permitted.

  • We currently use teacup as a templating engine, but this may change in the future.

  • Typically, we have one separate file for templates called, unsurprisingly,; within that module, there is a boilerplate header

    for name_ of teacup
      eval "#{name_.toUpperCase()} = teacup[ #{rpr name_} ]"

    that 'pulls' all teacup methods as uppercase variables into the module-local namespace.

  • Templates and views do not use teacup.renderable; instead, they typically have, near the top of the function, a line return RENDER => ...; this has proven to work better for our purposes (it's also less surprising than RENDERABLE).

  • There is some boilerplate near the top of each template and layout:

    O           = get_options request
    page_style  = O[ 'page-style'       ] ? 'plain'
    title       = O[ 'title'            ] ? 'welcome'

    and a module-local function

    get_options = ( request ) ->
      return request[ app_key ] ? {}

    providing a way to get view options from the request object. app_key is a text variable shared between the provider (the middleware stack) and the consumers (the templates and layouts) of per-request data; it should be set to a string that can be assumed to be unique within the given middleware stack (in the example, we chose alpha-one).

  • As a matter of convention, O[ 'layout' ] may contain a function that should be used instead of templates.layout.

  • Here is how templates are turned into views.—In app/main, there is a function view:

    view = ( view_name ) ->
      return ( request, response ) ->
        http_ok request, response
        response.write templates[ view_name ] request

    It is used like this:

    app.get '/',          view 'homepage'
    app.get '/welcome',   view 'welcome'
    app.get '/goodbye',   view 'goodbye'

    TAINT As becomes immediately apparent, there is no support to emit any other HTTP status code but 200 at this point in time—no redirects, no 404s are possible with this kind of setup; 'all sales are final'; when the given route matches, the templates is executed and its result are sent back with an HTTP 200 OK. Also, no next method to skip the view is present. Such functionality may or may not get integrated into the view function at some point in the future; right now the way to do it is to implement a middleware function yourself (it is easy) and not use view for this purpose.

User Authentication

Redirect, Bounce, and Back-To

There are three functions concerned with HTTP redirection: redirect, bounce , and back_to.

  • redirect request, response, location is just the plain Express response.redirect function (it is conceivable that some functionality might be added to this function so maybe prefer it over the Express method).

  • bounce request, response, location is like redirect, but it (1) issues a flash informing the user about the redirection (which your app may choose to silently ignore), and, (2) more importantly, sets a cookie value that contains the location from where the redirect was issued.

  • back_to request, response, location is like redirect, but it tries to retrieve the original location from the cookie set by bounce; if none can be found, it redirects to location. So say you go to a restricted are http://app/nogo, a bounce to /login will occur; after successfully entering credentials, you will go back_to /nogo. If you went straight to /login or you intentionally deleted the cookie in the process and the login middleware ends with the line back_to request, response, '/welcome', you will end up at a catch-all welcome page instead. (Implementation details discussed here may change; it is conceivably cleaner to store the bounce address in a session instead of in a cookie).