npm

Table of Contents

• Install
• Rationale
• API
• applicableOpts
• Example
• opts.cb
• Contributing
• Licence

Install

npm i detergent

Consume via a require():

const { det, opts, version } = require("detergent");

or as an ES Module:

import { det, opts, version } from "detergent";

or for web pages, as a production-ready minified script file (so-called "UMD build"), straight from CDN:

<script src="https://cdn.jsdelivr.net/npm/detergent/dist/detergent.umd.js"></script>

// in which case you get a global variable "detergent" which you consume like this:
const { det, opts, version } = detergent;

This package has three builds in dist/ folder:

⬆ back to top

Rationale

Detergent is a tool which cleans and prepares text so you can paste it safely into HTML template:

For starters, Detergent will:

• delete invisible Unicode characters
• collapse whitespace chunks longer than one space (considering newlines)
• strip HTML and recursively decode anything HTML-encoded

Then Detergent will optionally:

• encode all non-ASCII characters (for example, £ into £)
• improve English grammar style (for example, convert straight quotes to curly)

Adobe Photoshop and Illustrator both place ETX characters when you insert linebreaks using SHIFT+ENTER to break the line but keep the text within the same paragraph (that's opposed to normal line breaks using ENTER alone which breaks paragraphs). When a text with an ETX character is pasted into HTML template, it is invisible in the code editor but might surface up later as "�" when CMS or ESP or other platform attempts to read the code.

Detergent has optional features to improve the English style:

• widow word prevention adding   between last two words
• M-dash and N-dash recognition and automatic replacement where typographically appropriate
• Adding fancy apostrophes and curly quotes
• Adding missing spaces after full stops, commas and semicolons, except when it's a number.

Extra features are:

• You can skip the HTML encoding of non-Latin language letters. Useful when you are deploying Japanese or Chinese emails because otherwise, everything would be HTML-encoded.
• Detergent is both XHTML and HTML-friendly. You can set which way you want your <BR>'s to appear: with a closing slash (XHTML) or without (HTML), so your HTML code should be passing the W3C validator.
• Detergent handles the full range of Unicode code points. In other words, it's emoji-friendly.
• Detergent will use the named HTML entities (for example,   instead of  ) so you can read and recognise them. Not all named HTML entities work in all email clients, so we did the testing, found out which-ones don't render correctly and set those to be numeric.

⬆ back to top

API

The main function is exported in a plain object under key detergent, so please import it like that:

const { det } = require("detergent");
// or request everything:
const { det, opts, version } = require("detergent");
// this gives extra plain object `exportedOpts` with default options. Handy when
// developing front-ends that consume the Detergent.

det is the main function. See its API below.

opts is default options' object. You pass it (or its tweaked version) to det.

version returns same-named package.json key's value - the version of the particular copy of Detergent you've got.

⬆ back to top

API - det() Input

The det above is a function. You pass two input arguments to it:

⬆ back to top

API - det() options object

Here it is in one place:

det("text to clean", {
  fixBrokenEntities: true,
  removeWidows: true,
  convertEntities: true,
  convertDashes: true,
  convertApostrophes: true,
  replaceLineBreaks: true,
  removeLineBreaks: false,
  useXHTML: true,
  dontEncodeNonLatin: true,
  addMissingSpaces: true,
  convertDotsToEllipsis: true,
  stripHtml: true,
  stripHtmlButIgnoreTags: ["b", "strong", "i", "em", "br", "sup"],
  stripHtmlAddNewLine: ["li", "/ul"],
  cb: null,
});

The default set is a wise choice for the most common scenario - preparing text to be pasted into HTML.

You can also set the options to numeric 0 or 1, that's shorter than Boolean true or false.

⬆ back to top

API - det() output object

Function det returns a plain object, for example:

{
  res: "abc",
  applicableOpts: {
    fixBrokenEntities: false,
    removeWidows: false,
    convertEntities: false,
    convertDashes: false,
    convertApostrophes: false,
    replaceLineBreaks: false,
    removeLineBreaks: false,
    useXHTML: false,
    dontEncodeNonLatin: false,
    addMissingSpaces: false,
    convertDotsToEllipsis: false,
    stripHtml: false
  }
}

⬆ back to top

applicableOpts

Next generation web applications are designed to show only the options that are applicable to the given input. This saves user's time and also conserves mental resources — you don't even need to read all the labels of the options if they are not applicable.

Detergent currently has 14 option keys, 12 of them boolean. That's not a lot but if you use the tool every day, every optimisation counts.

I got inspiration for this feature while visiting competitor application https://typograf.github.io — it has 110 checkboxes grouped into 12 groups and options are hidden twice — first sidebar is hidden when you visit the page, second, option groups are collapsed.

Another example of overwhelming options set — Kangax minifier — https://kangax.github.io/html-minifier/ — it's got 26 options with heavy descriptions.

Detergent tackles this problem by changing its algorithm: it processes the given input and then makes a note, is particular option applicable or not, independently, is it enabled or not. Then, if it's enabled, it changes the result value.

For example, detergent's output might look like this — all options not applicable because there's nothing to do on "abc":

{
  res: "abc",
  applicableOpts: {
    fixBrokenEntities: false,
    removeWidows: false,
    convertEntities: false,
    convertDashes: false,
    convertApostrophes: false,
    replaceLineBreaks: false,
    removeLineBreaks: false,
    useXHTML: false,
    dontEncodeNonLatin: false,
    addMissingSpaces: false,
    convertDotsToEllipsis: false,
    stripHtml: false
  }
}

The options keys which have values of a type array (stripHtmlButIgnoreTags and stripHtmlAddNewLine) are omitted from applicableOpts report.

⬆ back to top

Example

The simplest possible operation - encoding using default settings:

const { det } = require("detergent");
let { res } = det("clean this text £");
console.log(res);
// > 'clean this text £'

Now, using custom settings object with one custom setting convertEntities (others are left default):

const { det } = require("detergent");
let { res } = det("clean this text £", {
  convertEntities: 0, // <--- zero is like "false", turns off the feature
});
console.log(res);
// > 'clean this text £'

⬆ back to top

opts.cb

One of the unique (and complex) features of this program is HTML tag recognition. We process only the text and don't touch the tags, for example, widow word removal won't add non-breaking spaces within your tags if you choose not to strip the HTML.

opts.cb lets you perform additional operations on all the string characters outside any HTML tags. We aim to tap detergent.io uppercase-lowercase functionality here but maybe you'll find additional uses.

Here's an example, where we have widow word removal, HTML tags and additionally, with help of opts.cb, turn all the letters uppercase (but not on HTML tags):

const { det } = require("detergent");
const { res } = det(`aAa\n\nbBb\n\ncCc`, {
  cb: (str) => str.toUpperCase(),
});
console.log(res);
// => "AAA<br/>\n<br/>\nBBB<br/>\n<br/>\nCCC"

⬆ back to top

Contributing

• If you see an error, raise an issue.
• If you want a new feature but can't code it up yourself, also raise an issue. Let's discuss it.
• If you tried to use this package, but something didn't work out, also raise an issue. We'll try to help.
• If you want to contribute some code, fork the monorepo via GitLab, then write code, then file a pull request on GitLab. We'll merge it in and release.

In monorepo, npm libraries are located in packages/ folder. Inside, the source code is located either in src/ folder (normal npm library) or in the root, cli.js (if it's a command-line application).

The npm script "dev", the "dev": "rollup -c --dev" builds the development version retaining all console.logs with row numbers. It's handy to have js-row-num-cli installed globally so you can automatically update the row numbers on all console.logs.

⬆ back to top

Licence

MIT License

Copyright (c) 2015-2020 Roy Revelt and other contributors

Passes unit tests from https://github.com/kemitchell/straight-to-curly-quotes.json, licenced under CC0-1.0