annotatedtext

A lightweight JavaScript library for converting markup documents into an annotated text format consumable by LanguageTool as AnnotatedText.

Usage

See API below for details.

Motivation

Provide an easier way of running LanguageTool on documents by separating the text from the markup.

Installation

npm:

npm install annotatedtext

API

compose(text, node, options = defaults)

Returns Annotated Text as described by LanguageTool's API:

{
  "annotation": [
    { "text": "A " },
    { "markup": "<b>" },
    { "text": "test" },
    { "markup": "</b>" }
  ]
}

Run the object through JSON.stringfy() to get a string suitable for passing to LanguageTool's data parameter.

This is the main function you'll use in implementing for different parsers.

import { compose } from "annotatedtext";
import { parse } from "remark-parse";
import { unified } from "unified";

const text = "This is a sentence.";
const processor = unified()
  .use(remarkparse, options.remarkoptions)
  .use(frontmatter, ["yaml", "toml"])
  .parse(text);

const annotatedtext = compose(text, parse);
JSON.stringify(annotatedtext);

• text: The text from the markup document in its original form.
• node: The root AST node that conforms to INode.
• options: (optional) See defaults.

build(text, parse, options = defaults)

Deprecated

Use compose instead.

Returns Annotated Text as described by LanguageTool's API:

{
  "annotation": [
    { "text": "A " },
    { "markup": "<b>" },
    { "text": "test" },
    { "markup": "</b>" }
  ]
}

Run the object through JSON.stringfy() to get a string suitable for passing to LanguageTool's data parameter.

This is the main function you'll use in implementing for different parsers.

import * as builder from ‘annotatedtext’;
const processor = unified().use(mark, { commonmark: true });
var annotatedtext = builder.build(text, processor.parse);
JSON.stringify(annotatedtext);

• text: The text from the markup document in its original form.
• parse: A function that parses a markup document and returns an abstract syntax tree.
• options: (optional) See defaults.

collecttextnodes(ast, options = defaults)

Returns an array of annotated text nodes used in the final annotated text object.

• ast: An abstract syntax tree.
• options: (optional) See defaults.

composeannotation(text, annotatedtextnodes, options = defaults)

• text: The text from the markup document in its original form.
• annotatedtextnodes: An array of an array of annotated text nodes such as produced by collecttextnodes.
• options: (optional) See defaults.

defaults

annotatedtext comes with the following default functions used throughout.

const defaults = {
  children: function (node) {
    return node.children;
  },
  annotatetextnode: function (node) {
    if (node.type === "text") {
      return {
        text: node.value,
        offset: {
          start: node.position.start.offset,
          end: node.position.end.offset,
        },
      };
    } else {
      return null;
    }
  },
  interpretmarkup: function (text = "") {
    return "";
  },
};

Functions can be overriden by making a copy and assigning a new function. For example, the tests use markdown and need to interpret new lines in the markup as new lines. The interpretmarkup function is overriden as:

var options = builder.defaults;
options.interpretmarkup = function (text) {
  let count = (text.match(/\n/g) || []).length;
  return "\n".repeat(count);
};

children(node)

Expected to return an array of child nodes.

annotatetextnode(node)

Expected to return a struture for a text ast node with at least the following:

• text is the natural language text from the node, devoid of all markup.
• offset contains offsets used to extract markup text from the original document.
  • start is the offset start of the text
  • end is the offset end of the text

{
  "text": "A snippet of the natural language text from the document.",
  "offset": {
    "start": 1,
    "end": 57
  }
}

If the node is not a text node, it must return null;

interpretmarkup(node)

Used to make sure LanguageTool knows when markup represents some form of whitespace. As mentioned above, the tests override this function to ensure new lines captured as markup are also visible to LanguageTool.

var options = builder.defaults;
options.interpretmarkup = function (text) {
  let count = (text.match(/\n/g) || []).length;
  return "\n".repeat(count);
};

Tests

Unit tests are also run via npm:

npm test

Implemented Parsers

The following packages wrap annotatedtext for specific parsers:

• annotatedtext-remark for markdown using remark-parse.
• annotatedtext-rehype for html using rehype-parse.

License

MIT © David L. Day