@xeger/web-tree-sitter

Web Tree-sitter

WebAssembly bindings to the Tree-sitter parsing library.

Setup

You can download the tree-sitter.js and tree-sitter.wasm files from the latest GitHub release and load them using a standalone script:

<script src="/the/path/to/tree-sitter.js"/>

<script>
  const Parser = window.TreeSitter;
  Parser.init().then(() => { /* the library is ready */ });
</script>

You can also install the web-tree-sitter module from NPM and load it using a system like Webpack:

const Parser = require('web-tree-sitter');
Parser.init().then(() => { /* the library is ready */ });

Basic Usage

First, create a parser:

const parser = new Parser;

Then assign a language to the parser. Tree-sitter languages are packaged as individual .wasm files (more on this below):

const JavaScript = await Parser.Language.load('/path/to/tree-sitter-javascript.wasm');
parser.setLanguage(JavaScript);

Now you can parse source code:

const sourceCode = 'let x = 1; console.log(x);';
const tree = parser.parse(sourceCode);

and inspect the syntax tree.

console.log(tree.rootNode.toString());

// (program
//   (lexical_declaration
//     (variable_declarator (identifier) (number)))
//   (expression_statement
//     (call_expression
//       (member_expression (identifier) (property_identifier))
//       (arguments (identifier)))))

const callExpression = tree.rootNode.child(1).firstChild;
console.log(callExpression);

// { type: 'call_expression',
//   startPosition: {row: 0, column: 16},
//   endPosition: {row: 0, column: 30},
//   startIndex: 0,
//   endIndex: 30 }

Editing

If your source code changes, you can update the syntax tree. This will take less time than the first parse.

// Replace 'let' with 'const'
const newSourceCode = 'const x = 1; console.log(x);';

tree.edit({
  startIndex: 0,
  oldEndIndex: 3,
  newEndIndex: 5,
  startPosition: {row: 0, column: 0},
  oldEndPosition: {row: 0, column: 3},
  newEndPosition: {row: 0, column: 5},
});

const newTree = parser.parse(newSourceCode, tree);

Parsing Text From a Custom Data Structure

If your text is stored in a data structure other than a single string, you can parse it by supplying a callback to parse instead of a string:

const sourceLines = [
  'let x = 1;',
  'console.log(x);'
];

const tree = parser.parse((index, position) => {
  let line = sourceLines[position.row];
  if (line) return line.slice(position.column);
});

Generate .wasm language files

The following example shows how to generate .wasm file for tree-sitter JavaScript grammar.

IMPORTANT: emscripten or docker need to be installed.

First install tree-sitter-cli and the tree-sitter language for which to generate .wasm (tree-sitter-javascript in this example):

npm install --save-dev tree-sitter-cli tree-sitter-javascript

Then just use tree-sitter cli tool to generate the .wasm.

npx tree-sitter build-wasm node_modules/tree-sitter-javascript

If everything is fine, file tree-sitter-javascript.wasm should be generated in current directory.

Running .wasm in Node.js

Notice that executing .wasm files in node.js is considerably slower than running node.js bindings. However could be useful for testing purposes:

const Parser = require('web-tree-sitter');

(async () => {
  await Parser.init();
  const parser = new Parser();
  const Lang = await Parser.Language.load('tree-sitter-javascript.wasm');
  parser.setLanguage(Lang);
  const tree = parser.parse('let x = 1;');
  console.log(tree.rootNode.toString());
})();

Running .wasm in browser

web-tree-sitter can run in the browser, but there are some common pitfalls.

Loading the .wasm file

web-tree-sitter needs to load the tree-sitter.wasm file. By default, it assumes that this file is available in the same path as the JavaScript code. Therefore, if the code is being served from http://localhost:3000/bundle.js, then the wasm file should be at http://localhost:3000/tree-sitter.wasm.

For server side frameworks like NextJS, this can be tricky as pages are often served from a path such as http://localhost:3000/_next/static/chunks/pages/index.js. The loader will therefore look for the wasm file at http://localhost:3000/_next/static/chunks/pages/tree-sitter.wasm. The solution is to pass a locateFile function in the moduleOptions argument to Parser.init():

await Parser.init({
  locateFile(scriptName: string, scriptDirectory: string) {
    return scriptName;
  },
});

locateFile takes in two parameters, scriptName, i.e. the wasm file name, and scriptDirectory, i.e. the directory where the loader expects the script to be. It returns the path where the loader will look for the wasm file. In the NextJS case, we want to return just the scriptName so that the loader will look at http://localhost:3000/tree-sitter.wasm and not http://localhost:3000/_next/static/chunks/pages/tree-sitter.wasm.

Can't resolve 'fs' in 'node_modules/web-tree-sitter'

Most bundlers will notice that the tree-sitter.js file is attempting to import fs, i.e. node's file system library. Since this doesn't exist in the browser, the bundlers will get confused. For webpack you can fix this by adding the following to your webpack config:

{
  resolve: {
    fallback: {
      fs: false
    }
  }
}