skyliner

    0.3.1 • Public • Published

    Skyliner

    Universal document outlining engine. Generate an outline of any text-based document!

    Skyliner is a regular expression based streaming document outlining engine. The goal is to support outlining as many different text-based formats as possible. It reads the input line-by-line and emits outline items.

    It was born out of frustration that Atom-IDE doesn't support half of the file formats I usually work with.

    It can be used in 2 ways:

    1. Through the command-line interface
    2. Via the Javascript API

    Eventually, the goal is to implement a plugin for Atom.

    Supported languages

    The following languages are currently supported:

    • clike (e.g. c, c++, header files, aliases: c, h, c++, h++, cpp, hpp)
    • csharp
    • go
    • ini
    • javascript (aliases: js, mjs)
    • json
    • latex (aliases: tex)
    • lua
    • markdown
    • php
    • rust
    • sh (including bash)
    • toml
    • wren
    • xml

    See below for how to add support for a new language.

    Usage

    Command Line Interface

    Install via npm:

    npm install --global skyliner

    Then, call skyliner like this:

    skyliner --lang <language_code> --input path/to/file

    For example:

    skyliner --lang javascript --input path/to/example.js

    You can also pipe content in via the standard input:

    curl https://example.com/example.js | skyliner --lang javascript

    Instead of the text-based output, you can also ask Skyliner to give you JSON instead:

    skyliner --json --lang latex <path/to/file.tex
    

    See below for a list of supported languages.

    Exit Codes

    The CLI may exit with a number of different exit codes depending on the problem. Below these are catalogued.

    Exit code Meaning
    0 Success
    1 General error
    2 Language definition not found

    API

    Skyliner also has a Javascript API you can use. To get started, install Skyliner with npm:

    npm install --save skyliner

    Then, import the Skyliner class and start generating outlines. Here's an example:

    import Skyliner from 'skyliner';
    
    (async () => {
    	"use strict";
    	
    	const skyliner = new Skyliner();
    	
    	console.log(await skyliner.outline("markdown", "# Heading 1\nThis is some text.\n"));
    })();

    This example will log an array of hierarchically-nested outline objects (see below for the spec on what they look like)

    You can also stream the outline objects directly, though they won't be auto-nested for you. Here's how you do that:

    import Skyliner from 'skyliner';
    
    (async () => {
    	"use strict";
    	
    	const skyliner = new Skyliner();
    	
    	for await (let outline_item of skyliner.outline_iterate("markdown", "# Heading 1\nThis is some text.\n"))) {
    		console.log(outline_item);
    	}
    })();

    skyliner.outline_iterate is an asynchronous generator that emits outline objects.

    Note also that the source to parse is passed directly to nexline, so you can pass it anything that nexline supports, such as:

    • A readable stream (e.g. fs.createReadStream(filepath))
    • A string containing source text to iterate over
    • A buffer of source text
    • A file handle (e.g. fs.openSync(filepath, "r"))
    • An array of any combination of the above

    Full automatically updated API documentation is coming soon.

    Specification

    An outline consists of an array of items. The form of each item looks like this:

    {
    	line,
    	index,
    	depth,
    	type,
    	text,
    	children,
    }

    Each property is described below:

    • line (Number): The line number the item appears on.
    • index (Number): The index within the line the item starts at.
    • depth (Number): How deeply nested the item is. Note that this isn't necessarily how deeply nested within the outline tree, but the document itself (e.g. brackets might mean it's more deeply nested in the document than the location in the outline).
    • type (string): The type of outline item this is. See the table below.
    • text (string): A textual string that describes the item.
    • children: (Array<Object>?): Optional. If specified, this is an array of child items.

    Additionally, the following property is added by Skyliner.outline():

    • depth_tree (Number): The depth in the tree at which the item is located (as opposed to the depth in the source syntax that was outlined).

    The following list of types can currently be returned:

    Type Meaning
    function A function
    class A class
    namespace A namespace or other logical grouping element
    interface An interface that defines the functionality of a class or other logical structure
    object An object or table containing multiple values
    property A property on a class or object
    heading A heading in some kind of text document (e.g. markdown)
    element A hierarchical element of some kind (e.g. XML / HTML)

    Adding a new language

    To add support for a new language, go to src/langs and create a new file (or copy an existing one) and create a definition for the new language. The filename must exactly match the name of the language you're adding.

    Skyliner works through a simple state machine based system. A file must have a default export of an object containing the different possible states the state machine can be in. All languages must at least define a "default" state, as this is the state that the engine begins in.

    States can have one or more rules defined therein as an object, where the keys are the names of the rules, and the values are the rules attached thereto.

    When you've added a new language definition file, you'll need to run this command to update the language list:

    npm run prepare

    This is because without statically defining the list of languages Skyliner would be incompatible with bundling systems like Rollup for example.

    Rule definition

    A rule is an object containing the following properties:

    • regex: RegExp - Regex to match with. This is the only required item.
    • group_index: Number - The index of the capturing group to use. By default the entire matched string is used by the Lexer.
    • depth_delta: Number - Delta value to add to the depth in the outline tree. Useful for handling brackets etc. to get the hierarchy in the generated outline correct.
    • depth_delta_after: Number - Same as depth_delta, but applied after any outline item is emitted and any other depth handling logic.
    • depth_set: Number - Set the depth to this value if this rule is chosen.
    • outline: string - The type of outline item to generate. If not specified, no outline item will be generated.
    • switch_state: string - The name of the alternate state to switch to. This will happen as soon as this item is chosen by the Lexer.
    • parent_type: string - The type of direct parent that this rule must have to trigger at all.
    • children: boolean - Whether this item can have children or not. If false, then this item isn't put on the stack for computing parent_type.
    • ends: string[] - A list of strings naming the rule names this block is responsible for ending. If specified, if this block is picked and the item on the top of the stack matches anything in this list and the depth (after applying any changes described above) is the same as that of the item on the top of the stack, it will be removed.

    Only regex is required.

    Writing Tests

    TODO: Write a section here on how to implement tests

    TODO

    This is a todo list of languages support for which support is planned. Feel free to pick any of these up after checking for an existing pull request. Don't feel limited to this list either - pull requests for other languages are also welcome.

    • html (for now use xml and make sure you close self-closing elements <xml_style />, because doing HTML will be very hard otherwise)
    • java
    • ruby
    • prolog
    • python
    • sql?
    • asm?
    • wasm?

    Contributing

    Contributions are very welcome - both issues and pull requests! Please mention in your pull request that you release your work under the MPL-2.0 (see below).

    Please don't forget to implement some tests if you've found & fixed a bug or implemented support for a new language :-)

    If you're feeling that way inclined, the sponsor button at the top of the page (if you're on GitHub) will take you to my Liberapay profile if you'd like to donate to say an extra thank you :-)

    License

    Skyliner is released under the Mozilla Public License 2.0. The full license text is included in the LICENSE file in this repository. Tldr legal have a great summary of the license if you're interested.

    Install

    npm i skyliner

    DownloadsWeekly Downloads

    1

    Version

    0.3.1

    License

    MPL-2.0

    Unpacked Size

    66.7 kB

    Total Files

    33

    Last publish

    Collaborators

    • sbrl