Contents
- Purpose
- Features
- Syntax
- Installation
- Usage
- FAQ
- Demonstration
- Core
- CLI
- Backend plugins
- Highlight plugins
- Miscellaneous
- Files
- Figures
- References
- To do
Purpose
Litdown is a literate programming tool that processes Markdown input into both documentation formats and extracts files embedded in the Markdown. Litdown can also be used to convert an existing project into Markdown source. Litdown is completely agnostic regarding programming languages, file formats, build systems, and version control systems.
Features
Bidirectional operation
Litdown operates in either extract or create mode. Extract mode converts literate Markdown source into generated documentation and files. Create mode is effectively the opposite of extract mode (although the round trip is lossy) and is intended to assist converting an existing project into literate Markdown source.
Multiple Markdown processing modules
Litdown currently supports using the Commonmark, Markdown-it, and Marked NPM modules to process literate Markdown source. Litdown does not intend to perform this step itself and this alows the user to determine which Markdown backend to use.
Litdown tries to be smart about which backend to use by checking available
backend plugins, installed modules, the user's list of preferred backends, and
a default list of preferred backends to determine which to use.
The -b
and -B
options can be used to modify and/or view this process.
See the Usage section for details.
Supporting a new or different backend module is as simple as creating a plugin. For details, see the Backend plugins section.
Non-executing
Litdown only creates files inside a previously non-existing directory under the current working directory. It never executes any embedded Javascript or system commands found in the literate Markdown source. This eliminates a bunch of potential security problems and places the responsibility and trust where it should be: between the developer and the user. If additional processing or compiling steps are required in your project, you should develop it in the form of a Makefile, script, or use a build system to perform it.
Litdown uses a Makefile to perform processing within its own source directory and a build script to perform steps that the Makefile cannot.
Self-hosting
Litdown (re)builds itself from the litdown.md file via the build script.
Table of contents
Litdown generates anchored headers and a linked table of contents from the headers in the literate Markdown source. The TOC is prepended to the source to create the README.md file, which is used to create the other documentation formats. Each header must be unique in order for the links to work properly. Anchor names are normalized versions of the headers and are inserted into the HTML. Authors may also leverage anchored headers to create cross-references in the form of internal links.
Templating
Litdown encourages the use of templates, which allow for content to be defined in one place and be embedded in multiple files or in multiple places in a file. See the template and template label syntax for more details.
Syntax
Litdown source is Markdown with a few specific conventions that facilitate its use as a literate programming tool.
Templates
A template consists of a header with a unique name which also contains at least one code block.
# Template name _"_backticks"Contents_"_backticks"
Template labels
Templates are transcluded into one or more templates by the use of a template label inside the destination template. The following would transclude the "Template name" template into the "Template labels" template.
_"Template name"
Notes about template labels
- If a template with the name given in the template label is not defined in the document, the template label is left as-is.
- Templates may contain successive template labels, but should never have a circular reference, as this would result in an infinite loop.
Embedded files
Files are templates with an associated file directive. The file directive is a special type of link with 2 requirements:
- Link target begins with '#'
- Link title begins with 'save:'
A file directive results in an internal link to the section and directs Litdown to create the file under the given path with the contents after resolving all template labels.
Standard Markdown link syntax is:
[text](href "title")
Litdown file directive syntax is:
[text](#path/to/file.ext "save:mode")
If the mode is omitted, and the title is just "save:"
, the default is 644
(rw-r--r--
).
Although the 'text' field can be anything, a good default is to make it the same as the 'href' field, except without the leading '#':
[path/to/file.ext](#path/to/file.ext "save:mode")
Internal links
Internal links are links to section headers in the same document. Note that the 'href' field starts with a '#'.
[text](#normalized-header-text "optional title")
Installation
Dependencies
Litdown depends on Node.js.
NPM
The official way to install Litdown is with NPM.
npm install -g litdown
Be sure to also install at least one of the supported backend modules:
Git
git clone https://github.com/qtfk/litdown
Archives
The following archive files are created automatically by the build script.
Building
Building Litdown isn't strictly necessary since everything is already built in the Git repository and the archives. However, you are welcome to rebuild it yourself. The build process uses a build script, which in turn uses the embedded Makefile. The files that are generated by this process are listed in the Resulting section.
Build dependencies
bin/build.sh
#!/bin/sh set -eo pipefail # exit on first failed command n=litdowncd $(dirname $0)/.. echo Running Litdown against $n.mdbin/$n.js -l $n.mdecho echo Processing $n/Makefilemake -C $necho echo Rearranging filesmv $n/$n.html $n/doc/mkdir $n/logmv $n/$n.json $n/log/echo echo Creating archivesrm -rf archivesmkdir archivestar cjf archives/$n.tbz $ntar czf archives/$n.tgz $nzip -qr archives/$n.zip $necho echo Replacing current filesrm -rf bin demo doc lib log tmv $n/* .rmdir $necho
Usage
Command line
# Usage _"_backticks"litdown -x [options] filelitdown -c [options] directory_"_backticks" # Options _"_backticks"-h, --help Show usage-v Increase verbosity-q Disable output_"_backticks" ## Mode options _"_backticks"-x Extract a Markdown file to a directory (default)-c Create a Markdown file from files in directory_"_backticks" ## Extract options _"_backticks"-l Save internal state to "litdown.json"-b name[,name] Preferred backend(s) (default: commonmark,marked,markdown-it)-B Show backend selection (use after -b to check)-s name[,name] Preferred syntax highlighter (default: highlightjs-cdn,none)-S Show syntax highlighter selection (use after -s to check)_"_backticks"
Browser
FAQ
-
Why does the CLI utility print the filenames in a seemingly "random" order?
Litdown creates the files in the same order that the file directives appear in the literate Markdown source, but in a non-blocking, asynchronous manner. Litdown prints each filename when the file has been successfully written.
-
Why doesn't Litdown support running code embedded in the literate Markdown source?
Read the Non-executing section.
Demonstration
GNU Autotools build system
This example uses the GNU build system, which has two major parts. The first part involves generating a "distribution", which to a user is normally an archive (gzipped tar file) containing the source code and necessary build system files, and requires the "autoconf" and "automake" tools. The second part uses the distribution to extract, compile and install the software, and requires a compiler ("gcc", "clang", ...) and the "make" tool.
The following steps should help you install these tools if your system does not already have them.
-
Install automake (and autoconf) to build the distribution. Since automake normally depends on autoconf in most package managers, it is not usually necessary to specify autoconf in the install command.
- Mac OSX + Homebrew:
brew install automake
- Debian/Ubuntu:
apt-get install automake
- Mac OSX + Homebrew:
-
Install a compiler and the "make" tool.
- Mac OSX:
xcode-select --install
- Debian/Ubuntu:
apt-get install build-essential
- Mac OSX:
Test-driven development
This example uses a test-driven development methodology, so each iteration involves the creation of a test and minimal build system files to get the test to compile and run. The first time each test runs, it should fail. Then the minimal amount of code is written so that the test succeeds. Each iteration is contained within a separate numbered directory to allow extracting the project at various stages without the use of a version control system. The example uses templates as a means to reduce repeating code between the iterations.
Demo templates
As discussed in the Templates section, templates enable reusing specific strings throughout one or more of the source code files but are not necessarily written to any file. It can be quite useful to put certain information in a template so that it is easy to find, change, or document. It also supports the "Don't Repeat Yourself" (DRY) concept. For example, the Version template allows including the "1.0" string into any source code file. Both of these templates below are included in the demo/1/configure.ac file file below.
Version
1.0
Email address
hello@hello.com
Write the first test
Given that we want to generate numbered iterations of each step in our development process, the files for this first step will be saved under the "demo/1" directory. In a regular project, it would probably be better to use a version control system (like Git) instead to save each iteration.
The "src" subdirectory will contain the source code files. Source code tests will go in files under the "src/t" subdirectory. Our tests will go in a C source file at "src/t/hello.c".
In order to instruct Litdown to create this structure and the C source code
file for the test, we use standard Markdown link syntax:
[text](href "title")
.
However for Litdown to recognize it as a directive to create the file, the
href must begin with "#" and the title must begin with "save:".
The "text" field becomes the hyperlink text.
The "href" field is converted into a local link to the heading with the same
name.
The "title" field is used to pass parameters to Litdown, but is ignored in the
woven output.
For example, the format used for our test's C source code file is
[demo/1/src/t/hello.c](#demo/1/src/t/hello.c "save:")
.
demo/1/src/t/hello.c
Our test includes some header files and a main function. In this case, template labels are used to have Litdown fetch the actual contents from other sections. Two template labels are used here: "demo/1/src/t/hello.c includes" and "demo/1/src/t/hello.c main body". Note that the labels may be anything you want, but they should uniquely reference a specific section. This is why the full path is used in these labels.
_"demo/1/src/t/hello.c includes" int
demo/1/src/t/hello.c main function body
The body of the main function requires a few variables to run the test, the test itself will be to run the currently nonexistent "src/hello" executable, perform any housekeeping steps, and prove that it runs successfully.
As you can see, the source of the main function body uses four more template labels. One of the benefits of the literate programming approach is that the source code can be broken into separate logical sections in a manner that supports human understanding.
_"demo/1/src/t/hello.c variables"_"demo/1/src/t/hello.c run the command"_"demo/1/src/t/hello.c clean up"_"demo/1/src/t/hello.c prove successful"
demo/1/src/t/hello.c variables
We'll save the command to be run in a string called "command". Note that the test will be run from the "src" directory and that we do not want to run some global "hello" executable, so we prepend "./" to the command.
char *command = "./hello";
demo/1/src/t/hello.c run the command
The popen function from the stdio.h header file is perfect for running another command and reading its output. Here we save the returned file pointer as "fp".
FILE *fp = popen (command, "r");
demo/1/src/t/hello.c clean up
We appropriately clean up by closing the "fp" file pointer and save the returned exit code to test whether the command ran successfully. We shift the returned exit code by 8 bits to get the actual exit value.
int e = pclose (fp) >> 8;
demo/1/src/t/hello.c prove successful
This assertion tests if the exit code was zero, which indicates that it ran successfully. However, since the "src/hello" executable doesn't exist yet (because its source doesn't even exist yet), it can't possibly be run. So we expect the assertion to fail.
demo/1/src/t/hello.c includes
This section demonstrates that the headings may be presented in any order desired by the author.
We used several functions that require the appropriate header files to be included from the standard library. The "stdio.h" header file provides declarations for the popen and pclose functions. The "assert.h" header file provides a declaration for the "assert" function.
Build the first test
Now that our first test is complete, we need to instruct the build system how to compile and run the test. The following three files get it done.
demo/1/src/Makefile.am
To inform the build system that we have a test and that we want to compile it
and run it when we execute the make check
command, we place the following
contents in the "src/Makefile.am" file.
The executables listed for "check_PROGRAMS" are compiled when make check
runs.
The executables listed for "TESTS" are ran when make check
runs.
The test is considered to have passed if the test executable exits with an exit
code of 0; otherwise the test failed.
The assert function aborts and returns non-zero if the test fails.
The GNU build system provides metrics on the outcome of the testing.
The files listed for "EXTRA_DIST" are added to the distribution when
make dist
is run.
The distribution is a gzipped tarball that is suitable for distributing a
project's source code to others.
It is better to share the distribution, because it automatically packages up
all of the files needed to easily compile and install the source code with the
familiar ./configure && make && sudo make install
commands.
Also it alleviates the "user" from installing automake and autoconf in order to
generate the configure script.
check_PROGRAMS = t/hello TESTS = $(check_PROGRAMS) EXTRA_DIST = t/hello.c
demo/1/Makefile.am
In the project root directory, we also need a Makefile.am file which merely
informs the build system to run make
inside the "src" subdirectory.
SUBDIRS = src
demo/1/configure.ac
It is possible to use the autoscan
command to generate a "configure.scan"
file which serves as a template for the "configure.ac" file.
In this case, we will just create the "configure.ac" file ourselves.
Regardless of how the "configure.ac" file is created, it is used by the
autoreconf -vi
command as a template to generate the configure script.
The following list addresses each line in the contents below:
- Defines the minimum version of autoconf to use.
- Initializes autoconf and defines a name, version number and email address for the project. Note that we use the template labels for the "Version" and "Email address" fields.
- Initializes automake with the following options:
- "foreign": Don't require standard GNU project files
- "subdir-objects": Enable the "src/Makefile.am" to address the tests in the "t" subdirectory
- "-Wall -Werror": Enable compiler warnings
- Generate the specified Makefiles
- Define where the source code is located
- Ensure that the specified header files are available on the user's system
- Check that the C compiler is available
- Check that the compiler can compile the C programming language
- Create the "configure" script and any other necessary files
AC_PREREQ([2.69])AC_INIT([Hello], [_"Version"], [_"Email address"])AM_INIT_AUTOMAKE([foreign subdir-objects -Wall -Werror])AC_CONFIG_FILES([Makefile src/Makefile])AC_CONFIG_SRCDIR([src])AC_CHECK_HEADERS([stdio.h assert.h])AC_PROG_CCAC_LANG_CAC_OUTPUT
Fail the first test
If you haven't run Litdown against the "litdown.md" file yet, you can do so now via "litdown litdown.md". This should produce the "litdown" directory in the current directory. Then run the following commands to generate and run the configure script, and run the test.
cd litdown/demo/1autoreconf -vi./configuremake check
The make check
output clearly indicates a failed test.
FAIL: t/hello============================================================================Testsuite summary for Hello 1.0============================================================================# TOTAL: 1# PASS: 0# SKIP: 0# XFAIL: 0# FAIL: 1# XPASS: 0# ERROR: 0============================================================================See src/test-suite.logPlease report to hello@hello.com============================================================================make[3]: *** [test-suite.log] Error 1make[2]: *** [check-TESTS] Error 2make[1]: *** [check-am] Error 2make: *** [check-recursive] Error 1
To investigate, we look at the contents in "src/test-suite.log":
=================================== Hello 1.0: src/test-suite.log=================================== # TOTAL: 1# PASS: 0# SKIP: 0# XFAIL: 0# FAIL: 1# XPASS: 0# ERROR: 0 .. contents:: :depth: 2 FAIL: t/hello============= sh: ./hello: No such file or directoryAssertion failed: (e == 0), function main, file t/hello.c, line 9.FAIL t/hello (exit status: 134)
Note the last 3 lines which state that the "./hello" file doesn't exist, and that the "e == 0" assertion failed.
Pass the first test
Now we write the minimal code needed to pass the first test. This requires one new file, "src/hello.c", and one modification to the "src/Makefile.am" file.
The rest of the files from the previous step are still needed but do not require any changes. These files just use a template label to pull the content from the previous step.
demo/2/src/hello.c
This is the minimal source for an executable that does nothing except run and exit with zero.
int
demo/2/src/Makefile.am
In order to instruct the build system to compile the new source file, we need to append a line to the "src/Makefile.am" file. Normally you might also see another line like "hello_SOURCES = hello.c", but this is not strictly necessary because make is smart enough to deduce that that the source for the "src/hello" executable is the "src/hello.c" file. Note that we have included the previous contents with a template label.
_"demo/1/src/Makefile.am"bin_PROGRAMS = hello
demo/2/src/t/hello.c
_"demo/1/src/t/hello.c"
demo/2/Makefile.am
_"demo/1/Makefile.am"
demo/2/configure.ac
_"demo/1/configure.ac"
Build and run the first test again
Run the following commands to observe the first test passing.
cd litdown/demo/2autoreconf -vi./configuremake check
The make check
output clearly indicates a passed test.
PASS: t/hello============================================================================Testsuite summary for Hello 1.0============================================================================# TOTAL: 1# PASS: 1# SKIP: 0# XFAIL: 0# FAIL: 0# XPASS: 0# ERROR: 0============================================================================make[1]: Nothing to be done for `check-am'.
Write the second test
So the first test really doesn't do anything, but it does get our entire project going. The next step is to get it to print "Hello!" to the screen. We start by inserting the second test to read the output between the calls to popen and pclose in the existing src/t/hello.c file. The rest of the files remain the same.
demo/3/src/t/hello.c
This file shares the same structure, but we change the main body template label and we will have to include another header file.
_"demo/3/src/t/hello.c includes" int
demo/3/src/t/hello.c main body
_"demo/1/src/t/hello.c variables"_"demo/1/src/t/hello.c run the command"_"demo/3/src/t/hello.c test the output"_"demo/1/src/t/hello.c clean up"_"demo/1/src/t/hello.c prove successful"
demo/3/src/t/hello.c test the output
The first test already proves that the executable exists and runs it via popen so that we can read its output. The second test should prove that the "src/hello" executable actually prints "Hello!" to STDOUT. This test establishes that the required output is "Hello!" followed by a newline and saves it in the expected variable. The match boolean represents whether the two strings match.
Each iteration of the loop compares the character read from the output, saved in the c char, to the same character in the expected string and saves that result to the match boolean. The loop starts with index integer i set to zero. It continues as long as all the characters read so far match, it hasn't reached end of file on the output, and we haven't reached the end of the expected string. At the end of each iteration, the next character is read from the output and the index i is incremented.
After the loop completes, if the output matched the expected string, two statements should be true: First, the index i should be equal to the length of the expected string. In other words, the entire expected string was compared. Second, the match boolean should still be true. If either of these conditions is false, then the output did not match the expected string.
As an aside, we could have placed both conditions inside a single assert, i.e.
assert (i == len && match);
.
However it is preferred to use separate asserts so that we can tell which one
failed.
Of course there are many possible algorithms to have tested this situation. This approach was chosen to minimize memory usage and be easy to understand, but it is not necessarily the most performant.
char *expected = "Hello!\n"; bool match = true; int i; char c = fgetc (fp); int len = strlen (expected); for (i = 0; match && c != EOF && i < len; i++)
demo/3/src/t/hello.c includes
The test the output code uses the strlen function and the bool type which requires that we include the string.h and stdbool.h header files, respectively.
_"demo/1/src/t/hello.c includes"
demo/3/src/hello.c
_"demo/2/src/hello.c"
demo/3/src/Makefile.am
_"demo/2/src/Makefile.am"
demo/3/Makefile.am
_"demo/2/Makefile.am"
demo/3/configure.ac
_"demo/2/configure.ac"
Build and run the second test
We can run the second test, which we again expect to fail initially, by running
autoreconf -vi; ./configure; make check
.
The output of make check
clearly indicates a failure.
FAIL: t/hello============================================================================Testsuite summary for Hello 1.0============================================================================# TOTAL: 1# PASS: 0# SKIP: 0# XFAIL: 0# FAIL: 1# XPASS: 0# ERROR: 0============================================================================See src/test-suite.logPlease report to hello@hello.com============================================================================make[3]: *** [test-suite.log] Error 1make[2]: *** [check-TESTS] Error 2make[1]: *** [check-am] Error 2make: *** [check-recursive] Error 1
The "src/test-suite.log" file specifies that the failing assertion was
i == len
.
This makes sense because it appears first and we can expect the loop to
terminate immediately since the first character from the output is the end of
file because there is no output yet.
Assertion failed: (i == len), function main, file t/hello.c, line 20.
Pass the second test
To pass the second test, we need only to change the src/t/hello.c file, replacing it with code that actually prints "Hello!" to STDOUT. The rest of the files remain the same.
demo/4/src/hello.c
The below is the minimal code that prints "Hello!" and a newline to STDOUT. The puts function prints a string and appends a newline, and is provided by the stdio.h header file.
int
demo/4/src/t/hello.c
_"demo/3/src/t/hello.c"
demo/4/src/Makefile.am
_"demo/2/src/Makefile.am"
demo/4/Makefile.am
_"demo/3/Makefile.am"
demo/4/configure.ac
_"demo/3/configure.ac"
Build and run the second test again
Now the make check
output shows a pass!
PASS: t/hello============================================================================Testsuite summary for Hello 1.0============================================================================# TOTAL: 1# PASS: 1# SKIP: 0# XFAIL: 0# FAIL: 0# XPASS: 0# ERROR: 0============================================================================make[1]: Nothing to be done for `check-am'.
We can further prove to ourselves that the program works by changing into the "demo/4/src" directory and running "./hello". You should see it output "Hello!"
$ cd demo/4/src$ ./helloHello!
Conclusion
This demonstration hopefully serves well as a simple example that support the various concerns and methodologies at play in a project. Some of those concerns include the choice of language, development and testing methodologies, tools, and so on. It is important is to realize that Litdown can be used to develop in any language and accommodate any methodology.
Core
Core file
lib/litdown.js
_"Core header"_"Core helper functions"_"CLI"_"Core function"_"Core options"_"Core footer"
Core function
{ try opt = opt ? : litdowndefaults; thisstate = readme: src toc: md: '# ' + opttocheader + '\n\n' blocks: contents: '_backticks': '```' '_tab': '\t' order: files: contents: {} order: mode: {} ; var backend = optbackend; // backend parsethisstate = backend; // prepend the toc to the readmethisstatetocmd += '\n';thisstatereadme = thisstatetocmd + thisstatereadme; // resolve filesthisstatefilesorder; // backend renderthisstate = backend;thisstatehtml = thisstatehtml ;thisstatehtml = thisstatetochtml + thisstatehtml;thisstatehtml = opthtmlheader + thisstatehtml + opthtmlfooter; // highlightthisstatehtml = opt; delete thisstatef;var r = thisstate;thisstate = {};ropt = opt;return r; catch e if ecode == 'MODULE_NOT_FOUND' var m = /''/; emessage += '. Install it via `npm install -g ' + m1 + '`.\n'; if optsilent return '<p>ERROR:</p><pre>' + + '</pre>'; throw e; }
Core options
litdownoptions = litdown { ; return litdown;}; litdowndefaults = silent: false toc: maxlevel: 2 header: 'Contents' top: '^' html: header: '<!DOCTYPE html>\n<html>\n<head>\n<style>\n' + '\tcode{\n' + '\t\tbackground-color: #f0f0f0;\n' + '\t\tpadding: 0px 2px;\n' + '\t\tborder: 1px solid #c0c0c0;\n' + '\t}\n' + '\tpre code{\n' + '\t\tdisplay: block;\n' + '\t}\n' + '\tpre{\n' + '\t\ttab-size: 4;\n' + '\t\twhite-space: pre-wrap;\n' + '\t}\n' + '\ta{\n' + '\t\ttext-decoration: none;\n' + '\t}\n' + '</style>\n</head>\n<body>\n' footer: '\t</body>\n</html>\n' lang_prefix: 'lang-' f: normalize: normalize encode: encode canary: canary header_state: header_state header_html: header_html link_state: link_state link_html: link_html code_state: code_state code_html: code_html ;
Core helper functions
var encodings = '&' 'amp' '<' 'lt' '>' 'gt' '"' 'quot'; { for var i = 0; i < encodingslength; i++ var re = encodingsi0 'g'; html = html; return html;} { var target; var key; for var i = 1; i < argumentslength; i++ target = argumentsi; for key in target if ObjectprototypehasOwnProperty objkey = targetkey; return obj;} { return s;} { if ! width width = processstdoutcolumns - 1 || 80; if ! indent indent = ''; var r = indent; var len = rlength; if text instanceof Array text = text; if text indent += ' '; text; r += '\n'; return r;} { return Math;} { var t = ; if stateblockscontentst == null stateblockscontentst = ''; stateblocksorder; if level <= tocmaxlevel for var i = 1; i < level; i++ statetocmd += ' '; statetocmd += '* [' + text + '](#' + t + ')\n'; return state;} { var t = ; var r = '<a name="' + t + '">' + text + '</a>'; if toc r += ' <a href="#' + + '">' + toctop + '</a>'; r = '<h' + level + '>' + r + '</h' + level + '>\n'; return r;} { if href && href href = href; if title && title if statefilescontentshref == null statefilescontentshref = ''; statefilesmodehref = title; statefilesorder; return state;} { if href && href href = '#' + ; var r = '<a href="' + href + '"'; if title r += ' title="' + title + '"'; r += '>' + text + '</a>'; return r;} { var t = stateblocksorderstateblocksorderlength - 1; if ! stateblockscontentst stateblockscontentst = code; return state;} { code = code; var r = '<pre><code'; if lang && prefix r += ' class="' + prefix + lang + '"'; r += '>' + + '\n</code></pre>\n'; return r;}
Core header
; {
Core footer
litdownstate = {};litdowncli = cli; if typeof module !== 'undefined' && typeof exports === 'object' moduleexports = litdown; else if typeof define === 'function' && defineamd ; else thislitdown = litdown; });
CLI
Litdown's command line interface is provided by the cli
function, which is
called by the CLI utility.
{ _"CLI modules"_"CLI helper functions"_"CLI configuration"_"CLI backend detection"_"CLI syntax highlighter detection"_"CLI usage"_"CLI process options"_"CLI error checking"_"CLI extract mode"_"CLI create mode" throw 'Invalid mode "' + cfgmode + '"!';}
CLI modules
var fs = ;var path = ;
CLI configuration
The cfg
object holds the configuration for the command line interface.
It is printed to STDOUT if the -v
option is used (at the bottom of
CLI process options).
It is also saved as state.cfg
in the litdown.json
file if the -l
option
is used.
var cfg = verbose: 1 encoding: 'utf8' backend: plugins: path preferred: 'commonmark' 'marked' 'markdown-it' highlight: plugins: path preferred: 'highlightjs-cdn' 'none' json: false mode: 'extract';
CLI backend detection
cfgbackendsupported = fs;if cfgbackendsupportedlength < 1 throw 'No backend plugins!';cfgbackendinstalled = cfgbackendsupported;for var i = 0; i < cfgbackendpreferredlength; i++ var n = cfgbackendpreferredi; if cfgbackendinstalled >= 0 cfgbackendselected = n; i = cfgbackendpreferredlength;
CLI syntax highlighter detection
cfghighlightsupported = fs;if cfghighlightsupportedlength < 1 throw 'No highlight plugins!';cfghighlightselected = cfghighlightpreferred0;
CLI usage
See also: Usage.
var usage = '\n' +'# Usage\n\n```\n' +'litdown -x [options] file\n' +'litdown -c [options] directory\n' +'```\n\n# Options\n\n```\n' +'-h, --help Show usage\n' +'-v Increase verbosity\n' +'-q Disable output\n' +'```\n\n## Mode options\n\n```\n' +'-x Extract a Markdown file to a directory (default)\n' +'-c Create a Markdown file from files in directory\n' +'```\n\n## Extract options\n\n```\n' +'-l Save internal state to "litdown.json"\n' +'-b name[,name] Preferred backend(s) (default: ' + cfgbackendpreferred + ')\n' +'-B Show backend selection (use after -b to check)\n' +'-s name[,name] Preferred syntax highlighter (default: ' + cfghighlightpreferred + ')\n' +'-S Show syntax highlighter selection (use after -s to check)\n' +'```\n';
CLI process options
var args = ;for var i = 0; i < argvlength; i++ var s = argvi; // Universal options // -h, --helpif s == '-h' || s == '--help' console; process; // -v else if s == '-v' cfgverbose++; // -q else if s == '-q' cfgverbose = 0; // Mode options // -x else if s == '-x' cfgmode = 'extract'; // -c else if s == '-c' cfgmode = 'create'; // Extract options // -l else if s == '-l' cfgjson = true; // -b else if s == '-b' var a = argvi + 1; if ! a a = ''; a = a; var b = ; for var j = 0; j < alength; j++ var n = aj; if ! n || n == '' ; return 0; else if cfgbackendsupported < 0 console; process; else if cfgbackendinstalled < 0 console; process; else b; cfgbackendpreferred = b; cfgbackendselected = cfgbackendpreferred0; i++; // -B else if s == '-B' ; return 0; // -s else if s == '-s' var a = argvi + 1; if ! a a = ''; a = a; var b = ; for var j = 0; j < alength; j++ var n = aj; if ! n || n == '' ; return 0; else if cfghighlightsupported < 0 console; process; else b; cfghighlightpreferred = b; cfghighlightselected = cfghighlightpreferred0; i++; // -S else if s == '-S' ; return 0; // Argument else args; ;
CLI error checking
if argslength < 1 console; return 1;if cfgbackendinstalledlength < 1 console; process;
CLI extract mode
if cfgmode == 'extract' var f = args0;;var n = path;fs; return 0;
CLI create mode
if cfgmode == 'create' var d = args0;var f = path + '.md';;if console; process;var list = ;var files = listfiles;var output = "# Files\n\n";var re = '^' + d + '/';for var i = 0; i < fileslength; i++ var fn = filesi; var mode = liststatfilesimode - 32768; if mode == '644' mode = ''; output += '* [' + fn + '](#' + fn + ' "save:' + mode + '")\n';for var i = 0; i < fileslength; i++ var fn = filesi; output += '\n## ' + fn + '\n\n```\n'; var d = fs; d = d; output += d; output = output + "\n```\n"; ;output += '\n';fs; return 0;
CLI helper functions
{ var r = files: dirs: stat: {} ; var f = fs; for var i = 0; i < flength; i++ if ! fi var n = path; var s = fs; if s rdirs; var t = ; for var j = 0; j < tfileslength; j++ rstattfilesj = tstattfilesj; rfiles; for var j = 0; j < tdirslength; j++ rdirs; else rstatn = s; rfiles; return r;} { d = path; f = f || {}; fs;} { var fn = path; if ! m m = 0644; else if m m = else throw 'Invalid file mode "' + m + '"!' + ' Must be three octal digits.'; ;} { try fs; catch e return e && ecode === 'EEXIST' ? true : false; } { console;} { console;} { return f } { if v >= n console } { } { }
CLI utility
The CLI utility is a small file that just loads the litdown module and passes the command line arguments to the cli function.
bin/litdown.js
#!/usr/bin/env node"use strict";;
Backend plugins
While the preferred backend is currently Commonmark due to its goal of standardization, Litdown's plugin system allows for the alternative use of either Markdown-it or Marked.
The backend plugin system is able to use any of the supported backend modules interchangeably. For example, it will automatically default to the one you have installed (if you have only one).
Backend plugins are written to smooth out any differences in the resulting
output from each of the backend modules.
This can be demonstrated via the backend benchmark script,
which processes litdown.md with each backend.
The diff
commands should not report any differences between the directories.
bin/backends.sh -kdiff -ru backends/{commonmark,marked}diff -ru backends/{commonmark,markdown-it}
The backend benchmark script can also be used to benchmark
the backend modules against each other on your system via bin/backends.sh
.
At this time, the rankings in terms of speed alone and given just one
iteration, are marked, commonmark, then markdown-it.
Note that you may wish to consider additional factors such as memory usage,
software license, philosophy, and so on in your choice of backend module.
commonmark real 0m0.111suser 0m0.111ssys 0m0.014s markdown-it real 0m0.147suser 0m0.126ssys 0m0.022s marked real 0m0.080suser 0m0.078ssys 0m0.013s
Backend plugins must have parse
and render
methods.
The parse
method accepts arguments src
, state
, and opt
.
The render
method accepts arguments state
and opt
.
The src
argument is the literate Markdown source.
The state
argument is the state object that the litdown function works on to
process the Markdown.
The opt
argument is the configuration of the litdown function.
The opt.f
object contains functions that aid the plugin's job.
The backend's parse method is called first to perform any initial setup and process the Markdown source into Litdown's state object. Then Litdown performs intermediate processing such as resolving the contents of files. Finally, the backend's render method is called to perform remaining processing, such as generating the HTML for the table of contents.
Backend plugin header
; {
Commonmark backend
Commonmark module
var commonmark = ;var parser = ;var renderer = ;
Commonmark parse function
{ src = src; stateast = parser; // traverse abstract syntax treevar walker = stateast;var e;var n;stateanchor = {};while e = walkernext if eentering n = enode; // Header if ntype === 'Header' var text = nfirstChildliteral; var level = nlevel; state = optf; var t = optf; if ! stateanchort stateanchort = text; nfirstChildliteral = '_' + '"_anchor:' + t + '"'; // Link else if ntype === 'Link' var title = ntitle; var href = ndestination; state = optf; if href ndestination = '#' + optf; // Code else if ntype === 'CodeBlock' var code = nliteral; //var lang = n.info; state = optf; return state; }
Commonmark render function
{ // use header_html? var h = opttocheader;var top = optf;statetochtml = renderer ; var html = renderer; delete stateast; // causes error when dumping state via -l to litdown.json // resolve anchorsvar re = '_"_anchor:([a-z0-9\-]+)"';var m;while m = re !== null var t = m1; var text = stateanchort; var r = '_"_anchor:' + t + '"' 'g'; html = html; // replace lang prefixre = '<code class="language-([^"]+)">';while m = re !== null html = html; statehtml = html; return state; }
lib/backend/commonmark.js
_"Backend plugin header"_"Commonmark module"_"Commonmark parse function"_"Commonmark render function"_"Backend plugin footer"
Markdown-it backend
Markdown-it module
var md = ;
Markdown-it parse function
{ var heading = '';var link = ''; var def = text: mdrendererrulestext; mdrendererrules { var text = tokensidxcontent; // Header if heading != '' var level = heading; heading = ''; state = optf; return optf; // Link else if link != '' var href = linkhref; var title = linktitle; link = ''; state = optf; return optf; return deftexttokens idx options env self;}; // Headermdrendererrules { heading = tokensidxtag; return '';};mdrendererrules { return '' } // Linkmdrendererrules { var a = tokensidxattrs; link = {}; for var i = 0; i < alength; i++ linkai0 = ai1; return '';}mdrendererrules { return '' } // Codemdrendererrules { var code = tokensidxcontent; var lang = tokensidxinfo; state = optf; return optf;}; src = src; statehtml = md; return state; }
Markdown-it render function
{ md = ; var heading = ''; var def = text: mdrendererrulestext; mdrendererrules { var text = tokensidxcontent; // Header if heading != '' var level = heading; heading = ''; return optf; return deftexttokens idx options env self;}; // Headermdrendererrules { heading = tokensidxtag; return '';};mdrendererrules { return '' } statetochtml = md; return state; }
lib/backend/markdown-it.js
_"Backend plugin header"_"Markdown-it module"_"Markdown-it parse function"_"Markdown-it render function"_"Backend plugin footer"
Marked backend
Marked module
var marked = ;
Marked parse function
{ var renderer = ; // Headerrenderer { state = optf; return optf;}; // Linkrenderer { state = optf; return optf;}; // Coderenderer { state = optf; return optf;}; src = src; statehtml = ; return state; }
Marked render function
{ var renderer = ; renderer { return optf;}; statetochtml = ; return state; }
lib/backend/marked.js
_"Backend plugin header"_"Marked module"_"Marked parse function"_"Marked render function"_"Backend plugin footer"
Backend plugin footer
var backend = parse: parse render: render ; if typeof module !== 'undefined' && typeof exports === 'object' moduleexports = backend; else if typeof define === 'function' && defineamd ; else thisbackend = backend; });
Highlight plugins
Highlight plugin header
; {
Highlightjs CDN highlight plugin
{ var p = 'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.4/';var css = p + 'styles/default.min.css';var js = p + 'highlight.min.js'; return html; }
lib/highlight/highlightjs-cdn.js
_"Highlight plugin header"_"Highlightjs CDN highlight plugin"_"Highlight plugin footer"
None highlight plugin
{ return html }
lib/highlight/none.js
_"Highlight plugin header"_"None highlight plugin"_"Highlight plugin footer"
Highlight plugin footer
if typeof module !== 'undefined' && typeof exports === 'object' moduleexports = highlight; else if typeof define === 'function' && defineamd ; else thishighlight = highlight; });
Miscellaneous
Entry point
index.js
moduleexports = ;
Make file
Makefile
n = litdown pdf = doc/$(n).pdf min = lib/$(n).min.js rm -f $(tex) rm -f $(pdf) rm -f $(min) rm -f doc/fig/*.png cd doc && pandoc ../$< -o $(n).pdf mkdir doc uglifyjs -c -m -o $@ $<
Notes
The PDF that pandoc creates from README.md has a couple problems:
- Doesn't convert the _backticks
sed -i '' 's/_''"_backticks"/```/g' $@
sed -i '' 's/^\\includegraphics{/\\includegraphics[height=0.75\\textwidth,width=0.75\\textwidth]{/g' $@
Makefile.old
n = litdown pdf = doc/$(n).pdf tex = doc/$(n).tex min = lib/$(n).min.js rm -f $(tex) rm -f $(pdf) rm -f $(min) rm -f doc/fig/*.png pandoc -s -o $@ $< sed -i '' 's/_''"_backticks"/```/g' $@ sed -i '' 's/^\\includegraphics{/\\includegraphics[height=0.75\\textwidth,width=0.75\\textwidth]{/g' $@ cd doc && pdflatex $n.tex >/dev/null && cd .. rm -f doc/$n.{aux,log,out} mkdir doc uglifyjs -c -m -o $@ $<
Benchmarks
bin/backends.sh
#!/bin/sh # purpose: runs each backend once against litdown.md # # usage: $1 [-k] # # options # -k Keep directories set -eo pipefailcd $(dirname $0)/..
Test input
t/circular1.md
# blah [blah](#blah "save:") _"_backticks"_"blah"_"_backticks"
t/circular2.md
# blah [blah](#blah "save:") _"_backticks"_"bleh"_"_backticks" # bleh _"_backticks"bleah ble blah bleh_"blah"_"_backticks"
Demo script
The main idea for the demonstration is for the reader to manually type out the necessary commands. However, the means to automate the entire demonstration can be appreciated.
bin/demo.sh
#!/bin/sh cd $(dirname $0)/../democd 1; run_test; view_log; cd ..cd 2; run_test; cd ..cd 3; run_test; view_log; cd ..cd 4; run_test; cd ..
Files
Embedded
The following files are embedded as templates within the litdown.md file.
- bin/backends.sh
- bin/build.sh
- bin/demo.sh
- bin/litdown.js
- demo/1/Makefile.am
- demo/1/configure.ac
- demo/1/src/Makefile.am
- demo/1/src/t/hello.c
- demo/2/Makefile.am
- demo/2/configure.ac
- demo/2/src/Makefile.am
- demo/2/src/hello.c
- demo/2/src/t/hello.c
- demo/3/Makefile.am
- demo/3/configure.ac
- demo/3/src/Makefile.am
- demo/3/src/hello.c
- demo/3/src/t/hello.c
- demo/4/Makefile.am
- demo/4/configure.ac
- demo/4/src/Makefile.am
- demo/4/src/hello.c
- demo/4/src/t/hello.c
- doc/fig/logical-diagram.dot
- index.js
- lib/backend/commonmark.js
- lib/backend/markdown-it.js
- lib/backend/marked.js
- lib/highlight/highlightjs-cdn.js
- lib/highlight/none.js
- lib/litdown.js
- Makefile
- t/circular1.md
- t/circular2.md
Resulting
Here are links to all of the files (generated by the Makefile, build script, or embedded in litdown.md).
- archives/litdown.tbz
- archives/litdown.tgz
- archives/litdown.zip
- bin/backends.sh
- bin/build.sh
- bin/litdown.js
- demo/1/Makefile.am
- demo/1/configure.ac
- demo/1/src/Makefile.am
- demo/1/src/t/hello.c
- demo/2/Makefile.am
- demo/2/configure.ac
- demo/2/src/Makefile.am
- demo/2/src/hello.c
- demo/2/src/t/hello.c
- demo/3/Makefile.am
- demo/3/configure.ac
- demo/3/src/Makefile.am
- demo/3/src/hello.c
- demo/3/src/t/hello.c
- demo/4/Makefile.am
- demo/4/configure.ac
- demo/4/src/Makefile.am
- demo/4/src/hello.c
- demo/4/src/t/hello.c
- doc/fig/logical-diagram.dot
- doc/litdown.html
- doc/litdown.pdf
- index.js
- lib/backend/commonmark.js
- lib/backend/markdown-it.js
- lib/backend/marked.js
- lib/highlight/highlightjs-cdn.js
- lib/highlight/none.js
- lib/litdown.js
- lib/litdown.min.js
- litdown.md
- log/litdown.json
- Makefile
- t/circular1.md
- t/circular2.md
Figures
Logical diagram
doc/fig/logical-diagram.dot
digraph { node[shape="box"] subgraph cluster_extract { label="Extract mode"; fontname="times bold"; color=black; style=dashed; emd -> eld; eld -> edir; edir -> er; edir -> ehtml; edir -> efiles; emd[label="Markdown (name.md)"]; eld[label="litdown [-x] name.md",shape=ellipse,fontname="times bold",style=filled]; edir[label="directory (name/)"]; er[label="README.md"]; ehtml[label="HTML (name.html)"]; efiles[label="files (*)"]; } subgraph cluster_space { label=""; color=white; a[label="",color=white]; } subgraph cluster_create { label="Create mode"; fontname="times bold"; color=black; style=dotted; cdir -> cld -> cmd; cdir[label="directory (name/)"]; cld[label="litdown -c name",shape=ellipse,fontname="times bold",style=filled]; cmd[label="Markdown (name.md)"]; }}
References
- Literate programming
- Markdown
- Node.js
- Node.js file system API
- Node.js path API
- Commonmark NPM
- Markdown-it NPM
- Marked NPM
- highlight.js
- Pandoc
- UglifyJS
- Test-driven development
- GNU build system
- Self-hosting
To do
commonmark issue with å!images assign figure # in the order seen use alt text as the caption link to the .dot file? how to convert doc/fig/*.dot > *.png via makefile w/o naming each one?calculate and output statistics about LOC, languages, etctest input test ~, ` code fences and 4 space indents in all backendscreate mode write to a stream? add option to include hidden, and/or specific globstest suite?highlighters other plugins? highlightjs <<< non-cdn? highlight command line tool line numbers?pandoc pdf: internal links, toc? man?in browser minify backend plugins? backend modules?parsing title? author? date? <<< is there a "standard" way? add references section to bottom? add numbering, footnotes, link to references...?multimedia images functions (mathml, tex, latex...) graphs (graphviz, gnuplot...)binary files in base64, json, buffer, xxd? 2 issues embed binary files in the Markdown source embed generated files and/or archive in an htmlinclude resolved files, original Markdown, in readme/html...compression? litdown -z file.md.gz litdown -cz dir >>> dir.md.gz like tar... z:gzip, Z:compress, j:bzip2, J:lzmaread stdin / write stdout? cat file.md |litdown -n file - litdown -c dir - >dir.md zcat file.md.gz |litdown -n file - litdown -c dir - |gzip >dir.md.gznpm init (package.json) optional dependencies? commonmark, marked, markdown-it, uglify-jscli options bundling full documentation: manpage? -d, --doc, --man add a --toc option to control the max level save/use a config file? ~/.litdown