Table of Contents generated with DocToc
Symbolic crumbs are (for now) only allowed in leading position—unlike file system routes and URLs, you
may not write things like
x— any provided implicit scope
./x— this scope
../x— the directly enclosing scope
.*/x??) — this or any enclosing scope
y/x– the scope named
y(which may itself be written
Not yet sure about these:
~/x— the current module scope (?)
/x— the global scope (?)
Doing it this way seems logical, but has the disadvantage that referring to the global scope is made easier than referring to the module scope.
We also have to keep in mind that requiring users to write
/x to refer to a global variable will break
the analogy between global variables and global keywords—unless we want people to use, say,
parse-integer. Maybe there should be some kind of 'standard vocabulary' namespace, i.e. all
names that can be referred to without any kind of licensing qualification.
Marked with a
$ (Dollar sign; interpolation mark).
Indirect crumbs take the (serialized) value of the referred variable instead of the name itself; therefore,
bar: 42; x: foo/$bar will set
x to the value of
Semantic indentation is known to be 'hard to parse'; it has typically been done 'outside the grammar', using tricks that are not normally available within classical parsing setups (think BNF, Yacc, Bison...). To quote a recent paper:
Several popular languages, such as Haskell, Python, and F#, use the indentation and layout of code as part of their syntax. Because context-free grammars cannot express the rules of indenta- tion, parsers for these languages currently use ad hoc techniques to handle layout. These techniques tend to be low-level and operational in nature and forgo the advantages of more declarative specifications like context-free grammars. For example, they are often coded by hand instead of being generated by a parser generator.
Arabika takes a somewhat novel approach in parsing semantic indentation.
It is known to be possible to simplify parsing significant whitespace when meaningful indentations are turned into regular parsing tokens; this is the approach both Python and the much less popular parboiled parsing library take.
I'm not quite sure how Python's somewhat opaque implementation works—whether it acts on strings or whether abstract tokens are inserted into a parse tree—but parboiled definitely inserts Unicode code points into the source to be parsed. In this particular case, implementors have chosen to recruit a number of lesser-used and otherwise 'illegal' Unicode codepoints (all of which have a status of 'reserved') to function as 'anchors' within the transformed source text:
del_error: u/fdea ins_error: u/fdeb resync: u/fdec resync_start: u/fded resync_end: u/fdee resync_eoi: u/fdef eoi: u/ffff indent: u/fdd0 dedent: u/fdd1
For a while, i have considered to use this exact same solution: pick some characters that 'should not' occur in 'regular' source code and use that to signal indentation structure.
But then, what to do if such codepoints should inadvertently crop up in a source file? Well, i thought, you could always escape such occurrances, do all the parsing stuff, and when the AST is there, you unescape all those occurrances. But... that's (1) a real nuisance to do, because you have a lot of tiny source snippets that are handled all across your grammar, and (2) whatever means you use to escape regular occurrances of such characters, there can not be any guarantee that such escape sequences do not already occur inside, say, a string literal (where they were intended to signify something completely different, and were not meant to be mutated by the parser). For this reason, escaping is not an option.
Having implemented some form of indentation parsing for the n-th iteration this time round, it occurred
to me that i dislike the use of 'weird' codepoints to signal indentation steps. Sure, the computer won't
mind whether that's a
u/4e01 or a
u/fdd0 in your string, but i do—and certainly so when i print out
that string for diagnostic purposes. Using reserved codepoints mean your terminal output will be littered
with lots of
�—you know, that
Replacement Character guy. This is ugly, uninformative, and also
misleading, as it could also indicate an encoding error. You'd have to translate that string before printing
it. Not good.
But then i realized i have been looking the wrong direction all the time: What if, instead of trying to hide our tokens, as it were, we made it part of the Official Syntax? I mean, Arabika and all that Parser Combinators stuff has long been intended to lead to modular, dynamically redefinable grammars that mainly function as high-level-to-high-level code translators, so, importantly:
(1) If a particular choice of meta-codepoints conflicts with what you want to use for other purposes in your source, you can always choose to use another dialect (of indentation parsing) to avoid that conflict.
(2) What we're doing here already is source translation, and as such it wouldn't hurt to keep it both out of the closet and readable. In other words, if
if x > 0x += 1print x
is the language you enjoy writing stuff in, and that gets turned into
if x > 0 x += 1; printx;
wouldn't you be interested in the fact that at some point that same program surfaces as
【if x > 0【x += 1〓print x】】
or maybe as
↳if x > 0↳x += 1↦print x↱↱
(3) Take note that although you're writing code in an indentation-based language, you can anytime
insert code that is bracketed instead of indented—it makes no difference to the parser whether a ststement
if x > 0【x += 1〓print x】 was bracketed by the parser or by the programmer.
Idea: Repurpose Semantic Versioning to indicate Language number, Dialect number, and Dialect Version number. For example, Arabica gets designated language #1 (zero reserved for special purposes), and, say, FlowMatic Forth as language #42.
Each language numbers its own dialects, but there may be some conventions—say, to put all the basic
numerical literals into a module called
4-number. That is, Arabika number literals are dealt with as
1.4.x, and FlowMatic Forth number literals as
SemVer literals with no or one dot always refer to the current language; in order to
use dialects from
another language, you must provide a tripartite expression with two dots:
use 4 # use dialect #4 of the current languageuse 4* # same as `4`, `@.4`, and `@.4.*`use @4 # see aboveuse ^4.3 # short for `^@.4.3`use ^@4.3 # use a version compatible with this language, dialect 4, release 3use 42.4* # use latest or available version of language 42, dialect 4use 42** # use latest or available version of language 42, complete official edition
Minimal npm module:
1234-mydialectpackage.jsonmain.js # or other supported language
Dependencies regarding dialects not listed under
package.json/dependencies, but under
package.json/flowmatic/needs (or similar; TBD).