Unicode Default Word Boundary
Implements the Unicode UAX #29 §4.1 default word boundary specification, for finding word breaks in multilingual text.
Use this to split words in text! Using UAX #29 is a lot smarter than the
\b
word boundary in JavaScript's regular expressions! Note that
character classes like \b
, \w
, \d
only work on ASCII
characters.
Usage
Import the module and use the split()
function:
const split = split; console;
Output:
[ 'The', 'quick', '(', '“', 'brown', '”', ')', 'fox', 'can’t', 'jump', '32.3', 'feet', ',', 'right', '?' ]
But that's not all! Try it with non-English text, like Russian:
[ 'В', 'чащах', 'юга', 'жил', 'бы', 'цитрус', '?', 'Да', ',', 'но', 'фальшивый', 'экземпляр', '!' ]
...Hebrew:
;
[ 'איך', 'בלש', 'תפס', 'גמד', 'רוצח', 'עז', 'קטנה', '?' ]
...nêhiyawêwin:
;
[ 'ᑕᐻ', 'ᒥᔪ ᑭᓯᑲᐤ', 'ᐊᓄᐦᐨ', '᙮' ]
...and many more!
More advanced use cases will want to use the findSpans()
function.
What doesn't work
Languages that do not have obvious word breaks, such as Chinese, Japanese, Thai, Lao, and Khmer. You'll need to use statistical or dictionary-based approaches to split words in these languages.
API Documentation
There are two exported function: split()
and findSpans()
.
split(text: string): string[]
split()
splits the text at word boundaries, returning an array of all
"words" from the text that contain characters other than whitespace.
See above for examples.
findSpans(text: string): Iterable<BasicSpan>
findSpans()
is a generator that yields successive basic spans from
the text. A basic span is a chunk of text that is guaranteed to
start at a word boundary and end at the next word boundary. In other
words, basic spans are indivisible in that there are no word
boundaries contained within a basic span.
A basic span has the following properties:
Note that unlike, split()
, findSpans()
does yield spans that
contain whitespace.
Example
Array.from(findSpans("Hello, world🌎!"))
Will yield spans with the following properties:
start: 0 end: 5 length: 5 text: 'Hello' start: 5 end: 6 length: 1 text: ',' start: 6 end: 7 length: 1 text: ' ' start: 7 end: 12 length: 5 text: 'world' start: 12 end: 14 length: 2 text: '🌎' start: 14 end: 15 length: 1 text: '!'
N.B.: findSpans()
may not yield plain JavaScript objects, as
shown above. The objects that findSpans()
yield will adhere to the
BasicSpan
interface, however what findSpans()
actually yields may
differ from simple objects.
Contributing and Maintaining
When maintaining this package, you might notice something strange.
index.ts
depends on ./src/gen/WordBreakProperty.ts
, but this file
does not exist! It is a generated file, created by reading Unicode
property data files, downloaded from Unicode's website.
These data files have been compressed and committed to this repository
in libexec/
:
libexec/
├── WordBreakProperty-12.0.0.txt.gz
├── compile-word-break.js
└── emoji-data-12.0.0.txt.gz
Note that compile-word-break.js
actually creates
./src/gen/WordBreakProperty.ts
!
./src/gen/WordBreakProperty.ts
How to generate When you have just cloned the repository, this file will be generated
when you run npm install
:
npm install
If you want to regenerate it afterwards, you can run the build script:
npm run build
License
TypeScript implementation © 2019 Eddie Antonio Santos. MIT Licensed.
The algorithm comes from UAX #29: Unicode Text Segmentation, an integral part of the Unicode Standard, version 12.0.