search-lite
Simple NodeJS backend service pre-indexing search for SQLite that updates when DB entries are added or updated.
By pre-indexing the search, search time is drastically reduced by simply evaluating matrices of integers, rather than comparing strings.
Data sources can be located in both the database and also a byte stream that is provided by the app when requested by search-lite. This is useful for indexing text files that are external to the database such as a source code repository or document collection.
By performing the string matching once in advance rather than every time a search is requested, the process is both faster and reduces the server load required to perform the search.
The result is a lightweight process capable of great results.
This library is for SQLite, thus the name search-lite
Getting started
Installing
From within your project directory where package.json is found, simply run this.
npm install @roycethebiker/search-liteThis library performs search analysis in advance of a search request by indexing the searchable data as it is saved to the DB.
How It Works
SQLite Triggers are added to tables for data that the developer makes searchable. When new data is added or an update to that table is made, the triggers then log that the record in that table needs to be scanned.
The scanning of the data marked for searching is broken down into tags of three or more alphanumeric blocks and saved for indexing.
When a search is requested, the tags are located and then referenced back to the table and record of origin.
How To Use It
Constructor
Create your SQLite DB instance and pass it to search-lite with the second option of true to use the internal event timer. Use false if to use an external event timer that is controlled by the app.
this.Sql = new sqlite.Database('./my_app.db', (err) => {
if (err) {
console.error('Could not connect to database', err)
} else {
console.info('Connected to database');
this.searchLite = new SearchLite(this.Sql, true);
}
});Constructor Arguments
| db: sqlite.Database | The SQLite DB instance |
| useInternalEventTrigger: boolean | True to let search-lite manage its own events, false to call externally |
| activeScanLimit?: number | Optional, to limit the number events to process during a scan |
| delimiterSpannersV?: string | Optional, to set the Delimiter Spanner |
Registration
By registering an event trigger, the DB is modified to create event records when an update or insert happens. The registration is used by the event processor to perform the scanning of the data.
NOTE: This current version requires that the searching table use a PRIMARY INDEX column named id.
Simple registration of an indexing for changes made to the users table.
this.searchLite.registerEvent('users',
[ 'username', 'firstname', 'lastname' ],
'users');Arguments
{
tableName: string, // Name of table to receive SQL triggering
columnNames: string[], // Array of column names to scan when triggered
category: string // Isolate search results by category
}The Search
Searching the index is done by asking with a list of tags as an array of strings. The return includes the table names and tuples for the records that match.
this.searchLite.search([[ 'perl', 'bash', 'grep', 'pipe' ], (results: any) => {
console.log('searched for %s has returned %s', w, JSON.stringify(results));
}, 'marquis');Managed Tables
The constructor will create search-lite tables if they don't exist. These tables are how the indexing is managed.
The Tables
All tables that search-lite creates are prefixed with searchLite and that is implied for the table below.
| Table name | Function |
|---|---|
| Triggers | Maps events back to the origin of the registration |
| Events | SQL changes execute triggers that record the origin of the event in this table |
| Tags | Normalizer for searchable text blocks |
| Tables | Normalizer for table names |
| Categories | Normalizer for categories |
| Locators | Matrix of know scan results |
Delimiter Spanners
Normal search tags are only generated from unbroken strings of alphnumeric words. Delimiter spanners allow special characters to be validated as part of the unbroken words. It also allows the text following the spanner to begin it's own tag.
By setting the hypen as a delimiter spanner.
searchLite.delimiterSpanners('-');The word CFM-109 becomes tags CFM, CFM-, CFM-1, CFM-10, and CFM-109, but also 109.
The word ABCD-1234-EFGH-5678 becomes 38 tags, all tags that start with ABC, all tags that start with 123, EFG, and one two tags that start with 567.
Upgrading
After upgrading use scanAllData() to rebuild all tags and locators.
Support
Send emails to osgnuru@gmail.com
Buttons generated by Shields IO
Contributing
Anyone is welcome to fork this project on GitLab search-lite
Create a fork and submit a pull request and it will be reviewed.
Authors and acknowledgment
SQLite3 team!!! 🍺🍺🍺
License
Copyright (C) 2021 Silicon Tao Technology Systems
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; GPL-2.0-only.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to
Free Software Foundation
51 Franklin Street, Fifth Floor
Boston, MA 02110
USA
Project status
This is a new project that is being tested in PROD.
Feature requests are added in GitLab as issues.
Developing This Project
One-time only install and setup.
npm install --global np
npm i -g typescript -D
npm i -g typings -D
tsc --inittsc is configured to compile in package.json in scripts.
To compile
tscCommit to GitLab.com
git commit
git pushPublishing is done by np
npTo publish a beta version for testing a branch in development.
npm run beta