@totemorg/totem

TOTEM

TOTEM provides a web service with the following features:

• endpoint routing
• http/https service
• denial-of-service protection
• secure connections for web clients
• client profiles
• PKI encryption and authentication
• fault protected run states
• indexing, uploading, downloading and cacheing static files
• crud interface to mysql and neo4j databases
• task queuing and regulation
• watchdog periodic file polling and services
• task sharding
• file streams
• data fetching using various protocols
• smartcard reader
• site skinning

By default, TOTEM provides dataset, file, store, graph, folder and command endpoints at:

/DATASET.TYPE ? QUERY
/AREA/FILE ? QUERY
/AREA/STORE ? QUERY
/AREA/GRAPH ? QUERY
/AREA/ ? QUERY
/COMMAND ? QUERY

where the optional TYPE:

db | csv | txt | html | json 

returns the requested dataset in the specified form.

Default TOTEM COMMANDs:

task | ping | login | agent | users

shard tasks, test connections, validate sessions, interface with agents, and show on-line users.

Install

npm install @totemorg/totem		# install base service
npm install @totemorg/PLUGIN	# install optional plugin

Start

npm run start 					# Start totem
npm run start PLUGIN PLUGIN...	# Start totem with optional plugins

Manage

npm run	startdbs				# Start database servers
npm run ?env					# Show env variables
npm run ?config					# show configuration
npm run verminor				# Roll minor version
npm run vermajor				# Roll major version
npm run redoc					# Regen documentation
npm run setprot					# Configure for protected mode
npm run setdebug				# Configure for debugging mode
npm run setoper					# Configure for operational mode
npm run setprod					# Configure for production mode

Usage

Acquire, optionally configure and start a TOTEM server:

const TOTEM = require("@totemorg/totem").config({
	key: value, 						// set key
	"key.key": value, 					// indexed set
	"key.key.": value					// indexed append
}, sql => {
	console.log( sql ? "look mom - Im running!" : "something evil is lurking" );
});

where configuration keys follow ENUMS deep copy conventions.

Program Reference

Open/Close

## Modules

TOTEM

Provides a barebones web service. This module documented in accordance with jsdoc.

String

Array

TOTEM

Provides a barebones web service. This module documented in accordance with jsdoc.

Requires: module:enums, module:jsdb, module:securelink, module:dogs, module:http, module:https, module:fs, module:constants, module:cluster, module:child_process, module:os, module:stream, module:vm, module:crypto, module:mime, module:xml2js, module:toobusy-js, module:json2csv, module:js2xmlparser, module:cheerio
Author: ACMESDS
Example

// npm test T1
	// Create simple service but dont start it.
	Log({
		msg: "Im simply a Totem interface so Im not even running as a service", 
		default_fetcher_endpts: TOTEM.nodeRouters,
		default_protect_mode: TOTEM.guard,
		default_cores_used: TOTEM.cores
	});

	

Example

// npm test T2
	// Totem service running in fault protection mode, no database, no UI; but I am running
	// with 2 workers and the default endpoint routes.

	config({
		mysql: null,
		guard: true,
		cores: 2
	}, sql => {

		Log( 
	`I'm a Totem service running in fault protection mode, no database, no UI; but I am running
	with 2 workers and the default endpoint routes` );

	});

	

Example

// npm test T3
	// A Totem service with no workers.

	config({
	}, sql => {
		Log( 
	`I'm a Totem service with no workers. I do, however, have a mysql database from which I've derived 
	my startup options (see the openv.apps table for the Nick="Totem1") with just the standard endpoints.
	 `
		);
	});

	

Example

// npm test T4
	// Only 1 worker, unprotected, a mysql database, and two endpoints.

	config({
		nodeRouters: {
			dothis: function dothis(req,res) {  //< named handlers are shown in trace in console
				res( "123" );

				Log("", {
					do_query: req.query
				});
			},

			dothat: function dothat(req,res) {

				if (req.query.x)
					res( [{x:req.query.x+1,y:req.query.x+2}] );
				else
					res( new Error("We have a problem huston") );

				Log("", {
					msg: `Like dothis, but needs an ?x=value query`, 
					or_query: req.query,
					or_user: req.client
				});
			}
		}
	}, sql => {
		Log("", {
			msg:
	`As always, if the openv.apps Encrypt is set for the Nick="Totem" app, this service is now **encrypted** [*]
	and has https (vs http) endpoints, here /dothis and /dothat endpoints.  Ive only requested only 1 worker (
	aka core), Im running unprotected, and have a mysql database.  
	[*] If my NICK.pfx does not already exists, Totem will create its password protected NICK.pfx cert from the
	associated public NICK.crt and private NICK.key certs it creates.`,
			my_endpoints: T.nodeRouters
		});
	});

	

Example

// npm test T5
	// no cores but a mysql database and an anti-bot shield

	config({
		"login.challenge.extend": 20
	}, sql => {
		Log("", {
			msg:
	`I am Totem client, with no cores but I do have mysql database and I have an anti-bot shield!!  Anti-bot
	shields require a Encrypted service, and a UI (like that provided by DEBE) to be of any use.`, 
			mysql_derived_parms: T.siteContext
		});
	});

	

Example

// npm test T6
	// Testing tasker with database, 3 cores and an additional /test endpoint.

	config({
		guard: false,	// ex override default 
		cores: 3,		// ex override default

		"nodeRouters.": {  // define endpoints
			test: function (req,res) {
				res(" here we go");  // endpoint must always repond to its client 
				if (isMaster)  // setup tasking examples on on master
					switch (req.query.opt || 1) {  // test example runTask
						case 1: 
							T.runTask({  // setup tasking for loops over these keys
								keys: "i,j",
								i: [1,2,3],
								j: [4,5]
							}, 
								// define the task which returns a message msg
								($) => "hello i,j=" + [i,j] + " from worker " + $.worker + " on " + $.node, 

								// define the message msg handler
								(msg) => console.log(msg)
							);
							break;

						case 2:
							T.runTask({
								qos: 1,
								keys: "i,j",
								i: [1,2,3],
								j: [4,5]
							}, 
								($) => "hello i,j=" + [i,j] + " from worker " + $.worker + " on " + $.node, 
								(msg) => console.log(msg)
							);
							break;

						case 3:
							break;
					}

			}
		}

	}, sql => {
		Log( "Testing runTask with database and 3 cores at /test endpoint" );
	});

	

Example

// npm test T7
	// Conduct db maintenance

	config({
	}, sql => {				
		Log( "db maintenance" );

		if (isMaster)
			switch (process.argv[3]) {
				case 1: 
					sql.query( "select voxels.id as voxelID, chips.id as chipID from openv.voxels left join openv.chips on voxels.Ring = chips.Ring", function (err,recs) {
						recs.forEach( rec => {
							sql.query("update openv.voxels set chipID=? where ID=?", [rec.chipID, rec.voxelID], err => {
								Log(err);
							});
						});
					});
					break;

				case 2:
					sql.query("select ID, Ring from openv.voxels", function (err, recs) {
						recs.forEach( rec => {
							sql.query(
								"update openv.voxels set Point=geomFromText(?) where ?", 
								[ `POINT(${rec.Ring[0][0].x} ${rec.Ring[0][0].y})` , {ID: rec.ID} ], 
								err => {
									Log(err);
							});
						});
					});
					break;

				case 3:
					sql.query( "select voxels.id as voxelID, cache.id as chipID from openv.voxels left join openv.cache on voxels.Ring = fileCache.geo1", function (err,recs) {
						Log(err);
						recs.forEach( rec => {
							sql.query("update openv.voxels set chipID=? where ID=?", [rec.chipID, rec.voxelID], err => {
								Log(err);
							});
						});
					});
					break;

				case 4:
					sql.query("select ID, geo1 from openv.cache where bank='chip'", function (err, recs) {
						recs.forEach( rec => {
							if (rec.geo1)
								sql.query(
									"update openv.cache set x1=?, x2=? where ?", 
									[ rec.geo1[0][0].x, rec.geo1[0][0].y, {ID: rec.ID} ], 
									err => {
										Log(err);
								});
						});
					});
					break;

				case 5: 
					var parms = {
	ring: "[degs] closed ring [lon, lon], ... ]  specifying an area of interest on the earth's surface",
	"chip length": "[m] length of chip across an edge",
	"chip samples": "[pixels] number of pixels across edge of chip"
					};
					//get all tables and revise field comments with info data here -  archive parms - /parms in flex will
					//use getfileds to get comments and return into

				case 6:
					var 
						RAN from "../randpr"),
						ran = new RAN({
							models: ["sinc"],
							Mmax: 150,  // max coherence intervals
							Mstep: 5 	// step intervals
						});

					ran.config( function (pc) {
						var 
							vals = pc.values,
							vecs = pc.vectors,
							N = vals.length, 
							ref = vals[N-1];

						vals.forEach( (val, idx) => {
							var
								save = {
									correlation_model: pc.model,
									coherence_intervals: pc.intervals,
									eigen_value: val,
									eigen_index: idx,
									ref_value: ref,
									max_intervals: ran.Mmax,
									eigen_vector: JSON.stringify( vecs[idx] )
								};

							sql.query("INSERT INTO openv.pcs SET ? ON DUPLICATE KEY UPDATE ?", [save,save] );	
						});
					});
					break;	
			}
	});		

	

Example

// npm test T8
	// Conduct neo4j database maintenance

	const $ from "../man/man.js");
	config();
	neoThread( neo => {
		neo.cypher( "MATCH (n:gtd) RETURN n", {}, (err,nodes) => {
			Log("nodes",err,nodes.length,nodes[0]);
			var map = {};
			nodes.forEach( (node,idx) => map[node.n.name] = idx );
			//Log(">map",map);

			neo.cypher( "MATCH (a:gtd)-[r]->(b:gtd) RETURN r", {}, (err,edges) => {
				Log("edges",err,edges.length,edges[0]);
				var 
					N = nodes.length,	
					cap = $([N,N], (u,v,C) => C[u][v] = 0 ),
					lambda = $([N,N], (u,v,L) => L[u][v] = 0),
					lamlist = $(N, (n,L) => L[n] = [] );

				edges.forEach( edge => cap[map[edge.r.srcId]][map[edge.r.tarId]] = 1 );

				//Log(">cap",cap);

				for (var s=0; s<N; s++)
					for (var t=s+1; t<N; t++) {
						var 
							{cutset} = $.MaxFlowMinCut(cap,s,t),
							cut = lambda[s][t] = lambda[t][s] = cutset.length;

						lamlist[cut].push([s,t]);
					}

				lamlist.forEach( (list,r) => {
					if ( r && list.length ) Log(r,list);
				});

			});
		});
	});	

String

• String
  • ~linkify(ref) ⇐ String
  • ~mailify(ref) ⇐ String
  • ~parseXML(cb) ⇐ String

String~linkify(ref) ⇐ String

Expands "*"-tagged string into html link.

Kind: inner method of String
Extends: String

Param	Type
ref	String

Example

"test [abc](http:/junk)".linkify()
	==> 'test <a href="http:/junk">abc</a>'

	

Example

"here is a link".linkify("test")
	==> '<a href="test">here is a link</a>'

String~mailify(ref) ⇐ String

Expands string into a html email link.

Kind: inner method of String
Extends: String

Param	Type
ref	String

Example

"contact me".mailify("user.one")
	'<a href="mailto:contact me">user.one</a>'

String~parseXML(cb) ⇐ String

Parse XML string into json then callsback cb(json).

Kind: inner method of String
Extends: String

Param	Type	Description
cb	function	callback( json

Array

• Array
  • ~gridify(noheader)
  • ~groupify(dot)
  • ~blog(req, keys, cb)
  • ~joinify(cb)
  • ~stopService()
  • ~uploadFile(client, srcStream, sinkPath, tags, cb)

Array~gridify(noheader)

Creates an html table from an array.

Kind: inner method of Array

Param	Type	Description
noheader	Boolean	switch to enable header processing

Example

[{x:1,y:2},{x:10,y:20}].gridify()
	==> "<table border='0' width='100%' ><tr ></tr><tr ></tr></table>"

	

Example

[{x:1,y:2},{x:10,y:20}].gridify({x:"user",y:"value"})
	==> "<table border='0' width='100%' ><tr ><th >user</th><th >value</th></tr><tr ><td class='intro' >1</td><td >2</td></tr><tr ><td class='intro' >10</td><td >20</td></tr></table>"

	

Example

[{x:1,y:2},{x:10,y:20}].gridify({x:"user",y:"value"},{border:1})
	==> "<table border='1' ><tr ><th >user</th><th >value</th></tr><tr ><td class='intro' >1</td><td >2</td></tr><tr ><td class='intro' >10</td><td >20</td></tr></table>"

Array~groupify(dot)

Groups [ "x.y.z", "u,v", "x" , ... ] list entries into "x(y(z)), u(v), x, ..." string.

Kind: inner method of Array

Param	Type	Description
dot	string	item seperator

Example

["a.b","x.y.z"].groupify(".")
	==> 'a(b),x(y(z))'

Array~blog(req, keys, cb)

Blogs each string in the list.

Kind: inner method of Array
See: totem:blogify

Param	Type	Description
req	Object	Request context
keys	List	list of keys to blog
cb	function	callback(recs) blogified version of records

Array~joinify(cb)

Joins a list with an optional callback cb(head,list) to join the current list with the current head.

Kind: inner method of Array

Param	Type
cb	function

Example

[	a: null,
		g1: [ b: null, c: null, g2: [ x: null ] ],
		g3: [ y: null ] ].joinify()

	==>
		"a,g1(b,c,g2(x)),g3(y)"

Array~stopService()

Stop the server.

Kind: inner method of Array

Array~uploadFile(client, srcStream, sinkPath, tags, cb)

Uploads a source stream srcStream to a target file sinkPath owned by the specified client; optional tags are tagged to the upload and the callback cb is made if the upload was successful.

Kind: inner method of Array

Param	Type	Description
client	String	file owner
srcStream	Stream	source stream
sinkPath	String	path to target file
tags	Object	hash of tags to add to file
cb	function	callback(file) if upload successful

Contacting, Contributing, Following

Feel free to

• submit and status TOTEM issues
• contribute to TOTEM notebooks
• revise TOTEM requirements
• browse TOTEM holdings
• or follow TOTEM milestones

License

MIT

---

© 2012 ACMESDS