test-result-normalizer

A data normalizer

Getting Started

Install the module with: npm install test-result-normalizer

The goal of this project is to apply smart normalizing on some of the component tests.

Objective

• Look at the data/init_data.json data to see a sample of different test data.
• The final result should be similar to the data/final_data.json.
• The task at hand is to apply smart normalizing process on a given payload that can find "patterns" and convert it to a normalized form.
• The data/nomalize.numbers contains the set of the reporting name/test name and the expected normalized form (shaded in green).
• The normalization will be applied at two levels -

  Name normalization

  Unit normalization along with value conversion

• The unit conversions are harder as it would need a way to identify the universe of incoming unit types and the normalized value. A good reference to start with this is: http://www.hl7.de/download/documents/ucum/ucumdata.html
• Many tests in components will not have a normalized form - in these cases, just copy it over to allResults.
• The name matching will need some fuzzy matches applied.

To train every time the training-set changes:

data/training-set.json is the main file which houses the training data. As our normalization pipeline become better over time, we will be likely moving this from a file into something more scalable like hbase/cassandra/etc.

The lib/train/limdu-trainer.js will train on this set and the output of this training is cached in data/limdu-trained-classifier-string.json

Here are the steps to build the limdu-trained-classifier-string.json each time there is a change to training-set.json

var TrainingSet = require('./lib/train/limdu-trainer');
var ts = new TrainingSet();
var cls = ts.createClassifier();
ts.train();

// Save the file
ts.saveStringAsTrainedSet(function(err, res) { console.log('Trained and saved!', err, res);});

To use the classifier:

var testNameNormalizer = require('./lib/TestNameNormalizer');

testNameNormalizer.normalizeName('Hemoglobin count is awesome'); // => 'hemoglobin level'
testNameNormalizer.normalizeName('Hemoglobin count is a1c in reading'); // => 'hemoglobin a1c'
testNameNormalizer.normalizeName('Hemogobulin count isin reading'); // => '' Typo but cannot yet predict
testNameNormalizer.normalizeName('count of rbc'); // => 'red blood cell count'
testNameNormalizer.normalizeName('count of blood cells'); // => ''
testNameNormalizer.normalizeName('count of blood cells that are red'); // => 'red blood cell count'

To use the unit/value normalizer:

var testUVNormalizer = require('./lib/TestUnitValueNormalizer');

testUVNormalizer.normalizeUnitAndValue('hcv viral load', 'IU/mL', '3'); // => { normalized: true, unit: 'IU/mL', value: 3 }
testUVNormalizer.normalizeUnitAndValue('white blood cell count', 'x10-3', '7.3'); // => { normalized: true, unit: 'mcL', value: 7300 }
testUVNormalizer.normalizeUnitAndValue('white blood cell count', 'K/uL', '4.62'); // => { normalized: true, unit: 'mcL', value: 4620 }
testUVNormalizer.normalizeUnitAndValue('neutrophils', 'x10^9/L', '3.32'); // => { normalized: true, unit: '10^9/L', value: 3.32 }
testUVNormalizer.normalizeUnitAndValue('neutrophils', '10(9)/L', '3.32'); // => { normalized: true, unit: '10^9/L', value: 3.32 }
testUVNormalizer.normalizeUnitAndValue('white blood cell count', 'invalidUnit', '4.62'); // => { normalized: false, unit: 'invalidUnit', value: '4.62' }