reach-ad-analyser-batch-processor

REACH Batch Article URL Analyser

Table of Contents:

• Local Development instructions
  • Environment service configuration
  • App Launch instructions
  • How To create a new environment file
  • Other environment variables
• Processing modes
• Code structure

Local Development instructions:

The application uses dotenv framework to inject environment variables within the application. which are passed with .env files in the root folder. As it stands there is within this project the folowing env files:

file	description
.env	local development
.env.demo	demo environment (currently not used)
.env.dev.Ad.Safety.analyser	ad safety analyser environment (currently not used)
.env.dev.appnexus	app nexus dev environment (currently not used)
.env.prod.appnexus	demo environment (currently not used)
.env.dev.BERTHA	bertha environment
.env.dev.stable	stable environment
.env.prod.reach	production environment
.env.sc.	(currently not used)
.env.telegraph	(currently not used)

Environment service configuration

Within each environment the following services must be configured:

• NLU: ``

App Launch instructions:

node app.js or npm start

to lauch with a specific environment, such as prod: NODE_ENV=prod.appnexus node app.js or bertha: NODE_ENV=dev.BERTHA node app.js

Which starts a server: http://localhost:6003 (see server log for port number)

How To create a new environment file:

To create a new environment file the following credentials are required: Obtain credentials obtained from the respective watson service:

• NLU:

    natural_language_understanding_apikey = << key from credentials in ibm console >> 
    natural_language_understanding_url = https://gateway-lon.watsonplatform.net/natural-language-understanding/api
    natural_language_understanding_version = 2019-02-01``

change url if there you the NLU instance is in different region than London.

• VR:

    visual_recognition_apikey = << key from credentials in ibm console >>  
    visual_recognition_url = https://gateway.watsonplatform.net/visual-recognition/api  
    visual_recognition_version = 2019-02-01  

change url if there you the VR instance is in different region than London.

• WDS:

    discovery_url = https://gateway-lon.watsonplatform.net/discovery/api  
    discovery_apikey = << key from credentials in ibm console >>  
    discovery_version = 2019-02-01  
    discovery_collectionid =  
    discovery_environmentid =

• Cloud storage bucket:
  Create a set of credentials for IBM cloud storage, with HMAC credentials, and convert the values to base 64 string with the following:

    cloud_storage_enpoint_url = https://s3.eu-gb.cloud-object-storage.appdomain.cloud  
    cloud_storage_apikey = << key from credentials in ibm console >>
    cloud_storage_resource_instance_id = << key from credentials in ibm console >>  
    cloud_storage_access_key_id = << key from credentials in ibm console >>  
    cloud_storage_secret_access_key = << key from credentials in ibm console >>  
    cloud_storage_bucket = << input storage bucket >>  
    cloud_storage_reports = << output storage bucket >>

• DB: Get the db credentials from the ibm console, connections are done as follows:

    postgreSQL_connectionString = postgres://user:[e-mail]:port/db?sslmode=verify-full
    postgreSQL_certificate_base64 = << pem ssl certificate string >>

Other environment variables

variable	description
write_to_db	enables writing nlu findings into the db, to be cached, the default value is present in the env file
read_from_db_cache	enables reading cached nlu findings from the db, the default value is present in the env file
write_to_log	create a CSV log file with a line for each analyzed article, containing the rating of the article
write_rules_to_log	adds also the rating of each rule. Applies only if write_to_log = true
write_to_cache	store result JSONs as files on server used as cache in future requests
analyze_images	enable/disable(s) analyzing images identified in HTML articles
recalculation_rate	recalculation rate for new rulesets
sleep_interval	time interval in seconds that the processor waits between new input files lookups, default values are in the env files
selected_process_mode	currently batch file processing mode is done through config in default.json, this defines the input/output format, and any filtering that might need doing. It defaults to default, and has with more modes
max_small_file_size	file size threshold in which it will process the whole file in one go instead of a stream, if not present it defaults to 20kb
articles_parallel	number of articles to process in parallel, if not present defaults to 30
NODE_ENV	used to change psql file to use test db for e2e tests
LOCAL_DEV	used to set the db to localhost, useful for local development

Processing modes:

Processing Mode config:

All existent processing modes are currently within the config/default.json under processMode.

The processing currently supports the following flags:

flag	description
name	name of the process mode
inputFormat	expected file format for the input
outputFormat	expected file format for the output file
saveArticles	saves articles as a list with format selected as above
saveReport	saves report from processed file, this includes the processed file, how many articles failed, total number processed, and status. Current saves report always in json
outputArticleErrors	if output format is json, it can output the error, for each article that fails, along with the input used, allowing better debugging
removeUnmatchedFilteredArticle	if true and there are matchers in articleFilterOutput and they return undefined (but not fail) it removes them from the output. the default is false
inputTransformation	uses Jexl framework to apply transforms to the input
articleFilterOutput	uses Jexl framework please see below for format.

Article Filter output format:

This part of the config allows allows to extract parts for the article response and filter it. It also allows to apply transforms to the data in the fields. It works by adding an array of key elements that should be present in the output. By adding them a key/value to output static values, or a matcher if we would like to filter parts of the original object only.

key	description
key	destination json key for the object (if format is csv, this value will be ommited)
value	value if we would like to output a static value associated with the key
matcher	Jexl matcher to filter objects
transforms	Jexl supports transforms to be added to a matcher, therefore, we add a static list of transformer fucntions in jexlTransforms.js class, and load them into jexl, to use them in order, just match the name of that function into the array

Please see default.json for transforms, and matcher examples and Jexl page.

If we need to add more transformer functions to the config, we add a new function into jexlTransforms

    const urlExtraneousRemoval = () => {
      return {
        name: 'urlExtraneousRemoval',
        method: urlString => URLStringBuilder.buildRemovingExtraneous(urlString)
      };
    };

Where the name returned is the name of the transform fucntion, and the method is the method to added to Jexl List of transforms. Then add the new transformer as an export

    module.exports = {
      urlExtraneousRemoval
    };

Then this function is available to be used in the transforms array for the inputTransformation array, articleFilterOutput.

Code structure:

The processor

The batch processor service is an express.js server app with no routing, its single goal is to pool for new files to be processed from ibm cloud storage. This works by starting the services in ./app.js and doing processor.init(). The service gets objects from ibm cloud storage by checking their name, and etag to inspect changes (in case the same file is uploaded multiple times).

• The server/processor/processor.js: is responsible for waiting and pooling new files from ibm cloudstorage to be analysed
• The server/processor/processorOrchestrator.js: is responsible for reading a file with multiple urls from ibm cloud storage, prepare a batch and save it, by either using a stream or an object put.
• The server/processor/controllers/processorOrchestrator.js: is responsible for reading a file with multiple urls from ibm cloud storage, and save it, by either using a stream or an object put.
• The server/processor/controllers/report/: contains controllers to create report or article streams.
• The server/processor/controllers/jsonFilter/: applies transfomrs and/or filtering to article objects as per processing mode configuration.
• The server/processor/controllers/objectOutputBuilder.js: is used to convert objects to their desired output format.
• The server/processor/controllers/storageCache.js: is used to build a local cache of processed cloud storage objects, as means to complement the db article_process table.

Once a batch ends up in the articleQueue file, it ends up being processed individually through brand-safety-tools orchestrator file.

Unit tests

Unit tests, are written using Jest framework, and can be run by doing npm test in the terminal. An HTML coverage report is available in link.

Folder structure goes as follows:

path	description
test	root test folder
test/data	sample files to run the batch processor locally, or to be used in the unit tests. These are purely for development/demo purposes
test/e2e	end to end tests, paths below attempt to mirror the path of the test file in server folder
test/helpers	files helped to setup the tests
test/mocks	reusable jest mock files
test/unit	unit tests

### Tests structure: A unit test should attempt to test one condition of the class/module it is testing. tests name should follow:

test('<method name>() <condition to test>, <expected return value>')

an example:

test('filter() with config, articleFilterOutput and removeUnmatchedFilteredArticle set to false, throws an error', () => {}

describe() should be aggregate tests in the following order of preference:

• aggregate tests within a test file.
• aggregate a complex/finicky logical scenario.
• aggregate tests around a method.
• a test file should never test more or have describes that test more than one file.