@exocet/pandora-elasticsearch

Pandora Elasticsearch

Addon to provide flow and setup functions to Elasticsearch persistence for Pandora.

npm install --save @exocet/pandora-elasticsearch

---

Concepts

Native ID

There are two approaches to index documents into Elasticsearch: using custom IDs or Elasticsearch generated IDs. Elasticsearch have some performance issues with custom IDs and how they suffer from lookup into Lucene index. To work around this issue this addon have two working modes per entity:

• Native ID off - With native ID off the data will be indexed with custom IDs, normally UUIDv4's generetad at application side (this feature is supported by this addon with zero effort).
• Native ID on - The addon operates with Elasticsearch generated IDs and is going to create an id field in the source of the documents to put the application generated custom IDs (here we can use the UUIDv4 auto generation feature too).

---

Setup

Available features:

• Create indexes with custom configuration and dynamic aliases
• Create mappings from entities definitions
• Provides input, output and AQS query parsers
• Flow steps: insert (index), update, delete, exists and search

To add this addon into your project, put the addon spec into your endpoint YAML:

kind: Pandora/endpoint
metadata:
  name: myEndpoint
spec:
  addons:
    - name: elasticsearch
      package: "@exocet/pandora-elasticsearch"
      flow: true
      setup: true
  configuration:
    elasticsearch: # Any configuration here will be passed to client configuration (1)
      node: http://127.0.0.1:9200
      searchOptions: # Optional options applied in all searches
        maxPageSize: 50 # Maximum allowed search page size
      indices:
        - entityName: myNamespace.myEntity # To create index and mapping for an entity, pass the entity name, the settings bellow are the defaults
          numberOfShards: 1
          numberOfReplicas: 1
          autoExpandReplicas: false
          refreshInterval: 1s
          maxResultWindow: 10000
          maxInnerResultWindow: 100
          checkOnStartup: false
        - entityName: myNamespace.myEntityWithCustomIndex
          index: my_entity_with_custom_index # Custom entity index name instead of autogenerated
        - index: my_index # You can create custom indexes
          mapping: # With custom mappings
            myField:
              type: keyword
          aliases: # With custom aliases
            my_alias: true # Normal alias
            my_filtered_alias: # Filtered alias (2)
              term:
                active: true

1. Client configuration
2. Filtered alias

After the setup the following property of the context will be available:

• .application.dbs.elasticsearch - The client connected to Elasticsearch

---

Hooks

The hooks created by this addon are:

• elasticsearchQueryParsers (.service.hooks.useHook('elasticsearchQueryParsers')) - Synchronous hook that parses the search AQS query into Elasticsearch search body.

   const [queryParsers] = this.service.hooks.useHook('elasticsearchQueryParsers');
   const queryParser = queryParsers(entityName);
   const {
   	body, // The bodybuilder instance with filters from AQS query (1)
   	includes // The included fields requested by AQS query
   } = queryParser(
   	aqsQuery,
   	{ // Optional options
   		body: bodybuilder(), // Existent instance of bodybuilder to build the filters (1)
   		nativeId: false // Indicates if the targed entity uses native id approach
   	}
   );

• elasticsearchOutputParsers (.service.hooks.useHook('elasticsearchOutputParsers')) - Synchronous hook that parses the search output from Elasticsearch (each hit object) into entity representation.

   const [outputParsers] = this.service.hooks.useHook('elasticsearchOutputParsers');
   const outputParser = outputParsers(entityName);
   const results = response.body.hits.hits.map(outputParser);

• elasticsearchInputParsers (.service.hooks.useHook('elasticsearchInputParsers')) - Synchronous hook that parses the input from entity data into Elasticsearch document source representation.

   const [inputParsers] = this.service.hooks.useHook('elasticsearchInputParsers');
   const inputParser = inputParsers(entityName);
   const documentToIndex = inputParser(entity);

1. bodybuilder

---

Flow

The provided flow type and steps are listed bellow:

• insert - This flow step index data into Elasticsearch:

   kind: Pandora/flowStep
   metadata:
     name: entityPersistence
     labels:
       operation: insert
   spec:
     type: elasticsearch # Flow step type
     options:
       operation: insert # Defines that is an insert operation
       inboundFrom: request.data # The path to the data to be indexed (acquired from execution context)
       idFrom: null # If you want to use a custom ID instead of UUIDv4 you can pass the path from the execution context to get the ID value
       indexName: null # If you're using custom index name, pass the name here
       nativeId: false # Enables or disables the Native ID use
       entity: # The entity to define the input parser hook
         namespace: myNamespace
         name: myEntity

• exists - This flow step check if an entity was indexed by ID:

   kind: Pandora/flowStep
   metadata:
     name: entityPersistence
     labels:
       operation: exists
   spec:
     type: elasticsearch
     options:
       operation: exists
       idFrom: request.data.id # The path of the ID
       outboundTo: null # The execution path of the context to put the found ID, is an useful flow step to put before update and delete flows to get the native ID before operating the entity
       nativeId: false # This is important, this option will instruct the flow step to search by _id field or the _source.id field
       indexName: null
       entity:
         namespace: myNamespace
         name: myEntity

• update - This flow step updates indexed entities:

   kind: Pandora/flowStep
   metadata:
     name: entityPersistence
     labels:
       operation: update
   spec:
     type: elasticsearch
     options:
       operation: update
       inboundFrom: request.data
       idFrom: request.data.id # The path of the ID to update entity
       indexName: null
       entity:
         namespace: myNamespace
         name: myEntity

• delete - This flow step deletes indexed entities:

   kind: Pandora/flowStep
   metadata:
     name: entityPersistence
     labels:
       operation: delete
   spec:
     type: elasticsearch
     options:
       operation: delete
       idFrom: request.data.id
       indexName: null
       entity:
         namespace: myNamespace
         name: myEntity

• search - This flow step searches for indexed entities using AQS:

   kind: Pandora/flowStep
   metadata:
     name: entityPersistence
     labels:
       operation: search
   spec:
     type: elasticsearch
     options:
       operation: search
       queryFrom: request.query # The execution context path that have the AQS querystring
       outboundTo: response # The execution context path to put the search result
       bodyFrom: null # The execution context path that have an existing bodybuilder instance to append the parsed AQS filters, pass null to indicate the flow step to generate a new instance
       indexName: null
       entity: # Defines the AQS parser and the output parser
         namespace: myNamespace
         name: myEntity