@redlink/squid

Squid - Supervising Quality in Information Discovery Frameworks

Squid is a tool that aims to help checking the quality of search frameworks. As of now it supports two types of tools, Solr and Elasticsearch. It provides two types of tests for quality checking:

• First N Results Test: Checks if a certain result is within a given range
• Similar to Query Test: Checks if the results of two queries are similar to each other

How it works

Squid can be installed locally by using

npm i -g @redlink/squid

Configuration

Squid takes a config file, which defines the adapter for the search framework and the tests that should be executed. The configuration is defined in JSON format as follows:

{
  "adapter": {
    "type": "...",
    ...
  },
  "tests": [
    {
      "type": "...",
      ...
    },
    ...
  ]
}

The adapter defines what search framework is used and provides a configuration for requests. The configuration for the adapters looks as follows:

{
  "type": "solr-adapter",
  "path": "https://solr-path.com/queryhandler",
  "credentials": {
    "username": "...",
    "password": "..."
  }
}

{
  "type": "elasticsearch-adapter",
  "host": "https://elastic-host.com/elastic",
  "index": "elastic-index",
  "credentials": {
    "username": "...",
    "password": "..."
  }
}

Read below how to use custom adapters.

Tests can look as follows:

{
  "type": "first-n-results-test",
  "query": {
    ...
  },
  "range": 5,
  "expectedResult": "my-expected-result",
  "field": "some-field-to-compare"
}

{
  "type": "similar-to-query-test",
  "query": {
    ...
  },
  "compareQuery": {
    ...
  },
  "range": 5,
  "idField": "some-id-field"
}

Solr Queries look as follows:

{
  "q": "some query",
  "fq": "optional parameter"
}

Elasticsearch are already specified in JSON and can be used the same way here, e.g.

{
  "query": {
    "match_phrase": {
      "field": "value"
    }
  }
}

Execution

Squid can then be executed like this:

squid execute <config-file> <directory-for-report-generation>

It takes the config file as first parameter, and it will generate a report in the specified directory. If none is specified, the report is added to the current working directory.

Results

Different tests produce different evaluations. The tests we have for now are:

FirstNResultsTest:

• Takes:
  • query: search query
  • expected result: the result you want
  • range: number of results to search in
  • field: field to compare (e.g. ID field)
• Produces:
  • score: 0 or 1, 0 if the result was not found an 1 if the result was found

SimilarToQueryTest

• Takes:
  • query: search query
  • compare query: query to compare the original query to
  • range: number of results to compare
  • id field
• Produces:
  • score: between 0 and 1
    How the score is calculated:
    • All results in result set of original query in result set of compare query: score = 1
    • No results of original query in result set of compare query: score = 0
    • Some results of original query in result set of compare query: For each result in the original query that is not in the result set of the compare query, (1/range) is subtracted from the original score of 1.
      • pos(result) < pos(compareQueryResult): score - range * (range - abs(diff(pos(result, compareQueryResult))))
      • pos(result) > pos(compareQueryResult): score + range * (range - abs(diff(pos(result, compareQueryResult))))

The report provides a summary of the evaluations. It contains the mean of all evaluation scores and the success rate of the evaluations. It also contains a list of all the evaluations.

Extending SQUID

SQUID can be extended with a custom adapter. For that, you need a local Node project with a class that extends SquidAdapter. It needs to implement the required methods execute(request: SquidRequest), which takes a SquidRequest and generates a SquidResponse, and configureAdapter(config: object), which should return an instance of your adapter. The export needs to be exports var adapter = new CustomAdapter(). An example of such a class would be:

class CustomAdapter extends SquidAdapter {
    
    // define your adapter properties (e.g. url, host, etc.)

    async execute(request: SquidRequest): Promise<SquidResponse> {
        // retrieve results for your query stored in request.query
        // transfrom results into SquidResponse
        // SquidResponse contains a list of SquidDocs consisting of field and field value of one result
    }

    configureAdapter(config?: object): SquidAdapter {
        // configure the properties of your adapter and return it
    }
}
// export your custom adapter like this: 
export var adapter = new CustomAdapter()

Using a custom adapter:

To use a custom adapter, provide the following configuration in your config file:

{
  "adapter": {
    "type": "custom",
    "file": "path/to/customAdapter"
  }
}

Important Notes:

• Do not use file extensions, e.g. use CustomAdapter instead of CustomAdapter.ts.
• If you are using TypeScript, provide the path to the transpiled version of your adapter, e.g. "dist/CustomAdapter".