@altiplano/taskserver

Altiplano tasks server

A language model tasks api server. Serve tasks from simple yaml files

Install:

yarn global add @altiplano/taskserver
# or 
npm install -g @altiplano/taskserver

Usage

Create a task

mkdir tasks
cd tasks
touch mytask.yaml

The task content:

name: fix_json
model: open-llama-7B-open-instruct.ggmlv3.q5_1
modelConf:
  - nCtx: 512
inferParams:
  - topK: 40,
  - topP: 0.1,
  - temp: 0,
  - repeatPenalty: 1
template: |-
  ### Instruction: Fix this invalid json:

  {prompt}
  ### Response: (Answer only with json)

A task is a model with it's config, some inference parameters and a template. Doc: models conf params, inference params

Run the tasks server

Command

Use the following command:

taskserver path/to/models/dir path/to/tasks/folder model_to_run-optional

Example:

taskserver /an/absolute/path/to/models/dir task open-llama-7B-open-instruct.ggmlv3.q5_1.bin

This will load the model at startup if a model name is provided

As library

Run your own server using the library:

#!/usr/bin/env node

import { argv, exit } from "process";
import { useTaskServer } from "../dist/server.js";


function _runserver(modelsDirPath, tasksDir, loadModel) {
  const { app } = useTaskServer(
    tasksDir,
    [],
    {
      modelsDirPath: modelsDirPath,
      loadModel: loadModel,
    },
    true,
  );
  app.listen(5143, () => {
    console.log("Tasks server running on port 5143");
  });
}

Parameters of useTaskServer:

• tasksDir: string required: the directory where the tasks are
• routes: Array<(r: Router) => void>: optional extra routes. Check the inferserver doc
• params: InferServerParams: the server params: one is required: modelsDirPath, the path to the models directory
• verbose: boolean: verbosity (default false)

Execute a task

Hit the /task/execute endpoint with a prompt and a task name:

curl -X POST -H "Content-Type: application/json" -d '{"prompt": "{\"a\":1,}", \
   "task": "code/json/fix"}' http://localhost:5143/task/execute