@polymath-ai/ingest

Polymath Ingest

This is a libraries that can be used to ingest data from a source and convert it into a format that can be used by the Polymath system.

Standalone ingestion tool

See: @polymath/cli

API

Extend Ingester for use in the CLI

You are able to extend the Ingester class to create your own importer.

1. Create a new class that extends Ingester
2. Implement the getStringsFromSource method a. getStringsFromSource should return an AsyncGenerator of partially filled out bits that is the full string of data from the source. b. The text property of the bit should be the full string of data from the source. The info property of the bit should be an object that contains any additional information about the bit. The info property is optional.

class MYRSS extends Ingester {

  constructor() {
    super();
  }

  async *getStringsFromSource(source: string): AsyncGenerator<Bit> {
    const feed = await (new RSSParser).parseURL(source);
    console.log(feed.title); // feed will have a `foo` property, type as a string

    for (const item of feed.items) {
      yield {
        text: item.content || "",
        info: {
          url: item.link || "",
          title: item.title || "",
        }
      };
    }
  }

}

E.g: > polymath ingest rss https://paul.kinlan.me/index.xml

Embed the Ingest process

It is possible to use the Ingest process directly in your own code, for example if you want to run a custom export as the embeddings are generated. The process will load the correct plugin and start processing it.

import { Ingest } from "@polymath/ingest";

const ingest = new Ingest({ /* options */ });

for await(const bit of ingest.run({ args, options, command })) {
  console.log(`${bit.url}: Embedding: ${bit.embedding}`);)  
}