rag-aiquest

📖 AIQuest

Aiquest is an npm package that streamlines the process of parsing websites, splitting content into manageable chunks, embedding these chunks into machine-friendly vectors, and subsequently storing and retrieving these embeddings from AWS. This documentation outlines its design and possibilities.

🚀 Table of Contents

• 🔧 Design
• 🔍 Installation
• 🛠️ Usage
  • 🕸️ Parsing
  • ✂️ Chunking
  • 🧬 Embedding
  • ☁️ Storing on AWS
  • 🔎 Retrieval
• 📝 Examples
• 🌟 Future Enhancements
• 🤝 Contribution
• 🐛 Bug Reporting

🔧 Design

rag-aiquest integrates several utilities under one package:

UnifiedParser: For parsing content from URLs,PDF or Text File.

ChunkUtility: To split the parsed content into chunks.

EmbeddingUtility: Utilizes the OpenAI API to embed the chunks into vectors.

VectorStoreAWS: A utility for AWS operations related to embedding storage.

Retrival: Provides functionality to retrieve knowledge and run QnA.

🔍 Installation

npm install rag-aiquest

🛠️ Usage

🕸️ Parsing

Use the UnifiedParser to parse content from a URL.

const parser = new UnifiedParser();
const parsedValue = await parser.parse('YOUR_URL_HERE');

✂️ Chunking

To split the parsed content into chunks:

const chunks = ChunkUtility.splitIntoChunks(parsedValue, chunkSize, overlapSize);

🧬 Embedding

Embed chunks using OpenAI API.

const embedding = new EmbeddingUtility('YOUR_OPENAI_API_KEY');
const embedded = await embedding.createEmbedding(chunks);

☁️ Storing on AWS

To upload the embedded model to AWS:

const aws = new VectorStoreAWS(AWS_ACCESS_KEY_ID, AWS_ACCESS_SECRET, AWS_BUCKET_NAME);
await aws.uploadEmbededModeltoAWS(embedded, 'YOUR_FILE_NAME');

🔎 Retrieval

To retrieve and query the knowledge:

const knowledge = await aws.getKnowledgeData('YOUR_FILE_NAME');
const retrive = new Retrival('YOUR_OPENAI_API_KEY');
const search = await retrive.QnARetrival(knowledge, 'YOUR_QUERY');
console.log(search.choices[0].message);

📝 Examples

As given in the provided code, you can easily integrate the utilities to parse, chunk, embed, store, and retrieve knowledge.

🌟 Future Enhancements

Compression: Improve storage efficiency by compressing embedded vectors.

Batch Processing: Enhance the library to handle batch processing of URLs.

Support for More Embeddings: Plan to add support for other embedding APIs.

🤝 Contribution

If you wish to contribute to rag-aiquest, please refer to the CONTRIBUTING.md file.

🐛 Bug Reporting

Feel free to open an issue on GitHub if you find any bug.

⭐ Feature Request

• Feel free to Open an issue on GitHub to request any additional features you might need for your use case.
• Connect with me on LinkedIn. I'd love ❤️️ to hear where you are using this library.

📋 Release Notes

Check here for release notes.

🛠 Prerequisites

• Active OpenAI API Key
• AWS BUCKET SECRET_KEY,Bucket Name, AUTH KEY

💌 Support

If you encounter any issues or require further assistance, please reach out to our support team at [e-mail].