DeepRails TypeScript SDK

Official TypeScript SDK for interacting with the DeepRails API v2.0.

DeepRails is a powerful developer tool with a comprehesive set of adaptive and accurate guardrails to protect against LLM hallucinations - deploy our Evaluate, Monitor, and Defend APIs in <15 mins for the best out-of-the-box guardrails in the market.

Installation

npm install deeprails

Quick Start

import { DeepRails } from 'deeprails';

// Initialize with your API token
const client = new DeepRails('YOUR_API_KEY');

// Create an evaluation
const evaluation = await client.createEvaluation({
  model_input: { user_prompt: 'Prompt used to generate completion' },
  model_output: 'Generated output',
  model_used: 'gpt-4o-mini',
  guardrail_metrics: ['correctness', 'completeness'],
});
console.log(`Evaluation created with ID: ${evaluation.eval_id}`);

// Create a monitor
const monitor = await client.createMonitor({
  name: 'Production Assistant Monitor',
  description: 'Tracking our production assistant quality',
});
console.log(`Monitor created with ID: ${monitor.monitor_id}`);

Features

• Simple API: Just a few lines of code to integrate evaluation into your workflow
• Comprehensive Metrics: Evaluate outputs on correctness, completeness, and more
• Real-time Progress: Track evaluation progress in real-time
• Detailed Results: Get detailed scores and rationales for each metric
• Continuous Monitoring: Create monitors to track AI system performance over time

Authentication

All API requests require authentication using your DeepRails API key. Your API key is a sensitive credential that should be kept secure.

# Best practice: Load token from environment variable
const token = process.env.DEEPRAILS_API_KEY as string;
const client = new DeepRails(token);

Evaluation Service

Creating Evaluations

try {
  const evaluation = await client.createEvaluation({
    model_input: { user_prompt: 'Prompt used to generate completion' },
    model_output: 'Generated output',
    model_used: 'gpt-4o-mini',
    guardrail_metrics: ['correctness', 'completeness'],
  });
  console.log(`ID: ${evaluation.eval_id}`);
  console.log(`Status: ${evaluation.evaluation_status}`);
  console.log(`Progress: ${evaluation.progress}%`);
} catch (e) {
  console.error(`Error: ${e}`);
}

Parameters

• model_input: Dictionary containing the prompt and any context (must include user_prompt)
• model_output: The generated output to evaluate
• model_used: (Optional) The model that generated the output
• run_mode: (Optional) Evaluation run mode - defaults to "smart"
• guardrail_metrics: (Optional) List of metrics to evaluate
• nametag: (Optional) Custom identifier for this evaluation
• webhook: (Optional) URL to receive completion notifications

Retrieving Evaluations

try {
  const eval_id = 'eval-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx';
  const evaluation = await client.getEvaluation(eval_id);

  console.log(`Status: ${evaluation.evaluation_status}`);

  if (evaluation.evaluation_result) {
    console.log('\nResults:');
    for (const [metric, result] of Object.entries(evaluation.evaluation_result)) {
      const score = (result as any).score ?? 'N/A';
      console.log(`  ${metric}: ${score}`);
    }
  }
} catch (e) {
  console.error(`Error: ${e}`);
}

Monitor Service

Creating Monitors

try {
  // Create a monitor
  const monitor = await client.createMonitor({
    name: 'Production Chat Assistant Monitor',
    description: 'Monitoring our production chatbot responses',
  });

  console.log(`Monitor created with ID: ${monitor.monitor_id}`);
} catch (e) {
  console.error(`Error: ${e}`);
}

Logging Monitor Events

try {
  // Add an event to the monitor
  const event = await client.createMonitorEvent({
    monitor_id: 'mon-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx',
    model_input: { user_prompt: 'Tell me about renewable energy' },
    model_output: 'Renewable energy comes from natural sources...',
    model_used: 'gpt-4o-mini',
    guardrail_metrics: ['correctness', 'completeness', 'comprehensive_safety'],
  });

  console.log(`Monitor event created with ID: ${event.event_id}`);
  console.log(`Associated evaluation ID: ${event.evaluation_id}`);
} catch (e) {
  console.error(`Error: ${e}`);
}

Retrieving Monitor Data

try {
  // Get monitor details
  const monitor = await client.getMonitor('mon-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx');
  console.log(`Monitor name: ${monitor.name}`);
  console.log(`Status: ${monitor.monitor_status}`);

  // Get monitor events
  const events = await client.getMonitorEvents({
    monitor_id: 'mon-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx',
    limit: 10,
  });

  for (const event of events) {
    console.log(`Event ID: ${event.event_id}`);
    console.log(`Evaluation ID: ${event.evaluation_id}`);
  }

  // List all monitors with filtering
  const monitors = await client.getMonitors({
    limit: 5,
    monitor_status: ['active'],
    sort_by: 'created_at',
    sort_order: 'desc',
  });

  console.log(`Total monitors: ${monitors.pagination.total_count}`);
  for (const m of monitors.monitors) {
    console.log(`${m.name}: ${m.event_count} events`);
  }
} catch (e) {
  console.error(`Error: ${e}`);
}

Available Metrics

• correctness: Measures factual accuracy by evaluating whether each claim in the output is true and verifiable.
• completeness: Assesses whether the response addresses all necessary parts of the prompt with sufficient detail and relevance.
• instruction_adherence: Checks whether the AI followed the explicit instructions in the prompt and system directives.
• context_adherence: Determines whether each factual claim is directly supported by the provided context.
• ground_truth_adherence: Measures how closely the output matches a known correct answer (gold standard).
• comprehensive_safety: Detects and categorizes safety violations across areas like PII, CBRN, hate speech, self-harm, and more.

Error Handling

The SDK throws DeepRailsAPIError for API-related errors, with status code and detailed message.

import { DeepRailsAPIError } from 'deeprails';

try {
  // SDK operations
} catch (e) {
  if (e instanceof DeepRailsAPIError) {
    console.error(`API Error: ${e.statusCode} - ${e.errorDetail}`);
  } else {
    console.error(`Unexpected error: ${e}`);
  }
}

Support

For questions or support, please contact [e-mail].