darkforest-blocker
TypeScript icon, indicating that this package has built-in type declarations

0.4.1 • Public • Published

DarkForest Blocker

A framework-agnostic middleware for blocking specific user agents at the application level. Supports Express.js, Next.js, Node.js HTTP, and Vite.

Features

  • 🛡️ Block specific user agents using preset categories or custom patterns
    • We use ai.robots.txt to provide 46 AI-related user-agents to block.
  • 🔄 Redirect blocked requests to a specified URL
  • 🎯 Customizable HTTP status codes
  • 🔌 Framework adapters for:
    • Express.js
    • Next.js (Middleware & API Routes)
    • Node.js HTTP/HTTPS
    • Vite Dev Server
    • Vercel Edge Middleware

Installation

npm install darkforest-blocker

Usage

Express.js

import express from 'express';
import { createExpressBlocker } from 'darkforest-blocker/express';

const app = express();

app.use(createExpressBlocker({
  // Use preset categories
  presetCategories: ['ai-search-bots', 'ai-crawl-bots'],
  // Optionally add custom patterns
  customBlockedUserAgents: ['additional-bot-pattern'],
  redirectUrl: 'https://example.com/blocked',  // Optional
  statusCode: 403  // Optional, defaults to 403
}));

Next.js

Middleware (app directory or pages/_middleware.ts)

import { createNextBlocker } from 'darkforest-blocker/next';

const { middleware } = createNextBlocker({
  presetCategories: ['ai-search-bots']
});

export default middleware;

API Routes

import { createNextBlocker } from 'darkforest-blocker/next';

const { apiHandler } = createNextBlocker({
  presetCategories: ['ai-search-bots'],
  customBlockedUserAgents: ['custom-bot']
});

export default function handler(req, res) {
  return apiHandler(req, res, () => {
    // Your API route logic here
    res.status(200).json({ message: 'Hello World' });
  });
}

Node.js HTTP

import http from 'http';
import { createNodeBlocker } from 'darkforest-blocker/node';

const blocker = createNodeBlocker({
  presetCategories: ['ai-search-bots', 'ai-crawl-bots']
});

const server = http.createServer((req, res) => {
  blocker(req, res, () => {
    // Your request handling logic here
    res.end('Hello World!');
  });
});

server.listen(3000);

Vercel Edge Middleware

// middleware.ts
import { createVercelMiddleware } from 'darkforest-blocker/vercel';

export const middleware = createVercelMiddleware({
  presetCategories: ['ai-search-bots'],
  redirectUrl: '/blocked',  // Optional
  statusCode: 403  // Optional, defaults to 403
});

Vite

// vite.config.ts
import { defineConfig } from 'vite';
import { createViteBlocker } from 'darkforest-blocker/vite';

export default defineConfig({
  plugins: [
    createViteBlocker({
      presetCategories: ['ai-search-bots']
    })
  ]
});

Configuration Options

Option Type Required Default Description
presetCategories string[] No* - Array of preset categories to block ('ai-search-bots', 'ai-crawl-bots')
customBlockedUserAgents string[] No* - Array of custom user agent strings or regex patterns to block
redirectUrl string No null URL to redirect blocked requests to
statusCode number No 403 HTTP status code for blocked requests
exemptPaths string[] No [] Array of path patterns that should be exempt from blocking

* Either presetCategories or customBlockedUserAgents must be provided

Preset Categories

The package includes predefined categories of user agents to block:

  • ai-search-bots: Common AI-powered search engine bots
  • ai-crawl-bots: AI-powered web crawlers and scrapers

You can use these categories alone or combine them with custom patterns:

createExpressBlocker({
  // Use both preset categories
  presetCategories: ['ai-search-bots', 'ai-crawl-bots'],
  // Add your own patterns
  customBlockedUserAgents: [
    'custom-bot',
    'specific-crawler'
  ]
});

Path Exemption

The exemptPaths option allows you to specify paths that should be exempt from blocking, even if the user agent matches a blocked pattern.

WARNING: If redirecting to any path /redirected_to on your site, you must add /redirected_to to exemptPaths! (Or else the user-agent will enter an infinite redirect loop)

createExpressBlocker({
  presetCategories: ['ai-search-bots'],
  exemptPaths: [
    '^/health$',        // Exact match for /health
    '^/api/public/.*',  // Any path starting with /api/public/
    '/callback$'        // Any path ending with /callback
  ]
});

Path patterns are treated as regular expressions, so you can use regex patterns to match multiple paths or create more complex matching rules.

Response Handling

When a request is blocked:

  1. If redirectUrl is provided:

    • The request will be redirected to the specified URL (i.e., an AI landing page)
  2. If no redirectUrl is provided:

    • Returns a JSON response with:
      • HTTP status code (default: 403)
      • Error message explaining the block

License

MIT

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Package Sidebar

Install

npm i darkforest-blocker

Weekly Downloads

4

Version

0.4.1

License

MIT

Unpacked Size

61 kB

Total Files

83

Last publish

Collaborators

  • darkestforest