Search results

@crawlify.io/robots

Parses robots.txt files to provide meaningful, useful output as well as reporting syntax errors.

• robots.txt
• parse
• seo
• meta
• crawl
• scraper
• robots

_j4wx

published 1.0.1 • 5 years agopublished 1.0.1 5 years ago

@firstandthird/robots

robots.txt text generator

• robots.txt

dawnerd

published 1.0.0 • 4 years agopublished 1.0.0 4 years ago

express-robots.txt

Express middleware to handle different robots.txt responses on different domains

• express
• middelware
• robots.txt

sua

published 1.0.3 • 9 years agopublished 1.0.3 9 years ago

robotstxt

a robotstxt parser for node.js

• robots.txt
• parser
• web
• crawl
• seo

franzenzenhofer

published 0.0.4-7 • 12 years agopublished 0.0.4-7 12 years ago

@myagizmaktav/sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

• parse
• sitemap
• xml
• robots.txt
• sitemaps
• crawlers
• webcrawler

thefoxsnow

published 3.2.8 • a year agopublished 3.2.8 a year ago

vuepress-plugin-robots

Robots.txt file generator plugin for vuepress

• vuepress-plugin
• robots.txt
• vue
• vuepress
• robots.txt generator for vuepress

hiyue

published 1.0.1 • 5 years agopublished 1.0.1 5 years ago

parse-robots

Promise based parser for robots.txt files.

• crawl
• web
• robots.txt
• seo
• robotstxt
• robots
• robots-txt
• sitemap

jaygoodfellow

published 3.2.0 • 4 years agopublished 3.2.0 4 years ago

robots-agent

robots.txt agent with cache.

• robots.txt
• agent
• cache

gajus

published 1.0.0 • 5 years agopublished 1.0.0 5 years ago

robots-txt-component

Lightweight robots.txt parsing component without any external dependencies for Node.js.

• robots.txt
• robots
• robots-parser
• robots-exclusion-standard
• javascript
• js
• node-js
• robots-txt
• crawler
• crawlers

ibragim64

published 0.0.5-dev • 2 years agopublished 0.0.5-dev 2 years ago

robogen

Robots.txt generator done right.

• robots.txt
• robots
• txt
• generator

dodekeract

published 0.0.2 • 7 years agopublished 0.0.2 7 years ago

@jamsite/plugin-robots-txt

Creates robots.txt page

• jamstack
• jamstack-plugin
• robots.txt

tsertkov

published 0.0.2 • 4 years agopublished 0.0.2 4 years ago

microfrontier

A web crawler frontier implementation in TypeScript backed by Redis. MicroFrontier is a scalable and distributed frontier implemented through Redis Queues.

• Web Crawler
• Frontier
• Search Engine
• URL
• Redis
• Queue
• Spider
• Robots.txt

adileo

published 1.0.2 • 2 years agopublished 1.0.2 2 years ago

@deskeen/web-builder-create-robots-txt

Web Builder Module - Create robots.txt file

• build
• website
• node.js
• robots.txt

deskeen

published 0.2.0 • 3 years agopublished 0.2.0 3 years ago

adibenmat-sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

• parse
• sitemap
• xml
• robots.txt
• sitemaps
• crawlers
• webcrawler

adibenmatdev

published 2.1.19 • 3 years agopublished 2.1.19 3 years ago

next-with-sitemap

Higher order Next.js configuration for generating sitemap.xml and robots.txt

• next.js
• sitemap
• robots.txt
• webpack

cansin

published 2.5.2 • 4 years agopublished 2.5.2 4 years ago

@ruhith/generate-robots-text

Generate Robots.txt file with your custom configs

• robots.txt
• cli
• react
• vue
• generate

ruhith

published 1.0.1 • 3 years agopublished 1.0.1 3 years ago

@jarronb/site-mapper

Site mapper utilizing puppeteer to generate a xml sitemap

• parse
• sitemap
• xml
• robots.txt
• sitemaps
• crawlers
• webcrawler

nearaj

published 1.0.5 • 4 years agopublished 1.0.5 4 years ago

@drorgl/sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

• parse
• sitemap
• xml
• robots.txt
• sitemaps
• crawlers
• webcrawler

drorgl

published 2.1.17 • 4 years agopublished 2.1.17 4 years ago

robot-webpack-plugin

robots.txt generation for webpack 3

• robots.txt
• webpack3

aankur

published 0.1.0 • 6 years agopublished 0.1.0 6 years ago

sitemap-ts-advanced

Sitemap generator for TypeScript projects

• Sitemap
• Typescript
• Generator
• Robots
• Sitemap.xml
• Robots.txt

manuelmeister

published 0.2.0 • 2 years agopublished 0.2.0 2 years ago