Search results

adibenmat-sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

• parse
• sitemap
• xml
• robots.txt
• sitemaps
• crawlers
• webcrawler

adibenmatdev

published 2.1.19 • 3 years agopublished 2.1.19 3 years ago

env-robotstxt

generate robots.txt based on env content for static websites

• seo
• env
• robots
• robots.txt
• html
• vercel env
• netlify env

chenghuichao

published 1.0.4 • 2 years agopublished 1.0.4 2 years ago

robots-txt-component

Lightweight robots.txt parsing component without any external dependencies for Node.js.

• robots.txt
• robots
• robots-parser
• robots-exclusion-standard
• javascript
• js
• node-js
• robots-txt
• crawler
• crawlers

ibragim64

published 0.0.5-dev • 2 years agopublished 0.0.5-dev 2 years ago

@mastixmc/sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers. (Extended version by mastixmc)

• parse
• sitemap
• xml
• robots.txt
• sitemaps
• crawlers
• webcrawler

mastixmc

published 3.2.0 • 5 years agopublished 3.2.0 5 years ago

@firstandthird/robots

robots.txt text generator

• robots.txt

dawnerd

published 1.0.0 • 5 years agopublished 1.0.0 5 years ago

@mixmaxhq/express-robots

Express middleware for generating your robots.txt

• express
• robots
• robots.txt

mixmax-codeship

published 1.0.0 • 5 years agopublished 1.0.0 5 years ago

express-seo-robots

Simple robots.txt file generation for express

• express
• seo
• robots.txt

john-doherty

published 1.0.2 • 3 years agopublished 1.0.2 3 years ago

@drorgl/sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

• parse
• sitemap
• xml
• robots.txt
• sitemaps
• crawlers
• webcrawler

drorgl

published 2.1.17 • 4 years agopublished 2.1.17 4 years ago

express-robots.txt

Express middleware to handle different robots.txt responses on different domains

• express
• middelware
• robots.txt

sua

published 1.0.3 • 9 years agopublished 1.0.3 9 years ago

@ruhith/generate-robots-text

Generate Robots.txt file with your custom configs

• robots.txt
• cli
• react
• vue
• generate

ruhith

published 1.0.1 • 3 years agopublished 1.0.1 3 years ago

vuepress-plugin-robots

Robots.txt file generator plugin for vuepress

• vuepress-plugin
• robots.txt
• vue
• vuepress
• robots.txt generator for vuepress

hiyue

published 1.0.1 • 5 years agopublished 1.0.1 5 years ago

parse-robots

Promise based parser for robots.txt files.

• crawl
• web
• robots.txt
• seo
• robotstxt
• robots
• robots-txt
• sitemap

jaygoodfellow

published 3.2.0 • 5 years agopublished 3.2.0 5 years ago

@jarronb/site-mapper

Site mapper utilizing puppeteer to generate a xml sitemap

• parse
• sitemap
• xml
• robots.txt
• sitemaps
• crawlers
• webcrawler

nearaj

published 1.0.5 • 4 years agopublished 1.0.5 4 years ago

@passmarked/seo

Rules related to checking for any SEO issues on the page given

• html
• passmarked
• passmark
• security
• standards
• test
• suite
• seo
• robots
• robots.txt

passmarked

published 1.0.9 • 8 years agopublished 1.0.9 8 years ago

robots

Parser for robots.txt

• robots.txt
• robotstxt
• parser

ekalinin

published 0.10.1 • 7 years agopublished 0.10.1 7 years ago

@ryan-sandy/no-robots

Deney all robots express middleware

• no
• robots.txt

ryan-sandy

published 1.0.3 • 6 years agopublished 1.0.3 6 years ago

express-robots

Express middleware for generating your robots.txt

• express
• robots
• robots.txt

ashaffer88

published 0.1.6 • 8 years agopublished 0.1.6 8 years ago

webcheck-robots

A plugin for Robots Exclusion Standard for webcheck

• robots.txt
• webcheck

atd

published 0.1.0 • 8 years agopublished 0.1.0 8 years ago

@deskeen/web-builder-create-robots-txt

Web Builder Module - Create robots.txt file

• build
• website
• node.js
• robots.txt

deskeen

published 0.2.0 • 3 years agopublished 0.2.0 3 years ago

robogen

Robots.txt generator done right.

• robots.txt
• robots
• txt
• generator

dodekeract

published 0.0.2 • 8 years agopublished 0.0.2 8 years ago