41 packages found

M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 3.0.54 months ago
M
Q
P

Description

A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input

Keywords

Publisher

published 0.17.212 days ago
M
Q
P

Description

Awesome generator robots.txt

Keywords

Publisher

published 8.0.314 days ago
M
Q
P

Description

Express middleware for generating your robots.txt

Keywords

Publisher

published 1.0.05 months ago
M
Q
P

Description

A webpack plugin to output a robots.txt file

Keywords

Publisher

published 7.0.07 months ago
M
Q
P

Description

Express middleware to serve and generate robots.txt

Keywords

Publisher

published 0.4.110 months ago
M
Q
P

Description

highlight.js syntax definition for Google's robot.txt format

Keywords

Publisher

published 0.9.15 months ago
M
Q
P

Description

Middleware for Express to add a robots.txt

Keywords

Publisher

published 1.0.14 years ago
M
Q
P

Description

Check user agents against robots.txt files

Keywords

Publisher

published 1.0.04 years ago
M
Q
P

Description

The grunt task for html-snapshots

Keywords

Publisher

published 1.0.22 years ago
M
Q
P

Description

It uses the user-agents.org xml file for detecting bots.

Keywords

Publisher

published 1.0.105 years ago
M
Q
P

Description

Higher order Next.js configuration for generating sitemap.xml and robots.txt

Keywords

Publisher

published 2.5.27 days ago
M
Q
P

Description

A proper robots.txt parser and combinator that works with eulalie

Keywords

Publisher

published 1.1.04 years ago
M
Q
P

Description

Deney all robots express middleware

Keywords

Publisher

published 1.0.3a year ago
M
Q
P

Description

Parser for robots.txt

Keywords

Publisher

published 0.10.13 years ago
M
Q
P

Description

Rules related to checking for any SEO issues on the page given

Keywords

Publisher

published 1.0.93 years ago
M
Q
P

Description

Promise based parser for robots.txt files.

Keywords

Publisher

published 3.2.02 months ago
M
Q
P

Description

Express middleware for generating your robots.txt

Keywords

Publisher

published 0.1.64 years ago
M
Q
P

Description

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Keywords

Publisher

published 1.0.13 years ago
M
Q
P

Description

Robots.txt file generator plugin for vuepress

Keywords

Publisher

published 1.0.110 months ago