npm

35 packages found

M
Q
P

Description

Awesome generator robots.txt

Keywords

Publisher

published 7.1.015 days ago
M
Q
P

Description

A webpack plugin to output a robots.txt file

Keywords

Publisher

published 6.0.015 days ago
M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 2.2.09 months ago
M
Q
P

Description

A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input

Keywords

Publisher

published 0.16.3a day ago
M
Q
P

Description

robots.txt agent with cache.

Keywords

Publisher

published 1.0.07 months ago
M
Q
P

Description

Middleware for Express to add a robots.txt

Keywords

Publisher

published 1.0.13 years ago
M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 3.0.210 months ago
M
Q
P

Description

Check user agents against robots.txt files

Keywords

Publisher

published 1.0.03 years ago
M
Q
P

Description

The grunt task for html-snapshots

Keywords

Publisher

published 1.0.2a year ago
M
Q
P

Description

A proper robots.txt parser and combinator that works with eulalie

Keywords

Publisher

published 1.1.03 years ago
M
Q
P

Description

Rules related to checking for any SEO issues on the page given

Keywords

Publisher

published 1.0.92 years ago
M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 2.1.1413 days ago
M
Q
P

Description

Deney all robots express middleware

Keywords

Publisher

published 1.0.35 months ago
M
Q
P

Description

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Keywords

Publisher

published 1.0.12 years ago
M
Q
P

Description

robots.txt middleware for Kerouac.

Keywords

Publisher

published 0.4.210 months ago
M
Q
P

Description

Parses robots.txt files to provide meaningful, useful output as well as reporting syntax errors.

Keywords

Publisher

published 1.0.15 months ago
M
Q
P

Description

Kerouac robots exclusion components for Bixby.js.

Keywords

Publisher

published 0.0.210 months ago
M
Q
P

Description

Express middleware to serve and generate robots.txt

Keywords

Publisher

published 0.4.14 days ago
M
Q
P

Description

A sitemapping tool for crawlers

Keywords

Publisher

published 0.3.13 years ago
M
Q
P

Description

Parser for robots.txt

Keywords

Publisher

published 0.10.12 years ago