npm

34 packages found

M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 2.2.07 months ago
M
Q
P

Description

A webpack plugin to output a robots.txt file

Keywords

Publisher

published 5.0.04 months ago
M
Q
P

Description

Awesome generator robots.txt

Keywords

Publisher

published 6.0.12 months ago
M
Q
P

Description

robots.txt agent with cache.

Keywords

Publisher

published 1.0.05 months ago
M
Q
P

Description

A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input

Keywords

Publisher

published 0.15.42 months ago
M
Q
P

Description

Check user agents against robots.txt files

Keywords

Publisher

published 1.0.03 years ago
M
Q
P

Description

Middleware for Express to add a robots.txt

Keywords

Publisher

published 1.0.13 years ago
M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 3.0.28 months ago
M
Q
P

Description

Robots.txt bounce middleware for koa

Keywords

Publisher

published 1.0.102 years ago
M
Q
P

Description

A proper robots.txt parser and combinator that works with eulalie

Keywords

Publisher

published 1.1.03 years ago
M
Q
P

Description

The grunt task for html-snapshots

Keywords

Publisher

published 1.0.2a year ago
M
Q
P

Description

Deney all robots express middleware

Keywords

Publisher

published 1.0.33 months ago
M
Q
P

Description

Rules related to checking for any SEO issues on the page given

Keywords

Publisher

published 1.0.92 years ago
M
Q
P

Description

robots.txt middleware for Kerouac.

Keywords

Publisher

published 0.4.28 months ago
M
Q
P

Description

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Keywords

Publisher

published 1.0.12 years ago
M
Q
P

Description

Express middleware to serve and generate robots.txt

Keywords

Publisher

published 0.4.010 months ago
M
Q
P

Description

Parses robots.txt files to provide meaningful, useful output as well as reporting syntax errors.

Keywords

Publisher

published 1.0.13 months ago
M
Q
P

Description

Kerouac robots exclusion components for Bixby.js.

Keywords

Publisher

published 0.0.28 months ago
M
Q
P

Description

Parse robot directives within HTML meta and/or HTTP headers.

Keywords

Publisher

published 0.4.02 years ago
M
Q
P

Description

Promise based parser for robots.txt files.

Keywords

Publisher

published 2.0.17 months ago