npm

34 packages found

M
Q
P

Description

Awesome generator robots.txt

Keywords

Publisher

published 6.0.123 days ago
M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 2.2.06 months ago
M
Q
P

Description

A webpack plugin to output a robots.txt file

Keywords

Publisher

published 5.0.03 months ago
M
Q
P

Description

robots.txt agent with cache.

Keywords

Publisher

published 1.0.04 months ago
M
Q
P

Description

Middleware for Express to add a robots.txt

Keywords

Publisher

published 1.0.13 years ago
M
Q
P

Description

Check user agents against robots.txt files

Keywords

Publisher

published 1.0.03 years ago
M
Q
P

Description

A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input

Keywords

Publisher

published 0.15.422 days ago
M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 3.0.27 months ago
M
Q
P

Description

Robots.txt bounce middleware for koa

Keywords

Publisher

published 1.0.102 years ago
M
Q
P

Description

The grunt task for html-snapshots

Keywords

Publisher

published 1.0.2a year ago
M
Q
P

Description

Deney all robots express middleware

Keywords

Publisher

published 1.0.32 months ago
M
Q
P

Description

Rules related to checking for any SEO issues on the page given

Keywords

Publisher

published 1.0.92 years ago
M
Q
P

Description

robots.txt middleware for Kerouac.

Keywords

Publisher

published 0.4.27 months ago
M
Q
P

Description

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Keywords

Publisher

published 1.0.12 years ago
M
Q
P

Description

Express middleware to serve and generate robots.txt

Keywords

Publisher

published 0.4.09 months ago
M
Q
P

Description

Parses robots.txt files to provide meaningful, useful output as well as reporting syntax errors.

Keywords

Publisher

published 1.0.12 months ago
M
Q
P

Description

Kerouac robots exclusion components for Bixby.js.

Keywords

Publisher

published 0.0.27 months ago
M
Q
P

Description

Parse robot directives within HTML meta and/or HTTP headers.

Keywords

Publisher

published 0.4.02 years ago
M
Q
P

Description

A sitemapping tool for crawlers

Keywords

Publisher

published 0.3.12 years ago
M
Q
P

Description

Robots.txt rules

Keywords

Publisher

published 0.1.03 years ago