npm

37 packages found

M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 2.2.0a year ago
M
Q
P

Description

robots.txt agent with cache.

Keywords

Publisher

published 1.0.09 months ago
M
Q
P

Description

Check user agents against robots.txt files

Keywords

Publisher

published 1.0.03 years ago
M
Q
P

Description

A proper robots.txt parser and combinator that works with eulalie

Keywords

Publisher

published 1.1.04 years ago
M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 2.1.142 months ago
M
Q
P

Description

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Keywords

Publisher

published 1.0.12 years ago
M
Q
P

Description

A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input

Keywords

Publisher

published 0.16.47 hours ago
M
Q
P

Description

Express middleware to serve and generate robots.txt

Keywords

Publisher

published 0.4.12 months ago
M
Q
P

Description

Awesome generator robots.txt

Keywords

Publisher

published 7.1.02 months ago
M
Q
P

Description

Robots.txt rules

Keywords

Publisher

published 0.1.03 years ago
M
Q
P

Description

A webpack plugin to output a robots.txt file

Keywords

Publisher

published 6.0.02 months ago
M
Q
P

Description

Parse robot directives within HTML meta and/or HTTP headers.

Keywords

Publisher

published 0.4.02 years ago
M
Q
P

Description

Deney all robots express middleware

Keywords

Publisher

published 1.0.37 months ago
M
Q
P

Description

Parser for robots.txt

Keywords

Publisher

published 0.10.12 years ago
M
Q
P

Description

Robots.txt generator done right.

Keywords

Publisher

published 0.0.23 years ago
M
Q
P

Description

Install robots.txt files on every site on a staging server, taking into account both PHP and node/stagecoach sites

Keywords

Publisher

published 0.1.22 years ago
M
Q
P

Description

Middleware for Express to add a robots.txt

Keywords

Publisher

published 1.0.13 years ago
M
Q
P

Description

The grunt task for html-snapshots

Keywords

Publisher

published 1.0.2a year ago
M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 3.0.2a year ago
M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers. (Extended version by mastixmc)

Keywords

Publisher

published 3.2.05 months ago