Newton's Programmatic Measurements

82 packages found

M
Q
P

robots.txt

exact match

Description

robots.txt middleware for express/connect to server up your robots.txt

Publisher

published 1.1.06 years ago
M
Q
P

Description

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Keywords

Publisher

published 3.0.59 months ago
M
Q
P

Description

A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input

Keywords

Publisher

published 0.17.24 months ago
M
Q
P

Description

Robots.txt parser.

Publisher

published 2.1.12 years ago
M
Q
P

Description

Express middleware for generating your robots.txt

Keywords

Publisher

published 1.0.010 months ago
M
Q
P

Description

Awesome generator robots.txt

Keywords

Publisher

published 8.0.35 months ago
M
Q
P

Description

Gatsby plugin that automatically creates robots.txt for your site

Keywords

Publisher

published 1.5.36 days ago
M
Q
P

Description

Express middleware to serve and generate robots.txt

Keywords

Publisher

published 0.5.0a month ago
M
Q
P

Description

Generates a robots.txt

Keywords

Publisher

published 1.1.1a year ago
M
Q
P

Description

robots.txt utilities

Keywords

Publisher

published 0.2.0a month ago
M
Q
P

Description

A webpack plugin to output a robots.txt file

Keywords

Publisher

published 7.0.0a year ago
M
Q
P

Description

Streaming parser for robots.txt files

Publisher

published 1.0.13 years ago
M
Q
P

Description

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

Keywords

Publisher

published 2.0.010 months ago
M
Q
P

Description

Validate urls against robots.txt rules.

Publisher

published 0.2.13 years ago
M
Q
P

Description

Middleware for Express to add a robots.txt

Keywords

Publisher

published 1.0.14 years ago
M
Q
P

Description

Check user agents against robots.txt files

Keywords

Publisher

published 1.0.04 years ago
M
Q
P

Description

A NuxtJS module thats inject a middleware to generate a robots.txt file

Publisher

published 2.4.2a year ago
M
Q
P

Description

highlight.js syntax definition for Google's robot.txt format

Keywords

Publisher

published 0.9.19 months ago
M
Q
P

Description

Hapi plugin to serve up robots.txt

Keywords

Publisher

published 3.2.0a year ago
M
Q
P

Description

Higher order Next.js configuration for generating sitemap.xml and robots.txt

Keywords

Publisher

published 2.5.24 months ago