Captain Sitemap

Captain-sitemaps is an npm package, that acts as a high-level streaming sitemap generator. It simplifies JSON sitemap creation, making it easy for developers to organize and manage site navigation data.

Install the captain-sitemap npm package using the following command:

Installation

npm install captain-sitemap

How To Use

const crawlWebsite = require('captain-sitemap');

// Specify the website URL to crawl
const websiteUrl = 'https://example.com';

// Excluded File Extensions
const excludeFiles = ['ico', 'pdf'];

// Excluded Folder Extensions 
const excludeFolders = ['/images/', '/downloads/'];

// Call the crawlWebsite function with the specified URL
crawlWebsite(websiteUrl)
  .then((urls) => {
    // Do something with the crawled URLs
    console.log('Crawled URLs:', urls);
  })
  .catch((error) => {
    console.error('Error:', error.message);
  });

Default Excluded File Extensions

let excludeDefaultArray = [
    'gif', 'jpg', 'jpeg', 'png', 'ico', 'bmp', 'ogg', 'webp', 'mp4', 'webm',
    'mp3', 'ttf', 'woff', 'woff2', 'eot', 'json', 'rss', 'atom', 'gz', 'zip',
    'rar', '7z', 'css', 'js', 'gzip', 'exe', 'svg', 'xml', 'pdf'
];

Add new Excluded File Extensions

const excludeFiles = ['ico', 'pdf',...];

Default Excluded Folder Extensions

let excludeSubdirectories = [
    '/_next/'
];

Add new Excluded Folder Extensions

const excludeFolders = ['/images/', '/downloads/',...];