Universal PDP Scrapper
This is a universal PDP scrapper that can scrape any product detail page and extract the following information:
product_name?: string;
images?: string[];
height?: string;
width?: string;
depth?: string;
material?: string;
price?: string;
sku?: string;
artist?: string;
type?: Types;
product_url?: string;
source?: string;
glbs?: string[];
glb_to_use?: string;
'supporting-surface'?: 'floor' | 'wall';
Under the hood, it uses OpenAI to extract estimated information from the product page and Google Custom Search JSON API / SerpAPI to extract images from product pages using Google Image Search coupled with Ikea's product search API and some custom scrappers for some popular furniture websites.
How to use
1. Install
npm i universal-pdp-scrapper
2. Usage
import { UniversalPDPScrapper } from 'universal-pdp-scrapper';
// Initialize the client and set the API keys
// You can set API Keys using environment variables as well: check [.env.sample](./.env.sample)
const client = new UniversalPDPScrapper({
openaiApiKey: '',
openaiOrgId: '',
openaiModelId: '',
googleApiKey: '',
googleCseId: ''
});
const result = await client.scrape('https://www.ikea.com/us/en/p/jokkmokk-table-and-4-chairs-antique-stain-50211104/');
console.log(result);
3. Demo
Here's a demo of running the scrapper in a server environment integrated with a React app