storescraper-ai

StoreScraper AI CLI

StoreScraper AI CLI is a powerful command-line tool for fetching, analyzing, and processing app reviews from Google Play Store and Apple App Store. It helps developers, product managers, and market researchers gain valuable insights from user feedback through AI-powered analysis.

The tool solves several key problems:

1. Automated review collection - Easily fetch reviews from both major app stores
2. AI-powered analysis - Extract actionable insights from user feedback
3. Bulk processing - Handle multiple apps efficiently
4. TAM analysis support - Extract app IDs from market research Excel files

This tool is aimed at:

• Mobile app developers seeking to improve their products
• Product managers tracking user sentiment
• Market researchers analyzing competitor apps
• QA teams identifying common issues

Overview

StoreScraper AI CLI is built with a modular architecture that separates concerns into distinct components:

Core Components

1. Command Layer (src/commands/)

   • Handles user input and command execution
   • Implements the CLI interface using Commander.js
   • Each command is in its own file for maintainability

2. Scraper Layer (src/utils/store-scraper.ts)

   • Fetches data from app stores using specialized scrapers
   • Handles platform detection and appropriate API calls
   • Normalizes data from different sources

3. AI Layer (src/ai/)

   • Processes and analyzes the fetched data using OpenAI
   • Implements use cases for different analysis types
   • Structures AI prompts and responses

4. Settings Layer (src/utils/settings.ts)

   • Manages user preferences and default settings
   • Provides persistence across command executions

Data Flow

User Input → Command Layer → Scraper Layer → Data Processing → AI Analysis → Output Generation

Key Technologies

• TypeScript - For type-safe code and better developer experience
• Commander.js - For CLI interface and command parsing
• Inquirer.js - For interactive prompts and user input
• OpenAI API - For AI-powered analysis of reviews
• Zod - For schema validation and type safety
• XLSX - For Excel file parsing in TAM analysis

Installation and Usage

Prerequisites

• Node.js (v14 or higher)
• npm (v6 or higher)
• OpenAI API key (for AI analysis features)

Global Installation (CLI)

npm install -g nanogiants-storescraper-ai

Local Installation (Library)

npm install nanogiants-storescraper-ai

Library Usage

You can use this package as a library in your Node.js, React, or NestJS applications:

import { StoreScraper } from 'nanogiants-storescraper-ai';

// Initialize with configuration
const scraper = new StoreScraper({
  openai: {
    apiKey: 'your-openai-api-key', // Required for AI features
    model: 'gpt-4-turbo' // Optional, defaults to gpt-4-turbo
  },
  defaultLanguage: 'en',
  defaultCountry: 'us',
  defaultReviewCount: 50
});

// Example: Fetch app info and reviews from Google Play
async function getAppInfo() {
  try {
    // Fetch app info and reviews
    const appInfo = await scraper.fetchGooglePlayApp('com.example.app', {
      reviewCount: 100,
      lang: 'en',
      country: 'us'
    });

    console.log(`App: ${appInfo.title}`);
    console.log(`Rating: ${appInfo.score}`);
    console.log(`Reviews: ${appInfo.reviews.length}`);

    // Analyze reviews with AI
    const analysis = await scraper.analyzeFeedback(
      appInfo.reviews.map(review => review.text),
      'english'
    );

    console.log('Analysis:', analysis);

    // Example: Bulk analysis of multiple apps from a CSV file
    const csvString = `Name;App Id;Mobile App Name
Company A;com.company.app;App Name
Company B;com.company.app2;App Name 2`;

    const result = await scraper.analyzeBulkFeedback(csvString, {
      outputPath: './output',
      platform: 'android',
      reviewCount: 50,
      language: 'en',
      country: 'us',
      generateDetailedReports: true
    });

    console.log(`Summary CSV saved to: ${result.summaryFilePath}`);
    console.log(`Processed ${result.summaryData.length} apps successfully.`);

    // Example: Validate Excel file before parsing
    const validationResult = scraper.validateTamExcel('./path/to/excel-file.xlsx');

    if (!validationResult.isValid) {
      console.error(`Error: ${validationResult.error}`);
      console.log('Available columns:', validationResult.columnNames.join(', '));
      return;
    }

    console.log('Excel file is valid!');
    console.log(`Name column: ${validationResult.nameColumn}`);
    console.log(`App column: ${validationResult.appColumn}`);
    if (validationResult.appNameColumn) {
      console.log(`App name column: ${validationResult.appNameColumn}`);
    }

    // Example: Parse Excel file to extract app IDs
    const parseResult = await scraper.parseTam('./path/to/excel-file.xlsx', {
      outputPath: './output',
      verbose: true
    });

    console.log(`Extracted ${parseResult.count} app IDs`);
    console.log(`Output saved to: ${parseResult.outputPath}`);

    // Access the extracted app data
    parseResult.data.forEach(app => {
      console.log(`${app.Name} - ${app['App Id']} (${app['Mobile App Name']})`);
    });
  } catch (error) {
    console.error('Error:', error);
  }
}

getAppInfo();

React Example

import { useState, useEffect } from 'react';
import { StoreScraper } from 'nanogiants-storescraper-ai';

function AppAnalyzer() {
  const [appInfo, setAppInfo] = useState(null);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState(null);

  // Initialize scraper (preferably in a service or context)
  const scraper = new StoreScraper({
    openai: {
      apiKey: process.env.REACT_APP_OPENAI_API_KEY
    }
  });

  async function analyzeApp(appId) {
    setLoading(true);
    try {
      const info = await scraper.fetchGooglePlayApp(appId);
      setAppInfo(info);
    } catch (err) {
      setError(err.message);
    } finally {
      setLoading(false);
    }
  }

  return (
    <div>
      {/* Your UI components */}
    </div>
  );
}

NestJS Example

import { Injectable } from '@nestjs/common';
import { StoreScraper } from 'nanogiants-storescraper-ai';
import { ConfigService } from '@nestjs/config';

@Injectable()
export class AppAnalysisService {
  private scraper: StoreScraper;

  constructor(private configService: ConfigService) {
    this.scraper = new StoreScraper({
      openai: {
        apiKey: this.configService.get<string>('OPENAI_API_KEY')
      }
    });
  }

  async analyzeApp(appId: string, platform: 'android' | 'ios') {
    if (platform === 'android') {
      return this.scraper.fetchGooglePlayApp(appId);
    } else {
      return this.scraper.fetchAppStoreApp(appId);
    }
  }

  async getFeedbackAnalysis(reviews: string[]) {
    return this.scraper.analyzeFeedback(reviews);
  }

  async validateExcelFile(filePath: string) {
    return this.scraper.validateTamExcel(filePath);
  }

  async parseExcelFile(filePath: string) {
    // First validate the Excel file
    const validationResult = await this.validateExcelFile(filePath);

    if (!validationResult.isValid) {
      throw new Error(`Invalid Excel file: ${validationResult.error}`);
    }

    // Then parse the Excel file
    return this.scraper.parseTam(filePath, {
      outputPath: './output/tam',
      verbose: false
    });
  }
}

### Local Installation

```bash
git clone https://github.com/nanogiants/storescraper-ai.git
cd storescraper-ai
npm install
npm link

Handling Secrets

This package requires an OpenAI API key for AI analysis features. There are several ways to provide this:

For CLI Usage:

1. Environment Variables: Set the OPENAI_API_KEY environment variable

   export OPENAI_API_KEY=your_api_key_here

2. Dotenv File: Create a .env file in your project root with your API key

   OPENAI_API_KEY=your_api_key_here
   

3. Settings Command: Use the settings command to save your API key

   storescraper settings

For Library Usage:

When using as a library, you must provide the API key in the configuration:

const scraper = new StoreScraper({
  openai: {
    apiKey: 'your_api_key_here'
  }
});

For security best practices:

• Never commit API keys to version control
• Use environment variables or secure secret management
• In frontend applications, consider using a backend proxy to make API calls

Examples

Check out the examples directory for complete working examples of how to use this package programmatically:

• Single App Analysis - Demonstrates how to fetch and analyze reviews for a single app (Deutsche Bahn)
• Bulk Analysis - Shows how to analyze multiple apps from a CSV file
• Simple Bulk Analysis - Demonstrates the simplified API for bulk analysis
• Parse TAM Excel - Shows how to extract app IDs from Excel files

Environment Setup

Create a .env file in the root directory with your OpenAI API key:

OPENAI_API_KEY=your_api_key_here

Available Commands

Feedback Command

Fetch and analyze reviews for a single app:

storescraper feedback [options]

Options:

• -p, --platform <platform>: Specify platform (android or ios)
• -c, --count <count>: Number of reviews to fetch (default: 50)
• -o, --output <path>: Output directory for the markdown file
• -l, --lang <lang>: Two letter language code (default: en)
• -C, --country <country>: Two letter country code (default: us)
• -s, --sort <sort>: Sort reviews by: newest, rating, or helpful (default: newest)
• -a, --analyze <boolean>: Analyze feedback with AI after fetching (true/false)

Example:

storescraper feedback -p android -c 100 -l de -C de -s newest -a true -o my-app

Output:

• Markdown file with app information and reviews
• Analysis markdown file (if analyze option is enabled)
• summary.csv file with the following columns:
  • score: Average review score of the app
  • total reviews: Total number of reviews in the store
  • name: Name of the app
  • id: App ID (package ID or bundle identifier)
  • link: URL to the app in the app store
  • company name: Developer/company name
  • min installs: Minimum number of installs (Android only, "N/A" for iOS)
  • max installs: Maximum number of installs (Android only, "N/A" for iOS)
  • description: App description
  • initial release: Initial release date
  • last release: Last update date
  • requested feature1, feature2, feature3: Top requested features from the AI analysis
  • issue1, issue2, issue3: Top issues identified in the AI analysis

Feedback Bulk Command

Process multiple apps from a CSV file:

storescraper feedback-bulk [options]

The command includes an interactive file picker that allows you to:

• Navigate through directories
• Go up to parent directories using the ".." option
• View only directories and CSV files
• Select a CSV file by pressing Enter

Options:

• -i, --input <path>: Path to the input CSV file with app IDs
• -c, --count <number>: Number of reviews to fetch (default: 50)
• -l, --lang <code>: Two letter language code (default: en)
• -C, --country <code>: Two letter country code (default: us)
• -s, --sort <sort>: Sort reviews by: newest, rating, or helpful (default: newest)
• -a, --analyze <boolean>: Analyze feedback with AI after fetching (true/false)
• -o, --output <path>: Output directory for the markdown files
• -d, --delay <ms>: Delay between requests in milliseconds (default: 1000)

Example:

storescraper feedback-bulk -i app-ids.csv -c 50 -l en -C us -a true -d 2000 -o output

Input CSV Format:

Name;App Id;Mobile App Name
App Name 1;com.example.app1;App Display Name 1
App Name 2;123456789;App Display Name 2

Output:

• Markdown files with app information and reviews for each app
• Analysis markdown files (if analyze option is enabled)
• A single summary.csv file with the following columns:
  • score: Average review score of the app
  • total reviews: Total number of reviews in the store
  • name: Name of the app (uses Mobile App Name if available)
  • id: App ID (package ID or bundle identifier)
  • link: URL to the app in the app store
  • company name: Developer/company name
  • min installs: Minimum number of installs (Android only, "N/A" for iOS)
  • max installs: Maximum number of installs (Android only, "N/A" for iOS)
  • description: App description
  • initial release: Initial release date
  • last release: Last update date
  • requested feature1, feature2, feature3: Top requested features from the AI analysis
  • issue1, issue2, issue3: Top issues identified in the AI analysis

Parse TAM Command

Extract app IDs from Excel files containing app URLs:

storescraper parse-tam [options]

Options:

• -i, --input <path>: Path to the input Excel file or directory
• -o, --output <path>: Path to save the output CSV file(s)
• -d, --delimiter <char>: Delimiter for the output CSV file (default: ;)
• -v, --validate-only: Only validate the Excel file without parsing (default: false)

Example:

storescraper parse-tam -i tam-data.xlsx -o output
storescraper parse-tam -i tam-directory -o output
storescraper parse-tam -i tam-data.xlsx -v  # Validate only

Expected Excel format:

• Column named "Name" or similar for company/app names
• Column named "Company Mobile App" or similar containing app URLs

The command will first validate the Excel file to ensure it has the required columns. If validation fails, it will display the available columns and suggest possible matches.

Output CSV format:

• Name: Company or organization name
• App Id: The app identifier (package name for Android, numeric ID for iOS)
• Mobile App Name: The name of the mobile application

Analyze Command

Analyze existing review text:

storescraper analyze [options]

Options:

• -f, --file <path>: Path to file containing feedback to analyze
• -o, --output <path>: Path to save the analysis results
• -l, --language <language>: Language for the analysis output (e.g., english, german, spanish)

Example:

storescraper analyze -f reviews.txt -o analysis.md -l german

Settings Command

Configure default settings:

storescraper settings [options]

Options:

• -r, --reset: Reset settings to default values

Output Structure

All output is saved in the output directory by default, with the following structure:

output/
  ├── app-name-1/
  │   ├── App-Name-1-android-timestamp.md  # Reviews
  │   └── analysis-timestamp.md            # AI analysis
  └── app-name-2/
      ├── App-Name-2-ios-timestamp.md      # Reviews
      └── analysis-timestamp.md            # AI analysis

Development

Project Setup

1. Clone the repository:

   git clone https://github.com/nanogiants/storescraper-ai.git
   cd storescraper-ai

2. Install dependencies:

   npm install

3. Create a local environment file:

   cp .env.example .env
   # Edit .env and add your OpenAI API key

4. Build the project:

   npm run build

5. Link the package locally:

   npm link

Project Structure

├── dist/               # Compiled JavaScript files
├── src/                # Source TypeScript files
│   ├── ai/             # AI-related functionality
│   │   ├── core/       # Core AI utilities
│   │   └── usecases/   # AI use cases
│   ├── commands/       # CLI commands
│   └── utils/          # Utility functions
├── output/             # Generated output files (gitignored)
├── .env                # Environment variables
└── tsconfig.json      # TypeScript configuration

Key Components

• Commands: Each command is implemented as a separate module in the src/commands directory.
• AI Usecases: AI analysis functionality is organized into use cases in the src/ai/usecases directory.
• Store Scrapers: The app store scraping functionality is in src/utils/store-scraper.ts.
• Settings: User settings management is handled in src/utils/settings.ts.

Development Workflow

1. Make changes to the TypeScript files in the src directory.
2. Build the project with npm run build.
3. Test your changes with npm link and running the CLI commands.
4. Commit your changes following the commit message conventions.

Commit Messages

This project follows the Karma commitlint style for commit messages:

<type>(<scope>): <subject>

Types:

• feat: A new feature
• fix: A bug fix
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code
• refactor: A code change that neither fixes a bug nor adds a feature
• perf: A code change that improves performance
• test: Adding missing tests or correcting existing tests
• chore: Changes to the build process or auxiliary tools

Example:

feat(feedback): add support for custom output formats

Adding a New Command

1. Create a new file in src/commands/ (e.g., my-command.ts).
2. Implement the command using the Commander.js pattern.
3. Register the command in src/index.ts.

Example:

// src/commands/my-command.ts
import { Command } from 'commander';

export function initMyCommand(program: Command): void {
  program
    .command('my-command')
    .description('Description of my command')
    .option('-o, --option <value>', 'Description of option')
    .action(async (options) => {
      // Command implementation
    });
}

// src/index.ts
import { initMyCommand } from './commands/my-command';
// ...
initMyCommand(program);

Adding a New AI Usecase

1. Create a new file in src/ai/usecases/ (e.g., my-usecase.ts).
2. Define the schema using Zod.
3. Implement the usecase class extending the base UseCase class.

Example:

import { z } from 'zod';
import { UseCase, UseCaseConfig } from '../core/usecase';

const MyResponseSchema = z.object({
  // Define your schema
});

export type MyResponse = z.infer<typeof MyResponseSchema>;

const myUsecaseConfig: UseCaseConfig<MyResponse> = {
  name: 'My Usecase',
  description: 'Description of my usecase',
  schema: MyResponseSchema,
  temperature: 0.2,
};

export class MyUsecase extends UseCase<MyResponse> {
  constructor() {
    super(myUsecaseConfig);
  }

  async execute(input: string): Promise<MyResponse> {
    return this.execute(input);
  }
}

OpenAI Integration Notes

When working with OpenAI's API and Zod schemas, note that OpenAI's implementation doesn't support certain Zod validation fields:

• 'minimum', 'maximum'
• 'minLength', 'maxLength'
• 'pattern', 'format'
• 'default'

Instead, include validation guidance in the description field.

Publishing a Release

1. Ensure all changes are committed and pushed to the develop branch.
2. Start a release using git flow:

   git flow release start <version>

3. Update version in package.json and create/update CHANGELOG.md.
4. Finish the release:

   git flow release finish <version>

5. Push the master branch and tags:

   git push origin master --tags

6. Publish to npm:

   npm publish

Changelog

v1.0.0 (Initial Release)

• Basic functionality to fetch reviews from Google Play Store and Apple App Store
• Interactive CLI with inquirer.js
• Support for configuring default settings
• AI-powered analysis of app reviews
• Multi-language support for both fetching and analyzing reviews
• Structured output directory organization

v1.1.0

• Added parse-tam command to extract app IDs from Excel files
• Added feedback-bulk command to process multiple apps from a CSV file
• Improved error handling and recovery
• Added delay option to prevent rate limiting
• Enhanced AI analysis formatting

Contributing

Contributions are welcome and appreciated! Here's how you can contribute:

1. Fork the repository on GitHub
2. Clone your fork locally
3. Create a new branch for your feature or bugfix
4. Make your changes and commit them with descriptive commit messages following the Karma commitlint style
5. Push your branch to your fork on GitHub
6. Open a pull request against the develop branch

Before submitting a pull request, please ensure:

• Your code follows the project's coding style
• You've added tests for new functionality
• All tests pass
• You've updated documentation as needed

Bug reports and feature requests are welcome on GitHub at https://github.com/nanogiants/storescraper-ai. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

MIT License Copyright (c) 2025 NanoGiants GmbH

Permission is hereby granted, free
of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to the
following conditions:

The above copyright notice and this permission notice
(including the next paragraph) shall be included in all copies or substantial
portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO
EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

Acknowledgements

• google-play-scraper - For Google Play Store data scraping
• app-store-scraper - For Apple App Store data scraping
• OpenAI - For AI-powered analysis capabilities
• Commander.js - For CLI interface
• Inquirer.js - For interactive prompts
• XLSX - For Excel file parsing

Troubleshooting

Common Issues

1. API Rate Limiting

   • Increase the delay between requests using the -d option
   • Reduce the number of reviews requested with -c

2. OpenAI API Errors

   • Ensure your API key is correctly set in the .env file
   • Check your OpenAI account has sufficient credits

3. CSV Parsing Issues

   • Ensure your CSV uses the correct delimiter (default is ;)
   • Check column names match expected format

4. Excel Parsing Issues

   • Ensure column names include "Name" and "Company Mobile App" or similar
   • Check that app URLs are in the expected format

Getting Help

If you encounter issues not covered here, please open an issue on GitHub with:

• Command you're trying to run
• Complete error message
• Your environment (OS, Node.js version)
• Sample data (if possible)