StoreScraper AI CLI

StoreScraper AI CLI is a powerful command-line tool for fetching, analyzing, and processing app reviews from Google Play Store and Apple App Store. It helps developers, product managers, and market researchers gain valuable insights from user feedback through AI-powered analysis.

The tool solves several key problems:

Automated review collection - Easily fetch reviews from both major app stores
AI-powered analysis - Extract actionable insights from user feedback
Bulk processing - Handle multiple apps efficiently
TAM analysis support - Extract app IDs from market research Excel files

This tool is aimed at:

Mobile app developers seeking to improve their products
Product managers tracking user sentiment
Market researchers analyzing competitor apps
QA teams identifying common issues

Overview

StoreScraper AI CLI is built with a modular architecture that separates concerns into distinct components:

Core Components

Command Layer (src/commands/)
- Handles user input and command execution
- Implements the CLI interface using Commander.js
- Each command is in its own file for maintainability
Scraper Layer (src/utils/store-scraper.ts)
- Fetches data from app stores using specialized scrapers
- Handles platform detection and appropriate API calls
- Normalizes data from different sources
AI Layer (src/ai/)
- Processes and analyzes the fetched data using OpenAI
- Implements use cases for different analysis types
- Structures AI prompts and responses
Settings Layer (src/utils/settings.ts)
- Manages user preferences and default settings
- Provides persistence across command executions

Data Flow

User Input → Command Layer → Scraper Layer → Data Processing → AI Analysis → Output Generation

Key Technologies

TypeScript - For type-safe code and better developer experience
Commander.js - For CLI interface and command parsing
Inquirer.js - For interactive prompts and user input
OpenAI API - For AI-powered analysis of reviews
Zod - For schema validation and type safety
XLSX - For Excel file parsing in TAM analysis

Installation and Usage

Prerequisites

Node.js (v14 or higher)
npm (v6 or higher)
OpenAI API key (for AI analysis features)

Global Installation (CLI)

npm install -g nanogiants-storescraper-ai

Local Installation (Library)

npm install nanogiants-storescraper-ai

Library Usage

You can use this package as a library in your Node.js, React, or NestJS applications:

import { StoreScraper } from 'nanogiants-storescraper-ai';

// Initialize with configuration
const scraper = new StoreScraper({
  openai: {
    apiKey: 'your-openai-api-key', // Required for AI features
    model: 'gpt-4-turbo' // Optional, defaults to gpt-4-turbo
  },
  defaultLanguage: 'en',
  defaultCountry: 'us',
  defaultReviewCount: 50
});

// Example: Fetch app info and reviews from Google Play
async function getAppInfo() {
  try {
    // Fetch app info and reviews
    const appInfo = await scraper.fetchGooglePlayApp('com.example.app', {
      reviewCount: 100,
      lang: 'en',
      country: 'us'
    });

    console.log(`App: ${appInfo.title}`);
    console.log(`Rating: ${appInfo.score}`);
    console.log(`Reviews: ${appInfo.reviews.length}`);

    // Analyze reviews with AI
    const analysis = await scraper.analyzeFeedback(
      appInfo.reviews.map(review => review.text),
      'english'
    );

    console.log('Analysis:', analysis);

    // Example: Bulk analysis of multiple apps from a CSV file
    const csvString = `Name;App Id;Mobile App Name
Company A;com.company.app;App Name
Company B;com.company.app2;App Name 2`;

    const result = await scraper.analyzeBulkFeedback(csvString, {
      outputPath: './output',
      platform: 'android',
      reviewCount: 50,
      language: 'en',
      country: 'us',
      generateDetailedReports: true
    });

    console.log(`Summary CSV saved to: ${result.summaryFilePath}`);
    console.log(`Processed ${result.summaryData.length} apps successfully.`);

    // Example: Validate Excel file before parsing
    const validationResult = scraper.validateTamExcel('./path/to/excel-file.xlsx');

    if (!validationResult.isValid) {
      console.error(`Error: ${validationResult.error}`);
      console.log('Available columns:', validationResult.columnNames.join(', '));
      return;
    }

    console.log('Excel file is valid!');
    console.log(`Name column: ${validationResult.nameColumn}`);
    console.log(`App column: ${validationResult.appColumn}`);
    if (validationResult.appNameColumn) {
      console.log(`App name column: ${validationResult.appNameColumn}`);
    }

    // Example: Parse Excel file to extract app IDs
    const parseResult = await scraper.parseTam('./path/to/excel-file.xlsx', {
      outputPath: './output',
      verbose: true
    });

    console.log(`Extracted ${parseResult.count} app IDs`);
    console.log(`Output saved to: ${parseResult.outputPath}`);

    // Access the extracted app data
    parseResult.data.forEach(app => {
      console.log(`${app.Name} - ${app['App Id']} (${app['Mobile App Name']})`);
    });
  } catch (error) {
    console.error('Error:', error);
  }
}

getAppInfo();

React Example

import { useState, useEffect } from 'react';
import { StoreScraper } from 'nanogiants-storescraper-ai';

function AppAnalyzer() {
  const [appInfo, setAppInfo] = useState(null);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState(null);

  // Initialize scraper (preferably in a service or context)
  const scraper = new StoreScraper({
    openai: {
      apiKey: process.env.REACT_APP_OPENAI_API_KEY
    }
  });

  async function analyzeApp(appId) {
    setLoading(true);
    try {
      const info = await scraper.fetchGooglePlayApp(appId);
      setAppInfo(info);
    } catch (err) {
      setError(err.message);
    } finally {
      setLoading(false);
    }
  }

  return (
    <div>
      {/* Your UI components */}
    </div>
  );
}

NestJS Example

import { Injectable } from '@nestjs/common';
import { StoreScraper } from 'nanogiants-storescraper-ai';
import { ConfigService } from '@nestjs/config';

@Injectable()
export class AppAnalysisService {
  private scraper: StoreScraper;

  constructor(private configService: ConfigService) {
    this.scraper = new StoreScraper({
      openai: {
        apiKey: this.configService.get<string>('OPENAI_API_KEY')
      }
    });
  }

  async analyzeApp(appId: string, platform: 'android' | 'ios') {
    if (platform === 'android') {
      return this.scraper.fetchGooglePlayApp(appId);
    } else {
      return this.scraper.fetchAppStoreApp(appId);
    }
  }

  async getFeedbackAnalysis(reviews: string[]) {
    return this.scraper.analyzeFeedback(reviews);
  }

  async validateExcelFile(filePath: string) {
    return this.scraper.validateTamExcel(filePath);
  }

  async parseExcelFile(filePath: string) {
    // First validate the Excel file
    const validationResult = await this.validateExcelFile(filePath);

    if (!validationResult.isValid) {
      throw new Error(`Invalid Excel file: ${validationResult.error}`);
    }

    // Then parse the Excel file
    return this.scraper.parseTam(filePath, {
      outputPath: './output/tam',
      verbose: false
    });
  }
}

### Local Installation

```bash
git clone https://github.com/nanogiants/storescraper-ai.git
cd storescraper-ai
npm install
npm link

Handling Secrets

This package requires an OpenAI API key for AI analysis features. There are several ways to provide this:

For CLI Usage:

Environment Variables: Set the OPENAI_API_KEY environment variable
```
export OPENAI_API_KEY=your_api_key_here
```
Dotenv File: Create a .env file in your project root with your API key
```
OPENAI_API_KEY=your_api_key_here
```
Settings Command: Use the settings command to save your API key
```
storescraper settings
```

For Library Usage:

When using as a library, you must provide the API key in the configuration:

const scraper = new StoreScraper({
  openai: {
    apiKey: 'your_api_key_here'
  }
});

For security best practices:

Never commit API keys to version control
Use environment variables or secure secret management
In frontend applications, consider using a backend proxy to make API calls

Examples

Check out the examples directory for complete working examples of how to use this package programmatically:

Single App Analysis - Demonstrates how to fetch and analyze reviews for a single app (Deutsche Bahn)
Bulk Analysis - Shows how to analyze multiple apps from a CSV file
Simple Bulk Analysis - Demonstrates the simplified API for bulk analysis
Parse TAM Excel - Shows how to extract app IDs from Excel files

Environment Setup

Create a .env file in the root directory with your OpenAI API key:

OPENAI_API_KEY=your_api_key_here

Available Commands

Feedback Command

Fetch and analyze reviews for a single app:

storescraper feedback [options]

Options:

-p, --platform <platform>: Specify platform (android or ios)
-c, --count <count>: Number of reviews to fetch (default: 50)
-o, --output <path>: Output directory for the markdown file
-l, --lang <lang>: Two letter language code (default: en)
-C, --country <country>: Two letter country code (default: us)
-s, --sort <sort>: Sort reviews by: newest, rating, or helpful (default: newest)
-a, --analyze <boolean>: Analyze feedback with AI after fetching (true/false)

Example:

storescraper feedback -p android -c 100 -l de -C de -s newest -a true -o my-app

Output:

Markdown file with app information and reviews
Analysis markdown file (if analyze option is enabled)
summary.csv file with the following columns:
- score: Average review score of the app
- total reviews: Total number of reviews in the store
- name: Name of the app
- id: App ID (package ID or bundle identifier)
- link: URL to the app in the app store
- company name: Developer/company name
- min installs: Minimum number of installs (Android only, "N/A" for iOS)
- max installs: Maximum number of installs (Android only, "N/A" for iOS)
- description: App description
- initial release: Initial release date
- last release: Last update date
- requested feature1, feature2, feature3: Top requested features from the AI analysis
- issue1, issue2, issue3: Top issues identified in the AI analysis

Feedback Bulk Command

Process multiple apps from a CSV file:

storescraper feedback-bulk [options]

The command includes an interactive file picker that allows you to:

Navigate through directories
Go up to parent directories using the ".." option
View only directories and CSV files
Select a CSV file by pressing Enter

Options:

-i, --input <path>: Path to the input CSV file with app IDs
-c, --count <number>: Number of reviews to fetch (default: 50)
-l, --lang <code>: Two letter language code (default: en)
-C, --country <code>: Two letter country code (default: us)
-s, --sort <sort>: Sort reviews by: newest, rating, or helpful (default: newest)
-a, --analyze <boolean>: Analyze feedback with AI after fetching (true/false)
-o, --output <path>: Output directory for the markdown files
-d, --delay <ms>: Delay between requests in milliseconds (default: 1000)

Example:

storescraper feedback-bulk -i app-ids.csv -c 50 -l en -C us -a true -d 2000 -o output

Input CSV Format:

Name;App Id;Mobile App Name
App Name 1;com.example.app1;App Display Name 1
App Name 2;123456789;App Display Name 2

Output:

Markdown files with app information and reviews for each app
Analysis markdown files (if analyze option is enabled)
A single summary.csv file with the following columns:
- score: Average review score of the app
- total reviews: Total number of reviews in the store
- name: Name of the app (uses Mobile App Name if available)
- id: App ID (package ID or bundle identifier)
- link: URL to the app in the app store
- company name: Developer/company name
- min installs: Minimum number of installs (Android only, "N/A" for iOS)
- max installs: Maximum number of installs (Android only, "N/A" for iOS)
- description: App description
- initial release: Initial release date
- last release: Last update date
- requested feature1, feature2, feature3: Top requested features from the AI analysis
- issue1, issue2, issue3: Top issues identified in the AI analysis

Parse TAM Command

Extract app IDs from Excel files containing app URLs:

storescraper parse-tam [options]

Options:

-i, --input <path>: Path to the input Excel file or directory
-o, --output <path>: Path to save the output CSV file(s)
-d, --delimiter <char>: Delimiter for the output CSV file (default: ;)
-v, --validate-only: Only validate the Excel file without parsing (default: false)

Example:

storescraper parse-tam -i tam-data.xlsx -o output
storescraper parse-tam -i tam-directory -o output
storescraper parse-tam -i tam-data.xlsx -v  # Validate only

Expected Excel format:

Column named "Name" or similar for company/app names
Column named "Company Mobile App" or similar containing app URLs

The command will first validate the Excel file to ensure it has the required columns. If validation fails, it will display the available columns and suggest possible matches.

Output CSV format:

Name: Company or organization name
App Id: The app identifier (package name for Android, numeric ID for iOS)
Mobile App Name: The name of the mobile application

Analyze Command

Analyze existing review text:

storescraper analyze [options]

Options:

-f, --file <path>: Path to file containing feedback to analyze
-o, --output <path>: Path to save the analysis results
-l, --language <language>: Language for the analysis output (e.g., english, german, spanish)

Example:

storescraper analyze -f reviews.txt -o analysis.md -l german

Settings Command

Configure default settings:

storescraper settings [options]

Options:

-r, --reset: Reset settings to default values

Output Structure

All output is saved in the output directory by default, with the following structure:

output/
  ├── app-name-1/
  │   ├── App-Name-1-android-timestamp.md  # Reviews
  │   └── analysis-timestamp.md            # AI analysis
  └── app-name-2/
      ├── App-Name-2-ios-timestamp.md      # Reviews
      └── analysis-timestamp.md            # AI analysis

Development

Project Setup

Clone the repository:

git clone https://github.com/nanogiants/storescraper-ai.git
cd storescraper-ai

Install dependencies:
```
npm install
```

Create a local environment file:

cp .env.example .env
# Edit .env and add your OpenAI API key

Build the project:
```
npm run build
```
Link the package locally:
```
npm link
```

Project Structure

├── dist/               # Compiled JavaScript files
├── src/                # Source TypeScript files
│   ├── ai/             # AI-related functionality
│   │   ├── core/       # Core AI utilities
│   │   └── usecases/   # AI use cases
│   ├── commands/       # CLI commands
│   └── utils/          # Utility functions
├── output/             # Generated output files (gitignored)
├── .env                # Environment variables
└── tsconfig.json      # TypeScript configuration

Key Components

Commands: Each command is implemented as a separate module in the src/commands directory.
AI Usecases: AI analysis functionality is organized into use cases in the src/ai/usecases directory.
Store Scrapers: The app store scraping functionality is in src/utils/store-scraper.ts.
Settings: User settings management is handled in src/utils/settings.ts.

Development Workflow

Make changes to the TypeScript files in the src directory.
Build the project with npm run build.
Test your changes with npm link and running the CLI commands.
Commit your changes following the commit message conventions.

Commit Messages

This project follows the Karma commitlint style for commit messages:

<type>(<scope>): <subject>

Types:

feat: A new feature
fix: A bug fix
docs: Documentation only changes
style: Changes that do not affect the meaning of the code
refactor: A code change that neither fixes a bug nor adds a feature
perf: A code change that improves performance
test: Adding missing tests or correcting existing tests
chore: Changes to the build process or auxiliary tools

Example:

feat(feedback): add support for custom output formats

Adding a New Command

Create a new file in src/commands/ (e.g., my-command.ts).
Implement the command using the Commander.js pattern.
Register the command in src/index.ts.

Example:

// src/commands/my-command.ts
import { Command } from 'commander';

export function initMyCommand(program: Command): void {
  program
    .command('my-command')
    .description('Description of my command')
    .option('-o, --option <value>', 'Description of option')
    .action(async (options) => {
      // Command implementation
    });
}

// src/index.ts
import { initMyCommand } from './commands/my-command';
// ...
initMyCommand(program);

Adding a New AI Usecase

Create a new file in src/ai/usecases/ (e.g., my-usecase.ts).
Define the schema using Zod.
Implement the usecase class extending the base UseCase class.

Example:

import { z } from 'zod';
import { UseCase, UseCaseConfig } from '../core/usecase';

const MyResponseSchema = z.object({
  // Define your schema
});

export type MyResponse = z.infer<typeof MyResponseSchema>;

const myUsecaseConfig: UseCaseConfig<MyResponse> = {
  name: 'My Usecase',
  description: 'Description of my usecase',
  schema: MyResponseSchema,
  temperature: 0.2,
};

export class MyUsecase extends UseCase<MyResponse> {
  constructor() {
    super(myUsecaseConfig);
  }

  async execute(input: string): Promise<MyResponse> {
    return this.execute(input);
  }
}

OpenAI Integration Notes

When working with OpenAI's API and Zod schemas, note that OpenAI's implementation doesn't support certain Zod validation fields:

'minimum', 'maximum'
'minLength', 'maxLength'
'pattern', 'format'
'default'

Instead, include validation guidance in the description field.

Publishing a Release

Ensure all changes are committed and pushed to the develop branch.
Start a release using git flow:
```
git flow release start <version>
```
Update version in package.json and create/update CHANGELOG.md.
Finish the release:
```
git flow release finish <version>
```
Push the master branch and tags:
```
git push origin master --tags
```
Publish to npm:
```
npm publish
```

Changelog

v1.0.0 (Initial Release)

Basic functionality to fetch reviews from Google Play Store and Apple App Store
Interactive CLI with inquirer.js
Support for configuring default settings
AI-powered analysis of app reviews
Multi-language support for both fetching and analyzing reviews
Structured output directory organization

v1.1.0

Added parse-tam command to extract app IDs from Excel files
Added feedback-bulk command to process multiple apps from a CSV file
Improved error handling and recovery
Added delay option to prevent rate limiting
Enhanced AI analysis formatting

Contributing

Contributions are welcome and appreciated! Here's how you can contribute:

Fork the repository on GitHub
Clone your fork locally
Create a new branch for your feature or bugfix
Make your changes and commit them with descriptive commit messages following the Karma commitlint style
Push your branch to your fork on GitHub
Open a pull request against the develop branch

Before submitting a pull request, please ensure:

Your code follows the project's coding style
You've added tests for new functionality
All tests pass
You've updated documentation as needed

Bug reports and feature requests are welcome on GitHub at https://github.com/nanogiants/storescraper-ai. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

MIT License Copyright (c) 2025 NanoGiants GmbH

Permission is hereby granted, free
of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to the
following conditions:

The above copyright notice and this permission notice
(including the next paragraph) shall be included in all copies or substantial
portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO
EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

Acknowledgements

google-play-scraper - For Google Play Store data scraping
app-store-scraper - For Apple App Store data scraping
OpenAI - For AI-powered analysis capabilities
Commander.js - For CLI interface
Inquirer.js - For interactive prompts
XLSX - For Excel file parsing

Troubleshooting

Common Issues

API Rate Limiting
- Increase the delay between requests using the -d option
- Reduce the number of reviews requested with -c
OpenAI API Errors
- Ensure your API key is correctly set in the .env file
- Check your OpenAI account has sufficient credits
CSV Parsing Issues
- Ensure your CSV uses the correct delimiter (default is ;)
- Check column names match expected format
Excel Parsing Issues
- Ensure column names include "Name" and "Company Mobile App" or similar
- Check that app URLs are in the expected format

Getting Help

If you encounter issues not covered here, please open an issue on GitHub with:

Command you're trying to run
Complete error message
Your environment (OS, Node.js version)
Sample data (if possible)

storescraper-ai

StoreScraper AI CLI

Overview

Core Components

Data Flow

Key Technologies

Installation and Usage

Prerequisites

Global Installation (CLI)

Local Installation (Library)

Library Usage

React Example

NestJS Example

Handling Secrets

For CLI Usage:

For Library Usage:

Examples

Environment Setup

Available Commands

Feedback Command

Feedback Bulk Command

Parse TAM Command

Analyze Command

Settings Command

Output Structure

Development

Project Setup

Project Structure

Key Components

Development Workflow

Commit Messages

Adding a New Command

Adding a New AI Usecase

OpenAI Integration Notes

Publishing a Release

Changelog

v1.0.0 (Initial Release)

v1.1.0

Contributing

License

Acknowledgements

Troubleshooting

Common Issues

Getting Help

/storescraper-ai/

Package Sidebar

Install

DownloadsWeekly Downloads

Version

License

Unpacked Size

Total Files

Last publish

Collaborators

Weekly Downloads