WatchTower AI — Luxury Watch Intelligence Platform

Complete watch sourcing and pricing automation platform with AI-powered scraping, real-time analytics, and messaging parser. Built with React frontend, Express API, PostgreSQL database, and self-healing scraper system.

🚀 Quick Start (Docker Compose)

Prerequisites

• Docker Desktop
• Node.js 20+ (for development)
• Optional: Residential proxy for scraping

2. Deploy the Complete System

```bash

Start all services (recommended for full system)

docker-compose up -d

Or start core services only (no scraper)

docker-compose up -d watchtower-app postgres redis opensearch

Or start with scraper enabled

docker-compose --profile scraper up -d ```

3. Initialize Database

```bash

Wait for services to start, then run migrations

docker-compose exec watchtower-app npm run db:migrate

Optional: Seed with sample data

docker-compose exec watchtower-app npm run db:seed ```

4. Browser Authentication (Optional)

For sites requiring authentication or to bypass blocks:

1. Start the browser authentication session: ```bash ./start-browser-session.sh ```

2. Open http://localhost:3001 and log in to required sites

3. The scraper will automatically use your authenticated sessions

See BROWSER_AUTH.md for detailed instructions.

5. Access Points

• Main Application: http://localhost:3000 (React frontend + API)
• API Documentation: http://localhost:3000/api/health
• Database: postgres://watchtower:watchtower@localhost:5432/watchtower (internal only)

🏗️ System Architecture

``` ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ React Frontend │◄──►│ Express API │◄──►│ PostgreSQL │ │ (Port 3000) │ │ (Port 3000) │ │ (Internal) │ └─────────────────┘ └──────────────────┘ └─────────────────┘ │ │ ▼ ▼ ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ Self-Healing │◄──►│ Redis │ │ OpenSearch │ │ Scraper │ │ (Queue) │ │ (Analytics) │ │ (Background) │ │ (Internal) │ │ (Internal) │ └─────────────────┘ └──────────────────┘ └─────────────────┘ │ │ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ │ Marketplace │ │ OpenSearch │ │ Scrapers │ │ Dashboards │ │ (Chrono24, eBay)│ │ (Internal) │ └─────────────────┘ └─────────────────┘ ```

Core Services

1. WatchTower App (`watchtower-app`)

• React Frontend: Modern UI migrated from Angular
• Express API: Comprehensive REST API with 20+ endpoints
• Port: 3000 (exposed)
• Features: Listings, sellers, analytics, messaging parser, FMV estimation

2. PostgreSQL Database (`postgres`)

• Purpose: Primary data store for listings, sellers, keywords, FMV data
• Port: 5432 (internal only)
• Data: Watch listings, seller profiles, price history, search analytics

3. Redis Queue (`redis`)

• Purpose: Job queue management with BullMQ
• Port: 6379 (internal only)
• Usage: Scraping jobs, background processing

4. OpenSearch (`opensearch`)

• Purpose: Search engine and analytics
• Port: 9200 (internal only)
• Features: Full-text search, price analytics, trend analysis

5. Self-Healing Scraper (`scraper` - optional)

• Purpose: AI-powered autonomous scraping with failure repair
• Technology: Puppeteer + DeepSeek AI
• Features: Chrono24, eBay scrapers with automatic debugging

🔧 Environment Configuration

Required Variables (.env)

```bash

Database

PG_URL=postgres://watchtower:watchtower@postgres:5432/watchtower

Queue System

REDIS_URL=redis://redis:6379

Search Engine

ES_URL=http://opensearch:9200

AI Services (for messaging parser and scraper repair)

DEEPSEEK_API_KEY=your-deepseek-api-key-here

Application

PORT=3000 NODE_ENV=development

Scraping Configuration

PROXY_URL=socks5://user:pass@host:port # Optional but recommended MAX_PAGE_DEPTH=3 RATE_BUDGET_URLS_PER_JOB=50 EMERGENCY_STOP=0

Pricing Configuration

BID_TARGET_PCT_OF_FMV=0.92 SMALL_NEGOTIATE_PCT=0.02

Alerts (Optional)

SLACK_WEBHOOK=https://hooks.slack.com/services/...

Security

SNAPSHOT_ENCRYPTION_KEY=64-character-hex-key-for-encrypting-html-snapshots ```

🎯 Key Features

1. Real-Time Market Intelligence

• Multi-Marketplace Scraping: Chrono24, eBay with anti-bot protection
• Price Analytics: FMV computation, deal scoring, trend analysis
• Seller Intelligence: Trust ratings, verification status, transaction history

2. AI-Powered Messaging Parser

• WhatsApp/Telegram Integration: Parse watch listings from chat messages
• DeepSeek AI: Advanced natural language processing
• Structured Output: Extract brand, model, reference, year, condition, price

3. Self-Healing Scraper System

• Autonomous Repair: AI detects and fixes scraping failures
• OODA Loop: Observe failures, Orient with context, Decide fixes, Act on repairs
• Zero Downtime: Automatic fallback to working versions

4. Browser Authentication System

• Web-Accessible Browser: Manual authentication through browser interface
• Session Sharing: Authenticated sessions shared with automated scrapers
• Bypass Protection: Overcome login requirements and anti-bot measures
• Multi-Site Support: PayPal, Chrono24, eBay authentication management

5. Advanced Analytics

• Fair Market Value (FMV): ML-based price estimation
• Deal Scoring: Identify undervalued listings
• Market Trends: Historical price analysis and forecasting

📊 API Endpoints

Core Data APIs

• `GET /api/listings` - Search and filter watch listings
• `GET /api/sellers` - Seller directory and profiles
• `GET /api/stats` - Dashboard statistics
• `GET /api/keywords/performance` - Search term analytics
• `GET /api/analytics/price-trends` - Price history data

AI-Powered Tools

• `POST /api/messages/parse` - Parse WhatsApp/Telegram messages
• `POST /mcp/normalize.attrs` - Extract structured data from text
• `POST /mcp/fmv.estimate` - Estimate fair market value
• `POST /mcp/keyword.optimize` - Optimize search terms

Operational

• `GET /api/health` - System health check
• `GET /api/metrics` - Operational metrics
• `GET /api/listings/:id/snapshot` - Archived HTML snapshots

🔄 Development Workflow

Local Development (Without Docker)

```bash

Start infrastructure only

docker-compose up -d postgres redis opensearch

Install dependencies

npm install

Set environment variables

export PG_URL=postgres://watchtower:watchtower@localhost:5432/watchtower export REDIS_URL=redis://localhost:6379

Run migrations

npm run db:migrate

Start React development server

npm run dev # Frontend on http://localhost:5175

Start API server (separate terminal)

npm run api:start # API on http://localhost:3000 ```

Production Deployment

```bash

Use production compose file

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

Or use the deployment script

./deploy.sh prod ```

Running Scraper Jobs

```bash

Start scraper with specific terms

docker-compose exec scraper node apps/scraper/src/index.js \ --site=chrono24 \ --terms="rolex daytona 116520 full set, ap royal oak 15500" \ --pageDepth=2

Enable self-healing mode

docker-compose exec scraper node apps/scraper/src/core/self-healing-scraper.js \ scrape chrono24 "rolex submariner" ```

Data Management

```bash

Run database migrations

docker-compose exec watchtower-app npm run db:migrate

Seed sample data

docker-compose exec watchtower-app npm run db:seed

Compute FMV values

docker-compose exec watchtower-app npm run fmv:compute

Train ML model

docker-compose exec watchtower-app npm run fmv:train

Clean fake data

docker-compose exec watchtower-app npm run db:cleanup-fake ```

🛡️ Security & Compliance

Data Protection

• Encrypted Snapshots: HTML snapshots encrypted at rest with AES-256-GCM
• Internal Networking: Database and queue services not exposed externally
• Rate Limiting: Configurable scraping limits and emergency stop
• Proxy Support: Residential proxy integration for anonymity

Scraping Ethics

• Robots.txt Respect: Configurable compliance modes
• Rate Limiting: Per-site request budgets
• Emergency Stop: Instant halt capability via `EMERGENCY_STOP=1`
• Legal Compliance: Built-in guardrails for responsible scraping

📈 Self-Healing Scraper Technology

Revolutionary AI-Powered Repair System

WatchTower AI features an autonomous scraper repair system that eliminates manual debugging:

How It Works

1. Failure Detection: Automatic detection of scraping failures
2. Context Collection: Comprehensive dossier creation with HTML, screenshots, logs
3. AI Analysis: DeepSeek API analyzes failures and generates repairs
4. Sandbox Testing: Isolated testing of repaired code
5. Automatic Deployment: Seamless deployment of verified fixes

Key Benefits

• 99.9% Uptime: Automatic repair of scraping failures
• Zero Manual Intervention: AI handles debugging and fixes
• Learning System: Each repair improves future performance
• Cost Effective: Eliminates manual debugging time

Technical Components

• ScraperOrchestrator: OODA loop controller
• ContextCollector: Comprehensive failure context gathering
• LLMRepairAgent: AI-powered debugging with DeepSeek
• SandboxEnvironment: Isolated testing environment
• LLMRepairToolkit: Function calling interface for AI

🎛️ Service Profiles & Options

Development Mode

```bash

Core services only

docker-compose up -d watchtower-app postgres redis

With analytics

docker-compose up -d watchtower-app postgres redis opensearch

Full development stack

docker-compose up -d ```

Production Mode

```bash

Full production deployment

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

With enhanced security and performance

POSTGRES_PASSWORD=secure-production-password docker-compose -f docker-compose.prod.yml up -d ```

Optional Services

```bash

Enable scraper service

docker-compose --profile scraper up -d ```

🔍 Monitoring & Operations

Health Checks

• Application: http://localhost:3000/api/health
• Database: Automatic connection validation
• Queue: Redis ping validation
• Search: OpenSearch cluster health

Metrics & Logging

• Operational Metrics: `/api/metrics` endpoint
• Scraper Health: Success rates, error tracking, index lag
• Performance: Response times, queue depth, database connections

Alerting (Optional)

```bash

Setup Slack alerts for undervalued listings

docker-compose exec watchtower-app npm run alerts:run

Daily market summary

docker-compose exec watchtower-app npm run alerts:daily ```

🎯 Use Cases

For Watch Dealers

• Inventory Sourcing: Find undervalued listings across marketplaces
• Pricing Intelligence: Real-time FMV computation and deal scoring
• Seller Verification: Trust ratings and marketplace badges
• Market Analysis: Trend analysis and price forecasting

For Collectors

• Deal Discovery: Automated detection of rare references
• Price Tracking: Historical price data and trend analysis
• Authentication Support: Seller trust signals and provenance data
• Market Intelligence: Comprehensive marketplace insights

For Developers

• API Integration: Comprehensive REST API for custom applications
• Scraper Framework: Extensible scraper system with AI repair
• Data Pipeline: ETL system for watch market data
• Analytics Platform: OpenSearch-based analytics and search

📚 Additional Resources

Documentation

• API Reference: http://localhost:3000/api/
• Database Schema: `packages/db/migrations/`
• Type Definitions: `packages/shared-types/`

Configuration Files

• Docker Compose: `docker-compose.yml` (development)
• Production Config: `docker-compose.prod.yml`
• Environment: `.env.example` (copy to `.env`)
• Build Config: `Dockerfile`, `vite.config.ts`

Scripts & Utilities

• Database: `npm run db:migrate`, `npm run db:seed`
• Scraping: `npm run scraper:start`
• Analytics: `npm run fmv:compute`, `npm run fmv:train`
• Alerts: `npm run alerts:run`

🤝 Contributing

Development Setup

1. Fork the repository
2. Create feature branch: `git checkout -b feature/amazing-feature`
3. Follow the local development workflow above
4. Run tests: `npm test`
5. Commit changes: `git commit -m 'Add amazing feature'`
6. Push branch: `git push origin feature/amazing-feature`
7. Open Pull Request

Testing

```bash

Run normalization tests

npm run test:normalize

Test scraper components

docker-compose exec scraper node apps/scraper/src/core/self-healing-scraper.js test ebay

API health check

curl http://localhost:3000/api/health ```

⚖️ Legal & Compliance

Important: This system is designed for legitimate market research and pricing intelligence. Users are responsible for:

• Compliance with marketplace Terms of Service
• Respecting robots.txt and rate limits
• Obtaining necessary permissions for data usage
• Following applicable data protection laws

The system includes built-in safeguards:

• Configurable rate limits and delays
• Emergency stop functionality
• Robots.txt respect modes
• Data minimization practices

WatchTower AI - Transforming luxury watch market intelligence through AI-powered automation.

Scraper Worker Usage (Chrono24/eBay)

🎛️ Scraper Engine Flexibility

WatchTower AI now supports multiple scraping engines, allowing you to choose the best approach for different use cases:

Available Engines

Browser Engine (Default)

• Technology: Puppeteer-based browser automation
• Best for: Sites requiring JavaScript execution, complex interactions
• Sites: Chrono24, eBay
• Features: Full browser simulation, CAPTCHA handling, authentication

Firecrawl Engine

• Technology: API-based scraping via Firecrawl service
• Best for: Fast, reliable data extraction without browser overhead
• Requirements: Firecrawl API key (set `FIRECRAWL_API_KEY` in environment)
• Features: Rate limiting, structured data extraction, HTML snapshots

External Engine

• Technology: Placeholder for custom URL→JSON API integration
• Best for: Integrating with existing scraping infrastructure
• Future: Will support custom endpoints returning structured listing data

Configuration

Environment Variables

```bash

Add to your .env file

FIRECRAWL_API_KEY=fc-xxxxxxxxxxxxxxxxxxxxxxxx ```

API Usage

```bash

Enqueue jobs with specific engine

POST /api/scraper/enqueue { "site": "chrono24", "terms": "rolex daytona 116520", "scraperType": "firecrawl" # browser, firecrawl, scrapegraph, or external }

Check available engines

GET /api/scraper/config ```

UI Usage

1. Navigate to Scraper Jobs in the admin interface
2. Select your preferred scraper type from the dropdown
3. Choose marketplace and search terms
4. Monitor job progress in real-time

Engine Selection Guidelines

• Use Browser Engine when:

  • Site requires JavaScript execution
  • Complex user interactions needed
  • Authentication or session management required

• Use Firecrawl Engine when:

  • Fast, reliable data extraction needed
  • API-based approach preferred
  • Cost-effective scaling required

• Use External Engine when:

  • Integrating with existing scraping infrastructure
  • Custom API endpoints available

Running Tests

To run the unit tests: ```bash source .venv/bin/activate && python -m tests.unit.test_fast_detector ```

To run the Kafka integration test: ```bash source .venv/bin/activate && python -m tests.integration.test_kafka_realtime ```

0) What this app is for

Goal: Give a luxury-watch dealer a real-time edge by (1) scraping marketplaces for floor prices & comps, (2) parsing shorthand listings from chat, and (3) ranking opportunities vs. a computed Fair Market Value (FMV)—all with Puppeteer (no marketplace APIs) and LLM strictly via MCP tools (no free-form model output). MCP is an open protocol for connecting LLMs to tools & data, which fits our requirement to keep all model I/O structured as tool calls. ([Chrome for Developers][1])

What watch buyers care about (drive UI & schema):

• Brand / model / reference (e.g., Rolex 116520)
• Condition & authenticity notes
• Scope of delivery: "box & papers / full set" materially increases value
• Service history / provenance
• Seller reputation & platform protections (e.g., Chrono24 Trusted/Certified, Buyer Protection) Use these as first-class filters and in the deal score. ([Model Context Protocol][2])

Compliance & ethics: Scraping risk varies by site ToS and access controls; design rate limits, robots handling modes, and emergency stop from day one. ([Browserless][3])

1) How the system works (high level)

``` Puppeteer Cluster (stealth+proxies) └─ Site adapters (Chrono24, eBay, …) └─ Extract canonical listing JSON └─ Normalize (rules → MCP tool `normalize.attrs`) └─ Store in Postgres → (optional) index to OpenSearch └─ FMV (rules v1 → ML v2 via LightGBM) └─ Dashboards + Alerts └─ Keyword optimizer (bandit) proposes next searches ```