YouTube Financial Video Analyzer

Part of STARS — Streaming Trading Analysis and Real‑time Systems. This service continuously collects finance‑focused YouTube videos, fetches long‑form transcripts reliably, runs DeepSeek analysis to produce structured market intelligence, and (optionally) generates audio voiceovers. It powers trader‑facing views like Video Analysis, Ticker Insights, Market Summary, Market Events, and the public Morning Report/Viewer.

• Who is it for? Active traders, analysts, and desk leads who want fast, consolidated insight from daily financial video flow (CNBC, Bloomberg, Yahoo Finance, Benzinga, and more). • What does it do? Ingests videos → obtains transcripts → converts them into structured, queryable signals → displays them in focused trading views.

How It Fits Into STARS (Streaming Trading Analysis and Real‑time Systems)

Within the STARS stack, this repository provides the "YouTube Ingest + Analyst" capability:

• Ingest: Discover and fetch recent videos from curated finance channels via YouTube Data API v3
• Transcript: Obtain robust long‑form transcripts through a Primary transcript service with Legacy fallback
• Analysis: Run DeepSeek on transcripts to produce JSON insights (tickers, themes, catalysts, sentiment, risks)
• Storage/Cache: Persist results and cache by videoId + analysis profile to avoid redundant computation
• Delivery: Expose APIs and a React UI used directly by traders and by other STARS modules

Output from this service can be consumed by:

• STARS dashboards/widgets (Ticker Insights, Market Summary)
• Morning Report generator (public read)
• Internal monitoring (Job Monitor, System tab)

What Traders Get (Views & Use Cases)

• Video Analysis: Per‑video breakdown (summary, key themes, tickers mentioned, sentiment, catalysts, risk notes). Ideal for scanning what matters in today's media flow.
• Ticker Insights: Aggregation of mentions and sentiment across recent videos. Filter by watchlist to surface names with rising chatter or fresh catalysts.
• Market Summary: Top macro themes, sector currents, and posture hints derived from aggregated video analyses.
• Market Events: Extracted events/catalysts from transcripts (earnings, guidance, M&A, layoffs, CPI/Fed/FOMC mentions) organized as a timeline.
• Morning Report & Viewer: A curated, public report generated on schedule. Traders can quickly see the day's video‑driven highlights with links back to detail views.
• Ad‑hoc Runs (private): Run your own analysis on demand. Results are private to you; great for desk prep or rapid deep‑dives.

End‑to‑End Flow (How It Works)

1. Collect Videos
   • Uses YouTube Data API v3 to fetch recent uploads from default finance channels (+ user extras for ad‑hoc runs).
2. Acquire Transcripts
   • PrimaryTranscriptService posts a job to a local transcription API (default http://192.168.1.11:8080) and polls intelligently.
   • Handles long videos (20+ min to 2+ hours) with escalating polling (30s → 2m) and generous timeouts.
   • Automatically falls back to a Legacy method when the primary is disabled or fails.
   • Details in docs/TRANSCRIPT_FETCHING.md.
3. Analyze with DeepSeek
   • Transcripts are sent to a single, resilient analysis client with retries/timeouts.
   • Produces JSON structured insights: tickers, sectors/themes, catalysts/events, sentiment, risks/uncertainties, and concise summaries.
4. Cache & Store
   • Writes normalized analysis to the DB and caches by videoId + ANALYSIS_PROFILE_VERSION (see User Workflow).
   • Cache hits are reused to minimize API usage and speed up UX.
5. Display & Distribute
   • React UI renders views described below.
   • Scheduler produces a public Morning Report; users can also create private reports via ad‑hoc runs.
   • Optional TTS (Jetson Orin) can create short voiceovers for summaries.

UI Modules (How Data Is Displayed)

• Video Analysis

  • Card/list of recent videos with status (Queued/Transcribing/Analyzing/Done) and quick badges for tickers.
  • Detail view includes: short summary, key takeaways, tickers/mentions, high/low‑confidence items, sentiment, catalysts/events, and links to the source video.
  • Where available, time‑coded highlights may be included by the analysis profile.

• Ticker Insights

  • Aggregates per‑ticker signals across a rolling window (e.g., last N videos/hours).
  • Shows mention frequency, average sentiment, recent catalysts, and sample quotes.
  • Filter by your watchlist; click through to the underlying videos.

• Market Summary

  • Macro themes and sector rotation hints derived from recent analyses.
  • Top narratives, risks on the tape, and a concise "What changed?" line for morning prep.

• Market Events

  • Structured events extracted from transcripts: earnings, guidance, macro prints (CPI, jobs), FOMC/Fed, M&A, product launches, layoffs, regulatory items.
  • Rendered as a simple timeline/table with links into Video Analysis for context.

• Morning Reports & Viewer

  • Public morning report generated by the scheduler from system default channels.
  • Morning Report Viewer provides a clean, shareable readout with links to details.
  • Ad‑hoc private reports are listed under the Reports tab for the owning user.

• System & Job Monitor (ops)

  • System: view/edit system defaults like monitored channels (admin only).
  • Job Monitor: see current/previous jobs; regular users see only their own; support/admin see all.

Architecture (High Level)

• Frontend: React + TypeScript + Tailwind (virtualized lists for responsiveness where needed)
• Backend: Node.js (Express) + SQLite (better‑sqlite3)
• Analysis: DeepSeek API via a centralized client with retry/timeout safeguards
• Transcript pipeline:
  • PrimaryTranscriptService → local transcription API (default http://192.168.1.11:8080)
  • Intelligent polling escalation: 30s → 2m after ~10m
  • 1h+ end‑to‑end timeouts, with client → backend timeout buffers
  • Automatic Legacy fallback
  • See docs/TRANSCRIPT_FETCHING.md for details
• Caching: video_analysis_cache to avoid re‑running analysis for the same video/profile
• Scheduler: automated morning batch run; manual ad‑hoc runs per user
• Optional TTS: tiny TTS model on Jetson Orin for audio summaries (see below)

User Management & Roles

• Authentication: cookie‑based sessions (httpOnly, SameSite=Lax; secure in production)
• Roles: free, paid, support (read‑only system visibility), admin
• Visibility:
  • Morning reports: public
  • Ad‑hoc reports: private to the owner
• API (summary):
  • POST /api/auth/register | /api/auth/login | /api/auth/logout | GET /api/auth/me
  • POST /api/user/reports/manual-run | GET /api/user/reports
  • GET /api/reports | GET /api/reports/:id
  • Admin/support: /api/admin/users, /api/jobs (mine=1 to filter)
• Quotas (by role): daily ad‑hoc run caps; user extra‑channels limits
• For complete flows and limits, see USERWORKFLOW.md

Transcript System (Primary + Legacy)

• Client endpoint: GET /api/transcript/:videoId
• Optional async usage:
  • POST /api/transcript/submit/:videoId
  • GET /api/transcript/status/:jobId
  • GET /api/transcript/result/:jobId
• Health: GET /api/transcript/health
• Primary service polls a local transcription API and gracefully handles long videos.
• Reliability: extended server/client HTTP timeouts and per‑poll request timeouts prevent premature aborts.
• Deep dive: docs/TRANSCRIPT_FETCHING.md

Tiny TTS on Jetson Orin (Optional)

• Purpose: Generate short voiceovers for summaries or morning reports.
• Deployment: Tiny TTS model runs on a Jetson Orin (edge device).
• Integration pattern:
  • Backend calls a local TTS HTTP endpoint (e.g., http://JETSON_ORIN_IP:PORT/tts)
  • POST JSON: { text, voice?, speed? } → returns audio (wav/mp3) or signed URL
• Suggested env (optional):
  • TTS_SERVICE_URL=http://JETSON_ORIN_IP:PORT
  • TTS_DEFAULT_VOICE=tiny
  • TTS_ENABLED=true
• Notes:
  • TTS is optional and not required for transcripts or analysis.
  • Keep audio generation out‑of‑band from long analyses.

Environment Configuration

Create a .env file in project root. Backend reads process.env directly; frontend uses Vite if exposing keys to the client.

Required (backend):

• DEEPSEEK_API_KEY=your_deepseek_key
• YOUTUBE_API_KEY=your_youtube_key

Primary Transcript Service (backend):

• PRIMARY_TRANSCRIPT_SERVICE_ENABLED=true
• PRIMARY_TRANSCRIPT_SERVICE_URL=http://192.168.1.11:8080
• PRIMARY_SERVICE_TIMEOUT_MS=3600000 # 1h total window for long videos
• PRIMARY_SERVICE_POLL_INTERVAL_MS=30000 # 30s initial
• PRIMARY_SERVICE_POLL_INTERVAL_EXTENDED_MS=120000 # 2m after escalation
• PRIMARY_SERVICE_POLL_ESCALATION_THRESHOLD_MS=600000 # escalate after ~10m
• PRIMARY_SERVICE_SUBMIT_TIMEOUT_MS=120000 # submit job timeout
• PRIMARY_SERVICE_POLL_REQUEST_TIMEOUT_MS=30000 # per‑status‑request timeout
• TRANSCRIPT_HTTP_TIMEOUT_BUFFER_MS=300000 # +5m buffer for client→backend fetch

Frontend (optional):

• VITE_YOUTUBE_API_KEY=...
• VITE_DEEPSEEK_API_KEY=...

CORS/Frontend URL:

• FRONTEND_URL=http://localhost:5173

Install & Run

1) Install deps

npm install

2) Start backend (loads .env and serves API)

npm run start-backend-only

or during development with frontend

yarn start-all or: npm run start-all

Dev combo starts:

• Backend at http://localhost:7319
• Frontend dev server at http://localhost:5173 (after backend health is up)

Usage Tips for Traders

• Morning run (scheduled): The public Morning Report updates during active hours and is visible without login.
• Manual (ad‑hoc) run: Log in, select channels or videos, and run analysis. Your report is private and listed under Reports.
• Cache‑first behavior: Re‑analyzing the same video with the same profile reuses cached results for speed.
• Watchlist workflow: Use Ticker Insights with a personal watchlist to surface names with rising mentions and sentiment shifts.

Troubleshooting

• Long videos time out or "fetch failed"

  • Check the local transcription API: curl http://192.168.1.11:8080/health
  • Inspect job lifecycle: submit/poll directly on the primary service
  • Increase PRIMARY_SERVICE_TIMEOUT_MS and/or TRANSCRIPT_HTTP_TIMEOUT_BUFFER_MS

• YouTube API 403/Quota

  • Verify key restrictions and that YouTube Data API v3 is enabled
  • Consider per‑second limits, retries, and exponential backoff

Default Channels

• Benzinga
• CNBC Television
• Yahoo Finance
• Bloomberg Television

Contributing

1. Fork the repo
2. Create a feature branch
3. Add tests/updates
4. Open a PR

License

MIT