Twatter

A standalone Python application for monitoring Twitter posts and financial news for stock-related sentiment analysis.

Overview

Twatter connects to a Kafka topic to process Twitter data, analyzes tweets to determine if they mention stocks/companies and their sentiment, and publishes results to another Kafka topic. It also periodically pulls financial news and grades from Financial Modeling Prep (FMP) API.

Features

• Consumes Twitter posts from Kafka topic `twitter_posts`
• Analyzes tweet content for stock/company mentions and sentiment
• Publishes results to Kafka topic `twitter_alerts`
• Uses Financial Modeling Prep (FMP) API for company lookup (focusing on US exchanges)
• Pulls latest stock grades/news every 5 minutes
• Analyzes sentiment of financial news
• Calculates upgrade/downgrade scores based on analyst actions
• Publishes results to Kafka topic `finnews_alerts`
• Includes validation tools for training data collection
• Uses Apache Arrow for historical training data storage
• Implements functional programming principles for robust error handling
• Features enhanced stock detection with advertisement filtering
• Uses LLM for context-aware company identification
• Tracks account reputation to prioritize reliable sources
• Implements tiered account system for trusted posters

Architecture

The system is designed with the following components:

1. Kafka Consumer: Processes incoming Twitter posts
2. Enhanced Stock Detector: Identifies stock/company mentions with improved accuracy
   • Fast Stock Detector: Provides quick filtering and advertisement detection
   • LLM Integration: Uses language models for context-aware company identification
   • Account Reputation Tracker: Maintains reputation scores for tweet sources
3. Sentiment Analyzer: Determines sentiment score of tweets and news
4. FMP API Client: Interfaces with Financial Modeling Prep API for ticker validation
5. News Processor: Handles periodic news fetching and deduplication
6. Kafka Producer: Publishes processed data to output topic
7. Training Data Collector: Stores and manages data for model improvement
8. Validation Interface: Tool for reviewing and correcting processed data

Data Flow

1. Twitter posts are consumed from Kafka topic `twitter_posts`
2. Account reputation is checked to determine processing priority
3. Fast advertisement detection is performed to filter promotional content
4. Posts are analyzed for stock/company mentions using cashtag extraction
5. For trusted accounts or non-ad content without cashtags, LLM analysis is performed
6. Identified tickers are validated against FMP API and common stocks list
7. Sentiment analysis is performed on validated stock mentions
8. Account reputation is updated based on post quality
9. Results are published to Kafka topic `twitter_alerts`
10. Every 5 minutes, latest financial news is pulled from FMP API
11. News items are deduplicated and analyzed for sentiment
12. News sentiment and upgrade/downgrade scores are calculated
13. Results are published to Kafka topic `finnews_alerts`

Development Approach

The system follows functional programming principles to ensure robust error handling and maintainability. Components are designed to be modular and testable, with clear interfaces between them.

Enhanced Stock Detection

The enhanced stock detection system implements a multi-stage approach:

1. Fast Advertisement Detection: Quickly identifies promotional content using pattern matching
2. Account Reputation Tracking: Maintains a reputation system for tweet sources
   • Trusted accounts: Automatically processed without advertisement checks
   • Probationary accounts: Being monitored for quality content
   • Standard accounts: Subject to full filtering process
3. Cashtag Extraction: Identifies direct stock mentions with $ symbol
4. LLM-based Company Detection: Uses language models to identify company mentions in natural language
5. Ticker Validation: Verifies tickers against both a common stocks list and FMP API

Running Tests

To run the unit tests: ```bash source .venv/bin/activate && python -m tests.unit.test_fast_detector ```

To run the Kafka integration test: ```bash source .venv/bin/activate && python -m tests.integration.test_kafka_realtime ```