Engineering Journey
From a single weekend bash script to an industrial-grade cloud-native media system. Each version was driven by a real problem — not a feature checklist.
The First Pipeline
Initial proof of concept: GitHub Trending → Gemini → Edge TTS → Remotion. A single Python script fetched the top trending repo, summarized it, generated audio, and rendered a video. Uploaded manually.
No filtering, no editorial judgment. The system would happily publish a JavaScript tutorial or a 'how to use Tailwind' guide as breaking developer news.
Full Architectural Rewrite
Complete rewrite introducing the editorial engine, parallel batch processing, and automated YouTube uploads via OAuth 2.0. The pipeline became a state machine rather than a script.
Selection was still based on an arbitrary 4-factor arithmetic score (freshness + popularity + trust + quality). Gemini would hallucinate context when source URLs returned 404s.
Newsroom Overhaul — 4 Sources
Expanded from GitHub Trending only to 4 independent sources: Hacker News API, Official Blog RSS feeds, GitHub Releases API, and Product Hunt RSS. Added the Signal Filter and a batch Gemini scoring system. Introduced source diversity cap (max 2 per source).
Deduplicator was URL-only — the same story could appear with different titles from different sources and pass through. Hype bonus made borderline meme content score too high.
Evidence-Driven Editorial Engine
Scrapped the 4-factor arithmetic score entirely. Replaced with an Evidence Builder (compiling objective facts: recency, engagement, source authority, cross-source coverage) and a two-pass Gemini evaluation system (Mission Check → Relative Ranking). Added 30-hour evaluation cache with viral re-scoring. Company diversity cap introduced.
Single-threaded evaluation took too long for large candidate batches (50+ items). The render + upload sequence was completely sequential with no parallelism.
Parallel Evaluation + Resumable Uploads
Pass 1 evaluation now runs in parallel via ThreadPoolExecutor (up to 5 concurrent Gemini chunks). YouTube upload switched to resumable 2 MB chunks with progress logging. Per-video timing reports introduced with proportional ASCII bar charts.
Still no Instagram or Facebook publishing. Rendered videos were only going to YouTube. No cloud storage bridge existed.
Migration from history.json to PostgreSQL + MongoDB Atlas
The pipeline's internal state (candidates, evaluation results, and run history) was migrated from a local JSON file to PostgreSQL. The presentation layer (published video records) was migrated to MongoDB Atlas. This decoupling allows the website to read published records even when the EC2 pipeline is offline.
Still no Instagram or Facebook publishing. Rendered videos were only going to YouTube. No cloud storage bridge existed.
Triple Broadcast + AWS S3 Bridge
Multi-platform publishing engine complete. AWS S3 temporary asset bridge: upload once, generate a 2-hour presigned URL, broadcast concurrently to Instagram Reels (Meta Graph API) and Facebook Pages (Meta Graph API), auto-delete. Concurrent pre-production (script + TTS in parallel across all batch candidates). 100% CPU core saturation via os.cpus().length Remotion concurrency.
What's next?
The v3 roadmap includes containerization with Docker Compose (see DOCKER_NOTES.md), PostgreSQL → MongoDB Atlas migration for unified data access, and the DevByte Wiki — a long-form article companion to the short-form video channel.