How It Works
The DEVLAR pipeline is a deterministic, five-stage state machine. It runs periodically via cron on an AWS EC2 instance, traversing from internet discovery to multi-platform broadcast entirely without human oversight.
Overview
Unlike typical AI wrappers that fetch one URL and paraphrase it, DEVLAR is modelled as a digital newsroom. Every stage has a single, well-defined responsibility, and failure in one stage never cascades — each gate protects the next. The pipeline starts with four independent discovery streams and fans in, running through deterministic hard-gates before a single LLM token is spent.
The editorial mission is strict: "DevByte covers products, releases, tools, developer infrastructure, AI breakthroughs, and major engineering announcements — not essays, tutorials, opinion pieces, or long-form discussions."
Zero Permanent Storage
The engine runs on ephemeral EC2 compute. No video files are stored permanently. Every rendered MP4 is relayed via a 2-hour presigned S3 URL, broadcast to all platforms concurrently, and then automatically deleted — storage costs remain absolutely flat at any scale.
01Content Discovery
The pipeline fans out to four independent collectors running concurrently. A failure in one source never blocks the batch — each collector is isolated and writes to its own output before they are merged downstream.
Polls top 50 stories. Items are scored for engagement velocity and tech-topic relevance before passing the signal filter.
Direct feeds from OpenAI, Google DeepMind, Anthropic, Meta, and AWS. No crawling needed — these are curated high-trust sources.
Watches /releases/latest on a curated list of tracked infrastructure repositories. Picks up major version bumps.
Developer tools, AI infrastructure, and productivity launches. Score- and recency-filtered before normalization.
02Ingestion Pipeline (Zero-Token Gates)
Before a single Gemini API call is made, every candidate must pass five deterministic hard-gates in sequence. These are rule-based — fast, cheap, and predictable. They exist so the expensive AI editorial layer only sees structurally sound, non-duplicate, technically relevant content.
Maps all four sources into a single unified candidate schema. Field names, date formats, and URL structures are standardized here.
Keyword whitelist + noise blacklist. Whitelists technical topics; blacklists tutorial, guide, opinion, essay, roundup, vercel, cloudflare.
Enforces schema presence. Drops items with missing titles, empty descriptions, or malformed URLs.
Story-level identity check across sources using canonical URL normalization and fuzzy title similarity. Prevents the same story appearing twice.
Hard 14-day recency cutoff. Any item older than 14 days is dropped without consulting the LLM.
03AI Newsroom — Two-Pass Gemini Editorial
Survivors from the ingestion pipeline enter the AI editorial layer. Instead of arbitrary composite scores, DEVLAR uses an evidence-driven approach: an Evidence Builder compiles objective facts, then Gemini Flash evaluates each item twice. Temperature is fixed at 0.0 and all responses are validated against a strict JSON schema.
"Code determines what is allowed. Gemini determines what is worth publishing."
Pass 1 — Mission Check
Items are batched in chunks of 10. Up to 5 chunks are sent concurrently via ThreadPoolExecutor. Gemini classifies each item as publish or reject with a concrete 1–2 sentence editorial reason grounded in factual evidence, not a numeric score.
Pass 2 — Relative Ranking
All approved items are ranked 1..N in strict order by newsworthiness. Large batches (50+ items) use segmented tournament ranking. Company diversity (max 2 per company) and source diversity (max 3 per source) constraints are enforced. The top 5 items form the production queue.
30-Hour Cache + Re-Scoring
Evaluation results are cached for 30 hours in data/evaluation_cache.json. Automatic re-scoring is triggered if a cached item goes viral (HN points surge >50%) or gains cross-source corroboration before the cache expires.
04Concurrent Video Production
The production phase is divided into two concurrency models. First, all script generation (Gemini) and TTS synthesis (Edge TTS) run in parallel for all queued stories simultaneously — all audio files are ready in ~10–15 seconds. Then Remotion renders each video sequentially, pinning 100% of the EC2 CPU cores per render.
8 category-specific prompt templates (update, free_alternative, hidden_gem, productivity, comparison, weekly_roundup, best_for, prompt_trick). Scripts are generated concurrently for all queued stories.
Microsoft Azure neural voices with sentence-level timing metadata. Timing data drives exact scene-cut boundaries in the renderer.
React video engine. Invokes --concurrency=${os.cpus().length} so every CPU core is pinned during render. Generates 1080×1920 MP4 with spring physics and dynamic timing.
Sanitizes Gemini script output. Checks structural integrity, field presence, and enforces the expected timing schema before TTS is called.
// Dynamic concurrency — pins ALL available CPU cores
const concurrency = os.cpus().length;
await renderMedia({ concurrency, ... });
// Timing example — from actual pipeline performance log
// Gemini Script : 6.2s █
// TTS Voice : 4.1s █
// Remotion : 58.4s ████████████████████
// Total : 142.0s05Triple Broadcast via S3 Bridge
Once a video is rendered, it takes two upload paths simultaneously. The direct YouTube upload uses the Data API v3 with OAuth 2.0 and 2 MB resumable chunks. For Instagram and Facebook, the video is uploaded once to AWS S3 and a 2-hour presigned URL is generated. Both Meta APIs receive that URL concurrently. As soon as both platforms confirm publication, the S3 object is automatically deleted.
YouTube Shorts
OAuth 2.0 Resumable2 MB chunked upload. Metadata (title, tags, description) injected dynamically from the script. Private staging, then immediate publication.
Instagram Reels
Meta Graph APIREELS media container → async status polling → instant publish. Delivered via the S3 presigned URL bridge.
Facebook Pages
Meta Graph APIPage Video endpoint. Asynchronous processing with status verification. Concurrent with Instagram via the same S3 presigned URL.