Triage Bot

Challenge

A solo operator running eight consumer products cannot personally read every piece of user feedback, classify it, decide whether it is a bug or a feature request, file it, and reply in each product’s voice. That work is real, constant, and exactly the kind of judgment most teams throw people at. The question was whether a single agent could do that judgment safely: autonomy where the cost of a mistake is low, a hard human gate where it is not.

Process

Triage Bot is a persistent Python service (asyncio, discord.py) running on one AWS EC2 instance. Feedback enters through Sentry widgets embedded in each product, fans out to per-project Discord channels, and flows through a single pipeline: a moderation pre-filter, a mandatory PII scrub (Microsoft Presidio) before any text reaches a model, then a Gemini 2.5 Flash-Lite classifier returning a schema-validated label, confidence, and severity.

The agent then acts on its own judgment. It files structured GitHub issues for bugs, dispatches draft-only auto-fix PRs for high-confidence low-severity ones, cross-posts feature requests to the right forum, and drafts a reply in that product’s exact brand voice, retrying itself when a draft trips a banned-phrase rule. Sensitive topics (refunds, legal, security, pricing) route to a human and are never auto-drafted. Adding a ninth product is a YAML entry, not a code change.

Safety lives in code, not policy. Auto-fix PRs are hardcoded draft-only; high and critical bugs are excluded from automation; prompt-injection attempts are classified as spam rather than followed; and every public reply passes an approval gate restricted to the owner’s Discord ID. The model choice was itself a cost decision: Gemini 2.5 Flash-Lite replaced Claude Haiku for roughly a 10x cut in per-message cost, with thinking_budget disabled after a measured token-starvation finding.

Result

Live and running in production. One service triages feedback for eight separate products, each with its own enforced voice and escalation routing, at a target cost under $10 per month. It is the portfolio’s clearest exhibit of an agent taking real, consequential action (opening real issues, dispatching real PRs) inside hard, code-level safety boundaries, not a demo that only answers questions. Promotion to unattended auto-reply is deliberately gated on a sustained approval-rate threshold that has not yet been reached.