How I Built celevent With an Agent (and Everything I Didn't Trust It With)

celevent finds Pittsburgh shows wherever they get announced (Bluesky posts, the live firehose, venue websites, City Paper, Ticketmaster), has an LLM decide which ones are real, and publishes them from @pittsburghevents.bsky.social and a custom feed. It posted 49 shows its first night.

An AI coding agent wrote most of the code. That's not the interesting part anymore. The interesting part is the gap between "the agent says it's done" and "it works on real data." This post is about that gap.

The spec

The first version of this project, calevent, was a toy: events were Bluesky posts in a rigid format (TITLE:, START:, #calevent). Nobody posts like that. Venues post flyers. Bands post "TONIGHT!!! doors 8." So the real spec was:

1. Collect from everywhere shows get announced.
2. Filter cheaply before anything expensive touches the data.
3. Have an LLM judge each candidate, but keep the actual accept/reject rules in code I can test.
4. Publish once per show, no matter how many places announce it.
5. Never double-post. Retries, restarts, and dry-run → live must be safe.

Points 3 and 5 were the ones I cared about most, because they're where an LLM-built system quietly goes wrong. A model in the loop is a source of confident nonsense, and an agent writing the code is a second one.

How I split the work

I didn't ask for "a bot." I asked for one stage at a time, each with a definition of done that wasn't "the agent said so":

• Pipeline core: collect → normalize → prefilter → dedupe → review → repost, with Postgres as the source of truth and dry-run as the default. Done meant integration tests passing against a real Postgres with a fake Bluesky client and a fake model.
• LLM backends: Claude and Ollama behind one interface, both returning the same schema-constrained JSON, so the review loop never knows which model it's talking to.
• Discovery and scraping: finding each venue's official account and events page, then scraping venues that don't post on Bluesky.
• Serving: a custom feed generator, OAuth with DPoP, health checks, metrics, alerts, a dashboard.
• Sources that only showed up after it ran: City Paper's guides and Ticketmaster, because real data showed what the first version missed.

Then I ran it on real data after every stage. That's where nearly every interesting bug came from.

What I rejected, and why

1. One search was eating the LLM budget

The first real runs searched Bluesky for every venue by name. One venue is called Spirit. Searching "Spirit" returned about 300 unrelated posts per run, and because venue names also count as location signals, all of them sailed through the cheap prefilter and burned LLM calls. The code was "correct." The data made it wrong. The fix was boring: use unambiguous venue names in searches. The lesson wasn't: look at what actually goes into the model, not what you meant to send it.

2. The model put the date in the wrong field

The local model (gpt-oss:20b) would sometimes leave event_date empty and write the date inside its free-text reason. Code then rejected a real show for having no date. I didn't want to prompt my way out of this ("PLEASE fill in the date field"), because prompts are suggestions. Instead, event_date became required in the schema, and the decision code falls back to an ISO date found in the reason. Rules in code, not in the prompt.

3. The discovery agent's answers are claims, not facts

The discovery step is itself an agent: it gets Bluesky account search, web search, and page fetch as tools, and reports each venue's official handle and events page. On the first real run, four venues failed because the agent just kept searching and never answered. It also sometimes reported URLs that didn't exist.

So: the final step only offers a "submit your answer" tool, which forces an answer instead of an endless loop. And nothing the agent says is trusted. A handle has to match the venue's domain, link the official site, or match the name on the live profile before the bot follows it. Every URL gets fetched before it's scraped. Invented URLs get dropped.

4. The bot started reposting itself

Once the bot was posting scraped shows, venue-name searches started finding the bot's own posts, and the review step happily approved them. Nothing in the spec said "don't review yourself" because it seemed too obvious to say. Now every source skips the bot's own account, and any earlier self-approvals are marked as such.

5. One venue, two spellings, two posts

Ticketmaster calls it "Thunderbird Cafe & Music Hall." The venue's site doesn't. Different names meant different event keys, so the same show posted twice. The fix compares the distinctive words in venue names and ignores filler like "cafe," "music," "hall," and "theatre." This is the kind of bug you only find by reading your own bot's timeline.

6. The tests passed on my machine and failed in CI every time

This one I found while writing this post. CI had been red on every single push, and the tests passed locally. The cause: locally, the Postgres driver wasn't installed, so the test helper silently fell back to a psql shim. In CI, the real driver ran.

One test skipped the reconnect delay by replacing the global setTimeout and treating any 1–60 second timer as a reconnect. The real Postgres driver starts its own 10-second idle timer. The test mistook it for a reconnect and shut the stream down early. The shim had no timers, so it never happened locally.

The fix was to pass the wait in as a parameter instead of hijacking a global. CI is green now, and it fails the build if line coverage drops below 85% (it's at 92.5%). The real lesson is the one I'd tell anyone using an agent: a green check on your machine is a claim, too. Verify it somewhere you don't control.

What I'd tell someone starting

• Give the agent stages with a testable "done," not a whole product.
• Run on real data after every stage. Every bug above came from real data, not from reading code.
• Keep decisions in code. The model extracts; tested code decides.
• Treat every model output as a claim, including the agent's own "it works."
• Make the safe thing the default: dry-run first, idempotent everywhere, caps on posting.

The agent made it possible to build all of this fast. Knowing where not to trust it is what made it work.

celevent's code is at github.com/martin473/celevent, with the full test suite and CI.