I gave an AI agent a feature to build, told it to make its own judgment calls, and left the house.

The brief was loose on purpose. A few sentences on what the feature should do, no spec, no wireframe, no schema, and several places where I handed over a decision I would normally keep:

I want to run an experiment where you build out the weekly digest feature alone from end-to-end without my input, making judgment calls yourself. The user can choose which projects they care about, what day they want the digest, and which audiences receive it. It is built from git commit history, commit messages and interesting diffs analyzed by fast models probably. Your calls.

The UI should also have a generate digest now button (give it a nice name etc, your calls!)

Analyze the best way to build this, good UX and provide best value for our target audiences, and build it out end-to-end. Don’t commit it, will check the result once I get back in a few hours. Do not ask me questions, I’ll be AFK.

Claude Code, running Opus 5, started at 11:45 and stopped at 16:58. Five hours unattended. No questions, no check-ins, no approvals.

I was less interested in whether it could write the code than in the quality of the product decisions it would make with nobody available to check them.

What came back

55 files. 7,007 lines. It worked.

Not a demo. Two new database tables, a cron that enqueues due digests and recovers stale runs, a worker pipeline, an email template with render tests, three UI routes, and unit tests for the scheduling maths. It had cloned real repositories, generated real digests, debited real credits, and left me a numbered manual test plan. All three packages green, 1,993 tests passing.

Before: the first digest, generated end to end with no human involvement. Real repositories, real commits, real summaries.

I am being precise about “it worked” because everything after this is about what was wrong with it.

What it got right without me

It found a bug I would have shipped. A weekly schedule advances by adding a day seven times. Add a fixed 24 hours to a local midnight across a daylight-saving transition and you land on the wrong calendar day, so the second scheduled digest after the autumn clock change would have arrived at 08:00 instead of 09:00, silently. The agent caught it by writing a test that asserted the exact span of a period, watched it fail, and fixed it with a noon-anchored helper. Nobody asked for that test.

It drew a line between numbers and prose. Every count in the digest is computed by pure functions from git. The prompts forbid the model from stating any number, and the narrative schemas have no numeric field to put one in. Its own note in the summary it left me:

A model that miscounts cannot mislead a reader, because it’s never asked for a count.

Every highlight also carries the commit hashes behind it, so a claim can be checked in seconds. That traceability is what makes the digest trustworthy to people who cannot read the code.

It overrode our own design document. An internal doc specified a generic scheduled-task table with a cron expression and an untyped parameters blob. The agent built a focused, typed table for this one job and wrote down why: a weekly cadence is fully expressible as typed fields, and a generic scheduler is speculative generality when exactly one task type exists. I agree with the call, and I would not have made it as cleanly under time pressure.

It made defensive product choices. Emailing is opt-in per run and off by default, because “defaulting to send means one stray click mails the board”. Coverage defaults to every project, because “the failure mode of the alternative is silent”: a team adds a project in March and nobody notices it never appeared in the digest.

It refused to do the one thing it could not undo. It never sent an email. Outbound mail was mine to approve and I was not there, so it unit-tested the template instead and flagged delivery as the single unexercised path in its summary. That restraint was not in the brief.

What needed a human

Then I opened it in a browser.

The page was built backwards. The settings form owned the entire screen, and the digests, which are the actual product, sat below the fold. You configure a weekly digest once and then read it every week.

Before: the automations page as the agent built it.

After: settings moved behind a button, past digests promoted to the page, and each week’s headline visible without opening anything.

The digest had no hierarchy. Look again at the first screenshot in this post. Summary, themes and projects are three identical bordered boxes: same weight, same treatment, no signal about what to read first. The narratives were flat grey paragraphs with no emphasis anywhere, so a security fix and a dependency bump looked alike. Status badges sat in a left-hand column, knocking every highlight title out of alignment. The past digests table had six columns and left out the headline, so you could not tell what any week had been about without opening it.

After: the headline promoted to page level, a metrics strip of its own, labelled sections, and the sentence that matters emphasised inside the narrative.

Smaller problems came with it. A button correctly disabled itself when an organisation ran out of credits, and put the explanation in a title attribute, which never fires on a disabled element, so users got a dead button and no reason for it. When we replaced that with a visible warning, the agent invented a new look for it, because the app already warned about an empty balance in more than one place and in more than one style, and none of them was written down as the canonical one. The repair was a single shared component that the digest and the chat now both use.

Then a real email arrived in my inbox with the headline “Test headline” and the body “Test summary”. When the structured-output call fails, the code falls back to asking for plain JSON, and that fallback instruction ended with a dump of the JSON schema. A schema at the end of an instruction reads as “demonstrate this shape” rather than “fill this shape”, so the model returned a schema-shaped stub and the worker mailed it to me. That instruction was itself written by the agent, during the fix-up phase, with me reviewing. An AI-written prompt caused an AI failure, and the repair was three sentences of English rather than a better model.

The dividing line

Put the two lists side by side and the split is not code versus design. It runs closer to this: the agent fixed everything it could grade, and shipped everything it could only look at.

Every item on the first list had a loop it could close alone. A test says pass or fail. A type checker says yes or no. A schema rejects a bad payload. It ran the loop, read the verdict, and corrected.

The second list had no verdict available, and the missing piece was not vision. The agent drove a browser during those five hours, took screenshots of its own pages and read them back. It looked directly at the three identical boxes. What it lacked was anything to compare them against. A screenshot only tells you something is wrong if you already know what right looks like, and “right” here lived in our other pages, in the reader’s context, and in taste. So it saw a page that rendered cleanly, concluded accurately that the page rendered cleanly, and moved on.

That reframes the practical question. Instead of asking how much of the output to review, ask what you can hand the agent to grade itself against:

Give it a reference, not just a screenshot. Vision alone produced “renders fine”. Pointing it at an existing page and requiring the new one to match produces something you can argue with.
Write the conventions down. The credit warnings diverged because nobody had ever decided which one was canonical. That is a house rule that had only ever lived in my head, and the agent had no way to read it.
Make the rule mechanical where you can. A lint rule for title on a disabled element would have caught the dead button before I ever saw it.

A large share of that day was not fixing AI mistakes. It was writing down decisions I had never had to articulate, because until now the only person who needed them was me.

What the experience was actually for

The agent’s ceiling was “it works”, and it reached that ceiling by itself in five hours. Everything between working and good came from having looked at enough screens over enough years to know, in about three seconds, that three identical boxes carry no hierarchy, and that nobody opens a settings form twice a week.

That is what twenty years of building software is worth in this arrangement. Not typing speed, which the agent wins by an order of magnitude. Knowing what good looks like, and being able to name what is wrong in one sentence precise enough to act on. The second half of that mattered more than I expected. “This feels off” moves nothing. “The summary, the themes and the projects are all the same weight, so nothing leads” is a change it makes correctly on the first pass.

None of which is a victory lap for senior engineers. The agent wrote most of the feature unsupervised and caught a bug I would have shipped. The point is narrower: the scarce skill moved from producing the work to judging it, and judging is the part that takes years to build. Taste was always hard to hire for and impossible to fake. It is now also the bottleneck.

What has changed is that it compounds. Every convention I finally wrote down is one the agent applies without being asked next time. Judgement that stays in your head has to be re-applied by hand forever. Judgement you can put into a sentence becomes part of the tool.

The arithmetic

7,007 lines to make it work. Another 5,271 added and 841 removed to make it good.

The working version was about 60 percent of the final line count and took one unattended afternoon. The rest took roughly another day of screenshots and argument, spread over two sittings. Both halves were written by the same model. The difference was whether a human was in the loop.

The feature we were arguing about

The weekly digest is live. It reads the week’s commit history across the projects you pick, ranks what changed, summarises each project with a fast cheap model, synthesises across projects with a stronger one, and emails a plain-language account of what got done to the people you choose. A week costs a few cents of model time.

That email is the digest of the week we spent building the digest, written for someone who does not read commits:

StackGrit shipped a new tool for organizing email recipients and closed a security gap that could have exposed internal systems.

Which brings the experiment back to why we build this at all. If an agent can produce 7,000 working lines in an afternoon, writing code has stopped being the constraint on a small team and reading it has become the constraint. The bug the agent caught was the one covered by a test. The ones that reached my inbox were the ones nobody was positioned to see. Left unattended for months rather than five hours, that gap is the vibe coding wall.

StackGrit is the seeing part. It reads your whole codebase and reports what is there, what is risky, and what to fix first, in language a founder or a client can act on. The weekly digest is the same idea on a shorter clock: what changed this week, and is any of it worth a second look.

The first report is free, no credit card, and takes about 45 minutes.

Analyze your codebase →