Give AI somewhere to prove its work

Earlier this year I led a team that used AI to replace a self-hosted FTP server with a managed file transfer platform on AWS.

It has run in production since cutover without an incident.

We used AI, mostly Claude with a mix of Codex, at every stage: research, build, and delivery.

In this post I’ll discuss what we did, and how we did it.

The problem

Clients had credentials and scripts pointed at the old box, some of them years old and written by people who have moved on. Break the endpoint and royalty statements stop arriving. The replacement had to keep serving those clients over FTP while everyone else moved to SFTP, keep each client’s files isolated in S3, and expose no admin surface to the internet.

I hold the AWS Developer Associate certification. It did not cover any of that.

Research by running the thing

What closed the gap was a prototype. Before any infrastructure code existed, I gave Claude the context and had the server running in Docker on my laptop, with Postgres for users and MinIO standing in for S3. I could provision a client and upload a file within minutes.

That gave me a loop. For every question about how the thing behaved, I could try it, watch what happened, then ask the model why.

Why does passive FTP hang without an advertised IP? Try it without one, watch the client stall, ask.

What happens to a user whose home directory does not exist yet? Try it.

The prototype gave me something concrete to reason about, and the model turned each experiment into an explanation and a pointer to what the community considers good practice. After a week, I could hold my own in a design conversation about a protocol I had never run in anger.

Understand the problem before you walk into the room, and a prototype is the fastest way I know to do that.

The same loop caught the constraint that shaped the whole architecture. Passive FTP requires a range of data ports, and the load balancer has a hard limit on how many it can forward to a single service. The model explained the limit; I asked it to point me at the AWS documentation it drew on, I read that documentation myself, and the first deploy confirmed it. The stack now refuses to deploy if you widen the port range past what the platform can serve. The model then helped me trace the consequences: that limit is why FTP runs in a single availability zone, and why the case for moving clients to SFTP was there from day one. A colleague’s questions at our first meetup sharpened the argument.

How we worked

Five people: me as the lead engineer; people who understood the business, requirements, and domain; and people more focused on security. Each of us knew enough AWS to follow the others, and each brought one area where the rest of us would have guessed wrong.

That mattered more than I expected. The model could write plausible infrastructure-as-code, but it would not apply least privilege unless we asked for it and then checked.

Knowing the vocabulary changes what you ask for, and knowing what good looks like changes what you notice when you review the result. I apply least privilege by default, but having someone whose job was to challenge and verify it gave us another line of defence, and it meant our first logging and alarms did not need to be perfect, just good enough to decide what we actually wanted from them.

My knowledge and confidence in using AI eliminated handoffs. Being able to do the work myself meant not waiting on a ticket to another department, and nobody took a design to a committee. I have written before about why design by committee leaves engineering change unfinished; this was the alternative.

No sprints, story points or stand-ups. It was a bounded problem with a small, focused group, and that let us shorten the feedback loop to: deploy something, meet up, change it, deploy again.

Between catchups, the work happened in a shared Slack group: a deploy result, a question about IAM, a link to a PR, in the open where anyone could answer. I have argued before that teams should communicate like a team fielding in cricket, and this is what that looked like on a small project.

The first version had a typed CLI instead of an admin console. Every trade-off like that had a documented upgrade path.

There was a deployed solution at every stage. Anyone could try it out in our non-prod environment. Anyone on the team could deploy the stack under their own name, CI deployed a throwaway stack per pull request; and cdk-nag flagged the security findings that a reviewer would otherwise hold in their head. Every change ran on real AWS before anyone reviewed it.

Build: what the model did and what we learned

Claude wrote the implementation: the infrastructure constructs and their tests, a typed admin client generated from the server’s OpenAPI spec, a smoke command that provisions a client and uploads a file, and a migration tool that moves a roster of users off the old server.

It did not decide what the system should be.

The architecture, the constraints it had to meet, and the acceptance criteria came from the prototype experiments and the conversations among the team members. The model wrote the operations playbook, the upload verification guide, and the pre-deploy checklist to those criteria. Because the same session that changed the code also updated the docs in the same commit, they stayed in sync.

The part I would show a sceptic is use-case evaluation. When someone asked whether a distributor’s upload client would work against the new stack, the model provisioned a user, ran the upload over the protocol that client uses, confirmed the bytes landed in S3 under that client’s prefix, and wrote up what it had done. That took minutes. Without AI, it would have taken an afternoon and a shared spreadsheet.

The first version was one stack file. It synthesised, it deployed, and it was messy. That is exactly where you want to be: make it work, then make it right. A working stack on real AWS taught us more in a day than a clean design on paper would have in a week, and it gave a colleague something concrete to react to. Their suggestion was to split it into constructs by concern: stateful resources, the service, ingress, admin access, the event hook, observability. The model had not proposed that, and once we did, the generated CloudFormation was cleaner as well, because the resource groupings now matched the code. Reviewers could read one construct at a time.

One thing cost nothing but could have. The alarm for authentication failures depends on a log filter matching specific phrases in the server’s output. The model wrote three phrases with confidence. Nobody had checked them against a real log line. It had produced something that looked testable without testing the assumption it depended on. If the phrases were wrong, the alarm would never fire, and nothing would tell you. I caught it while preparing for review, and it is the clearest example I have of why someone still has to read what the model writes.

For that review, I asked the model to build a decision inventory: every construct, what it does, why it exists, the alternatives we rejected, and the hardest question a reviewer could ask about it. I walked into the review able to answer from memory.

Delivery

Two things came out of the build that the business could see.

The first was a Slack channel that showed the business every file the moment it arrived. That came from the experience I wanted users to have. AI can’t do this for you on its own. I knew one of the weaknesses of the existing system was visibility, so I explored that idea with AI and used it to help implement and deliver it.

The second was that I did not deploy it to production. Another team did, and I was not in the room. What they had was the repo: the playbook, the checklist, the migration tool, and a verification script the model wrote against the upload guide. That script makes one probe upload per user in the migration roster, confirms each object lands under its own prefix, deletes the probes, and exits non-zero if any user fails. They deployed, migrated the roster, ran the script, and moved DNS. Nobody messaged me with a question. The old box is off.

That is the result I am proudest of. The same sessions that wrote the code wrote the docs; we followed those docs on ephemeral stacks every week, and they turned out to be enough for a team who had not built the thing to put it into production.

What I would tell another team

Give AI somewhere to prove its work. Every part of this project that went well had the same shape: the model did something, then ran it against a real system and showed the result. The Docker prototype on my laptop. The ephemeral stack under my name. The smoke test that provisions a client and uploads a file. The probe uploads before cutover. The one part that went wrong, the log filter, was the one place the model asserted without running anything. It is show, don’t tell applied to a model: a claim in a chat window is a tell, an upload that lands in S3 is a show.

Keep people on the structural decisions. Two important things the model did not do on its own were reorganise the code once it worked and challenge whether one of its own claims was true. Both are judgement, and both took a colleague or a reviewer.

Have the same session that changes the code change the docs. Docs written afterwards drift. Docs updated in the same commit start true, and ours carried a production deploy by people who had never seen the code.

Deploy early, meet, change it. Five people, little ceremony, and that loop shipped a platform that royalty statements now depend on.

How much code the model could write turned out to be the less interesting question. The one that mattered was whether we could provide an environment in which every important claim could be supported by evidence. We could, and that is why the old box is off.