Shipping faster with AI isn't engineering maturity. It's a demo that hasn't met year two yet.

Every AI rollout comes with a dashboard. PRs merged per week, time from ticket to prototype, model accuracy on the eval set. All of those numbers go up and to the right in the first quarter, and all of them are measuring the same thing: how fast you can produce something new. None of them measure what happens to that something eighteen months later. That's the gap. The demo version of a system is judged on how quickly it appeared. The production version is judged on whether it's still understandable, still stable, and still changeable after three teams, two reorgs, and a vendor price hike have happened to it. AI made the first part dramatically cheaper. It did nothing for the second part, and in some teams it made it worse. Here are three tests I use instead of velocity. None of them are new. That's the point. Test 1: can a new developer read it without an AI explaining it? Illustrative scenario: a team generates most of a service with an assistant. It works. Six months later, a new hire is asked to change how refunds are calculated. They open the code and find four helpers named processData, handleResult, transformPayload and finalizeResponse, each one 200 lines, each one doing a slightly different slice of refund logic. Their first move is to paste it into a chat window and ask what it does. That's the smell. If the only way to understand your codebase is to ask a model to summarize it, the code isn't documenting itself, and you've made comprehension a dependency on a tool. What to check: Can someone find where a business rule lives by reading file and function names alone? Are the domain concepts (Refund, Invoice, Tenant) visible as types, or buried in dictionaries passed between generic helpers? Would a code review catch a wrong rule, or would the reviewer also need to ask the model? # Hard to read without help def handle_result(data, cfg, flag=False): ... # Readable on its own def calculate_partial_refund(order: Order, returned_items: list[LineItem]) -> Money: ... AI-generated code isn't the problem. Unreviewed, unnamed, unstructured code is, and AI lets you produce it at a volume nobody can review. Test 2: does it run without someone babysitting it? A mature system is boring in production. A system that needs a person to restart a worker every Tuesday, re-run a stuck job, or manually clear a queue is not stable. It's being held up by a human who hasn't been written into the architecture diagram. Illustrative scenario: an ingestion pipeline built quickly with lots of generated glue code. No idempotency, no dead-letter queue, retries implemented three different ways in three places. It mostly works. "Mostly" means one engineer spends a few hours a week nudging it, and when that engineer goes on holiday, the on-call rotation discovers the runbook is "ask Sam". What to check: How many manual interventions did production need last month? Is anyone counting? Are failures handled by design (retries with limits, idempotent handlers, clear failure states), or by people? Could the system survive the departure of the person who knows its quirks? Test 3: can you swap a core dependency without rewriting the business logic? This one matters more now than it did three years ago, because the dependency most teams just added is the one most likely to change. Model providers change pricing, deprecate versions, and change behavior between releases. If your provider's SDK is imported in forty files and its response shape has leaked into your domain objects, switching is a rewrite. // Business logic coupled to one vendor const res = await openai.chat.completions.create({ ... }); order.summary = res.choices[0].message.content; // Business logic depends on a capability, not a vendor interface Summarizer { summarize(text: string): Promise; } order.summary = await summarizer.summarize(order.notes); The same applies to your database, your queue, your payment provider, and your auth vendor. The question isn't whether you'll ever swap them. It's whether the decision is yours to make, or already made for you by how deeply they're wired in. The reframe Velocity tells you how fast you can create code. Maturity tells you how cheaply you can live with it. AI changed the price of the first one. The second one is still priced in discipline: clear naming, stable failure handling, and clean boundaries around things you don't control. AI is a tool. A very good one. But the systems still standing in five years will be the ones that would have been well built without it. What's the one signal you trust more than velocity when you judge whether a codebase is actually healthy?