You don't need an ML engineer. You need a product engineer who doesn't trust the model.

There's a job-title collision happening right now that costs companies real money. A team decides to ship an AI feature, writes a req for an "ML engineer," and screens for the research skillset: model architectures, training pipelines, papers. Then the person they hire spends zero time training models, because the model comes from an API, and all of their time on problems the interview never touched: what happens when the model is wrong and how the user finds out, and what keeps the wrong answer from reaching them at all.

I've shipped several AI systems in the last year: a permit-readiness checker, a legal-change tracker, and a research-extraction pipeline that's indexed a 137,000-DOI catalog and fully extracted 2,233 papers into structured claims. The amount of machine learning I did, in the training-models sense, rounds to zero. The engineering around the model was the entire job. I've come to think of the role as its own discipline: the product engineer who doesn't trust the model.

What the work actually is

Strip away the demos and an AI feature is a probabilistic component embedded in a system that people need to behave deterministically. Every hard problem lives at that boundary, and none of them are solved by a better model.

Deciding what the model is allowed to do. In my law tracker, the model writes one thing: a plain-English explanation of a diff that plain code already computed. It never decides whether a law changed or produces the diff, and it never authors a date or a citation. That division of labor, model proposes within a lane and deterministic code owns the facts, is a product-architecture decision. No amount of model quality substitutes for making it, because the failure it prevents is categorical, not statistical.

Building the gate. Instructions aren't guarantees, so between the model and the user there has to be something with no imagination: a check that parses the load-bearing fields out of the output and verifies them against a source of truth. Citations get checked against the scholarly registries and numbers against the computation, legal references against the input they were supposedly drawn from. Anything that fails is quarantined with a reason, not retried until it slips through. This is bread-and-butter systems engineering, parsing and APIs and queues and idempotency, aimed at a new target.

Designing the abstention. The highest-leverage product decision in every system I've shipped is what happens when confidence runs out. The default in this industry is to answer anyway, because an empty state demos badly. The correct behavior for anything a professional relies on is to say "I can't confirm this, check it yourself," and making that a first-class outcome touches the UX, the API contract, and the data model. It's product work all the way down.

Measuring, so improvement is real. A small set of real cases with known answers, scored on every change. Mine have caught a newer, bigger model losing to an older, cheaper one, a bug in my own rules, and the true bottleneck hiding in a different layer than I assumed. Without the eval, all of that ships as vibes. Building one isn't research, it's a weekend of labeling and a script, but knowing it's the first thing to build is the expertise.

None of this shows up in an ML interview, and all of it decides whether the feature survives contact with users.

Why the confusion persists

The research skillset and the shipping skillset got conflated because five years ago you needed the former to have anything to ship. That inverted. The models are commodities behind APIs now, and differentiation moved into the system around them: the data plumbing, the gates, the evals, the failure UX. The industry's own post-mortems say the same thing: when AI initiatives stall, what kills them is integration, reliability, and trust, not model quality.

But hiring pipelines lag reality. So companies over-screen for skills the work no longer needs, under-screen for the ones it does, and are then surprised when the feature that aced the demo can't be trusted in production. Meanwhile the engineers who are good at this, who ask "where's the gate?" and "what does it do when it's unsure?", often don't call themselves AI engineers at all, because they've never trained a model and assume that disqualifies them.

The one question that sorts candidates

If you're hiring for AI product work, skip the architecture trivia and ask this: "The model returns a confident answer that is wrong, in a way that will cost the user money. Walk me through everything in your system that stands between that answer and the user."

The research-profile candidate talks about improving the model. The product-profile candidate starts listing layers: what gets parsed, what gets verified against what, where the confidence thresholds sit, what the user sees when the system abstains, what gets logged so the failure is findable later. The second answer is the job, and it's mostly what we used to call engineering. The new part is that one component of the system now lies fluently, and the work is arranging things so the lie never reaches a user.

1

Parse the load-bearing fields

Pull the citations, numbers, and references out of the model's output.

2

Verify against a source of truthgate

Citations against the scholarly registries, numbers against the computation, legal references against the input they were supposedly drawn from. Failures are quarantined with a reason, not retried until they slip through.

3

Confidence thresholds decide the outcome

A verdict when confidence holds, an abstention when it runs out.

4

The user sees the abstention

A first-class outcome in the UX, the API contract, and the data model, not an empty state hidden because it demos badly.

5

The failure gets logged

So it's findable later.

If you're an engineer wondering whether you're qualified for this work: if you've ever built input validation you trusted your job to, you're closer than most of the resumes in the pile.