Skip to content

Two AI Models Agreeing Doesn’t Make an Answer Correct

4 min read4 views★ featured[Developer Workflow]
Cover image for "Two AI Models Agreeing Doesn’t Make an Answer Correct"

An assistant proposes an explanation for a bug. You're unsure, so you ask another assistant. It agrees.

That can feel like independent confirmation. But agreement is useful only if the second answer brings independent evidence or a meaningfully different check. Two fluent explanations can still rest on the same wrong premise.

I would use a second model to broaden the investigation, then verify the claim against the system.

Identify the claim before checking it

“Model B agrees with Model A” is vague. What exact claim are we evaluating?

Consider a hypothetical slow endpoint. One assistant says the problem is a missing database index. Another says the same thing after reading the first answer. There are several distinct assertions hidden inside that recommendation:

The endpoint is waiting on database execution.
The expensive query filters on the proposed columns.
The planner can use the proposed index for this workload.
The index improves latency enough to justify its write cost.

Each assertion has a different check. A trace locates the wait. The actual SQL reveals the query. An execution plan shows how it runs. A representative measurement compares the result.

Agreement about the index is much weaker than completing those checks.

Don't feed the second reviewer the conclusion first

If the second assistant sees “We think the missing index is the cause; confirm,” you've framed its job around defending a hypothesis.

A more useful review brief would be:

Here is the request trace, generated SQL, schema, and query plan.
List plausible causes of the slow response.
For each cause, name an observation that would distinguish it.
Do not assume the database is the bottleneck.

You can later compare the proposed explanations. Differences are valuable because they identify places where the available evidence is insufficient.

Even this setup does not establish statistical independence between models. It simply reduces one obvious source of anchoring in the conversation.

Demand different evidence, not different wording

Suppose the claim concerns a library API. One assistant remembers a function name, and another recognizes the same name. The decisive next step is usually to inspect the installed version's documentation or exports and run a minimal example.

Suppose the claim concerns a race. A second prose explanation is less useful than a schedule showing the conflicting operations:

Request A reads "pending"
Request B reads "pending"
Request A performs external effect
Request B performs external effect
Both mark the row "done"

Now you can test whether the implementation permits that interleaving and whether a proposed fix prevents it.

The second assistant earns its place by constructing that check, finding an omitted assumption, or proposing a counterexample—not merely by sounding confident in a different voice.

Use a claim ledger for complicated changes

For a migration or architectural change, I would keep a short working note:

Claim: old and new application versions can coexist during deployment.
Evidence: compatibility test with both versions against the expanded schema.
Unresolved: old worker behavior when it sees the new job type.
Next check: run an old worker against a seeded new-type job.

This keeps “we discussed it” separate from “we verified it.” It also makes a handoff useful: another developer can inspect the evidence without reproducing an entire chat.

The note should be proportional to the change. A one-line fix does not require a research dossier. A change involving data loss or deployment compatibility deserves more than a consensus summary.

Preserve uncertainty when evidence is unavailable

Sometimes you cannot reproduce a production-only issue locally. Sometimes the necessary trace was never captured. The honest outcome is a ranked set of hypotheses and a plan to collect the missing observation.

A second model should not be used to manufacture certainty that the system data does not support.

For the slow endpoint, I would rather say “connection-pool waiting remains unmeasured” than install three speculative indexes and call the investigation complete. The former tells you exactly what to inspect next.

Decide where the second opinion helps

A second assistant can be useful for adversarial review, alternative designs, omitted edge cases, and explaining unfamiliar evidence. It can also add noise if every minor decision becomes a contest between generated opinions.

Use it where another perspective might reveal a meaningful mistake. Then resolve the important claims through documentation, code inspection, execution, or observed behavior.

The most reassuring answer is not that two models agree. It is that the claim survived a check that would have failed if the claim were wrong.

# Comments (0)

No comments yet. Be the first.

0/1000