Misplaced Trust: How Research Agents Judge Credibility

This is the first of a series of posts about something that keeps cropping up in my work: how can you trust what an agent does? First off, for the purposes of these posts, trust is whether you believe what an agent tells you and feel relaxed about the decisions they make. This is not the same as security, safety of the boundaries of authority - those are separate concerns. The trust question is: do you need to check, by hand, that your agent got the right answer in the right way?

I’ll start with a trust-related issue that plagues research agents: content credibility analysis. When an agent finds a document and relies on it to help answer a question, was its faith well placed?

It turns out that often the answer is “no”. Research agents frequently cite content they shouldn’t, a finding measured by several studies. For example, they will happily use sources known by media monitoring groups to be unreliable, or lean on supposed scientific “authorities” that have no institutional backing.

How do models evaluate content?

So how can a research agent decide what content is credible? There are essentially four ways:

1. Corroboration: are the document’s assertions consistent with the assertions within other documents?
2. Provenance: does the document come from a trustworthy source?
3. Quality: are there clues in the document’s format, composition and presentation that indicate its reliability, or otherwise?
4. Coherence: how much of what the agent believes would be changed by the claims in a document? An extraordinary claim that forces a cascade of revisions should require extraordinary evidence.

It’s not as if AI models don’t attempt to evaluate content - they do - and we can systematically check what they consider by running experiments under controlled conditions.

Take quality, for instance. Researchers have identified a handful of surface attributes which models seem to glom onto - rightly or wrongly - when evaluating credibility. Among them are:

• Directness: making a claim explicitly beats leaving it implicit in the text or spread among several texts;
• Academic dress: models love dense, academic prose formatted like a paper;
• Self-claimed authority: a document that proudly claims expertise or credentials is more likely to be believed - regardless of whether such claims are warranted.
• Junk styling: models penalise content that looks or is formatted like clickbait.

And the icing on the problem-cake: Generative Engine Optimisation (GEO) providers know all of this. The web is increasingly stuffed with content designed to be treated as authoritative by models. It’s getting harder for them to sort fact from fiction on the basis of content alone.

On coherency - you might hope that an incoherent claim would trigger a research agent to gather more data. It doesn’t. Research shows that models value quality clues above all others. A direct, confident yet incoherent claim will actually result in a research agent making fewer additional searches, not more.

We can also test whether deep research agents use corroboration to evaluate claims by building convincing-looking documents containing falsehoods and checking whether the agent will discover that they are at odds with claims in other documents. The consistent finding: they often do not; under some test conditions, the rejection rate of such content has been measured at only 15%.

Which methods actually work?

So, what techniques for agentic content credibility analysis have been found to work? Here are the big three:

1. Tiering sources. Maintaining a curated “tier list” of sources (authors, publishers, domains) and using these to gate what evidence the agent can see and which to prefer. Requires designers to curate the source tiers but works very well.
2. Corroborating against authoritative sources. Accepting lower-tiered documents only when they contain facts and claims that corroborate with content from high-tiered sources.
3. Verifying citations. When documents cite authorities, checking that the citations exist and verifying that the inference is licensed by the cited authority.

Note that none of the above relate to document quality! Perhaps counter-intuitively, none of the research I have read has found it to be a reliable guide to factuality.

The challenge with the above list is that two of the techniques - source tiering and corroboration with authority - both rely on the agent having a curated tier list of sources. This is something of a pain to have to construct by hand.

Fortunately, data scientists tackled this question back in the 2010s when they were building the first, web-scale knowledge bases. The best approach - which is just as relevant today - is to estimate claim truth and source reliability jointly and iteratively. Here’s a (simplified) sketch of the process:

The great advantage of this approach is that you don’t need to tier and rank your sources by hand, up front. Instead, you allow your agentic harness to learn this over time.

Quick aside: joint estimation is not a panacea; it doesn’t work well for the “long tail” of rare claims and niche sources. The solution, I believe, is to modify the approach to derive an uncertainty measure alongside the credibility measure for source - and to have this propagate through the research agent’s synthesis step. This is something I am working on, at present. (Ask if interested.)

Tips for building a trusted research agent

For now, if you are building a research agent, and you want to improve the credibility of the content it relies on, the key steps are:

1. Have your harness enforce a corroboration step when evaluating documents and claims.
2. Try to group and deduplicate documents according to their origin: for instance, when newswires replicate press releases, these should not all be returned to an agent since they will artificially inflate any corroboration calculation. The ideal (which is hard to achieve) would be to only allow documents with an independent origin to be returned to an agent by its search tools.
3. Tier your sources by authority - whether by hand or using the method outlined above. Have the tier define what the agent may or may not do with the source, for instance:
   1. lower tiers cannot drive citable figures, unless corroborated
   2. lower tiers cannot outvote a higher tier (regardless of how many lower tier sources agree on a fact)
   3. no search may terminate early having only found content from lower tier sources.
4. When there is conflict among the documents, don’t simply count corroborations to decide which is correct: weight the counts by source authority.
5. Force the harness to confirm that citations check out and any inferences are licensed.
6. Make the harness force the agent to actively seek disconfirming evidence for any “load-bearing” facts used to reach an answer. This is a way to force the agent to test for incoherency, something it will not do otherwise.

Taken together, these steps will significantly improve the content credibility analysis a research agent performs. This, in turn, should raise the trust you place in its outputs.