Insights

Unaided winged flight is a fairly rare skillset

By Will Hardman · · 5 min read

The title of this piece is a gem of advice given to a journalist (who was posing as a job-seeker) by the Department for Science, Innovation and Technology’s (DSIT) new “AI work assistant”.

Positive, encouraging and - needless to say - complete bullshit.

The writer had introduced themselves as Long John Silver’s parrot and the chatbot had proceeded to take the rest of the interaction seriously. In another example, a journalist from the Metro had the assistant build his cat a CV.

You may be surprised to learn that the service was built by the Government Digital Service directly supported by engineers from Anthropic.

How, you may ask, could Anthropic’s own engineers release an agent capable of such silliness?

Actually, even with well-designed guardrails, it’s surprisingly difficult to detect a piss-take like this. And, once we’ve examined why, we’re left with a an interesting take-away.

But first, what kinds of guardrail should this agent have in place? Three kinds:

  1. Moderation guardrails
  2. OOD drift detection
  3. A rubric-based review stage.

Having multiple stages of guardrails provides a “defence in depth”, reducing the risk that bad behaviours are not detected.

Let’s cover them, in turn.

Moderation guardrails

Moderation guardrails are applied to parse the users’ requests before the agent processes them. They have three roles:

  1. To redirect irrelevant requests.
  2. To shut down harmful requests.
  3. To detect relevant requests where the required functionality or supporting data is lacking. (We don’t want the model to hallucinate or respond off-policy, so better to redirect at this point.)

In both of the above examples, the user conversation concerned employment potential, relevant questions were asked and these were answerable by the AI.

You might wonder whether the engineers shouldn’t have designed a guardrail to detect when the user was taking the piss. After all, it is intended to be used by members of the public in a country that voted to name its new survey ship “Boaty-McBoat-Face”.

Possibly. Its remarkably difficult to string together a set of criteria for detecting a piss-take that wouldn’t, inadvertently, shut down perfectly innocent conversations from users with complex needs or unique circumstances.

Many factors in agent design involve finding a balance between behaviours. In this case, it’s probably better to accept a certain number of pranksters rather than reject some users who may well have a particular need of the assistant.

OOD drift detection

OOD stands for “out of distribution”. It’s a technique for detecting when an agent may be on shaky ground.

During operation, the work assistant will be referencing source materials. These can include things like guidance documents or case-banks of model answers, vetted by qualified reviewers. When the agent searches for these materials, they will be scored based on their relevancy to the current conversation.

It has long been known that if relevancy scores of retrieved information begins to dip - or the responses returned by the agent do not resemble previous responses - these are warning signs that the agent may be heading off the rails. Ideally, you hand the conversation off to a human but, if no humans back the service then the best you can do is mark the conversation for review later on.

Both the parrot and cat examples were structurally in-distribution for the model. The conversations were textbook. The absurdity is semantic, which is why an OOD-detector would be blind to it.

Rubric-based reviews

The goal of the work assistant is to produce good CVs for job hunters. The purpose of the review stage is ensure that the agent’s output conforms to expectations.

Typically, this is done by having an independent model score the outputs according to a set of carefully designed rubrics (a rubric is a checklist enumerating what “good” looks like).

The advice ought only be returned to the user if the reviewing model is confident that all of the rubrics are satisfied.

The key is to realise that a rubric-based evaluation can only score an output on its own terms. Is it properly structured? Is the correct language used? Are all the important bits an employer would want to see present?

The Metro journalist shared the CV that was produced for his cat. It would pass all the rubrics I would have written for it. The problem is that the rubric reviewer is structurally unable to determine whether any of the claims about the person on the CV are true, or not - just whether they are internally consistent and correctly presented.

So what do we learn?

I’ve just argued that a well designed set of defence-in-depth guardrails could not have been expected to detect a jokester. And yet, a human careers advisor, upon being told they were talking to an animal, would stop the conversation. What is the human doing that the agent is not?

The answer is that the human careers advisor spontaneously and proactively brings their “world model” (their expectations, norms, background beliefs and experiences) to bear on the task in front of them. An agent does not. It very likely has the relevant knowledge sitting in its weights (it knows perfectly well that parrots don’t job-hunt), but nothing in the agent’s design makes it consult that knowledge unprompted. Guardrails validate the content and form of an interaction, never its premises.

And so, when we discover these failures, we reach for a patch. (“New rule: only progress the conversation if the user is a human.”) But patching is not the same thing as contextual judgement: patches simply prevent a past failure from re-occurring; only proactive judgement generalises to prevent future failures.

In short: agents perform tasks, guardrails validate form, and only a proactively applied world model will call a halt because the jobseeker belongs to a fictional pirate. Perhaps we can spare Anthropic’s engineers their blushes after all.

Ready to move from pilot to production?

Book a free 30-minute consultation. You'll leave with a straightforward answer on where agentic AI can work for your organisation, and the first steps to get there.

Book a free 30-min consultation →