At ACM FAccT, we demonstrated why AI guardrails need the same scrutiny as models. Moving from static policies to context- and language-specific evaluations, our hands-on session proved that agentic guardrails equipped with tools like web search are vital for reliable, real-world deployment.
In June, we traveled to Montreal for the ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT), the premier venue for safe and responsible AI development; a conference where computer scientists, social scientists, policymakers, and lawyers share a room to ask not just how to build AI systems, but whether, when, and for whom. It was the right audience for our tutorial, Contextual Evaluation of LLM Guardrails Across Languages and Agentic Systems.
Evaluation is the throughline
Evaluation has been one of the main conversations in AI safety and a key step toward responsible deployment. The field is shifting from celebrating general capabilities to measuring real-world, domain- and language-specific performance, and cross-sector efforts like the EvalEval Coalition are now building the science and infrastructure for evaluating evaluations themselves. Despite the money spent on evals, benchmark saturation, and evaluation data going stale, one practical function endures: evaluation shapes guardrails. When an eval shows a model produces harmful or toxic content in a given context or language, one response is to design a guardrail around that exact failure.
Guardrails deserve the same scrutiny as models
Guardrails – the mechanisms that filter, flag, or constrain LLM inputs and outputs – shape what users actually see in chatbots, platforms, and public-facing services. Yet they get far less scrutiny than the models they govern. Historically, they were proprietary classifiers, visible only through unexplained refusals. Open-source guardrail models and policy-prompt guardrails makes independent evaluation possible.
Our argument at FAccT was simple: evaluating guardrails is as important as evaluating the LLMs they protect, and context- and language-specific evaluation results should inform guardrails that move beyond static taxonomies of harm toward dynamic policies.
The path from evaluation to guardrail
That path is what brought us to FAccT, because connecting the two is rarely straightforward. Our route: a community and language-informed evaluation of 120 refugee and asylum-focused scenario pairs across English, Farsi, Arabic, Kurdish-Sorani, and Pashto, scored by native-speaker evaluators from Respond Crisis Translation on six rights-based criteria. We published the results as the open MHRE evaluation data on Mozilla Data Collective, on terms set with the evaluators, and turned the recurring failures such as unsafe referrals, missing disclaimers, and stereotyped assumptions into concrete guardrail policies in English and Farsi.
Testing those policies exposed a structural gap: criteria like factuality and actionability cannot be judged from text alone. Does this NGO exist? Is this law current in this jurisdiction? Often text-only guardrails rubber-stamped responses they had no way to verify, hallucinated terms, and scored identical English and Farsi policies differently (check out our blogpost). So we formed a hypothesis: LLM-enabled guardrails need tools such as search, retrieval, and fact-checking to judge in a more reliable and trustworthy manner. That was the agentic guardrail we brought to Montreal to test.
What the hands-on session showed
The overall methodology and the case for contextual guardrails, with language- and context-dependent policies and tools, resonated with participants. Thirty-five participants ran our demo by selecting their own scenarios and policies and comparing agentic and non-agentic judges on the same responses.
Tools changed the supporting evidence more often than the final verdict: 90% of verdicts agreed across both modes. However, “agentic” behavior depended heavily on the underlying judge LLM. Claude Sonnet 4.6 used web search on every run (4.1 tool calls per run on average), whereas GPT-5 Nano rarely did (0.2 tool calls per run). When tools were invoked, they influenced outcomes in both directions: verifying the factual claims in an asylum-related response increased its score, while identifying a factual error that the tool-less judge had overlooked downgraded the verdict from pass to borderline.
Tooling that makes this practical
None of this experimentation is feasible if every comparison requires new engineering. Mozilla AI’s open-source any-guardrail gives a unified interface for choosing and swapping guardrails with custom policies, making the guardrail layer as configurable as the model. And Otari, Mozilla’s new open-source LLM gateway, lets you choose and switch the LLMs behind your judges seamlessly.
What’s next
We are continuing to refine our design and test whether tool access makes LLM-enabled guardrails more reliable and trustworthy across humanitarian, financial, and social-engineering use cases, with a wider array of tools and contextual scenarios, in English <-> Farsi and English <->Spanish. Results are coming; stay tuned, and hopefully see you at the next ACM FAccT!
Disclaimer on LLM use: Roya Pakzad used Claude Opus 4.8 to copyedit this post.