Note

Executive Summary: A peer-reviewed legal-RAG auditor (Akremi, 2026) achieved a 100% unsafe refusal rate by blocking 51.4% of all queries. This teardown examines why post-hoc structural validation (SHACL) breaks unit economics under enterprise TPRM standards (NIST AI RMF 1.0, ISO/IEC 42001), and how in-flight, sentence-level NLI verification resolves the resulting "hallucination tax."

What an academic legal-RAG auditor gets right, what it gets expensive, and why your enterprise buyer's TPRM questionnaire is about to ask you about both

If you are a CTO at a legal-tech company that has just started fielding Third-Party Risk Management (TPRM) questionnaires — anchored in frameworks like NIST AI RMF 1.0 (GOVERN 1.2 / MEASURE 2.6), ISO/IEC 42001, or the Shared Assessments SIG / CSA CAIQ v4 AI controls — from an AmLaw 100 firm or a bank's procurement team, you have probably noticed something uncomfortable: the questions have changed.

Two years ago the questionnaire asked where the data lives, who has access, and whether you hold SOC 2 Type II. Custody questions. Answerable questions. Now there is a second block of questions underneath mapping directly to AI risk measurement: "describe the controls that prevent the system from generating unsupported legal assertions," "what is your measured hallucination rate and how is it monitored in production," and "how do you demonstrate that a retrieved authority is currently in force."

Those are not custody questions. They are correctness questions, and they are much harder to answer, because the honest answer for most RAG pipelines is a shrug wrapped in a paragraph about prompt engineering.

The questions are getting sharper because the assurance side is reading the literature. And in July 2026 the literature acquired a very good, very quotable reference point: Akremi (2026), "Ontology-Driven Legal Rule Auditor for Secure, Trustworthy, and Governed RAG Systems," published in MDPI Computers (Vol. 15, Issue 8, Art. 471) out of Umm Al-Qura University.

It is a genuinely strong paper. It is also, read carefully, the clearest existing demonstration of why the standard approach to legal-RAG governance is economically unshippable — and it demonstrates this using its own headline numbers.

This post is a full breakdown of the paper: what it built, what it proved, where its architecture creates a production ceiling, and what an alternative control architecture looks like when you design — not yet build — against the same class of problem. I have no product to sell you at the end of it, and nothing past §7 is more than a design worked out on paper; I will be explicit about that distinction when we get there. The reason to read it is that the assurance teams gatekeeping your enterprise deals are converging on the ideas in this paper, and it is better to have opinions about them before someone asks.

1. The Reference Architecture: What the Akremi Paper Built

Let me describe it fairly, because it deserves that.

Most legal-RAG "governance" in the wild is a system prompt that says do not make things up and a citation regex. Akremi's system treats governance as an architectural layer with formal semantics, and the design is coherent end to end.

The ontology is a control, not documentation. An RDF/OWL ontology models legal instruments, provisions, jurisdictions, topics, and lifecycle states. Crucially it is not a passive schema — it is queried at request time to constrain what the retriever is allowed to see. Properties like belongsToJurisdiction are enforcement primitives.

Retrieval is gated before it happens. An NLU module infers jurisdiction and topic from the query, and the ontology gate filters the candidate chunk set before vector similarity is computed. If your query is about Saudi PDPL, the HIPAA chunks are not in the race.

Lifecycle validity is reasoned, not assumed. SWRL rules classify provisions as Active, Superseded, or Deprecated, with :isSupersededBy links between them. A superseded provision is structurally excluded from the evidence set. This is the part of the paper I like most, and I will come back to it, because it is the thing the paper does that I did not.

Two auditors bracket the model. An Input Gate screens for prompt injection and PII/PHI before generation (addressing vulnerabilities highlighted in the OWASP Top 10 for LLMs, specifically LLM01 and LLM06). An Output Validator runs pySHACL shapes against the generated response — including a CitationInContextShape that checks every citation in the output actually exists in the approved evidence set.

Everything is hashed. Each interaction produces a cryptographic hash record, mappable to W3C PROV-O provenance data. This directly satisfies the audit trail requirements of ISO/IEC 42001 Clause 9.1 and enterprise TPRM audit standards, producing exactly the verifiable artifact a procurement reviewer needs.

The stack is deliberately modest: Llama 3.1 8B quantised to Q4_K_M served via Ollama, Ontotext GraphDB for the ontology, Qdrant for vectors, BAAI/bge-m3 embeddings, pySHACL for validation. It runs on a single RTX 4070 with 8GB of VRAM. The evaluation corpus is four real instruments — Saudi PDPL, EU GDPR, US HIPAA, US HITECH.

The safety results are excellent:

Read that last row again.

2. The Headline Metric Nobody Quotes: 51.4% Abstention Rate

The system blocked 51.4% of all responses.

Not 51.4% of adversarial responses. Not 51.4% of responses that were actually wrong. 51.4% of everything the system was asked to do, it declined to answer.

To the paper's considerable credit, it does not hide this. Section 7.4 is candid about the mechanism: SHACL is a structural validator, and it discards entire responses over things like a synonym choice — the validator sees "ensures" where the shape expected "warrants" and fails the whole output. The paper also notes that the repair loop was "not separately instrumented or evaluated," which is a polite way of saying that regenerating a failed response introduces cache contamination and roughly doubles compute, so they did not test it.

For an academic contribution this is fine. The paper's claim is that ontology-governed retrieval plus declarative validation can make legal RAG safe, and it demonstrates that convincingly. Nobody is shipping this to lawyers.

But you are shipping to lawyers. So run the number.

A system that refuses half of all queries is not a product. It is a product with a coin flip in front of it. Every one of those refusals consumed a full generation pass — the tokens were produced, the GPU time was spent, the latency was incurred — and then the output was destroyed. You paid full price for zero user value, 51.4% of the time.

And here is the part that matters for your procurement conversation: your enterprise buyer will not read the abstention rate as a cost problem. They will read it as a safety benchmark.

Under empirical research from Stanford's Regulation, Evaluation, and Governance Lab (Magesh et al., 2025; Dahl et al., 2024), commercial legal RAG platforms show hallucination rates of 17% to 33% on specialized legal tasks. Under professional conduct rules (ABA Model Rule 1.1 Duty of Competence, SRA Principle 2, and the post-Mata v. Avianca federal sanctions regime), an enterprise buyer cannot tolerate unverified legal assertions.

"The literature achieves 100% unsafe refusal and 0% stale-law leakage" is now an explicit control expectation in enterprise TPRM questionnaires under NIST AI RMF MEASURE 2.6. You are going to be asked to match it. If the only architecture you know for matching it is post-hoc validation, you will match it by refusing half your queries, and your product will die of it.

So the useful question is not how do I hit those safety numbers. It is: is the abstention rate a property of governance, or a property of this particular enforcement architecture?

It is the second one. Here is why.

3. Root Cause 1: Validation Granularity vs. Contamination Granularity

The paper's control flow is: generate the whole response, then judge the whole response.

graph LR
    subgraph AKREMI ["Akremi (2026): post-hoc validation"]
        direction LR
        A1[Query] --> A2[Ontology gate]
        A2 --> A3[Retrieve filtered chunks]
        A3 --> A4[LLM generates full response]
        A4 --> A5[SHACL validator]
        A5 -->|Pass| A6[Release]
        A5 -->|Fail| A7["Block — 51.4% of responses"]
    end

Contamination, however, does not occur at the granularity of a response. It occurs at the granularity of a claim — usually one sentence, sometimes one clause. A 900-token answer with one unsupported sentence in it is not 900 tokens of garbage. It is roughly 40 tokens of garbage and 860 tokens of correct, useful, expensive work.

Post-hoc validation cannot express that distinction. Its only two verbs are release and destroy. So a single bad sentence costs you the entire response, and because the validator is structural it also costs you responses that were never wrong in the first place — just phrased unexpectedly.

This is the mechanism behind the 51.4%. It is not that the model was wrong half the time. It is that the enforcement boundary sits at the wrong granularity, and the blast radius of any single failure is the whole output.

Move the enforcement boundary and the number moves with it.

4. Root Cause 2: Inference Serving Layers Dictate Your Control Surface

There is a decision buried in the paper's methods section that constrains everything downstream, and it is easy to skim past: the system serves Llama 3.1 8B via Ollama.

Ollama does not expose KV cache state to the application layer.

That single fact forecloses an entire category of control. If you cannot reach into the generation state during inference, then you cannot:

• inspect what the model has committed to mid-generation
• selectively discard the attention state associated with a bad span
• resume generation from a known-clean point

Which means your only available remediation is generate everything, judge everything, destroy everything. The post-hoc architecture is not really a design choice in this paper. It is the only architecture available given the serving layer.

This generalises, and it is the single most portable lesson in the whole analysis: the compliance controls you are able to implement are bounded by the inference hosting you chose, and that choice is usually made for reasons that have nothing to do with compliance. If you are on a managed API endpoint — any of them — the generation state is opaque to you, and every control you can build is necessarily post-hoc. You will discover this at the worst possible moment, which is when an enterprise buyer asks you under NIST AI RMF MAP 1.5 to demonstrate inference-time control and you realise the answer requires an infrastructure migration.

Worth checking now rather than in a procurement cycle.

5. Root Cause 3: SHACL Validates Syntax and Structure, Not Meaning

The paper's CitationInContextShape checks that every citation appearing in the output exists in the approved evidence set. That is a real and valuable control — it kills the classic fabricated-citation failure mode, the one that has produced a documented trail of court sanctions.

It does not touch the harder failure mode.

Suppose the model cites Article 99 for one proposition and Article 94 for another. Both articles are genuinely in the approved evidence set. Both citations resolve. But the prose connecting them asserts a legal relationship that neither article supports — a conditional that is not there, a scope that is broader than the text, an obligation attached to the wrong party.

SHACL passes it. Every citation exists. Every shape is satisfied. The output is structurally immaculate and legally wrong.

The paper is honest about this too — Section 7.4 states that the validator "looks at the presence and metadata of the citations rather than the deep semantic logic of the English connecting them."

This is the OWASP LLM09 (Misinformation / Hallucination) trap in its purest form: citation-existence checks pass, retrieval-precision metrics pass, and document-level groundedness scores pass, while the actual legal proposition is fabricated. It only fails if a control is comparing this specific claim against this specific span of evidence and asking whether the first is entailed by the second under formal natural language inference.

6. Root Cause 4: Dense Vector Retrieval Fails on Identifier Collisions

This one is a retrieval-tier problem and it is specific to civil-law corpora, so it may not apply to you — but if you operate in Spain, Italy, France, Germany, or most of Latin America, it applies to you very much.

The paper's pre-retrieval gate filters on jurisdiction and topic. That is genuinely effective at the boundaries it can see: it will not let a HIPAA chunk contaminate a PDPL query.

Now consider a query about "Article 42" in a jurisdiction where the civil code, the commercial code, the labour code, and three procedural statutes each have an Article 42, all within the same topic space. The ontology gate has already done its job — correct jurisdiction, correct topic. It has no further discriminating metadata to apply. The disambiguation is handed to the embedding model.

Dense embeddings cannot do this. "Article 42" produces a near-identical vector regardless of which statute it belongs to, because the discriminating information is a discrete identifier, not a semantic property. Similarity degrades roughly as $O(1/n)$, where $n$ is the number of colliding articles — with four colliding articles you are choosing close to at random, and the model then generates fluent, confident, well-cited prose about the wrong statute. ColBERT-style late-interaction reranking does not rescue you, because the reranker is scoring the same undifferentiated token.

The failure is silent. There is no low-confidence signal, no refusal, no anomaly in the logs. The system retrieved something, it was topically plausible, and the answer reads perfectly. (See our documented reproduction of this failure mode across Spanish statutory law in Retrieval Integrity Under Statute Collision: Measurement and Control Specification.)

This is the retrieval-tier equivalent of the Article 99/94 problem: a failure mode that passes every control designed to catch a different failure mode.

7. An Alternative Control Architecture: In-Flight Verification

The following section models an alternative enforcement architecture, specifying the structural controls and formal properties derived from deterministic filtering and in-flight inference controls. (For the complete engineering control specification and state-machine formalization, see the companion case study: Generation-Tier Integrity in Legal RAG: An In-Flight Verification Control Specification.)

7.1 Moving the Enforcement Boundary Inside Generation (In-Flight NLI)

The first design targets the granularity mismatch directly. Instead of judging the response, judge each sentence as it is produced, so remediation is scoped to the failing sentence rather than the whole response.

graph LR
    subgraph OURS ["In-flight control"]
        direction LR
        B1[Query] --> B2[Statute-binding router]
        B2 --> B3[Retrieve bound chunks]
        B3 --> B4[LLM generates with in-flight audit]
        B4 --> B5[Per-sentence NLI]
        B5 -->|Entailed| B6[Stream to user]
        B5 -->|Contradiction| B7[Truncate KV cache, resume clean]
    end

The mechanism:

1. Self-hosted inference with cache access. vLLM or SGLang with RadixAttention rather than a managed endpoint. This is the enabling decision — everything else depends on being able to reach the KV cache.
2. Sub-span extraction. For each generated sentence, ColBERTv2 token-level MaxSim scores identify the specific ~50-word span of evidence that sentence is claiming support from. Not the document. Not the chunk. The span.
3. Sentence-level NLI. An 8B natural-language-inference auditor evaluates entailment between the sentence and that span, returning Entailment / Contradiction / Neutral. This is a semantic judgement — it is the control that catches the Article 99/94 problem, because it asks whether the claim follows from the evidence, not whether the citation exists.
4. Selective rollback. On Contradiction, the RadixAttention tree is navigated back to the last verified branch point. The contaminated branch is garbage-collected — the model's attention state no longer contains the bad sentence, so it cannot condition subsequent tokens. A corrective context prefix resumes generation from clean state.

The last point is the one that's easy to skip past on paper but is actually the crux: it is not enough to delete a bad sentence from the output string. If the bad sentence is still in the KV cache, the model continues reasoning from it, and you get a response that is superficially corrected and structurally still poisoned. Truncation is the difference between hiding contamination and eliminating it.

None of this has been run. Here is what the construction implies, with each line labelled by how it's derived rather than presented as a result:

The waste-reduction line is the one that matters conceptually, and it holds regardless of the exact lengths: if contamination is one sentence and you discard one sentence instead of the whole response, you save roughly the ratio between them. The architecture does not need to be clever to win there. It needs to operate at the right granularity. Whether it actually achieves 0% contamination and sub-second rollback in practice is an empirical question I have not yet answered.

7.2 Deterministic Disambiguation Before the Embedding Layer

The second build targets identifier collision, and the design principle is that some problems should not be handed to a similarity function at all. (See our companion empirical teardown in Retrieval Integrity Under Statute Collision: Measurement and Control Specification.)

• Entity extraction parses article numbers and statute keywords from the query.
• Collision detection checks the extracted identifier against a pre-computed index of every article number that appears in more than one statute in the corpus.
• Hard metadata filtering constrains the vector search space to a single statute before any similarity is computed: {statute_id = 'CdC', article_num = '42'}.

That is a boolean constraint with $O(1)$ precision. It is not a better ranker. It is the removal of the ranking problem.

The claim here is stronger than the usual empirical "our system did better" claim, because it's structural rather than statistical: a boolean equality filter on {statute_id, article_num} either matches the queried statute or it does not. There is no similarity score to get unlucky on. So for any query where the router correctly extracts the article number and statute keyword — a much easier problem than full semantic retrieval, since these are typically well-formed, regex-extractable tokens — the wrong-statute failure mode is not reduced, it is structurally impossible, by construction.

What I cannot tell you without building it: what fraction of real-world queries the extraction step actually handles cleanly, how much smaller the search space gets on a real corpus, what the true collision rate looks like once you compute it against an actual civil-law dataset, or what this costs in infrastructure once it's running against production volume. Those are empirical questions, and I am not going to attach numbers to them that I have not measured.

7.3 Economic Modeling: The Hallucination Tax

The paper reports p95 latency and stops. There is no cost model anywhere in it — no per-query cost, no cost of blocked responses, no scaling behaviour, no comparison against an ungoverned baseline.

For a system that discards 51.4% of its generations, that omission is doing a lot of work.

The framing I use is a hallucination tax: the ongoing OPEX penalty a validation architecture imposes per query, as a first-class number rather than an implementation detail.

$$C_{tax} = r \cdot C_{gen} + r \cdot T_{out} \cdot P_{out}$$

Where $r$ is the failure rate that triggers remediation, $C_{gen}$ is the cost of a full generation pass, $T_{out}$ is output tokens, and $P_{out}$ is price per output token. The structure of the formula is the argument: in a post-hoc architecture, the whole of $C_{gen}$ is multiplied by $r$, because the whole response is destroyed. In an in-flight architecture, $C_{gen}$ is replaced by the cost of the discarded span, which is one to two orders of magnitude smaller.

At $r = 0.514$ — the paper's own number — the post-hoc tax is a permanent surcharge of roughly half your inference budget, forever, growing linearly with volume. That is not a tuning problem. It is a structural property of the enforcement architecture, and no amount of prompt engineering touches it.

Plug in a range of query volumes, from 10K/day to 1M/day, across model tiers, and the shape of the formula does not change: the gap between post-hoc and in-flight widens with scale, because post-hoc waste scales with volume × response length while in-flight waste scales with volume × span length. That is a property of the algebra, not a claim about anyone's actual traffic.

If you take one operational habit from this post, take this one: put the governance overhead in your unit economics model explicitly. Most teams model inference cost against successful responses and treat blocked responses as a rounding error. At a 51% abstention rate they are not a rounding error; they are the larger line item.

8. Where the Ontology-Driven Approach Outperforms In-Flight Verification

I would not trust this analysis if it only ran one direction, so here is the other direction. Four things the paper does that the design in §7 does not — and this comparison holds regardless of whether §7 ever gets built, because it is a comparison of what each specification covers, not of implementation quality.

A formal, machine-readable governance specification. The RDF/OWL ontology with SWRL rules and SHACL shapes is a declarative artefact. You can hand it to an auditor. You can diff it across versions. You can reason about it independently of the code — and, notably, independently of whether it has been implemented, which is exactly the property this section of my own post cannot currently claim for itself. A router, an index, and an NLI auditor are procedural by nature, even on paper: they describe what the system does, not a specification an auditor can evaluate on its own terms. When a TPRM reviewer asks "show me your governance model," a SHACL file is a much better answer than a description of a pipeline. This is a real, structural advantage of the paper's approach, not just a maturity gap.

Lifecycle and temporal reasoning. The SWRL validity engine with :isSupersededBy supersession tracking is a real capability the design in §7 does not have an equivalent for. As designed, the statute-binding router would catch repealed provisions only incidentally — binding to the correct statute happens to exclude the wrong one, if that wrong one happens to be the repealed version. That is luck dressed as a control, not a mechanism, and it is a gap in the design itself, not something a build would fix. There is no hasEffectiveDate, no hasExpiryDate, no explicit supersession graph in what I have sketched. For a corpus where amendments are frequent, the paper's approach is the right one and mine is a gap I have not closed on paper, let alone in code.

Cross-jurisdictional comparison queries. "Compare HITECH and GDPR breach notification timelines" requires maintaining jurisdiction-separated evidence groups through generation so the model does not blend them. The paper handles this. The design in §7 does not — the router as sketched binds one statute at a time, and extending it to bind two in parallel while preserving separation is a real design problem I have not worked through.

Input-tier safety. The paper's Input Gate screens prompt injection and PII/PHI before anything else runs. The design in §7 has nothing at the input tier at all — it starts at retrieval and covers generation, and that is a straightforward, acknowledged hole. The paper's dual-auditor pattern is architecturally compatible with what's sketched in §7: an Input Gate ahead of the statute router would yield a clean three-tier stack — input safety → retrieval integrity → generation integrity — but that is itself an unbuilt, unverified extension on top of an unbuilt, unverified base.

The honest synthesis is that these look like complementary halves rather than competing designs, on paper. The paper solves what should be checked — it defines governance rules formally, in a way that survives audit, and it has actually been implemented and evaluated. The design in §7 is an argument about how enforcement might avoid destroying the product, worked out carefully but not yet tested against anything. Neither half, as it stands, is sufficient on its own: a formal governance specification with a 51% abstention rate does not ship, and an in-flight enforcement architecture that exists only as a diagram does not pass procurement either.

9. Six Unsolved Frontiers in Legal RAG Compliance

The genuinely open problems, stated as problems rather than as opportunities.

Number 1 is the one I would most like someone to solve, and I do not currently have a good approach to it.

10. Audit Playbook: Converting Architecture Into TPRM Questionnaire Answers

Practical residue, because the abstract argument is only useful if it changes what you write in an enterprise procurement form.

The TPRM correctness questions map onto a small number of architectural facts governed by NIST AI RMF 1.0, ISO/IEC 42001, and Shared Assessments SIG / CSA CAIQ v4 standards. If you can state these four things clearly, you are ahead of nearly everyone:

1. Where does your enforcement boundary sit? (NIST AI RMF MAP 1.5 / MEASURE 2.6) Pre-retrieval, post-retrieval, in-flight, or post-generation. This single answer determines your remediation cost, your abstention rate, and your latency profile. Most teams have never articulated it and discover their answer under questioning. If your boundary is post-generation, state your measured abstention rate and remediation cost explicitly.

2. Is your validation structural or semantic? (OWASP LLM09 / CAIQ AI.1.1) Citation-existence checking is structural. Claim-evidence entailment is semantic. Say which you do. If you only do the first, say that too — an accurate limited answer survives diligence far better than an inflated one, and a reviewer who catches you overstating one control will re-open all of them.

3. What is your control surface at inference time? (ISO/IEC 42001 Clause 9.1) This is really a question about your serving layer. Managed endpoint means opaque generation state means post-hoc only. Fine, if you know it and can say it. Not fine if you claim inference-time controls you cannot implement.

4. How do you handle identifier collision, if your corpus has any? (TPRM Retrieval Data Integrity) Compute the collision count in your own corpus first — that is a short script, not a project. If it is non-zero and you are relying on dense similarity, you have a silent failure mode with no observability, and it is better to find it yourself.

Underneath all four is a single reframing that I think is the actual takeaway. Governance is not a policy layer bolted onto a pipeline. It is a property of where in the pipeline you put the enforcement boundary, and what state that boundary is allowed to touch. Akremi (2026) demonstrates this negatively and rigorously: excellent rules, wrong boundary, 51.4%.

The paper is worth reading in full. It is careful, honest about its own limitations in a way that is unfortunately rare, and it is going to be cited in assurance frameworks for the next several years. Better to have read it.

Methodology Note & References

Methodology note: §§1–6 are a straight reading of the published paper (Akremi, 2026) and its own reported numbers — those figures are the paper's, not mine. §7 is a design, not a build: the architecture is worked out in enough detail to reason about its properties, but nothing in it has been implemented, run, or measured against a corpus, and every number in that section is labelled accordingly, either as a structural consequence of the design or as an explicitly-flagged illustrative estimate. Separately, I maintain an open-source deterministic evaluation harness, legal-audit-rag, for running pass/fail compliance checks against a RAG pipeline rather than LLM-as-judge scoring — it is not the source of any number in this post, and I am linking it here only because it is relevant to the general problem, not as evidence for §7's claims. Corrections to any of the above are welcome; I would rather be corrected than cited wrongly.

Referenced Standards and Literature

1. Akremi, A. (2026). "Ontology-Driven Legal Rule Auditor for Secure, Trustworthy, and Governed RAG Systems." Computers, 15(8), 471. https://doi.org/10.3390/computers15080471
2. Dahl, M., Magesh, V., Mirchandani, N., & Ho, D. E. (2024). "Large Language Models Fail on Legal Reasoning and Legal Retrieval." Stanford Regulation, Evaluation, and Governance Lab (RegLab) & Stanford Institute for Human-Centered Artificial Intelligence (HAI).
3. Magesh, V., et al. (2025). "Hallucination Rates in Commercial Legal Research AI: A Benchmark Evaluation." Stanford RegLab Working Paper.
4. National Institute of Standards and Technology (NIST) (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST Trustworthy and Responsible AI, NIST SP 1270.
5. International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) (2023). ISO/IEC 42001:2023 Information Technology — Artificial Intelligence — Management System.
6. OWASP Foundation (2025). OWASP Top 10 for Large Language Model Applications (v2025). Open Worldwide Application Security Project.
7. W3C (2013). PROV-O: The PROV Ontology. W3C Recommendation. https://www.w3.org/TR/prov-o/
8. Memon, A. (2026). legal-audit-rag: Deterministic Evaluation Harness for Retrieval-Augmented Generation Pipelines. GitHub repository.