Stay connected with KayaToday, follow us on Instagram and Facebook for the latest news and reviews delivered straight to you.
There is a category of enterprise AI failure that produces no error message, triggers no alert, and leaves no immediate trace. The system keeps running. Outputs keep generating. And somewhere around day nine of a multi-week workflow, the AI agent quietly stops following the compliance rules it was given at the start.
This is not a hypothetical edge case. Writing in VentureBeat, Ankit Anand, a managing consultant and enterprise data management architect, describes exactly this scenario: an AI agent deployed for a multi-day master data validation workflow ingests thousands of records over several days, and by day three the governance rules hardcoded into the original system prompt have been pushed out of active memory entirely. The system does not crash. An internal audit weeks later traces the compliance gap back to a precise moment of forgetting. That moment, Anand argues, is not a bug. It is a systemic architectural failure.
Why Attention Dilutes and Why That Is the Core Problem
To understand why this happens, it helps to understand what a large language model actually is. An LLM is a probabilistic generation engine, not a deterministic database. It does not store rules the way a relational database stores records. Instead, it processes everything inside a context window, a fixed-length sequence of tokens that the model attends to when generating its next output.
In a clean pilot environment, the context window is small and the model’s attention mechanism maps reliably to whatever instructions were placed in the system prompt. But as a long-running workflow scales into the hundreds of thousands of tokens, that attention dilutes. Researchers call this the “lost in the middle” phenomenon. The model loses the boundary between throwaway conversational context and the immutable governance rules that were supposed to anchor the entire operation.
The instinct among many infrastructure teams is to reach for retrieval-augmented generation, commonly known as RAG, as a fix. RAG pipelines use vector databases to retrieve semantically relevant text and feed it back into the model’s context. They are genuinely useful for knowledge retrieval tasks. But they cannot enforce state persistence, and they cannot override a probabilistic output with a hard operational constraint. If the model has drifted away from a compliance rule, the vector store will not catch it. It will simply retrieve text that seems relevant and hand it back to an engine that may or may not act on it.
Expanding the context window to a million tokens, a capability that several frontier model providers now offer, does not solve this either. It delays the collision between token volume and attention capacity, but it does not eliminate it. And it inflates cloud infrastructure costs in the process.
The Architecture That Actually Holds
The solution Anand proposes requires a conceptual shift in how enterprise teams think about AI system design. Business logic and governance rules must be treated as protected state that lives physically outside the text generation layer, not as instructions passed into the model’s context and trusted to persist.
In practice, this means neuro-symbolic separation. The neural network handles reasoning and language generation. A deterministic rule layer, sitting entirely outside the LLM context window, handles symbolic logic and constraint enforcement. The model produces a draft output. A deterministic engine then evaluates that output against immutable rules before any action is committed. If the output violates a constraint, the infrastructure blocks it. The model never gets to act on a forgotten rule because the rule is no longer inside the model’s head in the first place.
This architecture matters especially for workflows touching financial data or master data management, the exact domains where compliance failures carry the heaviest regulatory and reputational consequences. For enterprises in Malaysia and Singapore operating under frameworks governed by bodies such as Bank Negara Malaysia, the Securities Commission, or the Monetary Authority of Singapore, the stakes of silent AI drift are not abstract. A compliance gap traced back to an AI agent forgetting a rule on day nine is the kind of finding that surfaces in regulatory examinations and audit committee reports.
Three Places to Start Before the Next Audit
Anand offers three concrete actions for AI orchestrators and data infrastructure managers who want to assess their exposure without waiting for a six-month architecture overhaul.
The first is auditing for latent checkpointing. Every agent running multi-session workflows should be examined for how it re-validates initial constraints over time. Blanket assurances that system prompts “handle” governance are not sufficient. Engineers should be able to demonstrate exactly how the system checkpoints and re-validates rules on day four, day ten, and day thirty. If that verification mechanism does not exist, the architecture is vulnerable by design.
The second is moving compliance boundaries out of natural language prompts entirely. Critical constraints belong in deterministic policy engines, not in text instructions passed to a probabilistic model. API gateways should validate model outputs against hard-coded logic before those outputs reach production. Silent failures, where the AI drops a rule and the infrastructure lets the action proceed anyway, must become structurally impossible.
The third is enforcing state separation in orchestration. An agent’s scratchpad working memory should be treated as ephemeral. Governance rules should be treated as immutable state. The two must never be stored in the same layer, because the moment they share space inside an LLM context window, the rules become subject to the same probabilistic forgetting as everything else.
Why the Industry’s Current Obsession Is Pointing the Wrong Way
The broader enterprise AI conversation has been dominated by generative speed metrics: tokens per second, time-to-first-response, benchmark scores on reasoning tasks. These matter for user experience. They do not determine whether an AI deployment survives contact with a real audit.
What determines that is structural integrity. The enterprises that build architectures capable of sustaining memory constraints across long-running workflows will be the ones that can credibly demonstrate AI governance to regulators and boards. The ones that assume a larger context window or a heavier RAG pipeline will eventually close the gap are accumulating invisible risk with every passing day of deployment.
The failure mode Anand describes is silent precisely because it looks like normal operation right up until the audit. That is what makes it dangerous, and that is why fixing it demands architectural discipline rather than a product upgrade.
Read More: Nvidia Eyes $10 Billion Stake in Anthropic as the AI Lab Targets a $2 Trillion IPO Valuation
