Stay connected with KayaToday, follow us on Instagram and Facebook for the latest news and reviews delivered straight to you.
For eleven days, every light was green. Directed Acyclic Graphs completed without error. Snowflake tables loaded cleanly. Airflow showed no failures. And yet the audience segment counts feeding real advertising budget decisions were off by 40 percent. Nobody inside the system knew. A client figured it out first because their campaign was not performing.
The incident, described in detail by Siddharth Arun, a senior data engineer who worked at InMarket (formerly NinthDecimal) and now builds enterprise data migration pipelines at Salesforce, is not a cautionary tale about careless engineering. It is a structural critique of how the entire data industry thinks about pipeline health. Most monitoring tells you whether a job ran. Almost none of it tells you whether the output means anything.
One renamed field, eleven days of confident nonsense
InMarket operated at serious scale, processing over one petabyte of location and advertising event data. Mobile SDKs collected IDFA and AAID signals covering impressions, clicks, conversions, and location events. Raw events landed in AWS S3, Spark ETL jobs transformed them into structured audience datasets, Airflow orchestrated the workflow, and the final output fed measurement dashboards used by advertising clients making real budget calls.
The failure mechanism was almost insultingly simple. An upstream ad network silently renamed a field in its event payload. No migration notice, no warning, no versioning flag. The Spark join key that depended on that field name stopped matching records. Audience segment counts collapsed. But because the data still arrived on time, still landed in the right partitions, and still looked structurally like data, not a single alert fired. The system was checking whether the translation happened. It was not checking whether the translation was correct.
Arun describes a second incident that confirmed the pattern. A daily Airflow DAG completed successfully. The S3 partition for that date existed. But the actual data files had never arrived because the upstream SDK feed had dropped silently. Spark processed an empty partition, wrote an empty result, and marked the task green. Three days of audience data disappeared before anyone noticed, and again it was a client who spotted zeroes on a dashboard before the engineering team did.
The common thread across both failures is precise: the monitoring was designed to catch infrastructure problems, meaning crashed jobs, missing files, and timeout errors. It was never designed to catch data that arrived punctually, in the correct format, and was simply wrong. As Arun puts it, the system was checking for structural completeness rather than semantic correctness.
Why the industry defaults to the wrong question
This is not an InMarket-specific problem. Monte Carlo’s 2023 State of Data Quality Survey found that 68 percent of data teams report a mean time to detect data incidents of four hours or more, and that figure only covers incidents they eventually catch. Incidents where numbers are plausible but incorrect can go undetected indefinitely because nothing in the default toolchain is looking for them.
The reason the default is so persistent is partly architectural and partly cultural. Every major tool in the modern data stack, whether Airflow, Spark, dbt, or Snowflake, is optimised to answer the question of whether a job completed. That is a tractable, binary, automatable question. Whether the output is semantically correct requires knowing what the data is supposed to represent, which requires domain knowledge that does not live inside the tooling.
There is also a scale problem that makes silent failures especially dangerous in high-volume environments. When a system processes billions of events per day, a 40 percent drop in segment counts can hide comfortably inside normal statistical variance for days. The volume that makes the platform commercially valuable is precisely the volume that makes quality failures invisible until a human being notices something feels off downstream.
Arun also challenges a widely held assumption in data engineering culture, namely that ingesting more data is always better. More data means more storage cost, more compute cost, more schema complexity, and more surface area for silent quality failures. Ingesting everything on the premise that it might be needed later, he argues, is how organisations end up with data lakes nobody trusts and compute bills nobody can explain.
Fixing the right thing, in the right place
The remediation Arun describes is deliberately unglamorous. Schema validation at the ingestion boundary, meaning incoming event schemas are validated against a registry before any transformation runs. Most teams add validation deep in the pipeline after transformation has already occurred. Placing it at the front means a schema mismatch is caught in minutes rather than after eleven days of downstream propagation.
Beyond schema validation, the monitoring framework he now uses treats semantic correctness as a primary signal rather than an afterthought. The metrics that matter are audience segment counts as a business outcome rather than a system metric, data freshness measured in hours since last successful load, schema validation pass and fail rates at ingestion, partition completeness checks covering file count and byte count rather than just partition existence, and pipeline SLA completion windows.
The lakehouse architecture, increasingly popular across enterprises in the region and globally, does not change this calculus. A lakehouse full of bad data is still bad data. The architecture provides infrastructure capable of catching quality problems, but only if semantic discipline is built on top of it deliberately. The tooling is necessary but not sufficient.
For data and engineering teams in Malaysia and Singapore, where cloud-native data infrastructure has expanded rapidly across financial services, e-commerce, and digital advertising, the lesson is directly applicable. Regulators including Bank Negara Malaysia and the Monetary Authority of Singapore have both increased scrutiny of data governance and model risk in recent years, and the kind of silent semantic failure Arun describes is exactly the category of error that governance frameworks are designed to prevent but that purely infrastructure-focused monitoring will never surface.
The problem that agentic AI inherits
Arun closes with a forward-looking observation that carries real weight as the industry moves toward agentic AI workflows. Autonomous agents are increasingly being positioned to handle operational decisions that currently require human intervention, including schema drift response, backfill coordination, and quality remediation. But agents will inherit the same fundamental problem. They need to know when the numbers are wrong, not just when the system is down. An agent that can automatically respond to a crashed job but cannot detect a plausible-looking 40 percent error is not solving the problem. It is automating around it.
The discipline Arun is describing, understanding what a healthy row looks like, knowing what the business expects a number to mean, and identifying where upstream systems are fragile, is not something that changes with the tooling cycle. Tools change every two years. The underlying requirement to understand your data before you trust it does not. As more organisations in the region build agentic pipelines on top of data infrastructure that was never designed to catch semantic failures, that gap between structural monitoring and semantic correctness is going to become harder and more expensive to ignore.
Read More: Google Pixel 11 Review: When Refinement Becomes a Rut
