Our sibling practice DoThinkData builds change-data-capture pipelines — Debezium, Kafka Connect, the unglamorous plumbing that keeps copies of production data moving. A while back we built a Java wrapper that watches every connector through its two hardest phases, initialization and steady state, and exposes health as JMX metrics in Grafana. It exists because pipelines fail quietly: a stalled snapshot surfaces weeks later as “the reports look wrong”.
Then AI arrived, and the stakes changed. The same pipelines now feed RAG corpora, feature stores and analytics that models and agents read. A pipeline that fails quietly was an operations problem. A pipeline that fails quietly and incorrectly under AI is a correctness problem, a security problem and a trust problem at once.
What changes when the consumer is a model
- Correctness becomes security. A model trained or grounded on silently stale, duplicated or partial data makes wrong decisions with perfect confidence. Integrity checks and reconciliation at every hop stop being nice-to-haves.
- Provenance becomes a control. When every chunk in your corpus can trace back to a source, a timestamp and an owner, poisoning gets hard and audits get easy. When it cannot, neither is possible.
- Credentials become blast radius. A CDC user with blanket SELECT on everything feeds the model everything. Least privilege on the pipeline is least privilege on the AI.
- “Fresh enough” gets a definition. If access is revoked at the source, how fast does the retrieval corpus forget? Retention and refresh cadence are now part of your access-control story.
The healthy pattern is the same one we built for monitoring: make every hop observable, make failures loud, and keep an audit trail a model’s output can be checked against. Pipelines like these are where our secure data integration and LLM security practices meet — one system, one discipline.