AI Doesn’t Need to Be Hacked to Be Compromised

An AI system’s behavior depends on more than its model weights. It also depends on the information allowed to shape its answers, updates, evaluations, decisions, and actions.

Security teams should therefore treat AI data pipelines as a first-class attack surface. This does not make data integrity more important than every other AI security concern. Prompt injection, excessive permissions, insecure tool execution, and weak authorization still require separate controls. The point is that model-level defenses cannot reliably compensate for an untrusted data supply chain.

NIST’s Generative AI Profile treats information security and value-chain integration as lifecycle concerns for generative AI systems, emphasizing risk management throughout development, deployment, and operation. A 2025 joint guide from the NSA, CISA, FBI, and international partners similarly identifies data supply chains, maliciously modified data, and data drift as risks affecting the entire AI lifecycle.

Data poisoning is often discussed as a training-time attack. An adversary alters pre-training or fine-tuning examples so that a model learns a backdoor, develops a targeted bias, or behaves differently when it encounters a specific trigger.

Production AI systems create additional paths of influence.

A retrieval-augmented generation system can be manipulated when a malicious or unauthorized document enters its index. A support agent can inherit incorrect escalation rules from mislabeled ticket history. A feedback pipeline can repeatedly promote attacker-controlled interactions into future evaluation or training sets.

These cases do not all modify the model itself. Some change the context placed in front of it. Others change the information used to judge, fine-tune, or update it. OWASP’s 2025 guidance reflects this broader attack surface by separating supply-chain risk from data and model poisoning while recognizing that training data, embedding data, models, and deployment components can all be manipulated.

The shared problem is data authority: Which records are permitted to influence each part of the system, for what purpose, under what conditions, and with what evidence of origin?

A document being available to the system should not automatically make it authoritative.

A useful way to structure this problem is through a Trusted Data Boundary for AI. This is a proposed design pattern, not an established standard.

The pattern places a controlled boundary between unverified inputs and the datasets, indexes, or records authorized to influence an AI system.

The architecture separates raw inputs from approved data while preserving the provenance needed to investigate and revoke compromised sources.

On the untrusted side are external repositories, uploaded documents, public content, user feedback, third-party datasets, and raw operational records. Data enters an isolated landing area rather than moving directly into a production retrieval index, evaluation set, or training store.

At the boundary, the system applies controls appropriate to the data’s source, purpose, and potential impact.

Record who supplied the data, how it was obtained, which system produced it, and what level of confidence it receives.

Trust should not be treated as a single permanent label. A source may be approved for low-risk retrieval while remaining prohibited from fine-tuning, automated decision-making, or policy updates.

New or modified data should remain outside production indexes and training sets until the required checks succeed.

The depth of validation should match the potential impact. A public article used to answer general questions may require fewer controls than a record capable of changing payment instructions, permissions, safety decisions, or model behavior.

Technical validation can include file-type enforcement, malware scanning, schema checks, duplicate detection, statistical analysis, and integrity verification.

Those controls are necessary but insufficient. A perfectly formatted document can still be fraudulent. Systems must also determine whether the source was authorized to provide the information and whether that information is permitted to affect the intended process.

Store the original record, its cryptographic hash, source identity, transformation history, labels, approval status, and relationships to derived artifacts.

If a document is parsed, cleaned, divided into chunks, embedded, and added to a retrieval index, each resulting artifact should retain a link to the original source. Provenance that disappears during transformation provides little value during an investigation.

Data capable of changing payments, permissions, safety controls, account ownership, or system behavior should receive stronger validation and, where appropriate, human review.

Reviewers should have both the authority and context required to evaluate the change. A generic approval button does not create meaningful oversight if the reviewer cannot verify the underlying claim.

Teams should be able to determine which dataset, retrieval index, model, parser, prompt, and policy version contributed to a consequential output.

The 2025 joint AI data-security guidance recommends controls including provenance tracking, digital signatures for trusted revisions, and trusted infrastructure. Signatures can establish where data came from and whether it changed after signing. They cannot prove that the original information was accurate, authorized, or safe, so they remain one component of the boundary rather than a complete validation system.

Consider a company using an AI agent to process supplier invoices.

The system receives a PDF by email, extracts its contents through optical character recognition, identifies the vendor and payment amount, and submits structured fields to an accounting platform.

The model may perform every extraction step correctly and still produce a dangerous result.

An attacker could send a convincing invoice containing altered bank details. A supplier mailbox could be compromised. The document-processing service could retrieve files from an unauthorized source. Correctly reading a malicious document does not make the document legitimate.

Under the Trusted Data Boundary pattern, the PDF begins as untrusted input.

The ingestion system records its source, receipt time, cryptographic hash, and processing history. Extracted vendor details are compared against a controlled vendor master rather than previous invoices received through the same untrusted channel.

Changes to bank information, unfamiliar suppliers, unusual amounts, or discrepancies between the document and existing records place the transaction into quarantine.

Only verified fields proceed to the accounting system. High-impact changes require approval from someone with the authority and context to evaluate them. The raw document remains available as evidence, but it does not receive direct authority to modify payment instructions.

The distinction is important: the AI system may extract information from the document without being permitted to trust it.

This boundary also improves incident response. If a compromised supplier mailbox is discovered later, investigators can search provenance records for every document received from that source. They can identify affected transactions, extraction versions, approval decisions, and downstream systems without assuming that every model output produced during the period was compromised.

A trust boundary must exist in the architecture, not only in policy documents.

Approved and unapproved data should use separate storage locations, permissions, and service accounts. Production training jobs and retrieval-index builders should accept input only from approved stores. A downstream team should not be able to bypass review by pointing a job directly at a raw-data bucket.

Transformations also require lineage. If a document is parsed, cleaned, deduplicated, labeled, embedded, and indexed, every derived artifact should retain a relationship to its source and transformation history.

Removal matters as much as ingestion.

Teams need a process to revoke a compromised source, identify its derived records, rebuild affected indexes, locate model versions trained on its data, and determine whether those models require reevaluation or rollback.

Without revocation, a supposedly trusted record can continue influencing the system long after the organization learns that it was compromised.

Monitoring provides another layer. Statistical checks can identify unexpected changes in distributions, labels, sources, retrieval patterns, or model behavior. These checks may reveal accidental drift or deliberate poisoning, but anomaly detection cannot establish trust by itself. A sophisticated attacker may remain within expected statistical ranges, while legitimate changes may appear anomalous.

Monitoring can tell a team that something changed. Provenance and authorization controls help explain why the change occurred and whether it should have been allowed.

A trusted data boundary does not guarantee that approved data is correct.

A trusted supplier can be compromised. An authorized reviewer can make a mistake. An internally produced dataset can contain harmful assumptions, labeling errors, or outdated information.

The pattern is also difficult to apply to systems that continuously ingest volatile public content. In those environments, trust may need to be temporary, contextual, or limited to specific uses rather than granted to an entire source.

Operational failures remain possible. Trust labels become stale. Quarantine queues are bypassed to meet deadlines. Provenance metadata is lost during export. Human review becomes routine and ineffective. Teams approve a dataset but fail to preserve its chain of custody downstream.

The boundary also does not solve prompt injection, hallucination, excessive agency, insecure tool execution, or weak authorization. It controls what data is permitted to influence the system. Separate controls must govern what the system is permitted to do with that influence.

Teams operating AI systems should be able to answer four questions:

A team that cannot answer those questions does not have meaningful control over its AI data lifecycle.

Model security begins before a prompt reaches the model. By separating unverified inputs from authorized data, and preserving the lineage between them, teams can reduce silent poisoning, limit the reach of compromised sources, and make failures possible to investigate.

The core rule is simple: Data should not gain influence merely because it entered the pipeline.

National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. July 26, 2024.

National Security Agency, CISA, FBI, and international partners. AI Data Security: Best Practices for Securing Data Used to Train & Operate AI Systems. May 22, 2025.

OWASP GenAI Security Project. OWASP Top 10 for LLM Applications 2025. November 17, 2024.