Risk model
Untrusted user content competes with application instructions and can redirect the model toward unauthorized disclosure or actions.
Defensive AI security desk
Research the failure mode before it reaches production.
Searchable, source-linked dossiers for building safer AI systems. Each record separates prevention, detection, validation and known limits; no exploit payloads or universal-safety claims.
21dossiers
22primary and official sources
21 Aug 2026evidence checked
21 matching dossiers
01ApplicationPrompt injection containmentKeep model instructions from becoming an authorization boundary.v1 · last verified 21 Aug 2026
Prevention
Separate instructions from data, enforce permissions in deterministic code, minimize connected capabilities and require approval for consequential actions.
Detection
Log instruction conflicts, tool requests, denied operations and policy-filter outcomes without retaining sensitive prompt bodies.
Validation
Run direct, multilingual and obfuscated adversarial cases against every privileged path and assert that server-side policy still denies access.
Limitations
Input filtering and system prompts reduce exposure but do not reliably eliminate prompt injection.
Defensive workflow
Treat the model as an untrusted planner: validate its proposed action against user identity, resource scope and an explicit allowlist before execution.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
02ApplicationIndirect prompt injection in retrieved contentAssume documents, pages, messages and images can carry hostile instructions.v1 · last verified 21 Aug 2026
Risk model
An attacker plants instructions in content the system later retrieves, causing the model to treat external data as control text.
Prevention
Preserve source trust labels, isolate retrieved text, strip active content, narrow tools and block retrieved content from changing policy or recipients.
Detection
Flag instruction-like phrases in external context, unusual tool transitions and source-to-action chains that cross trust zones.
Validation
Seed defensive test documents across every ingestion format and confirm they cannot change tool parameters, destinations or secrets access.
Limitations
Content classifiers miss novel encodings and multimodal instructions, so containment must survive detection failure.
Defensive workflow
Fetch into a low-trust zone, sanitize and label, retrieve only the needed excerpt, then authorize any resulting action outside the model.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
03ApplicationSensitive information disclosureControl what enters context and what is allowed to leave it.v1 · last verified 21 Aug 2026
Risk model
Prompts, retrieval results, tool output, logs or model training data can expose personal data, credentials or confidential business information.
Prevention
Classify data, redact before inference, use scoped retrieval, keep secrets out of prompts and apply output data-loss controls.
Detection
Scan outputs for secret formats and protected entities, monitor unusual retrieval breadth and alert on repeated boundary probes.
Validation
Use synthetic canary secrets in each data zone and verify they never appear outside the authorized response path or logs.
Limitations
Pattern matching misses transformed or semantically inferred secrets and can over-block legitimate content.
Defensive workflow
Authorize retrieval per record, minimize the returned fields, redact context, generate, scan the response, then release or quarantine it.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
04ModelModel and dependency supply chainMake every model, adapter, dataset and runtime artifact traceable.v1 · last verified 21 Aug 2026
Risk model
Compromised repositories, unsafe serialization, mutable tags or opaque providers can introduce malicious code, backdoors or unexpected behavior.
Prevention
Pin immutable versions, verify signatures and hashes, prefer safe formats, inventory provenance and sandbox untrusted artifacts before promotion.
Detection
Monitor checksum drift, dependency advisories, unexpected files, capability changes and deviations from a signed release manifest.
Validation
Reproduce the build, scan artifacts, compare behavior to a trusted baseline and rehearse rollback before production admission.
Limitations
A clean signature proves artifact identity, not that the model is safe or that its training data is trustworthy.
Defensive workflow
Acquire into quarantine, verify provenance and integrity, run security and behavior gates, sign the approved bundle and deploy by digest.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
05DataTraining and fine-tuning data poisoningProtect the data path that changes model behavior.v1 · last verified 21 Aug 2026
Risk model
An attacker alters a small but influential subset of training or feedback data to create targeted errors, bias or hidden triggers.
Prevention
Track lineage, restrict write access, separate contributors, deduplicate data, review high-influence samples and preserve immutable snapshots.
Detection
Look for source concentration, label anomalies, trigger-linked behavior, distribution shifts and training runs that diverge from controls.
Validation
Compare clean-room and candidate models on held-out slices, influence-focused tests and known trigger families before release.
Limitations
Poisoning can mimic normal long-tail data, and no single statistical test establishes that a corpus is clean.
Defensive workflow
Ingest with provenance, quarantine anomalies, train reproducibly, compare against a signed baseline and require two-person approval for promotion.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
06DataRAG and vector-store poisoningA relevant passage is not automatically a trustworthy passage.v1 · last verified 21 Aug 2026
Risk model
Malicious or low-quality documents can rank highly in retrieval, steer answers, cross tenant boundaries or carry indirect instructions.
Prevention
Enforce tenant filters before similarity search, retain provenance, approve sources, limit write paths and separate instructions from retrieved evidence.
Detection
Monitor sudden rank changes, duplicate clusters, source reputation, retrieval diversity and documents that produce disproportionate answer influence.
Validation
Test cross-tenant isolation, poisoned-corpus scenarios, citation alignment and removal propagation from index to response.
Limitations
Embedding distance measures semantic similarity, not truth, authority or instruction safety.
Defensive workflow
Authorize and label documents at ingestion, retrieve within policy, rerank for authority, scan context, answer with citations and support deletion.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
07ApplicationImproper model-output handlingTreat generated output as untrusted input to every downstream system.v1 · last verified 21 Aug 2026
Risk model
Generated markup, code, queries or tool arguments can trigger injection, unsafe rendering or unauthorized execution when consumed directly.
Prevention
Use typed schemas, escaping, parameterized interfaces, allowlists and sandboxing; never execute raw model text.
Detection
Record parser failures, rejected fields, dangerous sink attempts and mismatches between requested and generated operations.
Validation
Fuzz every output-to-sink adapter with malformed structures and verify encoding at the final rendering or execution boundary.
Limitations
Schema validity does not establish business authorization or semantic safety.
Defensive workflow
Generate a typed proposal, parse strictly, authorize values, preview high-impact changes, execute through a narrow adapter and audit the result.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
08AgentExcessive agency and tool privilegeReduce what an agent can do, not only what it is told to do.v1 · last verified 21 Aug 2026
Risk model
An agent with unnecessary functions, broad credentials or unchecked autonomy can turn model error or manipulation into real-world impact.
Prevention
Give each task a scoped identity, narrow function set, bounded resources and explicit approval for irreversible or external actions.
Detection
Alert on new tool sequences, privilege denials, repeated retries, destination changes and actions outside the user's active intent.
Validation
Run scenario tests for compromised tools, hallucinated parameters and malicious context while checking that policy blocks the final action.
Limitations
Human approval can become a rubber stamp if reviewers lack context or receive too many prompts.
Defensive workflow
Plan in read-only mode, resolve exact resources, display the proposed effect, obtain risk-based approval and execute with a short-lived credential.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
09ApplicationSystem prompts, secrets and trust boundariesA hidden prompt is configuration, not a secret vault or policy engine.v1 · last verified 21 Aug 2026
Risk model
System instructions may be reconstructed or exposed, revealing confidential data or controls that fail once known.
Prevention
Keep credentials and sensitive business data outside prompts, enforce controls in code and design instructions to remain safe if disclosed.
Detection
Use canary markers, scan responses for protected configuration fragments and investigate repeated extraction-style conversations.
Validation
Assume the full prompt is public during threat modeling and verify that no data access or authorization decision changes.
Limitations
Refusal instructions can reduce accidental disclosure but cannot guarantee prompt confidentiality.
Defensive workflow
Store secrets in a vault, pass only scoped results, inspect output for leaks and rotate any credential that crosses the model boundary.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
10ModelJailbreak resilience testingMeasure policy bypass as a changing failure surface, not a solved filter.v1 · last verified 21 Aug 2026
Risk model
Adversarial phrasing, encoding, role-play or multimodal inputs can elicit behavior the deployment policy intends to block.
Prevention
Layer model alignment with input controls, output checks, least privilege and use-case-specific refusal and escalation rules.
Detection
Track policy-category failures, repeated reformulations, encoding changes and drift by model and prompt version.
Validation
Maintain a defensive red-team suite with benign controls, measure both bypass and over-refusal, and rerun after every model change.
Limitations
Passing a fixed jailbreak set does not establish resilience to adaptive or unseen attacks.
Defensive workflow
Define policy, generate safe adversarial transformations, test in isolation, triage failures, add regression cases and gate release on risk thresholds.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
11ApplicationUnbounded consumption and denial of serviceBound tokens, tools, time and spend at every level.v1 · last verified 21 Aug 2026
Risk model
Long contexts, recursive agents, repeated tool calls or many parallel requests can exhaust compute, quota or budget.
Prevention
Apply per-user and global quotas, token and step budgets, timeouts, concurrency caps, caching and circuit breakers.
Detection
Monitor cost per task, recursion depth, repeated near-identical calls, queue saturation and abnormal completion lengths.
Validation
Load-test worst-case contexts and tool loops, then confirm graceful cancellation and bounded financial exposure.
Limitations
Hard limits protect capacity but can interrupt legitimate long-running work and require product-specific tuning.
Defensive workflow
Estimate a task budget, reserve capacity, meter each step, stop at thresholds and return a resumable partial result instead of retrying blindly.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
12ModelModel extraction and replicationDetect systematic querying that reconstructs behavior or parameters.v1 · last verified 21 Aug 2026
Risk model
An attacker uses repeated API queries to approximate a model, infer decision boundaries or copy proprietary behavior.
Prevention
Use authentication, scoped quotas, output minimization, abuse controls and contractual boundaries appropriate to model value.
Detection
Look for broad coverage queries, synthetic grids, high-volume low-value sessions and coordinated accounts with shared patterns.
Validation
Replay known extraction strategies at safe scale and verify risk scoring, throttling and investigation evidence.
Limitations
Aggressive throttling can block legitimate evaluation, research or batch use and cannot stop all imitation.
Defensive workflow
Score sessions by coverage and repetition, challenge or throttle elevated risk, preserve privacy-safe evidence and review before account action.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
13DataMembership inference and training-data privacyLimit what model behavior reveals about whether a record was used.v1 · last verified 21 Aug 2026
Risk model
Confidence patterns or outputs can reveal whether a person's record appeared in training data, creating privacy risk.
Prevention
Minimize sensitive training data, control retention, reduce overfitting, consider privacy-preserving training and restrict detailed scores.
Detection
Audit privacy leakage on representative subgroups and monitor query patterns that compare closely related records.
Validation
Run membership-inference evaluations against train and holdout records under the same access available to a real caller.
Limitations
Privacy tests are attack-model specific, while stronger privacy mechanisms can trade off utility and cost.
Defensive workflow
Classify the dataset, remove unnecessary identifiers, train with documented privacy controls, test leakage and block release above threshold.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
14ModelAdversarial input and evasion robustnessMeasure how small input changes alter safety-critical decisions.v1 · last verified 21 Aug 2026
Risk model
Carefully chosen perturbations can move model predictions across decision boundaries while preserving apparent meaning or appearance.
Prevention
Use input constraints, robust training where evidence supports it, ensembles, abstention and downstream safety checks.
Detection
Monitor sensitivity to benign transformations, out-of-distribution scores and abrupt confidence shifts across equivalent inputs.
Validation
Test realistic perturbation families within the deployment threat model and report worst-case subgroup performance.
Limitations
Robustness to one perturbation norm or attack family does not transfer automatically to others.
Defensive workflow
Define valid input transformations, generate bounded tests, compare decision stability, route unstable cases to abstention and retain regressions.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
15HallucinationHallucination detection method stackCombine evidence, uncertainty and accountable review instead of trusting one score.v1 · last verified 21 Aug 2026
Risk model
A fluent answer can contain unsupported, contradicted, outdated or internally inconsistent claims that users may accept without checking.
Prevention
Limit generation to supported tasks, retrieve approved evidence, require citations, permit abstention and narrow output claims.
Detection
Use a layered stack of retrieval support, atomic claims, entailment, sample consistency, semantic uncertainty and human review.
Validation
Evaluate each layer and the combined decision policy on representative, time-split and adversarial examples with human labels.
Limitations
No method covers every failure class; detectors can share data, model and retrieval errors with the generator.
Defensive workflow
Decompose the answer, retrieve evidence, score support and uncertainty, block high-risk claims, then send unresolved high-impact items to review.
Hallucination-control comparison
| Method | What it detects | Failure condition | Operational use |
|---|---|---|---|
| Retrieval grounding | Checks whether a claim is supported by retrieved, approved evidence. | Fails when retrieval misses the right evidence, the corpus is stale or poisoned, or support is only superficial. | Require passage-level citations and reject answers with no attributable support. |
| Atomic claim verification | Splits long output into testable facts and scores each fact against evidence. | Misses errors when claims are decomposed badly, evidence is ambiguous, or the verifier shares the generator's bias. | Use for long-form reports and route unsupported high-impact claims to review. |
| NLI entailment checking | Classifies a claim as entailed, contradicted or not established by a cited passage. | Struggles with numerical reasoning, temporal qualifiers, long context and domains unlike its training data. | Treat undecidable results as a review state, not as support. |
| Self-consistency sampling | Flags statements that change across independently sampled answers. | Consistently repeated misconceptions look stable, and sampling increases cost and latency. | Use as a risk signal alongside evidence checks, never as a truth oracle. |
| Semantic entropy | Groups sampled answers by meaning and estimates uncertainty over those meanings. | Targets confabulation, not consistently wrong answers; meaning clustering can also introduce errors. | Escalate high-entropy questions to retrieval, refusal or human review. |
| Human review with calibrated abstention | Uses accountable subject-matter review when automated signals exceed a risk threshold. | Reviewers can be overloaded, anchored by fluent text, or lack access to authoritative evidence. | Define ownership, evidence requirements and stop conditions before deployment. |
Evidence
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile · NIST
- LLM09:2025 Misinformation · OWASP Gen AI Security Project
- SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models · Primary research
- Detecting hallucinations in large language models using semantic entropy · Nature
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation · Primary research
Version 1
last verified 21 Aug 2026
Initial defensive research review.
16HallucinationRetrieval grounding and citation alignmentMake every material claim traceable to the passage that supports it.v1 · last verified 21 Aug 2026
Risk model
A model can ignore retrieved evidence, cite an irrelevant passage or combine sources into a claim none of them supports.
Prevention
Retrieve from approved current corpora, preserve passage identifiers, constrain answers to evidence and allow a no-support outcome.
Detection
Score context relevance, claim support and citation coverage separately; flag citations that do not entail the attached claim.
Validation
Use answerable and unanswerable queries, stale evidence and distractor passages, then compare automated support labels with reviewers.
Limitations
Grounding cannot rescue a missing, incorrect or poisoned corpus and does not prove the source itself is true.
Defensive workflow
Classify the question, retrieve current approved passages, answer only supported claims, attach exact citations and abstain when coverage is insufficient.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
17HallucinationAtomic claim decompositionBreak long prose into claims that can each be checked.v1 · last verified 21 Aug 2026
Risk model
Document-level scores hide mixed outputs where several accurate statements surround one unsupported high-impact claim.
Prevention
Ask for bounded claims and structured fields, separate facts from interpretation and require evidence identifiers for factual fields.
Detection
Decompose sentences into independently verifiable units and label each supported, contradicted, unsupported or undecidable.
Validation
Measure decomposition completeness and claim labels against dual human annotation, especially for numbers, dates and negation.
Limitations
Decomposition can remove context, split claims incorrectly or create units whose truth depends on unstated assumptions.
Defensive workflow
Extract claims, preserve sentence context, retrieve evidence per claim, adjudicate labels and block the document if critical support is missing.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
18HallucinationEntailment-based evidence checkingDistinguish support from contradiction and missing evidence.v1 · last verified 21 Aug 2026
Risk model
Keyword overlap can make a citation appear relevant even when it contradicts or fails to establish the generated claim.
Prevention
Keep evidence passages short and attributable, normalize dates and units and require undecidable as an explicit outcome.
Detection
Run natural-language inference for each claim-passage pair and aggregate conservatively across independent sources.
Validation
Evaluate domain, language, numerical, temporal and negation slices; calibrate thresholds against human adjudication.
Limitations
NLI models inherit training-domain limits and may treat lexical similarity as evidence.
Defensive workflow
Pair each atomic claim with its cited passage, classify support, keep undecidable separate and escalate contradictions immediately.
Evidence
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models · ACL Anthology
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation · Primary research
- SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models · Primary research
Version 1
last verified 21 Aug 2026
Initial defensive research review.
19HallucinationSelf-consistency and semantic uncertaintyUse answer variation to find confabulation risk, not to vote truth into existence.v1 · last verified 21 Aug 2026
Risk model
When the model lacks stable knowledge, repeated generations can express incompatible meanings while each sounds confident.
Prevention
Allow abstention, retrieve evidence before generation and keep high-uncertainty tasks outside automated decision paths.
Detection
Sample multiple answers, cluster semantic equivalents and measure disagreement with SelfCheckGPT-style or semantic-entropy signals.
Validation
Calibrate on in-domain factual labels and test systematic falsehoods separately because consistent error can look certain.
Limitations
Sampling is costly, semantic clustering can be wrong and low uncertainty does not establish factuality.
Defensive workflow
Generate controlled samples, normalize meanings, compute disagreement, combine with evidence support and refuse or review above threshold.
Evidence
Version 1
last verified 21 Aug 2026
Initial defensive research review.
20HallucinationHallucination evaluation sets and driftMeasure the failures your product sees, then keep the test set moving with reality.v1 · last verified 21 Aug 2026
Risk model
Generic benchmarks can hide domain, language, time and workflow failures that dominate a real deployment.
Prevention
Create representative test strata, preserve fresh holdouts, document label policy and prevent evaluation examples from entering training.
Detection
Track factual precision, unsupported-claim rate, abstention, citation coverage and severity by domain, language and model version.
Validation
Use blinded dual review, adjudicate disagreement, report confidence intervals and rerun time-split holdouts after releases.
Limitations
Benchmarks become stale, can be contaminated and rarely predict every live failure mode.
Defensive workflow
Sample production-shaped prompts without personal data, label with evidence, freeze a release gate, monitor drift and refresh only with versioned review.
Evidence
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models · ACL Anthology
- TruthfulQA: Measuring How Models Mimic Human Falsehoods · Primary research
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation · Primary research
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile · NIST
Version 1
last verified 21 Aug 2026
Initial defensive research review.
21HallucinationHuman review and calibrated abstentionDefine when the system stops and who owns the unresolved claim.v1 · last verified 21 Aug 2026
Risk model
Automated checks can pass plausible errors, while unstructured review can become inconsistent, slow or overly trusting of fluent text.
Prevention
Set risk tiers, evidence requirements, reviewer qualifications, service levels and a safe abstention response before launch.
Detection
Route cases using claim severity, missing support, contradiction, uncertainty and detector disagreement rather than model confidence alone.
Validation
Measure reviewer agreement, correction rate, queue delay, anchoring and outcomes after release; audit a random sample of passes.
Limitations
Human review does not guarantee truth and can fail under overload, weak evidence or ambiguous accountability.
Defensive workflow
Apply automated gates, abstain on critical uncertainty, present claims beside evidence, record reviewer decisions and feed confirmed errors into regression tests.
Evidence
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile · NIST
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation · Primary research
- Detecting hallucinations in large language models using semantic entropy · Nature
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models · ACL Anthology
Version 1
last verified 21 Aug 2026
Initial defensive research review.