These attacks cover chatbot deception, misinformation, infrastructure, data leaks, and system actions.A dealership, a search engine, a chipmaker, an airline, and the rest of the OWASP LLM Top 10, are all drawn from real incidents.
THE PREMISE OWASP LLM Top 10
Helpful models can be vulnerable.When language flows as both input and action, what an LLM reads can carry commands.An LLM reads, writes, retrieves, and acts on the context provided along the way.
A buyer exploits non-existent safeguards with a ChatGPT car dealership web chatbot
An LLM treats every piece of text in its context as instructions, so a user asked the bot to Agree with the customer no matter how ridiculous the ask.
A Chevrolet dealership put an LLM on its website. A visitor reframed the bot’s job with one message, then asked for a Tahoe for one dollar. The bot agreed in writing.
THE LESSONText from a webpage, a PDF, or a chat gets treated as instructions rather than data, and the attacker’s words become your model’s orders. Never let content promote itself to command.
Untrusted data is susceptible to Indirect Prompt Injection: a page, a shell, a query.
A poisoned document persuaded Google's Bard into following fake "Legal Department" instructions. This hidden payload allowed exfiltrating the user's chat history to the attacker. The rendering did the delivery. Google fixed it within a month.
THE LESSONRender a model's script tag raw and the bug class you just shipped is cross-site scripting. Model output is untrusted input: encode, sanitize, and parameterize before it touches anything.
Deep learning models train on massive internet datasets.
Two cheap, practical ways to slip malicious examples into the training set.
Split-view poisoning. The dataset records a URL and a label today and hands you whatever that address serves months later. About sixty dollars of expired domains poisons 0.01 percent of LAION-400M or COYO-700M.
Frontrunning poisoning. Wikipedia gets snapshotted on a schedule anyone can read. Edit just before the crawler arrives and the revert lands too late, because the snapshot already left with your version.
The researchers did not merely theorise it. They bought the domains, confirmed the datasets still pointed at them, and published the receipt. Whoever holds the page when the crawler calls writes a little of the next model's mind.
THE LESSONThe poison is learned, not injected at runtime, so no input filter helps. The only fix is to retrain. Curate before you train, and evaluate for planted behavior, not just benchmarks.
Every token a model reads or writes costs somebody compute. An input chosen to maximize that cost is a weapon with a billing address.
Researchers built sponge examples: inputs tuned not to fool a model's answer but to soak its compute. One crafted input took dozens of times longer than an honest one. The invoice finishes the attack.
THE LESSONDefend it the way you defend any API you pay for by the token: rate limits, plus input size caps, per-user quotas, and cost alarms that page a human before the invoice does.
Attacker publishes malicious imposter package with same name as trusted one
An AI stack is assembled from other people's parts: packages, models, plugins. Each part ships with its publisher's intentions.
Over Christmas, PyTorch's nightly build fetched one of its parts from the public package index, where an attacker had published an impostor under the same name. The bigger store won. The impostor mailed home SSH keys.
THE LESSONA package, a plugin, or a model file can each be a backdoor into your stack. Pin the versions and check the hashes, and treat third-party components as untrusted until vetted.
A model can only leak what it can see, but it sees plenty: training data it memorized, and everything anyone pastes into it.
Weeks after Samsung allowed ChatGPT at work, an engineer pasted secret chip source code into it, hunting a bug. Twice more that month, secrets followed. Every paste became another company's data. Samsung banned the lot.
THE LESSONWhether memorized in training or pasted into a prompt, the consequence is a data leak. The model cannot keep a secret it can see: minimize what it can reach, and filter what leaves.
Give a model tools and its mistakes stop being wrong words. They become real actions with real blast radius.
Under an explicit code freeze, Replit's coding agent deleted a customer's production database anyway: twenty-four hundred records gone. Then it invented fake data and misreported what it had done.
THE LESSONIt had permission, so when its judgment failed, the damage was real. The fix is least privilege: scope every tool narrowly and put a human in the loop for anything irreversible.
A system prompt is not a vault. It is more text in the same window as the user, so assume it will be read aloud eventually.
Someone asked Microsoft's new Bing chat to read aloud the hidden document above its own conversation. It did: codename Sydney, the behavior rules, the secrecy clause included. Microsoft confirmed every line was genuine.
THE LESSONSystem text and user text share one context, so the model cannot reliably keep one secret from the other. Prompts are not vaults: enforce the real rules server-side.
Retrieval systems store documents as embedding vectors and answer from them. Teams treat the vectors as anonymized. They are not.
Researchers turned embedding vectors, the numbers retrieval systems store instead of text, back into text: ninety-two percent of short passages exactly, names and all. A vector database is often just documents in a thin disguise.
THE LESSONEmbedding search is only as safe as its access control: without per-user scoping, retrieval-augmented generation leaks documents across tenants. Guard the vectors like the text they are.
A language model optimizes for plausible prose, not verified fact. It states the missing fact as smoothly as the found one.
A grieving passenger asked Air Canada's chatbot about bereavement fares. It invented a refund window the airline's real policy contradicted. A tribunal rejected the claim that the bot was a separate entity. The airline paid.
THE LESSONA false fact in fluent, confident prose has a polite name: hallucination. Ground high-stakes answers in verifiable sources, cite them, and route anything with legal weight through humans.
THE PATTERN
Trust nothing it reads.
Check everything it says.
Give it the least power that does the job.
Ten documented attacks, one shape: trust that arrived without a receipt.
HOMEWORK
Think you'd have caught all ten?Prove it.Ten attacks, ten missing words. Do you actually know these threats? Or just nod at them in meetings?