Everyone ships a chatbot demo. Fewer people have tried to break their own. We spent a day running an authorized penetration test against our platform — prompt injection, authentication attacks, tenant isolation, brute force, webhook forgery — and the results were more useful than any amount of "we take security seriously."
The injection attempts failed for a structural reason, not a clever one: retrieved documents are framed as untrusted content the model must never obey, and — more importantly — anything shaped like an answer to a structured question never reaches a model at all. The attacks came back as "I didn't catch a valid Company ID." A parser doesn't have opinions you can talk it out of.
Token forgery hit signed JWT verification. Cross-tenant lookups hit a records allow-list. The MFA brute force died against a five-attempt cap — we confirmed it by burning the code on purpose and watching the correct code get rejected.
The honest part: the test found real gaps. PIN guessing had no lockout — now five wrong PINs locks the line for ten minutes. One webhook had signature checking silently disabled by an empty configuration value — now enforced and verified against forged requests. Both are the classic kind of bug that exists quietly until someone looks for it on purpose.
The lesson we'd offer anyone building on LLMs: test the seams, not the demo. The seams are where the model meets your data, where your channels meet your backend, and where your configuration meets reality. That's where our bugs were — and the demo would never have found them.