Will we reach a point where all text boxes are handled by a LLM? Even though most organizations are building on top of foundation models rather than trying to build their own. How can we build and maintain a security boundary with an “intelligent” system that can’t really think? How do concepts like prompt injection, multi-stage exploits, SQLi, etc. mean to a loan application chatbot? What can a non-deterministic system do in a deterministic world? Mandiant has exploited developer-assistance chatbots during its Red Team Assessments to gain privileges within a client environment. Its consultants have explored and bypassed protections built to restrict the scope of a financial services chatbot. How can these and other stories help improve the security of future applications built on top of GenAI and LLMs?