Tracking Threats, Incidents, and Vulnerabilities
Tracking Threats, Incidents, and Vulnerabilities
Researchers found that styling prompts as poetry can significantly undermine the effectiveness of language models’ safety guardrails. By testing 25 popular models, they discovered that poetic prompts increased the likelihood of unsafe responses by an average of 35%. Google’s Gemini 1.5 Pro was the most susceptible, with a 100% success rate in bypassing restrictions with poetic prompts.
Researchers from Radware discovered a new exploit chain called “ZombieAgent” that leverages ChatGPT’s long-term memory and connector features to enable more severe indirect prompt injection (IPI) attacks. By planting malicious instructions in ChatGPT’s memory, attackers can persistently exfiltrate sensitive information from connected platforms. OpenAI has addressed this exploit by restricting ChatGPT’s ability to modify URLs, but further structural fixes are needed to enhance the security of AI agents.
https://www.darkreading.com/endpoint-security/chatgpt-memory-feature-prompt-injection
Prompt injection is a security challenge in AI, where attackers manipulate AI responses using malicious instructions in user inputs. As AI gains more capabilities and access to sensitive data, protecting users from these risks is crucial. OpenAI employs a multi-layered defense approach, including safety training, monitoring, security protections, user controls, red-teaming, and a bug bounty program. Users are advised to limit data access, verify agent actions, provide explicit instructions, and stay informed about security. Ongoing research aims to enhance AI robustness against these attacks and ensure safe interactions.
