ai

Cybercriminals exploit large language models (LLMs) for illicit activities, using uncensored models or creating their own to bypass safety features. They employ these models for tasks like generating phishing emails, scanning vulnerabilities, and writing malware. Techniques like jailbreaking and obfuscation allow them to manipulate legitimate LLMs for harmful purposes. Cybercriminals also face risks of scams when accessing fraudulent LLM services. As LLM technology evolves, it serves as a tool for enhancing traditional cyber threats.

https://blog.talosintelligence.com/cybercriminal-abuse-of-large-language-models/

New report warns that AI presents significant data-breach risks, with 99% of organizations exposing sensitive information to potential AI misuse. Inadequate security measures, such as unverified apps and poor MFA practices, increase vulnerability. Recommendations include reducing risk exposure, continuous monitoring, and leveraging AI for security enhancements. No organization is fully prepared for these threats.

https://www.bleepingcomputer.com/news/security/ai-is-a-data-breach-time-bomb-reveals-new-report/

OpenAI's recent report highlights misuse of generative AI tools, primarily from actors in China, involved in producing misinformation and engaging in cyber activities. Specific examples include creating social media posts to influence political discourse and unauthorized cyber intrusions. The report emphasizes the ongoing arms race between developers and malicious users, amidst concerns about regulatory gaps in the US.

https://www.zdnet.com/article/how-global-threat-actors-are-weaponizing-ai-now-according-to-openai/

AI can weaken cybersecurity through three main methods:

1. Slopsquatting – Spreading malware via hallucinated software libraries recommended by AI, often targeting users who mistype URLs.
2. Prompt Injection – Attackers inject malicious prompts into AI applications, potentially leading to unauthorized information access or code execution.
3. Data Poisoning – Manipulating training data to skew AI model outputs, which poses risks for various industries.

These tactics exploit the vulnerabilities of AI systems, emphasizing the need for increased vigilance and adapted security measures.

https://www.bigdatawire.com/2025/04/25/three-ways-ai-can-weaken-your-cybersecurity/

Slopsquatting is a new supply chain attack using AI-generated nonexistent package names, making it easier for attackers to exploit developer trust in AI tools. Research shows significant hallucination rates in code generation models, with 19.7% of suggested packages being fake. This poses serious risks as developers may unknowingly install these malicious packages, especially with the growing trend of “vibe coding” where developers rely heavily on AI for code creation. Security measures must adapt to identify and mitigate these threats effectively to protect software ecosystems.

https://socket.dev/blog/slopsquatting-how-ai-hallucinations-are-fueling-a-new-class-of-supply-chain-attacks

Microsoft's blog discusses a straightforward jailbreak method, Context Compliance Attack (CCA), effective against many AI systems. CCA manipulates AI by exploiting reliance on client-supplied conversation history, allowing for context manipulation with minimal effort. Models maintaining conversation state, like Copilot and ChatGPT, are safe from this attack. Microsoft suggests enhancements like cryptographic signatures and server-side history to bolster AI safety. The implications of CCA stress the need for comprehensive security considerations in AI system designs, encouraging discussions on further mitigation strategies.

https://msrc.microsoft.com/blog/2025/03/jailbreaking-is-mostly-simpler-than-you-think/

Nearly 12,000 API keys and passwords were discovered in the Common Crawl dataset used to train AI models, raising concerns about insecure coding practices. Researchers found 11,908 valid secrets after examining 400 terabytes of data from billions of web pages. Among these were AWS and MailChimp keys, often hardcoded into HTML and JavaScript. Vulnerabilities include potential misuse for phishing and data exfiltration. The study highlights the challenge of removing sensitive information from large datasets despite pre-processing efforts.

https://www.bleepingcomputer.com/news/security/nearly-12-000-api-keys-and-passwords-found-in-ai-training-dataset/

DeepSeek AI may secretly transfer U.S. user data to the Chinese government, raising national security concerns. Cybersecurity experts found embedded code suggesting direct links to Chinese-controlled servers, potentially exposing users' identities and online activities. This situation mirrors past worries over other Chinese tech companies, prompting calls for banning DeepSeek on government devices.

https://abcnews.go.com/US/deepseek-coding-capability-transfer-users-data-directly-chinese/story?id=118465451

Over 57 nation-state threat groups, including those from China, Iran, North Korea, and Russia, are using Google-powered AI, notably Gemini, for cyber operations. These groups primarily use AI for research, troubleshooting code, and creating content. Iranian group APT42 utilizes Gemini extensively for phishing and reconnaissance, while Chinese APTs leverage it for network infiltration tactics. Russian actors focus on converting malware, and North Koreans use it for job applications to infiltrate Western companies. Google highlights the urgent need for public-private cooperation to enhance cyber defenses.

https://thehackernews.com/2025/01/google-over-57-nation-state-threat.html