Day 8/10: Your AI Feature Is Also a Backdoor
• Limit who can access your AI tools
• Log everything the LLM sees + says
• Hide system prompts from users
AI isn’t just smart — it’s a surface.
Follow @nickbalancom to build and protect.
#AIThreats #PromptInjection #WebSecurity #LLMFeatures #nickbalancom
AI Coding Assistants Can be Both a Friend & a Foe
New research shows that GitLab's AI assistant, Duo, can be tricked into writing malicious code and even leaking private source data through hidden instructions embedded in developer content like merge requests and bug reports.
How? Through a classic prompt injection exploit that inserts secret commands into code that Duo reads. This results in Duo unknowingly outputting clickable malicious links or exposing confidential information.
While GitLab has taken steps to mitigate this, the takeaway is clear: AI assistants are now part of your attack surface. If you’re using tools like Duo, assume all inputs are untrusted, and rigorously review every output.
Read the details: https://arstechnica.com/security/2025/05/researchers-cause-gitlab-ai-developer-assistant-to-turn-safe-code-malicious/
#AIsecurity #GitLab #AI #PromptInjection #Cybersecurity #DevSecOps #CISO #Infosec #IT #AIAttackSurface #SoftwareSecurity #CISO
𝟸𝟶+ 𝙰𝙸 𝚌𝚑𝚊𝚝𝚋𝚘𝚝𝚜 𝚌𝚘𝚞𝚕𝚍𝚗’𝚝 𝚑𝚘𝚕𝚍 𝚝𝚑𝚎 𝚕𝚒𝚗𝚎 — prompt injections slipped through in what we're calling the Yearbook Attack.
https://www.straiker.ai/blog/weaponizing-wholesome-yearbook-quotes-to-break-ai-chatbot-filters
#AISecurity #SecureAI #AICybersecurity #AIThreats #GenAISecurity #AITrust #PromptInjection
GitHub MCP pomaga wykradać dane z prywatnych repozytoriów – kolejna sprytna technika prompt injection w wydaniu indirect
W naszym ostatnim tekście przybliżyliśmy problemy opisane przez firmę Legit, które pozwalały na wykorzystanie asystenta Duo do kradzieży poufnych informacji z prywatnych repozytoriów lub zmuszania modelu LLM do wykonywania niepożądanych przez prawowitego właściciela projektu działań. Jak można się było domyślić, poruszana tematyka jest związana raczej z architekturą rozwiązań niż z...
Angriff über #GitHub MCP Server: Zugriff auf private Daten | Developer https://www.heise.de/news/Angriff-ueber-GitHub-MCP-Server-Zugriff-auf-private-Daten-10399081.html #ArtificialIntelligence #PromptInjection
"The curse of prompt injection continues to be that we’ve known about the issue for more than two and a half years and we still don’t have convincing mitigations for handling it.
I’m still excited about tool usage—it’s the next big feature I plan to add to my own LLM project—but I have no idea how to make it universally safe.
If you’re using or building on top of MCP, please think very carefully about these issues:
Clients: consider that malicious instructions may try to trigger unwanted tool calls. Make sure users have the interfaces they need to understand what’s going on—don’t hide horizontal scrollbars for example!
Servers: ask yourself how much damage a malicious instruction could do. Be very careful with things like calls to os.system(). As with clients, make sure your users have a fighting chance of preventing unwanted actions that could cause real harm to them.
Users: be thoughtful about what you install, and watch out for dangerous combinations of tools."
https://simonwillison.net/2025/Apr/9/mcp-prompt-injection/
#AI #GenerativeAI #LLMs #Chatbots #CyberSecurity #MCP #PromptInjection
A vulnerability in GitHub MCP lets malicious Issues hijack AI agents to leak data from private repos. Invariant calls this a “toxic agent flow” and shows it can exfiltrate sensitive info via prompt injection. GitHub alone can't fix it—mitigation needs system-level controls.
GitHub Copilot Ecosystem Hit by Critical MCP Server Security Flaw
#AI #GitHub #AISecurity #MCP #Vulnerability #Coding #DataLeak #PromptInjection #Cybersecurity #DevSecOps #AICoding #Microsoft
https://medium.com/@brunosaetta/prompt-injection-e-manipolazione-delle-ai-bee5610ee6e8
Le AI possono essere manipolate. E noi con loro. Esploriamo i rischi (reali) della prompt injection: da bug invisibili a strumenti di propaganda personalizzata. Dove si possono "mettere le mani" in un LLM?
#AI #Manipolazione #PromptInjection #EticaDigitale #LLM
El lado del mal - Hacking Gitlab Duo: Remote Prompt Injection, Malicious Prompt Smuggling, Client-Side Attacks & Private Code Stealing. https://www.elladodelmal.com/2025/05/hacking-gitlab-duo-hacking-gitlab-duo.html #GitLab #Duo #AI #IA #Hacking #LLM #PromptInjection #MemoryPoisoning #Smuggling #XSS #HTMLInjection
Day 3/10: Fight LLM Prompt Injection in Web Forms
AI doesn’t just answer. It listens.
• Never feed raw input into your prompt
• Clean everything users type
• Review AI output like it came from a stranger
Your forms are smart now. So are the risks.
Follow @nickbalancom for AI-ready security.
#PromptInjection #SecureAI #WebSecurity #CyberTips #nickbalancom
Whenever you're talking to your AI assistant, make sure it's in a space where no other party can inject any type of sounds. Because inaudible background noise may contain instructions that do a prompt injection on your LLM: https://repello.ai/blog/turning-background-noise-into-a-prompt-injection-attacks-in-voice-ai
The AI Security Storm is Brewing: Are You Ready for the Downpour?
1,360 words, 7 minutes read time.
We live in an age where artificial intelligence is no longer a futuristic fantasy; it’s the invisible hand guiding everything from our morning commute to the recommendations on our favorite streaming services. Businesses are harnessing its power to boost efficiency, governments are exploring its potential for public services, and our personal lives are increasingly intertwined with AI-driven conveniences. But as this powerful technology becomes more deeply embedded in our world, a darker side is emerging – a growing storm of security risks that businesses and governments can no longer afford to ignore.
Think about this: the global engineering giant Arup was recently hit by a sophisticated scam where cybercriminals used artificial intelligence to create incredibly realistic “deepfake” videos and audio of their Chief Financial Officer and other executives. This elaborate deception tricked an employee into transferring a staggering $25 million to fraudulent accounts . This isn’t a scene from a spy movie; it’s a chilling reality of the threats we face today. And experts are sounding the alarm, with a recent prediction stating that a massive 93% of security leaders anticipate grappling with daily AI-driven attacks by the year 2025. This isn’t just a forecast; it’s a clear warning that the landscape of cybercrime is being fundamentally reshaped by the rise of AI.
While AI offers incredible opportunities, it’s crucial to understand that it’s a double-edged sword. The very capabilities that make AI so beneficial are also being weaponized by malicious actors to create new and more potent threats. From automating sophisticated cyberattacks to crafting incredibly convincing social engineering schemes, AI is lowering the barrier to entry for cybercriminals and amplifying the potential for widespread damage. So, let’s pull back the curtain and explore the growing shadow of AI, delving into the specific security risks that businesses and governments need to be acutely aware of.
One of the most significant ways AI is changing the threat landscape is by supercharging traditional cyberattacks. Remember those generic phishing emails riddled with typos? Those are becoming relics of the past. AI allows cybercriminals to automate and personalize social engineering schemes at an unprecedented scale. Imagine receiving an email that looks and sounds exactly like it came from your CEO, complete with their unique communication style and referencing specific projects you’re working on. AI can analyze vast amounts of data to craft these hyper-targeted messages, making them incredibly convincing and significantly increasing the chances of unsuspecting employees falling victim. This includes not just emails, but also more sophisticated attacks like “vishing” (voice phishing) where AI can mimic voices with alarming accuracy.
Beyond enhancing existing attacks, AI is also enabling entirely new forms of malicious activity. Deepfakes, like the ones used in the Arup scam, are a prime example. These AI-generated videos and audio recordings can convincingly impersonate individuals, making it nearly impossible to distinguish between what’s real and what’s fabricated. This technology can be used for everything from financial fraud and corporate espionage to spreading misinformation and manipulating public opinion. As Theresa Payton, CEO of Fortalice Solutions and former White House Chief Information Officer, noted, these deepfake scams are becoming increasingly sophisticated, making it critical for both individuals and companies to be vigilant .
But the threats aren’t just about AI being used to attack us; our AI systems themselves are becoming targets. Adversarial attacks involve subtly manipulating the input data fed into an AI model to trick it into making incorrect predictions or decisions. Think about researchers who were able to fool a Tesla’s autopilot system into driving into oncoming traffic by simply placing stickers on the road. These kinds of attacks can have serious consequences in critical applications like autonomous vehicles, healthcare diagnostics, and security systems .
Another significant risk is data poisoning, where attackers inject malicious or misleading data into the training datasets used to build AI models. This can corrupt the model’s learning process, leading to biased or incorrect outputs that can have far-reaching and damaging consequences. Imagine a malware detection system trained on poisoned data that starts classifying actual threats as safe – the implications for cybersecurity are terrifying.
Furthermore, the valuable intellectual property embedded within AI models makes them attractive targets for theft. Model theft, also known as model inversion or extraction, allows attackers to replicate a proprietary AI model by querying it extensively. This can lead to significant financial losses and a loss of competitive advantage for the organizations that invested heavily in developing these models.
The rise of generative AI, while offering incredible creative potential, also introduces its own unique set of security challenges. Direct prompt injection attacks exploit the way large language models (LLMs) work by feeding them carefully crafted malicious inputs designed to manipulate their behavior or output . This can lead to the generation of harmful, biased, or misleading information, or even the execution of unintended commands . Additionally, LLMs have the potential to inadvertently leak sensitive information that was present in their training data or provided in user prompts, raising serious privacy concerns. As one Reddit user pointed out, there are theoretical chances that your data can come out as answers to other users’ prompts when using these models.
Beyond these direct threats, businesses also need to be aware of the risks lurking in the shadows. “Shadow AI” refers to the unauthorized or ungoverned use of AI tools and services by employees within an organization. This can lead to the unintentional exposure of sensitive company data to external and potentially untrusted AI services, creating compliance nightmares and introducing security vulnerabilities that IT departments are unaware of.
So, what can businesses and governments do to weather this AI security storm? The good news is that proactive measures can significantly mitigate these risks. For businesses, establishing clear AI security policies and governance frameworks is paramount. This includes outlining approved AI tools, data handling procedures, and protocols for vetting third-party AI vendors. Implementing robust data security and privacy measures, such as encryption and strict access controls, is also crucial. Adopting a Zero-Trust security architecture for AI systems, where no user or system is automatically trusted, can add another layer of defense. Regular AI risk assessments and security audits, including penetration testing by third-party experts, are essential for identifying and addressing vulnerabilities. Furthermore, ensuring transparency and explainability in AI deployments, whenever possible, can help build trust and facilitate the identification of potential issues. Perhaps most importantly, investing in comprehensive employee training on AI security awareness, including recognizing sophisticated phishing and deepfake techniques, is a critical first line of defense.
Governments, facing even higher stakes, need to develop national AI security strategies and guidelines that address the unique risks to critical infrastructure and national security. Implementing established risk management frameworks like the NIST AI Risk Management Framework (RMF) and the ENISA Framework for AI Cybersecurity Practices (FAICP) can provide a structured approach to managing these complex risks. Establishing clear legal and regulatory frameworks for AI use is also essential to ensure responsible and secure deployment. Given the global nature of AI threats, promoting international collaboration on AI security standards is crucial. Finally, focusing on “security by design” principles in AI development, integrating security considerations from the outset, is the most effective way to build resilient and trustworthy AI systems.
The AI security landscape is complex and constantly evolving. Staying ahead of the curve requires a proactive, multi-faceted approach that combines technical expertise, robust policies, ethical considerations, and ongoing vigilance. The storm of AI security risks is indeed brewing, but by understanding the threats and implementing effective mitigation strategies, businesses and governments can prepare for the downpour and navigate this challenging new terrain.
Want to stay informed about the latest developments in AI security and cybercrime? Subscribe to our newsletter for in-depth analysis, expert insights, and practical tips to protect yourself and your organization. Or, join the conversation by leaving a comment below – we’d love to hear your thoughts and experiences!
D. Bryan King
Sources
Disclaimer:
The views and opinions expressed in this post are solely those of the author. The information provided is based on personal research, experience, and understanding of the subject matter at the time of writing. Readers should consult relevant experts or authorities for specific guidance related to their unique situations.
Related Posts
#adversarialAttacks #AIAudit #AIBestPractices #AICompliance #AICybercrime #AIDataSecurity #AIForNationalSecurity #AIGovernance #AIInBusiness #AIInCriticalInfrastructure #AIInGovernment #AIIncidentResponse #AIMisuse #AIModelSecurity #AIMonitoring #AIRegulations #AIRiskAssessment #AIRiskManagement #AISafety #AISecurity #AISecurityAwareness #AISecurityFramework #AISecurityPolicies #AISecuritySolutions #AISecurityTrends2025 #AIStandards #AISupplyChainRisks #AIThreatIntelligence #AIThreatLandscape #AIThreats #AITraining #AIVulnerabilities #AIAssistedSocialEngineering #AIDrivenAttacks #AIEnabledMalware #AIGeneratedContent #AIPoweredCyberattacks #AIPoweredPhishing #artificialIntelligenceSecurity #cyberSecurity #cybersecurityRisks #dataBreaches #dataPoisoning #deepfakeDetection #deepfakeScams #ENISAFAICP #ethicalAI #generativeAISecurity #governmentAISecurity #largeLanguageModelSecurity #LLMSecurity #modelTheft #nationalSecurityAIRisks #NISTAIRMF #privacyLeaks #promptInjection #shadowAI #zeroTrustAI
El lado del mal - Taxonomía de Fallos de Seguridad en Agentic AI: Memory Poisoning Attack con Cross-Domain Prompt Injection Attack (XPIA) https://www.elladodelmal.com/2025/05/taxonomia-de-fallos-de-seguridad-en.html #IA #AI #PromptInjection #XPIA #AgenticAI #InteligenciaArtificial #Hacking #Ciberseguridad #hardening
This is an awesome, scary, black hat presentation.
Prompt injection to the next level.
Practical, easy to understand, jargon-free, actionable demonstrations to make it really clear.
I wish more talks were like this.
AI-powered features are the new attack surface! Check out our new blog in which LMG Security’s Senior Penetration Tester Emily Gosney @baybedoll shares real-world strategies for testing AI-driven web apps against the latest prompt injection threats.
From content smuggling to prompt splitting, attackers are using natural language to manipulate AI systems. Learn the top techniques—and why your web app pen test must include prompt injection testing to defend against today’s AI-driven threats.
#CyberSecurity #PromptInjection #AIsecurity #WebAppSecurity #PenetrationTesting #LLMvulnerabilities #Pentest #DFIR #AI #CISO #Pentesting #Infosec #ITsecurity
El lado del mal - Cómo saltarse los AI Guardrails con Invisible Characters & Adversarial Prompts para hacer Prompt Injection & Jailbreak Smuggling https://www.elladodelmal.com/2025/05/como-saltarse-los-ai-guardrails-con.html #AI #Hacking #IA #jailbreak #PromptInjection #AzurePromptShield #LlamaGuard #Smuggling #AIGuardrails
Indirect prompt injection attacks exploit LLMs by embedding malicious instructions in external content. Learn how they work & how to protect AI systems: https://jpmellojr.blogspot.com/2025/05/indirect-prompt-injection-attacks.html #AIsecurity #LLM #PromptInjection
Straiker Ascend AI
Red teaming on autopilot. It’s continuous, intelligent, and relentless.
Straiker Defend AI
Defences purpose-built for agentic applications, and their evolving risks.
https://youtube.com/shorts/YG8Id5tLqxw?feature=share
Securing the future so you can imagine it.
#SecureAIwithAI #Straiker #AIAppSec #LLMSecurity #AgenticAI #AIredteam #PromptInjection #AIguardrails #GenAISecurity #RuntimeSecurity #SecureAIwithAI
El lado del mal - Llama Protections: LlamaFirewall con PromptGuard 2, LlamaGuard 4, AlignmentCheck, CodeShield + AutoPatchBench & CyberSecEval 4 https://www.elladodelmal.com/2025/04/llamafirewall-con-promptguard-2.html #Llama #LLM #Hardening #Ciberseguridad #PromptInjection #Jailbreak #Llama4 #CodeShield #IA #OpenSource #IA
