This new data poisoning tool lets artists fight back against generative AI
A new tool lets artists add invisible changes to the pixels in their art before they upload it online so that if it’s scraped into an AI training set, it can cause the resulting model to break in chaotic and unpredictable ways.
https://www.technologyreview.com/2023/10/23/1082189/data-poisoning-artists-fight-generative-ai/
Wer sich über die vielen tollen Informationsangebote im Internet freut, sollte wissen:
#KI 🤖 randaliert im Netz – #Admins halten dagegen, damit wir Menschen ungestört surfen können.
Lest mal, wie Admins ihre absolut frustrierende aber unsichtbare Abwehrarbeit gegen KI beschreiben – im Blog von @campact:
👉 https://blog.campact.de/2025/05/ki-randaliert-im-netz-admins-halten-dagegen/
❗Nicht vergessen: 25. Juli ist #SysAdminDay
#FediAdmins #KIScraping #AI #AIScraping #TDM #AdminLeiden #MastoAdmin #DataPoisoning #aitxt #GPT #TDMRep
The AI Security Storm is Brewing: Are You Ready for the Downpour?
1,360 words, 7 minutes read time.
We live in an age where artificial intelligence is no longer a futuristic fantasy; it’s the invisible hand guiding everything from our morning commute to the recommendations on our favorite streaming services. Businesses are harnessing its power to boost efficiency, governments are exploring its potential for public services, and our personal lives are increasingly intertwined with AI-driven conveniences. But as this powerful technology becomes more deeply embedded in our world, a darker side is emerging – a growing storm of security risks that businesses and governments can no longer afford to ignore.
Think about this: the global engineering giant Arup was recently hit by a sophisticated scam where cybercriminals used artificial intelligence to create incredibly realistic “deepfake” videos and audio of their Chief Financial Officer and other executives. This elaborate deception tricked an employee into transferring a staggering $25 million to fraudulent accounts . This isn’t a scene from a spy movie; it’s a chilling reality of the threats we face today. And experts are sounding the alarm, with a recent prediction stating that a massive 93% of security leaders anticipate grappling with daily AI-driven attacks by the year 2025. This isn’t just a forecast; it’s a clear warning that the landscape of cybercrime is being fundamentally reshaped by the rise of AI.
While AI offers incredible opportunities, it’s crucial to understand that it’s a double-edged sword. The very capabilities that make AI so beneficial are also being weaponized by malicious actors to create new and more potent threats. From automating sophisticated cyberattacks to crafting incredibly convincing social engineering schemes, AI is lowering the barrier to entry for cybercriminals and amplifying the potential for widespread damage. So, let’s pull back the curtain and explore the growing shadow of AI, delving into the specific security risks that businesses and governments need to be acutely aware of.
One of the most significant ways AI is changing the threat landscape is by supercharging traditional cyberattacks. Remember those generic phishing emails riddled with typos? Those are becoming relics of the past. AI allows cybercriminals to automate and personalize social engineering schemes at an unprecedented scale. Imagine receiving an email that looks and sounds exactly like it came from your CEO, complete with their unique communication style and referencing specific projects you’re working on. AI can analyze vast amounts of data to craft these hyper-targeted messages, making them incredibly convincing and significantly increasing the chances of unsuspecting employees falling victim. This includes not just emails, but also more sophisticated attacks like “vishing” (voice phishing) where AI can mimic voices with alarming accuracy.
Beyond enhancing existing attacks, AI is also enabling entirely new forms of malicious activity. Deepfakes, like the ones used in the Arup scam, are a prime example. These AI-generated videos and audio recordings can convincingly impersonate individuals, making it nearly impossible to distinguish between what’s real and what’s fabricated. This technology can be used for everything from financial fraud and corporate espionage to spreading misinformation and manipulating public opinion. As Theresa Payton, CEO of Fortalice Solutions and former White House Chief Information Officer, noted, these deepfake scams are becoming increasingly sophisticated, making it critical for both individuals and companies to be vigilant .
But the threats aren’t just about AI being used to attack us; our AI systems themselves are becoming targets. Adversarial attacks involve subtly manipulating the input data fed into an AI model to trick it into making incorrect predictions or decisions. Think about researchers who were able to fool a Tesla’s autopilot system into driving into oncoming traffic by simply placing stickers on the road. These kinds of attacks can have serious consequences in critical applications like autonomous vehicles, healthcare diagnostics, and security systems .
Another significant risk is data poisoning, where attackers inject malicious or misleading data into the training datasets used to build AI models. This can corrupt the model’s learning process, leading to biased or incorrect outputs that can have far-reaching and damaging consequences. Imagine a malware detection system trained on poisoned data that starts classifying actual threats as safe – the implications for cybersecurity are terrifying.
Furthermore, the valuable intellectual property embedded within AI models makes them attractive targets for theft. Model theft, also known as model inversion or extraction, allows attackers to replicate a proprietary AI model by querying it extensively. This can lead to significant financial losses and a loss of competitive advantage for the organizations that invested heavily in developing these models.
The rise of generative AI, while offering incredible creative potential, also introduces its own unique set of security challenges. Direct prompt injection attacks exploit the way large language models (LLMs) work by feeding them carefully crafted malicious inputs designed to manipulate their behavior or output . This can lead to the generation of harmful, biased, or misleading information, or even the execution of unintended commands . Additionally, LLMs have the potential to inadvertently leak sensitive information that was present in their training data or provided in user prompts, raising serious privacy concerns. As one Reddit user pointed out, there are theoretical chances that your data can come out as answers to other users’ prompts when using these models.
Beyond these direct threats, businesses also need to be aware of the risks lurking in the shadows. “Shadow AI” refers to the unauthorized or ungoverned use of AI tools and services by employees within an organization. This can lead to the unintentional exposure of sensitive company data to external and potentially untrusted AI services, creating compliance nightmares and introducing security vulnerabilities that IT departments are unaware of.
So, what can businesses and governments do to weather this AI security storm? The good news is that proactive measures can significantly mitigate these risks. For businesses, establishing clear AI security policies and governance frameworks is paramount. This includes outlining approved AI tools, data handling procedures, and protocols for vetting third-party AI vendors. Implementing robust data security and privacy measures, such as encryption and strict access controls, is also crucial. Adopting a Zero-Trust security architecture for AI systems, where no user or system is automatically trusted, can add another layer of defense. Regular AI risk assessments and security audits, including penetration testing by third-party experts, are essential for identifying and addressing vulnerabilities. Furthermore, ensuring transparency and explainability in AI deployments, whenever possible, can help build trust and facilitate the identification of potential issues. Perhaps most importantly, investing in comprehensive employee training on AI security awareness, including recognizing sophisticated phishing and deepfake techniques, is a critical first line of defense.
Governments, facing even higher stakes, need to develop national AI security strategies and guidelines that address the unique risks to critical infrastructure and national security. Implementing established risk management frameworks like the NIST AI Risk Management Framework (RMF) and the ENISA Framework for AI Cybersecurity Practices (FAICP) can provide a structured approach to managing these complex risks. Establishing clear legal and regulatory frameworks for AI use is also essential to ensure responsible and secure deployment. Given the global nature of AI threats, promoting international collaboration on AI security standards is crucial. Finally, focusing on “security by design” principles in AI development, integrating security considerations from the outset, is the most effective way to build resilient and trustworthy AI systems.
The AI security landscape is complex and constantly evolving. Staying ahead of the curve requires a proactive, multi-faceted approach that combines technical expertise, robust policies, ethical considerations, and ongoing vigilance. The storm of AI security risks is indeed brewing, but by understanding the threats and implementing effective mitigation strategies, businesses and governments can prepare for the downpour and navigate this challenging new terrain.
Want to stay informed about the latest developments in AI security and cybercrime? Subscribe to our newsletter for in-depth analysis, expert insights, and practical tips to protect yourself and your organization. Or, join the conversation by leaving a comment below – we’d love to hear your thoughts and experiences!
D. Bryan King
Sources
Disclaimer:
The views and opinions expressed in this post are solely those of the author. The information provided is based on personal research, experience, and understanding of the subject matter at the time of writing. Readers should consult relevant experts or authorities for specific guidance related to their unique situations.
Related Posts
#adversarialAttacks #AIAudit #AIBestPractices #AICompliance #AICybercrime #AIDataSecurity #AIForNationalSecurity #AIGovernance #AIInBusiness #AIInCriticalInfrastructure #AIInGovernment #AIIncidentResponse #AIMisuse #AIModelSecurity #AIMonitoring #AIRegulations #AIRiskAssessment #AIRiskManagement #AISafety #AISecurity #AISecurityAwareness #AISecurityFramework #AISecurityPolicies #AISecuritySolutions #AISecurityTrends2025 #AIStandards #AISupplyChainRisks #AIThreatIntelligence #AIThreatLandscape #AIThreats #AITraining #AIVulnerabilities #AIAssistedSocialEngineering #AIDrivenAttacks #AIEnabledMalware #AIGeneratedContent #AIPoweredCyberattacks #AIPoweredPhishing #artificialIntelligenceSecurity #cyberSecurity #cybersecurityRisks #dataBreaches #dataPoisoning #deepfakeDetection #deepfakeScams #ENISAFAICP #ethicalAI #generativeAISecurity #governmentAISecurity #largeLanguageModelSecurity #LLMSecurity #modelTheft #nationalSecurityAIRisks #NISTAIRMF #privacyLeaks #promptInjection #shadowAI #zeroTrustAI
Like cutting down a forest without growing new trees, the AI corporations seem to be consuming the natural raw material of their money-making machines faster than it can be replenished.
Natural, human-generated information, be they works of art, or conversations about factual things like how to write software, are the source of training data for Large Language Models (LLMs), which is what people are calling “artificial intelligence” nowadays. LLM shops spend untold millions on curating the information they harvest to ensure this data is strictly human-generated and free of other LLM-generated content. If they do not do this, the non-factual “hallucinations” (fictional content) that these LLMs generate may come to dominate the factual human-made training data, making the answers that the LLMs generate increasingly more prone to hallucination.
The Internet is already so full of LLM-generated content that it has become a major problem for these companies. The new LLMs are more and more often trained on fictional LLM-generated content that passes as factual and human-made, which is rapidly making LLMs less and less accurate as time goes on — a viscous downward spiral.
But it gets worse. Thanks to all of the AI hype, everyone is asking questions of LLMs nowadays and not of other humans. So the source of these LLMs training data, web sites like StackOverflow and Reddit, are now no longer recording as many questions from humans to other humans. If that human-made information disappears, so does the source of natural resources that make it possible to build these LLMs.
Even worse still, if there are any new innovations in science or technology, unless humans are asking question to the human innovators, the LLMs can’t learn about these things innovations either. Everyone will be stuck in this churning maelstrom of AI “slop,” asking only questions that have asked by millions of others before, and never receiving any true or accurate answers on new technology. And nobody, neither the humans nor the machines, will be learning anything new at all, while the LLMs become more and more prone to hallucinations with each new generation of AI released to the public.
I think we are finally starting to see the real limitations of this LLM technology come into clear view, the rate at which it is innovating is simply not sustainable. Clearly pouring more and more money and energy into scaling up these LLM project will not lead to increased return-on-investment, and will definitely not lead to the “singularity” in which machine intelligence surpasses human intelligence. So how long before the masses finally realize they have been sold nothing but a bill of goods by these AI corporations?
#tech #AI #Slop #LLM #GenerativeAI #ChatGPT #OpenAI #Google #Gemini #MetaAI #StackOverflow #Reddit #DataPoisoning
Florida International University: “Poisoned” AI models can unleash real-world chaos. Can these attacks be prevented?. “The majority of AI systems we encounter today — from ChatGPT to Netflix’s personalized recommendations — are only ‘intelligent’ enough to pull off such impressive feats because of the extensive amounts of text, imagery, speech and other data they are trained on. If […]
Generar contingut amb IA per contrarrestar l'excés de cerques amb IA. Què pot sortir malament?
Al blog: Bloquejar cerques d'IA embrutant (també) dades
AI struggles with less common data: Inconsistent results for Valletta Bastions (actual mean height: 25m) highlight issues with insufficient training data. We also touch on AI poisoning.
https://www.alanbonnici.com/2025/03/ai-got-it-wrong-missing-information-or.html
#AI #DataBias #Valletta #TTMO #ArtificialIntelligence #hallucination #Mistakes #TestingAI #InsufficientData #DataPoisoning
A message you do not want to see when loading "just data" in your AI / ML framework. Beware!
Description in comments.
#ctf #HackTheBox #AI #ML #CyberSecurity #TensorFlow #DataPoisoning
How does AI handle insufficient information? 🤔 We tested an AI with questions about the Eiffel Tower, Big Ben, and the bastions of Valletta. The AI gave inconsistent answers when training data is limited or unclear. We also touch on AI poisoning, where AI models can be misled by fake data
▶️ https://buff.ly/yRDWPTf
#AI #InsufficientData #DataPoisoning #EiffelTower #BigBen #Valletta #TestingAI #Accuracy #TTMO
Hi #Admins 👋,
Can you give me quotes that explain your fight against #AIScraping? I'm looking for (verbal) images, metaphors, comparisons, etc. that explain to non-techies what's going on. (efforts, goals, resources...)
I intend to publish your quotes in a text on @campact 's blog¹ (DE, German NGO).
The quotes should make your work🙏 visible in a generally understandable way
¹ https://blog.campact.de/author/friedemann/
#TDM #MastoAdmin #DataPoisoning #aitxt #GPT #TDMRep #Kudurru #Nightshade #Glaze #FediAdmins
Liebe #Admins 👋,
für meine @campact -Kolumne¹ suche ich #Techies, die der Welt da draußen den fight #Admins vs. #KIScraping erklären. Mit welchen Bildern, Metaphern, Vergleichen beschreibt ihr, was da abgeht? (Zeitaufwand, Sinn, Ressourcen, Tools…)
Die Zitate sollen allgemeinverständlich eure Arbeit🙏 sichtbar machen.
(namentlich, pseudoym, anonym → gern angeben)
🙏🙏🙏
¹https://blog.campact.de/author/friedemann/
#TDM #AdminLeiden #MastoAdmin #DataPoisoning #aitxt #TDMRep #Nightshade #Glaze #FediAdmins
“We find that replacement of just 0.001% of training tokens with medical misinformation results in harmful models more likely to propagate medical errors. Furthermore, we discover that corrupted models match the performance of their corruption-free counterparts on open-source benchmarks routinely used to evaluate medical LLMs. Using biomedical knowledge graphs to screen medical LLM outputs, we propose a harm mitigation strategy…”
#LLM #misinformation #datapoisoning
https://www.nature.com/articles/s41591-024-03445-1
"The adoption of large language models (LLMs) in healthcare demands a careful analysis of their potential to spread false medical knowledge. Because LLMs ingest massive volumes of data from the open Internet during training, they are potentially exposed to unverified medical knowledge that may include deliberately planted misinformation. Here, we perform a threat assessment that simulates a data-poisoning attack against The Pile, a popular dataset used for LLM development. We find that replacement of just 0.001% of training tokens with medical misinformation results in harmful models more likely to propagate medical errors. Furthermore, we discover that corrupted models match the performance of their corruption-free counterparts on open-source benchmarks routinely used to evaluate medical LLMs. Using biomedical knowledge graphs to screen medical LLM outputs, we propose a harm mitigation strategy that captures 91.9% of harmful content (F1 = 85.7%). Our algorithm provides a unique method to validate stochastically generated LLM outputs against hard-coded relationships in knowledge graphs. In view of current calls for improved data provenance and transparent LLM development, we hope to raise awareness of emergent risks from LLMs trained indiscriminately on web-scraped data, particularly in healthcare where misinformation can potentially compromise patient safety."
https://www.nature.com/articles/s41591-024-03445-1?utm_source=substack&utm_medium=email
#AI #GenerativeAI #LLMs #Healthcare #ThePile #Healthcare #AISafety #DataPoisoning #Misinformation #AITraining
Safeguarding #OpenData: #Cybersecurity essentials and skills for #data providers by Publications Office of the #EuropeanUnion
This webinar provides an overview of the fundamentals of open data and the complexity in terms of cybersecurity.
https://www.youtube.com/watch?v=6kPiY_8hRwQ
#InfoSec #Security #DataPoisoning #DataTampering #Privacy #Risk #OpenSource #Technology
Second up was Weeks et al.'s "A First Look at Toxicity Injection Attacks on Open-domain Chatbots", exploring how easy it is to inject toxicity into chatbots after deployment. (https://www.acsac.org/2023/program/final/s155.html) 3/4
#Chatbots #AI #DataPoisoning
Does anyone know of an existing open source project working on AI model poisoning or style cloaking, in the vein of #glaze and #nightshade?
I'm interested in this tech but they both seem to be proprietary, and I'd like to see if there is any work being done on the open source side of things.
#Nightshade is an offensive #DataPoisoning tool, a companion to a defensive style protection tool called #Glaze, which The Register covered in February last year.
Nightshade poisons #ImageFiles to give indigestion to models that ingest data without permission. It's intended to make those training image-oriented models respect content creators' wishes about the use of their work. #LLM #AI
How artists can poison their pics with deadly Nightshade to deter #AIScrapers
https://www.theregister.com/2024/01/20/nightshade_ai_images/
/imagine salt-and-thorium mini-reactors designed by #cyberpunk and #solarpunk at #Microsoft.
Widespread LLM usage was Chernobyl of the internet.
https://www.superversive.co/blog/synthetic-chernobyl
#AI #LLMs #DataPoisoning #ModelCollapse #ecosystem #internet #DataGovernance #FirstPrinciples #ProductManagement #LinkedIn #B2B
