ACL 2024 Tutorial:
Vulnerabilities of Large Language Models to Adversarial Attacks

Schedule

Our tutorial will be held on World Ballroom B (Level 23). Slides may be subject to updates.

Reading List

The full list of papers can be found in our recent survey: Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks, below we highlight a small subset of the papers where bold papers will be discussed in detail during our tutorial.

Prerequisites

• Attention is all you need, Vaswani et al., 2017.

Section 2: NLP and Security Background

• Delving into Transferable Adversarial Examples and Black-box Attacks (Liu et al., 2016)
• HotFlip: White-Box Adversarial Examples for Text Classification (Ebrahimi et al., 2017)
• Simple Black-Box Adversarial Attacks on Deep Neural Networks (Narodytska et al., 2017)
• Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers (Gao et al., 2018)
• Sensitivity of Adversarial Perturbation in Fast Gradient Sign Method (Liu et al., 2019)
• Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment (Jin et al., 2020)
• LLM Censorship: A Machine Learning Challenge or a Computer Security Problem? (Glukhov et al., 2023)
• Identifying and Mitigating the Security Risks of Generative AI (Barrett et al., 2023)
• Training language models to follow instructions with human feedback (Ouyang et al., 2023)

Section 3: Text-only Attacks

• Universal and Transferable Adversarial Attacks on Aligned Language Models (Zou et al., 2023)
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (Liu et al., 2023)
• AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs (Liao et al., 2024)
• Jailbreaking Black Box Large Language Models in Twenty Queries (Chao et al., 2023)
• Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study (Liu et al., 2023)
• Low-Resource Languages Jailbreak GPT-4 (Yong et al., 2023)
• Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (Huang et al., 2023)
• Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations (Wei et al., 2023)
• Many-shot Jailbreaking (Anil et al., 2024)
• GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (Yu et al., 2023)
• How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs (Zeng et al., 2024)

Section 4-1: Multi-modal Attacks (Image -> Text)

• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (Shayegani et al., 2024)
• Visual Adversarial Examples Jailbreak Aligned Large Language Models (Qi et al., 2024)
• Are aligned neural networks adversarially aligned? (Carlini et al., 2023)
• Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons (Noever et al., 2021)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (Greshake et al., 2023)
• Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs (Bagdasaryan et al., 2023)
• FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (Gong et al., 2024)
• JailBreakV-28K: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks (Luo et al., 2024)
• The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative (Tan et al., 2024)
• Adversarial Attacks on Multimodal Agents (Wu et al., 2024)
• Misusing Tools in Large Language Models With Visual Adversarial Examples (Fu et al., 2023)

Section 4-2: Multi-modal Attacks (Text -> Image)

• A Pilot Study of Query-Free Adversarial Attack against Stable Diffusion (Zhuang et al., 2023)
• Asymmetric Bias in Text-to-Image Generation with Adversarial Attacks (Shahgir et al., 2024)
• SneakyPrompt: Jailbreaking Text-to-image Generative Models (Yang et al., 2023)
• Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery (Wen et al., 2023)
• Evaluating the Robustness of Text-to-image Diffusion Models against Real-world Attacks (Gao et al, 2023)
• Black Box Adversarial Prompting for Foundation Models (Maus and Chao et al., 2023)
• Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models? (Tsai et al., 2024)
• Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts (Chin et al., 2024)
• To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now (Zhang et al., 2024)
• MMA-Diffusion: MultiModal Attack on Diffusion Models (Yang et al., 2023)

Section 5: Additional Attacks

• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (Greshake et al., 2023)
• PoisonedRAG: Knowledge Poisoning Attacks to Retrieval-Augmented Generation of Large Language Models (Zou et al., 2024)
• Phantom: General Trigger Attacks on Retrieval Augmented Language Generation (Chaudhari et al., 2024)
• Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems (Qi et al., 2024)
• SEEING IS BELIEVING: BLACK-BOX MEMBERSHIP INFERENCE ATTACKS AGAINST RETRIEVAL AUGMENTED GENERATION (Li et al., 2024)
• From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application? (Pedro et al., 2023)
• RatGPT: Turning online LLMs into Proxies for Malware Attacks (Beckerich et al., 2023)
• The Intelligent Agent NLP-based Customer Service System (Changran, 2021)
• FedMLSecurity: A Benchmark for Attacks and Defenses in Federated Learning and LLMs (Han et al., 2024)
• Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (Fang et al., 2020)
• A robust analysis of adversarial attacks on federated learning environments (Nair et al., 2023)
• A Systematic Review of Federated Generative Models (Vedadi Gargary et al., 2024)

Section 6: Causes

• Jailbroken: How Does LLM Safety Training Fail? (Wei et al., 2023)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (Qi et al., 2023)
• LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked (Helbling et al., 2023)
• LLM Censorship: A Machine Learning Challenge or a Computer Security Problem? (Glukhov et al., 2023)
• Automatically Auditing Large Language Models via Discrete Optimization (Jones et al., 2023)
• Pretraining language models with human preferences (Korbak et al., 2023)
• Baseline Defenses For Adversarial Attacks Against Aligned Language Models (Jain et al., 2023)
• Certifying LLM Safety against Adversarial Prompting (Kumar et al., 2023)
• Interpretability and Transparency-Driven Detection and Transformation of Textual Adversarial Examples (IT-DT) (Sabir et al., 2023)
• Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attacks (Zhang et al., 2023)
• Towards building a robust toxicity predictor (Bespalov et al., 2023)
• Exploring the Universal Vulnerability of Prompt-based Learning Paradigm (Xu et al., 2022)
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (Ganguli et al., 2022)
• Red Teaming Language Models with Language Models (Perez et al., 2022)

Section 7: Defenses

• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (Bai et al., 2022)
• Direct preference optimization (DPO): Your language model is secretly a reward model (Rafailov et al., 2023)
• Large Language Model Unlearning (Yao et al., 2023)
• Baseline Defenses for Adversarial Attacks Against Aligned Language Models (Jain et al., 2023)
• Certifying LLM Safety against Adversarial Prompting (Kumar et al., 2023)
• SMOOTHLLM: Defending Large Language Models Against Jailbreaking Attacks (Robey et al., 2023)