In the rapidly evolving field of artificial intelligence, large language models (LLMs) are increasingly being used to evaluate and filter their own outputs, serving as “judges” to ensure safety and prevent harmful content from reaching users. But what happens when those same judges can be tricked by something as simple and playful as an emoji?

ICSI researchers Zhipeng Wei, Yuqi Liu, and N. Benjamin Erichson recently published a new study, Emoji Attack: A Method for Misleading Judge LLMs in Safety Risk Detection, that identifies a vulnerability in AI safety systems. Their work demonstrates that inserting emojis into text can exploit a subtle bias in Judge LLMs, causing harmful content to slip past safety filters.

Discovering a Vulnerability

Have you ever noticed how spammers can sneak past filters by replacing letters with symbols—like using ‘fr33’ instead of ‘free’? The research team discovered that a similar trick can be used to fool AI systems designed to screen for harmful content.

These screening systems, known as Judge LLMs, are trained to evaluate whether AI-generated responses can be considered safe or unsafe. But the researchers discovered a flaw: when emojis are inserted into words—especially positive or misleading emojis—they distort not only how the word is broken up into sub-parts but also how its meaning is interpreted. This is because modern AI models process words by breaking them into smaller pieces (sub-tokens), and slight changes to these pieces that are split can confuse the model’s internal understanding of what the word means, a phenomenon called token segmentation bias. Some emojis introduced additional noise, further confusing the AI into thinking a harmful message is neutral or even positive. To a human, the sentence might still clearly be offensive or dangerous, but the LLM sees a warped version that passes its safety filters.

Demonstrating the Emoji Attack

To test whether this poses a real problem, the team developed the Emoji Attack, a technique that inserts emojis into words to distort how the Judge LLM sees those words internally, tricking it into thinking the content is harmless.

By combining this emoji trick with prompts specifically designed to bypass restrictions (called “jailbreak prompts”), they were able to get generative AI models to produce clearly harmful content that the Judge LLMs failed to flag. Even though the response would be obvious to a human reviewer, the emojis caused just enough confusion in the AI’s interpretation for it to mistakenly approve the content.

In experiments across six leading Judge LLMs, the Emoji Attack:

  • Reduced “unsafe” detection rates by up to 75% for ShieldLM.
  • Achieved an average 15.8% reduction across practical jailbreak scenarios.
  • Demonstrated that mixing characters and emojis to carry out the attack makes filtering strategies nearly ineffective.

The figure below shows that emoji-based attacks consistently trick Judge LLMs into misclassifying harmful content as safe—even more effectively than other common jailbreak techniques.

Why This Matters

The implications of this research are profound. As AI systems become gatekeepers for their own content, subtle vulnerabilities like token segmentation bias raise urgent questions:

  • Can we trust Judge LLMs to recognize harmful content if minor tweaks fool them?
  • How should safety researchers evaluate robustness in an era where adversarial creativity is as simple as an emoji?

The team hopes this work will guide the next generation of robust Judge LLMs, capable of recognizing semantic meaning even in the face of obfuscation tricks.

Team