Cracking the Code: My Deep Dive into Red Teaming LLMs

Red Teaming: The Art of Simulated Adversarial Testing

At its core, Red Teaming is the practice of rigorously testing security
systems, policies, and even people. It involves simulating the tactics,
techniques, and procedures (TTPs) used by real-world adversaries to
uncover vulnerabilities. In the exciting and rapidly evolving world of
Large Language Models (LLMs), red teaming applies these same principles
to scrutinize and strengthen LLM safety and security safeguards.

LLMs are increasingly equipped with sophisticated defenses like prompt
filtering, data sanitization, content moderation, and access controls,
all designed to prevent harmful or unethical outputs. Complementing
these technical measures are crucial policies encompassing governance
frameworks, user education, data minimization, and regulatory
compliance, which guide responsible development and deployment.

My Journey into the LLM Red Teaming Frontier

Recently, I had the opportunity to get a hands-on look at red teaming
and the intricate behaviors of LLMs when I participated in OpenAI's
Red-Teaming Challenge
on Kaggle. This hackathon, sponsored by
OpenAI, tasked participants with probing their newly
released gpt-oss-20b open-weight model. The mission? To uncover any
undetected vulnerabilities and harmful behaviors, ranging from deceptive
alignment and reward-hacking to sabotage and evaluation awareness.

As a complete newcomer to both red teaming in general and LLM-specific
red teaming, I faced a significant learning curve. With the hackathon
running for just one month from start to finish, there was limited time
to fully grasp the nuances before diving headfirst into the challenge.
The strategies employed in LLM red teaming are remarkably diverse and
far-reaching. I began by conducting a quick survey of existing
literature and public efforts to understand the landscape.

Learning from the Pioneers: Early Explorations of gpt-oss-20b

It wasn't surprising to discover that others had already begun probing
the gpt-oss-20b model for vulnerabilities immediately after its release.
One particularly insightful and well-documented approach I encountered
was from researchers at Wharton AI Labs, titled "Call Me a Jerk."
Their work highlighted how large language models exhibit "parahuman"
responses to persuasion techniques, even though they are not human.
Another notable example is "Pliny the Liberator," a well-known LLM
jailbreaker, who announced on X (formerly Twitter) his success in
cracking GPT-OSS. His post, "OPENAI: PWNED 🤗 GPT-OSS: LIBERATED,"
included screenshots showcasing the model generating instructions for
creating methamphetamine, Molotov cocktails, VX nerve agent, and
malware.

A Tale of Two Strategies: Our Approach to the Challenge

Teaming up with my friend Martin Puskas, we set out to develop our own
offensive strategies against gpt_oss-20b. Martin proposed a truly clever
and original approach. He hypothesized that as powerful language models
become integrated into more accessible and free applications running on
provider servers, teenagers will emerge as one of the largest and most
technically sophisticated user demographics. He envisioned a future
where teenagers, having grown up with this technology, might struggle to
imagine a world without LLMs -- much like previous generations
experienced with television, the internet, and mobile phone technology.

Martin's innovative agent framework maps teenage behavioral patterns
onto the "vocabulary of a teenage enthusiast" -- the linguistic and
social boundaries that define teenage digital interaction. This allowed
us to demonstrate systematic exploitation patterns that could
organically emerge in real-world deployment scenarios. Through a process
of systematic, gradual escalation, starting with seemingly innocent
fireworks chemistry questions, Martin's child persona successfully
extracted comprehensive explosive information from gpt-oss-20b,
including precise black powder ratios (75% potassium nitrate, 15%
charcoal, 10% sulfur), detonation mechanics, and high explosive
specifications.

My own approach, while more conventional, also yielded eye-opening
results. I uncovered instances of Evaluation Awareness and Deception, along
with dangerous recommendations for the use of shell scripting commands.
It was both fascinating and concerning to observe the LLM initially
resist these behaviors, only to eventually succumb to the attacks and
respond in ways that were inconsistent with its internal policies and
safeguards.

The Path Forward: Layered Defense for a Safer Future

This experience underscored a critical truth: LLM safety ultimately
requires a layered, holistic approach. By uniting defense-in-depth
strategies with ongoing, vigilant oversight, we can ensure that LLMs
remain trustworthy and safe for all users, especially the next
generation who will shape and be shaped by this transformative
technology.