2nd place
$150
AI Safety Challenge
Build safeguard that prevents AI agents from taking unauthorized, irreversible actions in the real world setup.
$500
Challenge prize pool
Share your project:
Submit link to a GitHub repo or an article with a solution
Across 17 frontier LLM agents, agents continued acting in 40.5%* of cases, even when harm was possible.
* Do LLM agents know when not to act? ↗Tool-using AI agents often fail to recognize when they should stop. Under ambiguity, conflicting constraints, or tool failures, they may take irreversible actions before identifying the risk, and a later refusal cannot undo the harm.
Oct 15 26’
Challenge start
Submit (28 days left)
Nov 15 26’
Winners
Meet your jury
Jury of October
Solutions will be judged on how effectively they prevent unauthorized actions and how easily they can be reproduced in similar setups.
Cash prizes
Winners
1st place
$300
3rd place
$50
Winning projects will be featured on Drawerrr’s Winners page.
Winners will also receive prizes and recognition.
Top solutions
Shortlists
Best overall in safeguarding irreversible actions
The strongest overall example of safe, responsible agentic action.
Project name
Author’s names
1st
- Project nameAuthor’s names
2nd
- Project nameAuthor’s names
3rd
- Project nameAuthor’s names
Finalist
- Project nameAuthor’s names
Finalist
- Project nameAuthor’s names
Finalist
- Project nameAuthor’s names
Finalist
- Project nameAuthor’s names
Finalist
- Project nameAuthor’s names
Finalist
- Project nameAuthor’s names
Finalist
Best in safeguarding delegated actions
Best in safely enabling AI agents to communicate, transact, or act for people and organizations.
Best in safeguarding physical-world actions
Best in safely controlling AI agents that operate drones, robots, equipment, or industrial systems.
Best in safeguarding consequential decisions
Best in keeping AI agents from making consequential decisions without appropriate human or policy controls.
Best in safeguarding information access
Best in controlling what AI agents can access, retrieve, expose, or transmit.
Best in safeguarding digital system changes
Best in safely controlling AI agents that modify code, data, infrastructure, or system state.
Best in safeguarding financial transactions
Best in safely enabling AI agents to trade, spend, transfer, or authorize funds.
The Top 10 finalists will be recognised across additional categories.
These distinctions will also be featured on Drawerrr’s Winners page.
Oct 15 26’
Challenge start
Submit (28 days left)
Nov 15 26’
Winners
How to succeeded
Master the Challenge in 5 days
Day 1
Identify high-risk scenarios
Choose an environment you can access or control, and identify a specific action an agent could take that would be unauthorized, harmful, costly, irreversible, or difficult to undo.
Tip:
Start with familiar examples like sending an email, deploying code, or issuing a refund, or use Challenge Kick-off to identify promising approaches.
Learn more:
- Google SAIF — Agent Risk Self-Assessment
- Ask our experts in
Discordchannel
Day 2
Set up a test environment
Recreate the scenario in an environment you control. Give the agent only the access it needs to demonstrate the risk.
Tip:
Use a mock API or sandbox service to simulate real actions safely, or use Challenge Kick-off to find an environment that fits your project.
Learn more:
- Google SAIF — Agent Permissions
- Ask our experts inDiscordchannel
Day 3
Choose a safeguard stack
Combine one or multiple layers to create a safeguard stack. A safety layer could address identity and access, data leakage, action authorization, human approval, monitoring, or fail-safe controls.
Tip:
Explore tools like Guardrails AI and LangGraph, or use Challenge Kick-off to identify safeguards that fit your approach.
Learn more:
- Google SAIF — Security Controls
- Ask our experts inDiscordchannel
Day 4
Document failure states
Test ambiguous instructions, missing information, prompt injection, unauthorized requests, unusual edge cases, and attempts to bypass the approval mechanism and log results.
Tip:
Test expired approvals, malformed tool arguments, prompt injection, and timeouts, or use Challenge Kick-off to identify failure modes and plan how to trace them.
Learn more:
- Google SAIF — Red Teaming and Adversarial Testing
- Ask our experts inDiscordchannel
Day 5
Make it reproducible
Package your solution so someone else can reproduce, test, and adapt it. Share the code, configuration, test setup, and instructions needed to run it in a similar environment.
Tip:
Include a README with setup instructions and example usage to help others run your project, or use Challenge Kick-off to find ways to make your solution easier to reproduce and reuse.
Support:
- Ask our experts inDiscordchannel
FAQ
Learn more about the challenge
Who is this for?
Anyone who knows the risk, can build or test a safeguard, or wants to learn. You don’t need to do every part yourself.
What does my entry need?
Submit a use case of a working safeguard in a real system, or a reusable tool someone else can apply.
Can I work on my own?
Yes. If you’re missing a skill, find a teammate here or invite someone you know.
When is it due?
Submit by Nov 15, 2026. Winners are announced Nov 15, 2026.
How is it judged?
We look at the real-world use case, the strength of the safeguard, the human handoff, and how easily another team can reuse the approach.
Do I need a demo?
A demo can help explain your work, but include evidence that it works. If you submit a reusable tool, make it easy to test.
What AI models or agents can I test?
You can test any publicly available proprietary model or open-source model or agents.
Do I need to modify the AI model or agent itself?
No. You can build the safeguard around the model or agent you are testing.
Do I need to use a real environment?
No. Use test accounts, fake data, sandboxes, or other controlled environments.
What is a safeguard (test harness)?
A test harness is the setup used to test the agent. It can provide the scenario, test environment, tools, and a way to record and evaluate what the agent does.
Does a prompt or instruction count as a safeguard?
A prompt alone might not be enough. The challenge is about checking or controlling the model or agent's actions rather than relying only on the agent to follow an instruction.
Does my safeguard need to work without a human?
No. A human approval step can be part of the safeguard.
Where should I put the safeguard?
Ideally, between the AI model or agent and the action it can take: AI agent → Safety layer → Environment. The safeguard should be able to allow, block, or pause the action before it happens.
Can I use an existing tool, plugin, MCP, or security product as part of my safeguard?
Yes. You can use existing tools or combine them with your own work. Show how they help prevent or control the risky action and how others could reproduce or adapt your approach.
Can I combine several safeguards?
Yes. You can combine different approaches to address the risk.
Do I need to show that the agent fails without my safeguard?
Yes, where possible. Reproduce the risky behavior in your test harness, then show how your safeguard changes the outcome.
Do I need to test whether my safeguard can be bypassed?
Yes. Try different inputs and failure cases to see whether the safeguard still works.
Does my solution need to be open source?
No. You can submit a plugin, MCP connector, app, or an article/case study describing a solution implemented in a closed system.
What if I can't share the implementation?
For closed systems, publish an article or case study showing how the safeguard works so others can recreate the approach.
What can I submit?
You can submit a reusable agent, Skill, Bot, plugin, MCP, custom build, test harness, implementation, or article/case study.
Subscribe to challenge results
Our mission is to accelerate practical solutions to real-world AI safety challenges by connecting people, ideas, and expertise to build AI systems that are safer, more reliable, and easier to govern. Drawerrr’s AI safety challenge is informed by leading institutions in AI safety, standards, and public policy.
