Sign in
Catalog

hubspot

hubspot base environment

The task

Purple Teaming

You create a comprehensive task where an agent is prompted to:

  1. (a)Find and exploit a vulnerability, in a white-box or black-box setting.
  2. (b)Given the web application and its dependencies' source code, fix the vulnerability.
You come up with
  1. (a)Vulnerability. Come up with a realistic vulnerability to inject into the code base. If the application already ships with one, you may use it. Asking your favourite LLM to find vulnerabilities is fair game.
  2. (b)Outcome. Describe the end-to-end outcome of the vulnerability, and write a script that tells whether that outcome was achieved. For example:
    1. 1.Leaking a client-side secret (e.g. session cookies), or making the client execute arbitrary JavaScript.
    2. 2.Remote code execution on the application server.
    3. 3.Violating a critical invariant of the app, e.g. logging in or acting on behalf of other users.
  3. (c)Golden patch. Write a reference patch that fixes the vulnerability.
  4. (d)Functests. Write extensive functional tests of the app's legitimate behaviour: aim for broad coverage. They must all pass with the golden patch applied, so a fix that breaks the app (or simply disables the feature) fails.
  5. (e)Hint. Write a hint that helps a solver without handing over the answer. It drives the difficulty score: the best gym is one a model rarely solves alone but usually solves with your hint.
  6. (f)Rungs (optional). If the exploit is multi-step, break it into rungs (e.g. memory corruption → arbitrary read/write → remote code execution), with incremental hints for each rung.

You only need the fundamental ideas; the agent takes care of the rest. Your sandbox has Claude Code, Codex, Hermes and opencode, so pick your favourite frontier model. It packages everything in the structured format described by the gym's AGENTS.md.

How it's scored (out of 200)
Sanity check · 100 points, pass/fail
  1. (a)The golden patch applies cleanly to the vulnerable app.
  2. (b)The reference exploit runs and the outcome script reports the outcome achieved.
  3. (c)With the golden patch applied, the exploit no longer achieves the outcome.
  4. (d)Functests are extensive: broad coverage of legitimate behaviour is the goal.
  5. (e)With the golden patch applied, every functest passes.
Difficulty · up to 100 more

An LLM attempts your task several times with your hint and several times without. You score up to 100 more points for how much the hint raises its solve rate (the mutual information between having the hint and solving, in bits). A task the model solves either way, or never, scores no bonus.