Add exclude_think param to MultiTurnEnv

[kyleavery]

Description

This doesn't change existing behavior, it just adds a new parameter to MultiTurnEnv called exclude_think. This is meant for training reasoning models on tasks with many turns. Inspired by Kevin: Multi-Turn RL for Generating CUDA Kernels.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Test improvement

Testing

• All existing tests pass when running uv run pytest locally.
• New tests have been added to cover the changes

Checklist

• My code follows the style guidelines of this project as outlined in AGENTS.md
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

Additional Notes

Example - Default behavior (second turn)

Prompt:

[
  {
    "content": "Answer questions about the following paragraph:\n\n\"Perhaps the first public statement on the matter to be both serious and strong came from Catherine II (Catherine the Great), whose liberal views were consistent with her acceptance of the Enlightenment. In her Nakaz of 1767, the empress expressed disdain for the death penalty, considering it to be improper, adding: \"In the usual state of the society, the death penalty is neither useful nor needed\". However, an explicit exception was still allowed for the case of someone who, even while convicted and incarcerated, \"still has the means and the might to ignite public unrest\". This specific exception applied to mutineers of Pugachev's Rebellion in 1775. Consistent with Catherine's stance, the next several decades marked a shift in public perception against the death penalty\"\n\nWhat's the fifth sentence of the paragraph?",
    "role": "user"
  },
  {
    "role": "assistant",
    "content": "<think>\nOkay, let's see. The user is asking for the fifth sentence of the given paragraph. First, I need to break down the paragraph into individual sentences to identify the fifth one.\n\nThe paragraph starts with \"Perhaps the first public statement...\". Then, \"In her Nakaz of 1767...\" that's the second. The third sentence is \"However, an explicit exception was still allowed...\". The fourth sentence is \"This specific exception applied to mutineers of Pugachev's Rebellion in 1775.\" Then the fifth sentence would be \"Consistent with Catherine's stance, the next several decades marked a shift in public perception against the death penalty.\" \n\nWait, let me check again. Let me count each sentence:\n\n1. Perhaps the first public statement on the matter to be both serious and strong came from Catherine II...\n2. In her Nakaz of 1767, the empress expressed disdain for the death penalty...\n3. However, an explicit exception was still allowed for the case of someone who, even while convicted and incarcerated...\n4. This specific exception applied to mutineers of Pugachev's Rebellion in 1775.\n5. Consistent with Catherine's stance, the next several decades marked a shift in public perception against the death penalty.\n\nYes, that seems correct. The user probably wants the fifth sentence as the answer. I should make sure there's no confusion with the structure. The paragraph is structured into sentences separated by periods. So the fifth sentence is indeed the last one listed here.\n</think>\n\nThe fifth sentence of the paragraph is:  \n**\"Consistent with Catherine's stance, the next several decades marked a shift in public perception against the death penalty.\"**"
  },
  {
    "role": "user",
    "content": "OK how about the first sentence?"
  }
]

Response:

{
  "role": "assistant",
  "tool_calls": [],
  "content": "<think>\nAlright, the user just asked about the first sentence of the paragraph. Let me recall the original paragraph.\n\nThe paragraph starts with: \"Perhaps the first public statement on the matter to be both serious and strong came from Catherine II (Catherine the Great), whose liberal views were consistent with her acceptance of the Enlightenment.\"\n\nSo the first sentence is that one. The user might be checking if I can identify the first sentence correctly. They might be testing my ability to parse the paragraph structure. Maybe they're studying for a test or just curious about the structure of the text.\n\nI should confirm that the first sentence is indeed the one I mentioned. Let me count the sentences again. The first sentence is the opening one, then the next sentences follow. The user might have asked this after previously asking about the fifth sentence, so they're probably looking for a pattern or to verify each sentence's position. \n\nI need to make sure I'm not making a mistake here. The first sentence is definitely the one that introduces Catherine's statement. No other sentence comes before that. The user might be verifying my understanding of sentence numbering in the paragraph. So the answer is straightforward, but I should present it clearly.\n</think>\n\nThe first sentence of the paragraph is:  \n**\"Perhaps the first public statement on the matter to be both serious and strong came from Catherine II (Catherine the Great), whose liberal views were consistent with her acceptance of the Enlightenment.\"**"
}

Example - Behavior when exclude_think is True (second turn)

Prompt:

[
  {
    "content": "Answer questions about the following paragraph:\n\n\"Perhaps the first public statement on the matter to be both serious and strong came from Catherine II (Catherine the Great), whose liberal views were consistent with her acceptance of the Enlightenment. In her Nakaz of 1767, the empress expressed disdain for the death penalty, considering it to be improper, adding: \"In the usual state of the society, the death penalty is neither useful nor needed\". However, an explicit exception was still allowed for the case of someone who, even while convicted and incarcerated, \"still has the means and the might to ignite public unrest\". This specific exception applied to mutineers of Pugachev's Rebellion in 1775. Consistent with Catherine's stance, the next several decades marked a shift in public perception against the death penalty\"\n\nWhat's the fifth sentence of the paragraph?",
    "role": "user"
  },
  {
    "role": "assistant",
    "content": "The fifth sentence of the paragraph is:  \n**\"Consistent with Catherine's stance, the next several decades marked a shift in public perception against the death penalty.\"**"
  },
  {
    "role": "user",
    "content": "OK how about the first sentence?"
  }
]

Response:

{
  "role": "assistant",
  "tool_calls": [],
  "content": "<think>\nAlright, the user just asked about the first sentence of the paragraph. Let me recall the original paragraph.\n\nThe paragraph starts with: \"Perhaps the first public statement on the matter to be both serious and strong came from Catherine II (Catherine the Great), whose liberal views were consistent with her acceptance of the Enlightenment.\"\n\nSo the first sentence is that one. The user might be checking if I can identify the first sentence correctly. They might be testing my ability to parse the paragraph structure. Maybe they're studying for a test or just curious about the structure of the text.\n\nI should confirm that the first sentence is indeed the one I mentioned. Let me count the sentences again. The first sentence is the opening one, then the next sentences follow. The user might have asked this after previously asking about the fifth sentence, so they're probably looking for a pattern or to verify each sentence's position. \n\nI need to make sure I'm not making a mistake here. The first sentence is definitely the one that introduces Catherine's statement. No other sentence comes before that. The user might be verifying my understanding of sentence numbering in the paragraph. So the answer is straightforward, but I should present it clearly.\n</think>\n\nThe first sentence of the paragraph is:  \n**\"Perhaps the first public statement on the matter to be both serious and strong came from Catherine II (Catherine the Great), whose liberal views were consistent with her acceptance of the Enlightenment.\"**"
}

[kyleavery]

i'm not sure if there would be other metadata in the message dict, but i updated the code to deepcopy just in case.

[kyleavery]

Included in #536