# run the grader with a test reference and sample
payload = {
  "grader": grader,
  "item": {},
  "model_sample": "0.9"
}
response = requests.post(
    "https://api.openai.com/v1/fine_tuning/alpha/graders/run",
    json=payload,
    headers=headers
)
print("run response:", response.text)
```

### Model grader constraints

- Only the following models are supported for the `model` parameter\`
  - `gpt-4o-2024-08-06`
  - `gpt-4o-mini-2024-07-18`
  - `gpt-4.1-2025-04-14`
  - `gpt-4.1-mini-2025-04-14`
  - `gpt-4.1-nano-2025-04-14`
  - `o1-2024-12-17`
  - `o3-mini-2025-01-31`
  - `o3-2025-04-16`
  - `o4-mini-2025-04-16`
- `temperature` changes not supported for reasoning models.
- `reasoning_effort` is not supported for non-reasoning models.

### How to write grader prompts

Writing grader prompts is an iterative process. The best way to iterate on a
model grader prompt is to create a model grader eval. To do this, you need:

1.  **Task prompts**: Write extremely detailed prompts for the desired task,
    with step-by-step instructions and many specific examples in context.
2.  **Answers generated by a model or human expert**: Provide many high quality
    examples of answers, both from the model and trusted human experts.
3.  **Corresponding ground truth grades for those answers**: Establish what a
    good grade looks like. For example, your human expert grades should be 1.

Then you can automatically evaluate how effectively the model grader
distinguishes answers of different quality levels. Over time, add edge cases
into your model grader eval as you discover and patch them with changes to the
prompt.

For example, say you know from your human experts which answers are best:

```text
answer_1 > answer_2 > answer_3
```

Verify that the model grader's answers match that:

```text
model_grader(answer_1, reference_answer) > model_grader(answer_2, reference_answer) > model_grader(answer_3, reference_answer)
```

### Grader hacking

Models being trained sometimes learn to exploit weaknesses in model graders,
also known as âgrader hackingâ or âreward hacking." You can detect this by
checking the model's performance across model grader evals and expert human
evals. A model that's hacked the grader will score highly on model grader evals
but score poorly on expert human evaluations. Over time, we intend to improve
observability in the API to make it easier to detect this during training.

## Python graders

This grader allows you to execute arbitrary python code to grade the model
output. The grader expects a grade function to be present that takes in two
arguments and outputs a float value. Any other result (exception, invalid float
value, etc.) will be marked as invalid and return a 0 grade.

```json
{
  "type": "python",
  "source": "def grade(sample, item):\n    return 1.0",
  "image_tag": "2025-05-08"
}
```

The python source code must contain a grade function that takes in exactly two
arguments and returns a float value as a grade.

```python
from typing import Any

def grade(sample: dict[str, Any], item: dict[str, Any]) -> float:
    # your logic here
    return 1.0
```

The first argument supplied to the grading function will be a dictionary
populated with the modelâs output during training for you to grade.
`output_json` will only be populated if the output uses `response_format`.

```json
{
    "choices": [...],
    "output_text": "...",
    "output_json": {},
    "output_tools": [...]
}
```

The second argument supplied is a dictionary populated with input grading
context. For evals, this will include keys from the data source. For fine-tuning
this will include keys from each training data row.

```json
{
    "reference_answer": "...",
    "my_key": {...}
}
```

Here's a working example:

```python
import os
import requests