# run the grader with a test reference and sample
payload = {
  "grader": grader,
  "item": {
     "reference_answer": "fuzzy wuzzy had no hair"
  },
  "model_sample": "fuzzy wuzzy was a bear"
}
response = requests.post(
    "https://api.openai.com/v1/fine_tuning/alpha/graders/run",
    json=payload,
    headers=headers
)
print("run request_id:", response.headers["x-request-id"])
print("run response:", response.text)
```

**Tip:**  
If you don't want to manually put your grading function in a string, you can
also load it from a Python file using `importlib` and `inspect`. For example, if
your grader function is in a file named `grader.py`, you can do:

```python
import importlib
import inspect

grader_module = importlib.import_module("grader")
grader = {
    "type": "python",
    "source": inspect.getsource(grader_module)
}
```

This will automatically use the entire source code of your `grader.py` file as
the grader which can be helpful for longer graders.

### Technical constraints

- Your uploaded code must be less than `256kB` and will not have network access.
- The grading execution itself is limited to 2 minutes.
- At runtime you will be given a limit of 2Gb of memory and 1Gb of disk space to
  use.
- There's a limit of 2 CPU coresâany usage above this amount will result in
  throttling

The following third-party packages are available at execution time for the image
tag `2025-05-08`

```text
numpy==2.2.4
scipy==1.15.2
sympy==1.13.3
pandas==2.2.3
rapidfuzz==3.10.1
scikit-learn==1.6.1
rouge-score==0.1.2
deepdiff==8.4.2
jsonschema==4.23.0
pydantic==2.10.6
pyyaml==6.0.2
nltk==3.9.1
sqlparse==0.5.3
rdkit==2024.9.6
scikit-bio==0.6.3
ast-grep-py==0.36.2
```

Additionally the following nltk corpora are available:

```text
punkt
stopwords
wordnet
omw-1.4
names
```

## Multigraders

> Currently, this grader is only used for Reinforcement fine-tuning

A `multigrader` object combines the output of multiple graders to produce a
single score. Multigraders work by computing grades over the fields of other
grader objects and turning those sub-grades into an overall grade. This is
useful when a correct answer depends on multiple things being trueâfor example,
that the text is similar _and_ that the answer contains a specific string.

As an example, say you wanted the model to output JSON with the following two
fields:

```json
{
  "name": "John Doe",
  "email": "john.doe@gmail.com"
}
```

You'd want your grader to compare the two fields and then take the average
between them.

You can do this by combining multiple graders into an object grader, and then
defining a formula to calculate the output score based on each field:

```json
{
  "type": "multi",
  "graders": {
    "name": {
      "name": "name_grader",
      "type": "text_similarity",
      "input": "{{sample.output_json.name}}",
      "reference": "{{item.name}}",
      "evaluation_metric": "fuzzy_match",
      "pass_threshold": 0.9
    },
    "email": {
      "name": "email_grader",
      "type": "string_check",
      "input": "{{sample.output_json.email}}",
      "reference": "{{item.email}}",
      "operation": "eq"
    }
  },
  "calculate_output": "(name + email) / 2"
}
```

In this example, itâs important for the model to get the email exactly right
(`string_check` returns either 0 or 1) but we tolerate some misspellings on the
name (`text_similarity` returns range from 0 to 1). Samples that get the email
wrong will score between 0-0.5, and samples that get the email right will score
between 0.5-1.0.

You cannot create a multigrader with a nested multigrader inside.

The calculate output field will have the keys of the input `graders` as possible
variables and the following features are supported:

**Operators**

- `+` (addition)
- `-` (subtraction)
- `*` (multiplication)
- `/` (division)
- `^` (power)

**Functions**

- `min`
- `max`
- `abs`
- `floor`
- `ceil`
- `exp`
- `sqrt`
- `log`

## Limitations and tips

Designing and creating graders is an iterative process. Start small, experiment,
and continue to make changes to get better results.

### Design tips

To get the most value from your graders, use these design principles:

- **Produce a smooth score, not a pass/fail stamp**. A score that shifts
  gradually as answers improve helps the optimizer see which changes matter.
- **Guard against reward hacking**. This happens when the model finds a shortcut
  that earns high scores without real skill. Make it hard to loophole your
  grading system.
- **Avoid skewed data**. Datasets in which one label shows up most of the time
  invite the model to guess that label. Balance the set or upâweight rare cases
  so the model must think.
- **Use an LLMâasâa-judge when code falls short**. For rich, openâended answers,
  ask another language model to grade. When building LLM graders, run multiple
  candidate responses and ground truths through your LLM judge to ensure grading
  is stable and aligned with preference. Provide few-shot examples of great,
  fair, and poor answers in the prompt.