# Source: https://docs.openpipe.ai/features/criteria/alignment-set.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openpipe.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Criterion Alignment Sets

> Use alignment sets to test and improve your criteria.

Alignment sets are a collection of LLM input/output pairs that are judged by both the criterion LLM judge and a human.
The performance of the criterion LLM judge is then measured by how well it matches the judgements of the human judge. We recommend importing and judging at least 30 rows to ensure the alignment stats are meaningful.

## Importing an Alignment Set

You can import an alignment set from either an OpenPipe dataset or a JSONL file. Alignment sets can be added to an existing criterion or imported when a new criterion is created.

<Frame><img src="https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-alignment-set.png?fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=53d622a1ff7fe1bf5debc305bd54af83" alt="" data-og-width="1184" width="1184" data-og-height="1152" height="1152" data-path="images/features/criteria/alignment-set/import-alignment-set.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-alignment-set.png?w=280&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=591c2d778e48a8245fccf2ac5ea2fd7f 280w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-alignment-set.png?w=560&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=729ec03d466075f707911915c9f0f52b 560w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-alignment-set.png?w=840&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=c5f3813b3052ba6947af599994480298 840w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-alignment-set.png?w=1100&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=5f48dd323673ae5f0ae4b130cd30fa85 1100w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-alignment-set.png?w=1650&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=cead9c14a698385bbc55364a02136b7b 1650w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-alignment-set.png?w=2500&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=848803b5b7c50dad0ebbe92af5a7345a 2500w" /></Frame>

### Importing from a Dataset

When importing from a dataset, you select a number of rows to be randomly sampled from the dataset of your choice to imported into the criterion alignment set. The inputs of each of these rows will be copied directly from the rows in the dataset without any changes. By default, the outputs will also be copied from the original dataset. However, if you set **Output Source** to be an LLM model, the outputs will be generated by the LLM model based on the dataset inputs.

<Frame><img src="https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-dataset.png?fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=252490124c262a3e2a39963cb811a707" alt="" data-og-width="1082" width="1082" data-og-height="564" height="564" data-path="images/features/criteria/alignment-set/import-from-dataset.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-dataset.png?w=280&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=fdaf15e0b43bdad3d8a4057972e01802 280w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-dataset.png?w=560&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=e1ea6770f5ca5fa20eaa8a21c2604ef4 560w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-dataset.png?w=840&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=17c942f5d53d2401c49d6f7c9e03278e 840w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-dataset.png?w=1100&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=5a6f39ae628a410d332c74de773efb09 1100w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-dataset.png?w=1650&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=0fe8f704f11f56016aee7f79b79b1ecb 1650w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-dataset.png?w=2500&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=02ab05f97c508f1bd7072d3031c2d007 2500w" /></Frame>

### Importing from a JSONL File

You can also import an alignment set from a JSONL file. Uploads are limited to 10MB in size,
which should be plenty for an alignment set.

<Frame><img src="https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-upload.png?fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=b1e17e9165d32117bac5dc739a5bd34a" alt="" data-og-width="1100" width="1100" data-og-height="872" height="872" data-path="images/features/criteria/alignment-set/import-from-upload.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-upload.png?w=280&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=16bd18df908274b3718793a0d073161e 280w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-upload.png?w=560&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=b1e74b750ebcdf0861a418b073c16cc4 560w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-upload.png?w=840&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=18b0d458da713ebc087389a2f45f1517 840w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-upload.png?w=1100&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=9beb047413b2df2dcb40050c5809c15e 1100w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-upload.png?w=1650&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=004c01e53f7cacdf848501002dd70f79 1650w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/import-from-upload.png?w=2500&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=5814822cbbe4e671c5cbed8e71882b9b 2500w" /></Frame>

The schema of the JSONL file is exactly the same as an OpenAI-compatible [JSONL fine-tuning file](/features/datasets/uploading-data#openai-fields), but also supports an optional `judgement` field for each row. `judgement` can be either `PASS` or `FAIL`, depending on whether the row should pass or fail the criterion.

#### Example

```jsonl  theme={null}
...
{"judgement": "PASS", "messages":[{"role":"system","content":"You are a helpful assistant"},{"role":"user","content":"What is the capital of Tasmania?"},{"role":"assistant","content":null,"tool_calls":[{"id":"","type":"function","function":{"name":"identify_capital","arguments":"{\"capital\":\"Hobart\"}"}}]}],"tools":[{"type":"function","function":{"name":"identify_capital","parameters":{"type":"object","properties":{"capital":{"type":"string"}}}}}]}
{"judgement": "FAIL", "messages":[{"role":"system","content":"You are a helpful assistant"},{"role":"user","content":"What is the capital of Sweden?"},{"role":"assistant","content":null,"tool_calls":[{"id":"","type":"function","function":{"name":"identify_capital","arguments":"{\"capital\":\"Beijing\"}"}}]}],"tools":[{"type":"function","function":{"name":"identify_capital","parameters":{"type":"object","properties":{"capital":{"type":"string"}}}}}]}
{"messages":[{"role":"system","content":"You are a helpful assistant"},{"role":"user","content":"What is the capital of Sweden?"},{"role":"assistant","content":null,"tool_calls":[{"id":"","type":"function","function":{"name":"identify_capital","arguments":"{\"capital\":\"Stockholm\"}"}}]}],"tools":[{"type":"function","function":{"name":"identify_capital","parameters":{"type":"object","properties":{"capital":{"type":"string"}}}}}]}
...
```

## Alignment Stats

Alignment stats are a simple way to understand how well your criterion is performing.
As you refine your criterion prompt, you're alignment stats will improve as well.

<Frame><img src="https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/alignment-stats.png?fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=e7d93f7d33061789ecd3989744c89c2d" alt="" data-og-width="742" width="742" data-og-height="224" height="224" data-path="images/features/criteria/alignment-set/alignment-stats.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/alignment-stats.png?w=280&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=3d270a2a18daf35e6686dbc39d84fb78 280w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/alignment-stats.png?w=560&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=21970348c0dbcdeda3f2d25caae91987 560w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/alignment-stats.png?w=840&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=cff6302a2a816b70eaff194b25239dba 840w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/alignment-stats.png?w=1100&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=92010468fbc44d88d6c6c41d521d0a8e 1100w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/alignment-stats.png?w=1650&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=3b2d4a4fae0361e864a968de9cfa26bf 1650w, https://mintcdn.com/openpipe/yLyh_RHELnvU-7tP/images/features/criteria/alignment-set/alignment-stats.png?w=2500&fit=max&auto=format&n=yLyh_RHELnvU-7tP&q=85&s=043f9cedb0f1814914a0004a1220bc2e 2500w" /></Frame>

* **Precision** indicates the fraction of rows that the LLM judge labeled as failing that a human judge also labeled as failing. It's an indicator of how reliable the LLM judge's FAIL label is.
* **Recall** indicates the fraction of rows that a human judge labeled as failing that the LLM judge also labeled as failing. It's an indicator of how reliable the LLM judge's PASS label is.
* **F1 Score** is the harmonic mean of precision and recall. As either score improves, the F1 score will also improve.

To ensure your alignment stats are meaningful, we recommend labeling at least 30 rows,
but in some cases you may need to label more in order to get a reliable statistic.