# Define datasets

> Design, version, and validate datasets used for optimization runs.

The optimizer evaluates candidate prompts against datasets stored in Opik. If you are brand new to datasets in Opik, start with [Manage datasets](/evaluation/manage_datasets); this page highlights specific tips to get you started.

Datasets are a crucial component of the optimizer SDK, serving as a key component to run and evaluate (score) each dataset item using optimizers to develop a better outcome. Without datasets, it's not possible to steer the optimizer on what is good and bad.

## Dataset schema

Every item is a JSON object. Required keys depend on your prompt template; optional keys help with analysis. Schemas are optional—define only the fields your prompt or metrics actually consume.

| Field                                  | Purpose                                                    |
| -------------------------------------- | ---------------------------------------------------------- |
| `inputs` (e.g., `question`, `context`) | Values substituted into your `ChatPrompt` placeholders.    |
| `answer` / `label`                     | Ground truth used by metrics.                              |
| `metadata`                             | Arbitrary dict for tagging scenario, split, or difficulty. |

## Create or load datasets

<Steps>
  <Step title="Create via SDK">
    ```python
    import opik

    client = opik.Opik()
    dataset = client.get_or_create_dataset(name="agent-opt-support")
    dataset.insert([
        {"question": "Summarize Opik.", "answer": "Opik is an LLM observability platform."},
        {"question": "List two optimizer types.", "answer": "MetaPrompt and HRPO."},
    ])
    ```
  </Step>

  <Step title="Upload from file">
    * Prepare a CSV or Parquet file with column headers that match your prompt variables.
    * Load the file via Python (e.g., pandas) and call `dataset.insert(...)` or related helpers from the [Dataset SDK](https://www.comet.com/docs/opik/python-sdk-reference/evaluation/Dataset.html).
    * Verify in the UI that rows include `metadata` if you plan to filter by scenario.
  </Step>

  <Step title="Use built-in samples">
    The optimizer SDK provides ready-made datasets for quick experiments:

    ```python
    from opik_optimizer import datasets
    hotpot = datasets.hotpot(count=300)
    tiny = datasets.tiny_test()
    ```

    These datasets live in `sdks/opik_optimizer/src/opik_optimizer/datasets` and mirror the notebook examples. Most helpers accept common slice controls like:

    * `split` (e.g., `"train"`, `"validation"`)
    * `count` and `start` (slice size + offset after shuffling)
    * `seed` (deterministic shuffle)
    * `filter_by` (filter rows before slicing)
      Some helpers also expose `prefer_presets` to override dataset-defined presets.
  </Step>
</Steps>

## Filter dataset rows

Use `filter_by` to select a subset of rows before slicing. Filters support:

* exact match (`{"task_id": "e57337a4"}`)
* membership (`{"type": {"bridge", "comparison"}}`)
* callables (`{"task_id": lambda value: value.startswith("e57")}`)

```python
from opik_optimizer.datasets import arc_agi2, hotpot

dataset = arc_agi2(
    split="train",
    count=1,
    prefer_presets=False,
    filter_by={"task_id": "e57337a4"},
)

dataset = hotpot(
    split="validation",
    count=100,
    filter_by={"type": {"bridge", "comparison"}},
)
```

## Use Hugging Face datasets

If you already have a Hugging Face dataset, you can ingest it into Opik and use it with any optimizer:

```python
from datasets import load_dataset
import opik

hf = load_dataset("your-org/your-dataset", split="train")
records = [
    {"question": row["question"], "answer": row["answer"]}
    for row in hf.select(range(200))
]

client = opik.Opik()
dataset = client.get_or_create_dataset("your-hf-train")
dataset.insert(records)
```

You can also wrap Hugging Face datasets with a custom helper by following the patterns in `sdks/opik_optimizer/src/opik_optimizer/datasets` (using `DatasetSpec` + `DatasetHandle`) if you want the same `split`/`count`/`filter_by` interface as the built-in datasets.

## Train/validation splits

**Overfitting** occurs when an optimized prompt performs well on the examples it was trained on but fails to generalize to new, unseen data. To prevent this, split your dataset into separate sets:

* **Training dataset** (70-80%): Used by the optimizer to generate prompt improvements
* **Validation dataset** (20-30%): Used to evaluate and rank candidate prompts during optimization, helping select prompts that generalize well
* **Test dataset** (optional, separate): Held out completely until after optimization to measure final real-world performance

The optimizer uses the training set for learning and the validation set for selection, ensuring the best prompt works beyond the training examples.

```python
import opik

client = opik.Opik()