# Heuristic metrics

> Describes all the built-in heuristic metrics provided by Opik

Heuristic metrics are rule-based evaluation methods that allow you to check specific aspects of language model outputs. These metrics use predefined criteria or patterns to assess the quality, consistency, or characteristics of generated text. They come in two flavours:

* **Token or string heuristics** – operate on a single turn and compare the candidate output to a reference or handcrafted rule.
* **Conversation heuristics** – analyse whole transcripts to spot issues like degeneration or forgotten facts across assistant turns.

### String and token heuristics

| Metric                  | Description                                                                         |
| ----------------------- | ----------------------------------------------------------------------------------- |
| BERTScore               | Contextual embedding similarity; robust alternative to Levenshtein.                 |
| ChrF                    | Character n-gram F-score (supports chrF and chrF++).                                |
| Contains                | Checks if the output contains a specific substring (case-sensitive or insensitive). |
| CorpusBLEU              | Calculates a corpus-level BLEU score across many candidates.                        |
| Equals                  | Checks if the output exactly matches an expected string.                            |
| GLEU                    | Estimates fluency and grammatical correctness on a 0–1 scale.                       |
| IsJson                  | Ensures the output can be parsed as JSON.                                           |
| JSDivergence            | Jensen–Shannon similarity between token distributions.                              |
| JSDistance              | Raw Jensen–Shannon divergence between token distributions.                          |
| KLDivergence            | Kullback–Leibler divergence between token distributions.                            |
| LanguageAdherenceMetric | Checks whether text adheres to an expected language code.                           |
| LevenshteinRatio        | Computes the normalised Levenshtein similarity between output and reference.        |
| Readability             | Reports Flesch Reading Ease and Flesch–Kincaid grade levels.                        |
| RegexMatch              | Validates the output against a regular expression pattern.                          |
| ROUGE                   | Calculates ROUGE variants (rouge1, rouge2, rougeL, rougeLsum).                      |
| SentenceBLEU            | Calculates a single-sentence BLEU score against one or more references.             |
| Sentiment               | Scores sentiment using NLTK's VADER lexicon (compound, pos/neu/neg).                |
| SpearmanRanking         | Spearman's rank correlation for two equal-length rankings.                          |
| Tone                    | Flags tone issues such as negativity, shouting, or forbidden phrases.               |

### Conversation heuristics

| Metric                    | Description                                                                                                         |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| Conversation Degeneration | Detects repetition and low-entropy responses over a conversation (implemented by `ConversationDegenerationMetric`). |
| Knowledge Retention       | Checks whether the last assistant reply preserves user-provided facts from earlier turns.                           |

> \[!TIP]
> These metrics operate on a single transcript without requiring a gold reference. If you
> need BLEU/ROUGE/METEOR-style comparisons, compose a custom `ConversationThreadMetric`
> that wraps the single-turn heuristics (`SentenceBLEU`, `ROUGE`, `METEOR`).

## Score an LLM response

You can score an LLM response by first initializing the metrics and then calling the `score` method:

```python
from opik.evaluation.metrics import Contains

metric = Contains(name="contains_hello", case_sensitive=True)

score = metric.score(output="Hello world !", reference="Hello")

print(score)
```

## Metrics

### Equals

The `Equals` metric can be used to check if the output of an LLM exactly matches a specific string. It can be used in the following way:

```python
from opik.evaluation.metrics import Equals

metric = Equals()

score = metric.score(output="Hello world !", reference="Hello, world !")
print(score)
```

### Contains

The `Contains` metric can be used to check if the output of an LLM contains a specific substring. It can be used in the following way:

```python
from opik.evaluation.metrics import Contains

metric = Contains(case_sensitive=False)

score = metric.score(output="Hello world !", reference="Hello")
print(score)
```

### RegexMatch

The `RegexMatch` metric can be used to check if the output of an LLM matches a specified regular expression pattern. It can be used in the following way:

```python
from opik.evaluation.metrics import RegexMatch

metric = RegexMatch(regex="^[a-zA-Z0-9]+$")

score = metric.score("Hello world !")
print(score)
```

### IsJson

The `IsJson` metric can be used to check if the output of an LLM is valid. It can be used in the following way:

```python
from opik.evaluation.metrics import IsJson

metric = IsJson(name="is_json_metric")

score = metric.score(output='{"key": "some_valid_sql"}')
print(score)
```

### LevenshteinRatio

The `LevenshteinRatio` metric measures how similar the output is to a reference string on a 0–1 scale (1.0 means identical). It is useful when exact matches are too strict but you still want to penalise large deviations.

```python
from opik.evaluation.metrics import LevenshteinRatio

metric = LevenshteinRatio()

score = metric.score(output="Hello world !", reference="hello")
print(score)
```

### BLEU

The BLEU (Bilingual Evaluation Understudy) metrics estimate how close the LLM outputs are to one or more reference translations. Opik provides two separate classes:

* `SentenceBLEU` – Single-sentence BLEU
* `CorpusBLEU` – Corpus-level BLEU
  Both rely on the underlying NLTK BLEU implementation with optional smoothing methods, weights, and variable n-gram orders.

You will need nltk library:

```bash
pip install nltk
```

Use `SentenceBLEU` to compute single-sentence BLEU between a single candidate and one (or more) references:

```python
from opik.evaluation.metrics import SentenceBLEU

metric = SentenceBLEU(n_grams=4, smoothing_method="method1")