# Source: https://docs.zenml.io/user-guides/starter-guide/manage-artifacts.md

# Manage artifacts

Data sits at the heart of every machine learning workflow. Managing and versioning this data correctly is essential for reproducibility and traceability within your ML pipelines. ZenML takes a proactive approach to data versioning, ensuring that every artifact—be it data, models, or evaluations—is automatically tracked and versioned upon pipeline execution.

![Walkthrough of ZenML Artifact Control Plane (Dashboard available only on ZenML Pro)](https://3621652509-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F75OYotLPi8TviSrtZTJZ%2Fuploads%2Fgit-blob-646b6b8aa99d1a223f2984e2cb23725b0a357a64%2Fdcp_walkthrough.gif?alt=media)

This guide will delve into artifact versioning and management, showing you how to efficiently name, organize, and utilize your data with the ZenML framework.

## Managing artifacts produced by ZenML pipelines

Artifacts, the outputs of your steps and pipelines, are automatically versioned and stored in the artifact store. Configuring these artifacts is pivotal for transparent and efficient pipeline development.

### Giving names to your artifacts

Assigning custom names to your artifacts can greatly enhance their discoverability and manageability. As best practice, utilize the `Annotated` object within your steps to give precise, human-readable names to outputs:

```python
from typing import Annotated
import pandas as pd
from sklearn.datasets import load_iris

from zenml import pipeline, step

# Using Annotated to name our dataset
@step
def training_data_loader() -> Annotated[pd.DataFrame, "iris_dataset"]:
    """Load the iris dataset as pandas dataframe."""
    iris = load_iris(as_frame=True)
    return iris.get("frame")


@pipeline
def feature_engineering_pipeline():
    training_data_loader()


if __name__ == "__main__":
    feature_engineering_pipeline()
```

{% hint style="info" %}
Unspecified artifact outputs default to a naming pattern of `{pipeline_name}::{step_name}::output`. For visual exploration in the ZenML dashboard, it's best practice to give significant outputs clear custom names.
{% endhint %}

Artifacts named `iris_dataset` can then be found swiftly using various ZenML interfaces:

{% tabs %}
{% tab title="OSS (CLI)" %}
To list artifacts: `zenml artifact list`
{% endtab %}

{% tab title="Cloud (Dashboard)" %}
The [ZenML Pro](https://zenml.io/pro) dashboard offers advanced visualization features for artifact exploration.

<figure><img src="https://3621652509-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F75OYotLPi8TviSrtZTJZ%2Fuploads%2Fgit-blob-159ed1941642a0ce9d03a7c2e894406d9949ee37%2Fdcp_artifacts_list.png?alt=media" alt=""><figcaption><p>ZenML Artifact Control Plane.</p></figcaption></figure>

{% hint style="info" %}
To prevent visual clutter, make sure to assign names to your most important artifacts that you would like to explore visually.
{% endhint %}
{% endtab %}
{% endtabs %}

### Versioning artifacts manually

ZenML automatically versions all created artifacts using auto-incremented numbering. I.e., if you have defined a step creating an artifact named `iris_dataset` as shown above, the first execution of the step will create an artifact with this name and version "1", the second execution will create version "2", and so on.

While ZenML handles artifact versioning automatically, you have the option to specify custom versions using the [`ArtifactConfig`](https://sdkdocs.zenml.io/latest/core_code_docs/core-model.html#zenml.model.artifact_config). This may come into play during critical runs like production releases.

```python
from typing import Annotated
import pandas as pd
from zenml import step, ArtifactConfig

@step
def training_data_loader() -> (
    Annotated[
        pd.DataFrame, 
        # Add `ArtifactConfig` to control more properties of your artifact
        ArtifactConfig(
            name="iris_dataset", 
            version="raw_2023"
        ),
    ]
):
    ...
```

The next execution of this step will then create an artifact with the name `iris_dataset` and version `raw_2023`. This is primarily useful if you are making a particularly important pipeline run (such as a release) whose artifacts you want to distinguish at a glance later.

{% hint style="warning" %}
Since custom versions cannot be duplicated, the above step can only be run once successfully. To avoid altering your code frequently, consider using a [YAML config](https://docs.zenml.io/user-guides/production-guide/configure-pipeline) for artifact versioning.
{% endhint %}

After execution, `iris_dataset` and its version `raw_2023` can be seen using:

{% tabs %}
{% tab title="OSS (CLI)" %}
To list versions: `zenml artifact version list`
{% endtab %}

{% tab title="Cloud (Dashboard)" %}
The Cloud dashboard visualizes version history for your review.

<figure><img src="https://3621652509-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F75OYotLPi8TviSrtZTJZ%2Fuploads%2Fgit-blob-440f3a5aed556b449ab8f7fae80e83ba808458f3%2Fdcp_artifacts_versions_list.png?alt=media" alt=""><figcaption><p>ZenML Data Versions List.</p></figcaption></figure>
{% endtab %}
{% endtabs %}

### Add metadata and tags

If you would like to extend your artifacts and runs with extra metadata or tags you can do so by following the patterns demonstrated below:

```python
from zenml import step, log_metadata, add_tags


# In the following step, we use the utility functions `log_metadata` and `add_tags`.
# Since we are calling these functions directly from a step, both will attach
# the additional information to the current run.
@step
def annotation_approach() -> str:
    log_metadata(metadata={"metadata_key": "metadata_value"})
    add_tags(tags=["tag_name"])
    return "string"


# There are other ways to attach this information to different versions of your
# artifacts as well. For instance, you will see a step with a single output below.
# If you modify the call to include the `infer_artifact` flag, these functions
# will attach this information to the artifact version instead.
@step
def annotation_approach() -> str:
    log_metadata(metadata={"metadata_key": "metadata_value"}, infer_artifact=True)
    add_tags(tags=["tag_name"], infer_artifact=True)
    return "string"
```

{% hint style="info" %}
There are multiple ways to interact with tags and metadata in ZenML. If you would like to how to use this information in different scenarios please check the respective guides on [tags](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/tagging) and [metadata](https://docs.zenml.io/how-to/model-management-metrics/track-metrics-metadata).
{% endhint %}

## Comparing metadata across runs (Pro)

The [ZenML Pro](https://www.zenml.io/pro) dashboard includes an Experiment Comparison tool that allows you to visualize and analyze metadata across different pipeline runs. This feature helps you understand patterns and changes in your pipeline's behavior over time.

### Using the comparison views

The tool offers two complementary views for analyzing your metadata:

#### Table View

The tabular view provides a structured comparison of metadata across runs:

![Comparing metadata values across different pipeline runs in table view.](https://3621652509-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F75OYotLPi8TviSrtZTJZ%2Fuploads%2Fgit-blob-4a1778f91787e3b86e7c6eb40f65a93e9b52e867%2Ftable-view.png?alt=media)

This view automatically calculates changes between runs and allows you to:

* Sort and filter metadata values
* Track changes over time
* Compare up to 20 runs simultaneously

#### Parallel Coordinates View

The parallel coordinates visualization helps identify relationships between different metadata parameters:

![Comparing metadata values across different pipeline runs in parallel coordinates view.](https://3621652509-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F75OYotLPi8TviSrtZTJZ%2Fuploads%2Fgit-blob-0c52194430d75ac7f0b5e0a958315b7812cf33c1%2Fcoordinates-view.png?alt=media)

This view is particularly useful for:

* Discovering correlations between different metrics
* Identifying patterns across pipeline runs
* Filtering and focusing on specific parameter ranges

### Accessing the comparison tool

To compare metadata across runs:

1. Navigate to any pipeline in your dashboard
2. Click the "Compare" button in the top navigation
3. Select the runs you want to compare
4. Switch between table and parallel coordinates views using the tabs

{% hint style="info" %}
The comparison tool works with any numerical metadata (`float` or `int`) that you've logged in your pipelines. Make sure to log meaningful metrics in your steps to make the most of this feature.
{% endhint %}

### Sharing comparisons

The tool preserves your comparison configuration in the URL, making it easy to share specific views with team members. Simply copy and share the URL to allow others to see the same comparison with identical settings and filters.

{% hint style="warning" %}
This feature is currently in Alpha Preview. We encourage you to share feedback about your use cases and requirements through our Slack community.
{% endhint %}

## Specify a type for your artifacts

Assigning a type to an artifact allows ZenML to highlight them differently in the dashboard and also lets you filter your artifacts better.

{% hint style="info" %}
If you don't specify a type for your artifact, ZenML will use the default artifact type provided by the materializer that is used to\
save the artifact.
{% endhint %}

```python
from typing import Annotated
from zenml import ArtifactConfig, save_artifact, step
from zenml.enums import ArtifactType

# Assign an artifact type to a step output
@step
def trainer() -> Annotated[MyCustomModel, ArtifactConfig(artifact_type=ArtifactType.MODEL)]:
    return MyCustomModel(...)


# Assign an artifact type when manually saving artifacts
model = ...
save_artifact(model, name="model", artifact_type=ArtifactType.MODEL)
```

## Consuming external artifacts within a pipeline

While most pipelines start with a step that produces an artifact, it is often the case to want to consume artifacts external from the pipeline. The `ExternalArtifact` class can be used to initialize an artifact within ZenML with any arbitrary data type.

For example, let's say we have a Snowflake query that produces a dataframe, or a CSV file that we need to read. External artifacts can be used for this, to pass values to steps that are neither JSON serializable nor produced by an upstream step:

```python
import numpy as np
from zenml import ExternalArtifact, pipeline, step

@step
def print_data(data: np.ndarray):
    print(data)

@pipeline
def printing_pipeline():
    # One can also pass data directly into the ExternalArtifact
    # to create a new artifact on the fly
    data = ExternalArtifact(value=np.array([0]))

    print_data(data=data)


if __name__ == "__main__":
    printing_pipeline()
```

Optionally, you can configure the `ExternalArtifact` to use a custom [materializer](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/handle-custom-data-types) for your data or disable artifact metadata and visualizations. Check out the [SDK docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifacts.html#zenml.artifacts.external_artifact) for all available options.

{% hint style="info" %}
Using an `ExternalArtifact` for your step automatically disables caching for the step.
{% endhint %}

## Consuming artifacts produced by other pipelines

It is also common to consume an artifact downstream after producing it in an upstream pipeline or step. As we have learned in the [previous section](https://docs.zenml.io/user-guides/tutorial/fetching-pipelines#fetching-artifacts-directly), the `Client` can be used to fetch artifacts directly inside the pipeline code:

```python
from uuid import UUID
import pandas as pd
from zenml import step, pipeline
from zenml.client import Client


@step
def trainer(dataset: pd.DataFrame):
    ...

@pipeline
def training_pipeline():
    client = Client()
    # Fetch by ID
    dataset_artifact = client.get_artifact_version(
        name_id_or_prefix=UUID("3a92ae32-a764-4420-98ba-07da8f742b76")
    )

    # Fetch by name alone - uses the latest version of this artifact
    dataset_artifact = client.get_artifact_version(name_id_or_prefix="iris_dataset")

    # Fetch by name and version
    dataset_artifact = client.get_artifact_version(
        name_id_or_prefix="iris_dataset", version="raw_2023"
    )

    # Pass into any step
    trainer(dataset=dataset_artifact)


if __name__ == "__main__":
    training_pipeline()
```

{% hint style="info" %}
Calls of `Client` methods like `get_artifact_version` directly inside the pipeline code makes use of ZenML's [late materialization](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/load-artifacts-into-memory) behind the scenes.
{% endhint %}

If you would like to bypass materialization entirely and just download the data or files associated with a particular artifact version, you can use the `.download_files` method:

```python
from zenml.client import Client

client = Client()
artifact = client.get_artifact_version(name_id_or_prefix="iris_dataset")
artifact.download_files("path/to/save.zip")
```

Take note that the path must have the `.zip` extension, as the artifact data will be saved as a zip file. Make sure to handle any exceptions that may arise from this operation.

## Managing artifacts **not** produced by ZenML pipelines

Sometimes, artifacts can be produced completely outside of ZenML. A good example of this is the predictions produced by a deployed model.

```python
# A model is deployed, running in a FastAPI container
# Let's use the ZenML client to fetch the latest model and make predictions

from zenml.client import Client
from zenml import save_artifact

# Fetch the model from a registry or a previous pipeline
model = ...

# Let's make a prediction
prediction = model.predict([[1, 1, 1, 1]])

# We now store this prediction in ZenML as an artifact
# This will create a new artifact version
save_artifact(prediction, name="iris_predictions")
```

You can also load any artifact stored within ZenML using the `load_artifact` method:

```python
from zenml import load_artifact

# Loads the latest version
load_artifact("iris_predictions")
```

{% hint style="info" %}
`load_artifact` is simply short-hand for the following Client call:

```python
from zenml.client import Client

client = Client()
client.get_artifact("iris_predictions").load()
```

{% endhint %}

Even if an artifact is created externally, it can be treated like any other artifact produced by ZenML steps - with all the functionalities described above!

{% hint style="info" %}
It is also possible to use these functions inside your ZenML steps. However, it is usually cleaner to return the artifacts as outputs of your step to save them, or to use External Artifacts to load them instead.
{% endhint %}

### Linking existing data as a ZenML artifact

Sometimes, data is produced completely outside of ZenML and can be conveniently stored on a given storage. A good example of this is the checkpoint files created as a side-effect of the Deep Learning model training. We know that the intermediate data of the deep learning frameworks is quite big and there is no good reason to move it around again and again, if it can be produced directly in the artifact store boundaries and later just linked to become an artifact of ZenML.\
Let's explore the Pytorch Lightning example to fit the model and store the checkpoints in a remote location.

```python
import os
from zenml.client import Client
from zenml import register_artifact
from pytorch_lightning import Trainer
from pytorch_lightning.callbacks import ModelCheckpoint
from uuid import uuid4

# Define where the model data should be saved
# use active ArtifactStore
prefix = Client().active_stack.artifact_store.path
# keep data separable for future runs with uuid4 folder
default_root_dir = os.path.join(prefix, uuid4().hex)

# Define the model and fit it
model = ...
trainer = Trainer(
    default_root_dir=default_root_dir,
    callbacks=[
        ModelCheckpoint(
            every_n_epochs=1, save_top_k=-1, filename="checkpoint-{epoch:02d}"
        )
    ],
)
try:
    trainer.fit(model)
finally:
    # We now link those checkpoints in ZenML as an artifact
    # This will create a new artifact version
    register_artifact(default_root_dir, name="all_my_model_checkpoints")
```

{% hint style="info" %}
The artifact produced from the preexisting data will have a `pathlib.Path` type, once loaded or passed as input to another step.
{% endhint %}

Even if an artifact is created and stored externally, it can be treated like any other artifact produced by ZenML steps - with all the functionalities described above!

For more details and use-cases check-out detailed docs page [Register Existing Data as a ZenML Artifact](https://docs.zenml.io/how-to/data-artifact-management/complex-usecases/registering-existing-data).

## Logging metadata for an artifact

One of the most useful ways of interacting with artifacts in ZenML is the ability to associate metadata with them. [As mentioned before](https://docs.zenml.io/user-guides/tutorial/fetching-pipelines#artifact-information), artifact metadata is an arbitrary dictionary of key-value pairs that are useful for understanding the nature of the data.

As an example, one can associate the results of a model training alongside a model artifact, the shape of a table alongside a `pandas` dataframe, or the size of an image alongside a PNG file.

For some artifacts, ZenML automatically logs metadata. As an example, for `pandas.Series` and `pandas.DataFrame` objects, ZenML logs the shape and size of the objects:

{% tabs %}
{% tab title="Python" %}

```python
from zenml.client import Client

# Get an artifact version (e.g. pd.DataFrame)
artifact = Client().get_artifact_version('50ce903f-faa6-41f6-a95f-ff8c0ec66010')

# Fetch it's metadata
artifact.run_metadata["storage_size"].value  # Size in bytes
artifact.run_metadata["shape"].value  # Shape e.g. (500,20)
```

{% endtab %}

{% tab title="OSS (Dashboard)" %}
The information regarding the metadata of an artifact can be found within the DAG visualizer interface on the OSS dashboard:

<figure><img src="https://3621652509-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F75OYotLPi8TviSrtZTJZ%2Fuploads%2Fgit-blob-9143ff57505cce9fd81583a7319624436584d552%2Fdashboard_artifact_metadata.png?alt=media" alt=""><figcaption><p>ZenML Artifact Control Plane.</p></figcaption></figure>
{% endtab %}

{% tab title="Cloud (Dashboard)" %}
The [ZenML Pro](https://zenml.io/pro) dashboard offers advanced visualization features for artifact exploration, including a dedicated artifacts tab with metadata visualization:

<figure><img src="https://3621652509-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F75OYotLPi8TviSrtZTJZ%2Fuploads%2Fgit-blob-f1004f678aff9ce7b38638bcd6447303b8663aa6%2Fdcp_metadata.png?alt=media" alt=""><figcaption><p>ZenML Artifact Control Plane.</p></figcaption></figure>
{% endtab %}
{% endtabs %}

A user can also add metadata to an artifact directly within a step using the `log_metadata` method:

```python
from typing import Tuple
from typing import Annotated
import numpy as np
from sklearn.base import ClassifierMixin
from zenml import step, log_metadata, ArtifactConfig

@step
def model_finetuner_step(
    model: ClassifierMixin, dataset: Tuple[np.ndarray, np.ndarray]
) -> Annotated[
    ClassifierMixin, ArtifactConfig(name="my_model", tags=["SVC", "trained"])
]:
    """Finetunes a given model on a given dataset."""
    model.fit(dataset[0], dataset[1])
    accuracy = model.score(dataset[0], dataset[1])

    
    log_metadata(
        # Metadata should be a dictionary of JSON-serializable values
        metadata={"accuracy": float(accuracy)},
        # Using infer_artifact=True automatically attaches metadata to the
        # artifact produced by this step. Since this step has only one output,
        # we don't need to specify the artifact_name
        infer_artifact=True
        # If the step had multiple outputs, we would need to specify which one:
        # artifact_name="my_model", infer_artifact=True

        # A dictionary of dictionaries can also be passed to group metadata
        #  in the dashboard
        # metadata = {"metrics": {"accuracy": accuracy}}
    )
    return model
```

For further depth, there is an [advanced metadata logging guide](https://docs.zenml.io/how-to/model-management-metrics/track-metrics-metadata) that goes more into detail about logging metadata in ZenML.

Additionally, there is a lot more to learn about artifacts within ZenML. Please read the [dedicated data management guide](https://docs.zenml.io/how-to/data-artifact-management) for more information.

## Code example

This section combines all the code from this section into one simple script that you can use easily:

<details>

<summary>Code Example of this Section</summary>

```python
from typing import Optional, Tuple
from typing import Annotated

import numpy as np
from sklearn.base import ClassifierMixin
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from zenml import ArtifactConfig, pipeline, step, log_metadata
from zenml import save_artifact, load_artifact
from zenml.client import Client


@step
def versioned_data_loader_step() -> (
    Annotated[
        Tuple[np.ndarray, np.ndarray],
        ArtifactConfig(
            name="my_dataset",
            tags=["digits", "computer vision", "classification"],
        ),
    ]
):
    """Loads the digits dataset as a tuple of flattened numpy arrays."""
    digits = load_digits()
    return (digits.images.reshape((len(digits.images), -1)), digits.target)


@step
def model_finetuner_step(
    model: ClassifierMixin, dataset: Tuple[np.ndarray, np.ndarray]
) -> Annotated[
    ClassifierMixin,
    ArtifactConfig(name="my_model", tags=["SVC", "trained"]),
]:
    """Finetunes a given model on a given dataset."""
    model.fit(dataset[0], dataset[1])
    accuracy = model.score(dataset[0], dataset[1])
    log_metadata(metadata={"accuracy": float(accuracy)})
    return model


@pipeline
def model_finetuning_pipeline(
    dataset_version: Optional[str] = None,
    model_version: Optional[str] = None,
):
    client = Client()
    # Either load a previous version of "my_dataset" or create a new one
    if dataset_version:
        dataset = client.get_artifact_version(
            name_id_or_prefix="my_dataset", version=dataset_version
        )
    else:
        dataset = versioned_data_loader_step()

    # Load the model to finetune
    # If no version is specified, the latest version of "my_model" is used
    model = client.get_artifact_version(
        name_id_or_prefix="my_model", version=model_version
    )

    # Finetune the model
    # This automatically creates a new version of "my_model"
    model_finetuner_step(model=model, dataset=dataset)


def main():
    # Save an untrained model as first version of "my_model"
    untrained_model = SVC(gamma=0.001)
    save_artifact(
        untrained_model, name="my_model", version="1", tags=["SVC", "untrained"]
    )

    # Create a first version of "my_dataset" and train the model on it
    model_finetuning_pipeline()

    # Finetune the latest model on an older version of the dataset
    model_finetuning_pipeline(dataset_version="1")

    # Run inference with the latest model on an older version of the dataset
    latest_trained_model = load_artifact("my_model")
    old_dataset = load_artifact("my_dataset", version="1")
    latest_trained_model.predict(old_dataset[0])


if __name__ == "__main__":
    main()
```

This would create the following pipeline run DAGs:

**Run 1:**

<img src="https://3621652509-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F75OYotLPi8TviSrtZTJZ%2Fuploads%2Fgit-blob-3d1a65bc95017295a44e609647cf1eb9680408a3%2Fartifact_management_1.png?alt=media" alt="Create a first version of my_dataset" data-size="original">

**Run 2:**

<img src="https://3621652509-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F75OYotLPi8TviSrtZTJZ%2Fuploads%2Fgit-blob-29ac08892bc04c243896144d709e9753140c3418%2Fartifact_management_2.png?alt=media" alt="Uses a second version of my_dataset" data-size="original">

</details>

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>