# Deploy your finetuned model on AWS Marketplace

> Learn how to deploy your finetuned model on AWS Marketplace.

<AuthorsContainer
  authors={[
    {
      name: "Youran Qi",
      imageSrc: "https://fern-image-hosting.s3.amazonaws.com/cohere/929cb1c-youran-headshot.jpg"
    }
  ]}
/>

<CookbookHeader href="https://github.com/cohere-ai/cohere-developer-experience/blob/main/notebooks/finetuning/Deploy%20your%20own%20finetuned%20command-r-0824.ipynb" />

## Deploy Your Own Finetuned Command-R-0824 Model from AWS Marketplace

This sample notebook shows you how to deploy your own finetuned HuggingFace Command-R model [CohereForAI/c4ai-command-r-08-2024](https://huggingface.co/CohereForAI/c4ai-command-r-08-2024) using Amazon SageMaker. More specifically, assuming you already have the adapter weights or merged weights from your own finetuning of [CohereForAI/c4ai-command-r-08-2024](https://huggingface.co/CohereForAI/c4ai-command-r-08-2024), we will show you how to

1. Merge the adapter weights to the weights of the base model, if you bring only the adapter weights
2. Export the merged weights to the TensorRT-LLM inference engine using Amazon SageMaker
3. Deploy the engine as a SageMaker endpoint to serve your business use cases

> **Note**: This is a reference notebook and it cannot run unless you make changes suggested in the notebook.

### Pre-requisites:

1. **Note: This notebook contains elements which render correctly in Jupyter interface. Open this notebook from an Amazon SageMaker Notebook Instance or Amazon SageMaker Studio.**
2. Ensure that IAM role used has **AmazonSageMakerFullAccess**
3. To deploy this ML model successfully, ensure that:
   1. Either your IAM role has these three permissions and you have authority to make AWS Marketplace subscriptions in the AWS account used:
      1. **aws-marketplace:ViewSubscriptions**
      2. **aws-marketplace:Unsubscribe**
      3. **aws-marketplace:Subscribe**
   2. or your AWS account has a subscription to the packages for [Cohere Bring Your Own Fine-tuning](https://aws.amazon.com/marketplace/pp/prodview-5wt5pdnw3bbq6). If so, skip step: [Subscribe to the bring your own finetuning algorithm](#subscribe)

### Contents:

1. [Subscribe to the bring your own finetuning algorithm](#subscribe)
2. [Preliminary setup](#setup)
3. [Get the merged weights](#merge)
4. [Upload the merged weights to S3](#upload)
5. [Export the merged weights to the TensorRT-LLM inference engine](#export)
6. [Create an endpoint for inference from the exported engine](#endpoint)
7. [Perform real-time inference by calling the endpoint](#inference)
8. [Delete the endpoint (optional)](#delete)
9. [Unsubscribe to the listing (optional)](#unsubscribe)

### Usage instructions:

You can run this notebook one cell at a time (By using Shift+Enter for running a cell).

<a name="subscribe" />

## 1. Subscribe to the bring your own finetuning algorithm

To subscribe to the algorithm:

1. Open the algorithm listing page [Cohere Bring Your Own Fine-tuning](https://aws.amazon.com/marketplace/pp/prodview-5wt5pdnw3bbq6).
2. On the AWS Marketplace listing, click on the **Continue to Subscribe** button.
3. On the **Subscribe to this software** page, review and click on **"Accept Offer"** if you and your organization agrees with EULA, pricing, and support terms. On the "Configure and launch" page, make sure the ARN displayed in your region match with the ARN you will use below.

<a name="setup" />

## 2. Preliminary setup

Install the Python packages you will use below and import them. For example, you can run the command below to install `cohere` if you haven't done so.

```sh
pip install "cohere>=5.11.0"
```

```python
import cohere
import os
import sagemaker as sage

from sagemaker.s3 import S3Uploader
```

Make sure you have access to the resources in your AWS account. For example, you can configure an AWS profile by the command `aws configure sso` (see [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html#cli-configure-sso-configure)) and run the command below to set the environment variable `AWS_PROFILE` as your profile name.

```python