# Enabling Large CSV Uploads

By default, Opik supports CSV file uploads up to 20MB for dataset creation. For self-hosted deployments that need to process larger CSV files (up to 2GB), you can enable the large CSV upload feature with additional configuration.

## Overview

When enabled, this feature allows:

* **CSV files up to 2GB** in size
* **Asynchronous processing** - files are processed in the background after upload

## Configuration Steps

### 1. Enable the Feature Toggle

Set the following environment variable for the Opik backend service:

```bash
TOGGLE_CSV_UPLOAD_ENABLED: true
```

### 2. Increase Idle Timeout

Large file uploads require more time to transfer. Increase the server idle timeout:

```bash
SERVER_IDLE_TIMEOUT: 10m
```

The default timeout is 30 seconds, which is insufficient for large file uploads. We recommend setting it to 10 minutes for files up to 2GB.

### 3. Configure Nginx (Kubernetes/Helm Deployments)

If you're using the Helm chart deployment, add the following configuration to your `values.yaml`:

```yaml
component:
  frontend:
    # Increase client body size limit to 2GB
    clientMaxBodySize: "2g"
    
    # Increase proxy timeouts for large file uploads
    upstreamConfig:
      proxy_read_timeout: 600s
      proxy_connect_timeout: 600s
      proxy_send_timeout: 600s
      client_max_body_size: 2g
```

### 4. Ensure Adequate Disk Space

The backend service temporarily buffers uploaded CSV files to disk before processing them. Ensure your backend pods/containers have:

* **Minimum 50GB of disk space** available
* **Sufficient IOPS** for concurrent file operations

### 5. Optional: Adjust Batch Size

You can optionally configure the batch size for CSV processing:

```bash
BATCH_OPERATIONS_DATASETS_CSV_BATCH_SIZE: 1000
```

The default batch size is 1000 rows per batch. Adjust this based on your:

* Available memory
* Row complexity (number of columns, data size)
* Desired processing speed

## Docker Compose Deployments

For Docker Compose deployments, the configuration is slightly different:

### 1. Update docker-compose.yml

Add the environment variables to the backend service:

```yaml
services:
  backend:
    environment:
      - TOGGLE_CSV_UPLOAD_ENABLED=true
      - SERVER_IDLE_TIMEOUT=10m
      - BATCH_OPERATIONS_DATASETS_CSV_BATCH_SIZE=1000
```

### 2. Update Nginx Configuration

The nginx configuration files already include the 2GB limit for local deployments. No additional changes are needed for `nginx_default_local.conf` or `nginx_local_be_local.conf`.

## Kubernetes/Helm Deployment Example

Here's a complete example for Helm chart deployments:

```yaml