# File Uploads
Source: https://speakeasy.com/api-design/file-uploads

File uploads can be confusing to work with at first because it takes a bit of a
mental shift to think about. 

Firstly, a file is usually not just a file, it also has metadata needs to go
with it and that can be hard to keep track of.

Secondly, it is not really a file upload, simply a resource or collection of
resources with a `Content-Type` of something other than the usual JSON or XML.

## URL design 

To visualize how file uploads could be designed into an API, let's see how
images could be added for two different use-cases.

A user could have an avatar sub-resource, which might look like this:

```
/users/<username>/avatar
```

This can then be uploaded and retrieved on the same URL, making it a consistent
API experience with any other type of resource. 

Multiple images could be needed for product thumbnails, and that can be a
sub-collection of the product.

```
/product/<uuid>/thumbnails
```

A collection of resources could be available, and a particular thumbnail could
be retrieved or deleted using regular semantics like `GET` and `DELETE` on the
particular resource URL.

```
/product/<uuid>/thumbnails/<image-uuid>
```

## POST or PUT

There is no particular [HTTP method](/api-design/http-methods) specific to file
uploads, instead we use the appropriate hTTP method for the resource or
collection being worked with.

For the example of a single avatar for each user, the URL is already known, and
it does not make any difference whether this is the first avatar they have
uploaded, or they have remade the same request 10 times in a row after an
intermitted internet connection messed up the first few. This should use a
`PUT`, because that means "The end result should be this, regardless of what is
there right now."

```
PUT /users/<username>/avatar
```

When working with a collection, the URL of the resource is not known until it
has been created. For this reason a `POST` would be more appropriate.

```
POST /product/<uuid>/thumbnails
```

How these uploads work could vary depending on the use case, so let's look at
the most popular methods.

## Different methods of file upload

There are a few popular approaches to file uploads in APIs:

1. Uploading a file by itself, like adding an avatar for an existing user.
2. Uploading a file with metadata in the same request, like a video file with a title, description, and geodata.
3. Importing a file from a URL, like a user's avatar from Facebook.

It's not entirely unreasonable to consider an API using all of these approaches
for different use cases throughout the API depending on the specifics. Lets
learn how these things work, and talk about when to use one over the other.

### Method A: Direct file uploads

When no metadata is needed to be uploaded with a request, a direct file upload
is beautifully simple.

- Uploading a CSV of emails being imported to send a tree sponsorship email to.
- A new logo for a funding partner.
- A replacement avatar for a user profile.

In all of these situations, the file is the only thing that needs to be uploaded
and they also have a handy content type that can go right into the HTTP request
to let the API know what's coming.

```http
PUT /users/philsturgeon/image HTTP/2
Authentication: Bearer <token>
Content-Type: image/jpeg
Content-Length: 284

<raw image content>
```

Any file can be uploaded this way, and the API can infer the content type from
the `Content-Type` header. The API can also infer the user from the token, so 
the request does not need to include any user information.

The API will then save the file, and return a response with a URL to the file
that was uploaded. This URL can be used to access the file in the future, and
can be used to link the file to the user that uploaded it.

The response here will have a simple body:

```json
{
  "url": "https://cdn.example.org/users/philsturgeon.jpg",
  "links": {
    "self": "https://example.org/api/images/c19568b4-77b3-4442-8278-4f93c0dd078",
    "user": "https://example.org/api/users/philsturgeon"
  }
}
```

That `user` was inferred from the token, and the `url` is the resulting URL to
the avatar that has been uploaded. Normally this would be some sort of Content
Delivery Network (CDN) URL, but it could be a direct-to-S3 URL, or a URL to a Go
service that handles file uploads. splitting file uploads to a separate service
and hosting them elsewhere keeps the API server free to do more productive work
than reading and writing files.

### Method B: Upload from URL

Depending on how the client application works, uploading from a file might not
be the preferred approach. A common pattern is mobile clients uploading user
images directly from the photo libraries on the mobile device, and the web teams
were pulling avatars from Facebook or Twitter profiles after they have done a
"social login" flow.

This is common because its harder for the web application to access the raw
content of a file using just browser-based JavaScript. At some point a server
needs to be involved to read that, so whether they have uploaded via cloudinary
or some other upload service, the API server is going to need to take a URL and
download the file.

The same endpoint that handled the direct upload can serve this same logic, with
the `Content-Type` header changed to `application/json` and the body of the
request containing a URL to the file.

```http
PUT /users/philsturgeon/image HTTP/2
Authentication: Bearer <token>
Content-Type: application/json

{
  "url" : "https://facebook.com/images/dfidsyfsudf.png"
}
```

The API will then download the file from the URL, save it, and return a response
with a URL to the file that was uploaded. This URL can be used to access the file
in the future, and can be used to link the file to the user that uploaded it.

```json
{
  "url": "https://cdn.example.org/users/philsturgeon.jpg",
  "links": {
    "self": "https://example.org/api/images/c19568b4-77b3-4442-8278-4f93c0dd078",
    "user": "https://example.org/api/users/philsturgeon"
  }
}
```

Supporting both might not be necessary, but if they are, just support both the
image types needed and the JSON alternative of that. HTTP makes that
incredibly easy to do thanks to being able to switch `Content-Type`.

### Method 3: Separate metadata resource

The above examples are great for simple file uploads, but what if there is a need to
upload metadata with the file? This is where things get a bit more complex. 

One approach would be multipart forms, but they're pretty complex to work with
and not ideal for large files. If sending a massive video file, it's not ideal
to have to send the title, description, and tags in the same request as the
video file. If the video file upload fails, it would have to be re-uploaded with
the video file and all of the metadata again.

The way YouTube handles uploads via API are an interesting examples of splitting
out metadata and a video file. They use a two-step process which focuses on
metadata first, which allows for the metadata to be saved and the video can then
be retried and uploaded without losing the metadata.

The YouTube Data API (v3) approach to [Resumable
Uploads](https://developers.google.com/youtube/v3/guides/using_resumable_upload_protocol)
works like this.

First, they make a POST request to the video upload endpoint with the metadata
in the body of the request:

```http
POST /upload/youtube/v3/videos?uploadType=resumable&part=snippet,status HTTP/1.1
Host: www.googleapis.com
Authorization: Bearer <token>
Content-Length: 278
Content-Type: application/json; charset=UTF-8

{
  "snippet": {
    "title": "My video title",
    "description": "This is a description of my video",
    "tags": ["cool", "video", "more keywords"],
    "categoryId": 22
  },
  "status": {
    "privacyStatus": "public",
    "embeddable": true,
    "license": "youtube"
  }
}
```

The response then contains a `Location` header with a URL to the video upload
endpoint:

```http
HTTP/1.1 200 OK
Location: https://www.googleapis.com/upload/youtube/v3/videos?uploadType=resumable&upload_id=xa298sd_f&part=snippet,status,contentDetails
Content-Length: 0
```

Then to upload the video it's back to direct file uploads. The video file can be
uploaded to the URL provided in the `Location` header, with the content type set
to `video/*`:

```http
PUT https://www.googleapis.com/upload/youtube/v3/videos?uploadType=resumable&upload_id=xa298sd_f&part=snippet,status,contentDetails HTTP/1.1
Authorization: Bearer AUTH_TOKEN
Content-Length: <file length>
Content-Type: video/mp4

<BINARY_FILE_DATA>
```

What's cool about this approach, is that URL _could_ be part of the main API,
or it _could_ be a totally different service. It could be a direct-to-S3 URL,
Cloudinary, or some other service that handles file uploads. 

Larger companies will be more prone to building a service to handle such files
coming in, whilst smaller teams might want to keep things simple and let the API
do the heavy lifting. The larger the file, the more likely it will be desirable
to split that off, as having the API handle these huge files - even if the
uploads are chunked - will keep the HTTP workers busy. Maintaining those
connections might slow down a Rails-based API for a long time, for example, so
having another service would help there.

## Best practices

### Check Content-Type and Content-Length

It is worth noting that the `Content-Type` header is not always reliable, and
should not be trusted. If expecting an image, check the first few bytes of the
file to see if it is a valid image format. If expecting a CSV, check the first
few lines to see if it is a valid CSV. **Never trust input.**

The only thing worth mentioning on that request is the addition of
`Content-Length`, which is basically the size of the image being uploaded. A
quick check of `headers['Content-Length'].to_i > 3.megabytes` will let us
quickly reply saying "This image is too large", which is better than waiting
forever to say that. Sure, malicious folks could lie here, so backend code
will need to check the image size too. **Never trust input.**

Protecting against large files is important, as it can be a denial of service
attack. If users are allowed to upload files, they could upload a 10GB file and
fill up disk space. This is why it's important to check the size of the
file before writing it to disk. 

To make sure it seems to be the right type, and to make sure it's not too large, read the file in chunks. This can be done with a simple `File.open` and
`File.read` in Ruby, or similar in other languages. The file is read in chunks,
and then written to a file on disk. This is a good way to handle large files, as
it's not trying to load the whole file into memory at once.

```ruby
def update
  if headers['Content-Type'] != 'image/jpeg'
    render json: { error: 'Invalid content type' }, status: 400
    return
  end

  if headers['Content-Length'].to_i > 3.megabytes
    render json: { error: 'File is too large' }, status: 400
    return
  end

  file = File.open("tmp/#{SecureRandom.uuid}.jpg", 'wb') do |f|
    f.write(request.body.read)
  end

  # Do something with the file
end
```

### Securing File Uploads

Allowing file uploads can introduce all sorts of new attack vectors, so it's worth being very careful about the whole thing. 

One of the main issues with file uploads is directory traversal attacks. If users are allowed to upload files, they could upload a file with a name like `../../etc/passwd`, which could allow them to read sensitive files on the server.

Uploading from a URL could allow for [Server-Side Request Forgery (SSRF)](https://owasp.org/API-Security/editions/2023/en/0xa7-server-side-request-forgery/) attacks, where an attacker could upload a file from a URL that points to a sensitive internal resource, like an AWS metadata URL, or something like `localhost:8080` which allows them to scan for ports on the server.

The [OWASP File Upload Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html) has a lot of good advice on how to secure file uploads, including:

- Limiting the types of files that can be uploaded.
- Limiting the size of files that can be uploaded.
- Storing files in a location that is not accessible via the web server.
- Renaming files to prevent directory traversal attacks.
- Checking the file type by reading the first few bytes of the file.
- Checking the file size before writing it to disk.
- Checking the file for viruses using a virus scanner.

## Summary

Think about what sort of file uploads are needed, how big the files are, where
they're going, and what sort of clients will be using the API. 

The YouTube approach is a bit complex, but a combination of 1 and 2 usually take care of the
job, and help avoid complicated multipart uploads. 

As always, build defensively, and never trust any user input at any point.