# Filtering Collections
Source: https://speakeasy.com/api-design/filtering-responses

When building a REST API, the ability to filter potentially large collections of
data is essential, and sorting can reduce a huge amount of work for both the
client and server.

## What is Filtering?

Filtering allows API users to request only the data they need, "filtering out"
irrelevant things on the server-side instead of making them do it themselves on
the client-side. Reducing the amount of data being returned and transferred
reduces server resources and improves performance, which reduces carbon
emissions and saves money.

## How to Filter

The most straightforward way to filter resources is by using query parameters.
These are appended to the URL to refine the results of an API request. For
example:

```bash
GET /products?category=books&price_lt=20
```

In this case, the request filters products where the `category` is "books", and
the `price` field is less than 20. The query string is easy for both the API
designer and users to understand, making it a natural choice for filtering data.

Naming conventions and deciding if or how to use operators will vary depending on
the implementation, but there are a few common practices and standards to consider.

### Simple Filtering

Starting with the most basic, filter by a single parameter using a query
parameter with a sensible name.

```bash
GET /products?category=books&status=available
```

In these examples, the query parameter `category` or `status` is used to remove
any products that don't match those exact values.

The query parameters in some APIs might be a little busy, as there could be not
just sorting and pagination, but people do things changing output structures,
selecting which properties should be returned, or all kinds of functionality
which are not filtering.

To avoid confusion, it's a good idea to use a consistent naming scheme, like
`filter_category` or better yet a "filter array", e.g.:

```bash
GET /products?filter[category]=books&filter[status]=available
```

This makes it clear that these are filtering parameters, keeping it separate from
pagination, sorting, or any response modifiers which may be present.

Sometimes, users want to combine multiple filters. This is generally done by
adding more parameters to the URL:

```bash
GET /orders?filter[status]=shipped&filter[customer_id]=123
```

Using multiple filters is always considered a logical `AND` and the filters
should be combined. Supporting a logical `OR` is trickier to represent in a
query string, but one common convention is to allow multiple values for a single
parameter with a comma-separated list:

```bash
GET /products?category=books,electronics
```

This would return products in either the "books" or "electronics" categories.

### Declaring Operators

Simple value matching is the most common form of filtering, but it might not be
enough depending on the use-cases clients expect. For example, filtering for books with a
price of `20` will ignore any books that cost `19.99`, which is probably not
very helpful.

```bash
GET /products?filter[price]=20
```

To solve this, use operators to specify the type of comparison, like
"less than", "greater than", or "not equal". These are usually implemented with
suffixes or specific words added to the parameter name. For example, `GET
/products?price_gt=50` would retrieve products where the price is greater than
50. Other common operators include:

- `_lt` for less than (e.g., `price_lt=20`)
- `_gt` for greater than (e.g., `price_gt=100`)
- `_gte` and `_lte` for greater than or equal to, and less than or equal to, respectively.

Some people are tempted to try and use operators as a prefix for the value, like
`GET /products?price=<20` but that gets fairly awkward if trying 'less than or
equal': `GET /products?price=<=20`, everything needs to be escaped, and its
impossible to read.

Sticking with the filter array approach, this can be made a little more
readable:

```bash
GET /products?filter[price][lt]=20
GET /products?filter[price][gt]=99
GET /products?filter[price][gte]=100
```

This is a little more verbose, but it's much easier to read and understand.

### Advanced Filtering

Instead of trying to invent a new approach, there are standards that can be
used to make this easier for everyone, like
[FIQL](https://datatracker.ietf.org/doc/html/draft-nottingham-atompub-fiql-00),
[RSQL](https://github.com/jirutka/rsql-parser), or
[OData](https://www.odata.org/getting-started/basic-tutorial/#queryData).

As an example, OData is a widely used standard that provides a consistent way to
query and manipulate data. It uses a specific syntax for filtering, which might
look like this:

```bash
GET /products?$filter=category eq 'books' and price lt 50
```

Here, `$filter` is the standard keyword for filtering, and `eq` is used for
equality, while `lt` means less than. You can combine multiple filters using
`and`, just like in the example above.

FIQL is a compact, text-based query language used for filtering. It uses
operators such as `==` for equality, `!=` for not equal, `<` and `>` for less
than and greater than, and `;` for AND logic. For example, a FIQL filter might
look like this:

```bash
GET /products?filter=category==books;price<20
```

This is a concise way to express complex filtering logic, making it useful for
more advanced APIs.

Another option is RSQL, which is a slightly more modern version of FIQL that is gaining popularity:

```bash
GET /products?filter=category==books,price<50
```

RSQL uses a comma to separate filters, which is a little more readable than the
semicolon and doesn't need to be URL encoded. It can make some amazing queries
like `last_name==foo*,(age=lt=55;age=gt=5)`.

Whichever of these formats is picked it will have pros and cons, but the most
important thing is to pick a standard instead of reinventing the wheel, to leverage existing libraries and tools on both the client-side and the
server-side. It's important to reuse existing tools for things like this instead
of wasting infinite time building and maintaining custom solutions
instead of solving genuine problems for users.

## What is Sorting?

What order to return resources in a collection?

- Oldest first or newest first?
- Alphabetical based on the name?
- Highest price to lowest price?

Whichever is picked at first may be a sensible default, but it's likely that users
will want to change this.

For APIs, sorting is the process of arranging resources in a specific order
based on user inputs.

## How to Sort

Sorting is usually done with a query parameter:

```bash
GET /products?sort=name
```

This sorts products by the `name` property, and by default that will be in ascending order.

Most APIs will also allow clients to specify the order, which is usually done with another query parameter:

```bash
GET /products?sort=price&order=desc
```

Here if we just had `sort=price` it would be reasonable to assume the client
wanted the cheapest results, but if we're looking for the most expensive
products, we can add `order=desc` to return the most expensive first.

This convention is very closely related to the SQL `ORDER BY` clause, which
takes a database property and an order in exactly the same way. Unlike a
database query the API does not have to allow clients to sort by every single
property, it could be restricted to a few common use-cases and make sure they are
well optimized.

## Best Practices

### Consistency and Documentation

When designing filters for the REST API, it's important to make sure they are
intuitive and consistent. Use clear, descriptive names for the parameters. For
example, `price_lt` is much easier to understand than something vague like
`lower_price`. Providing solid documentation is equally important. Developers
should be able to quickly find information on the available filters and how to
use them.

### Validation and Error Handling

Validation is also critical. If a user tries to apply a filter with invalid data
(like `price=abc`), the API should return a helpful error message rather than
just failing silently or returning incorrect results. Be sure to handle edge
cases as well, such as empty values or invalid characters in the query string.

_Learn more about [error handling in REST APIs](/api-design/errors)._

### Performance Considerations

The more that clients are allowed to customize their requests, the harder it
becomes to set up caching rules and optimize database queries that might be
produced.

Anyone using an SQL database will know that the more complex the query, the
harder it is to optimize. If clients are allowed to send in completely arbitrary
queries, it's going to be very hard to optimize the database because it won't be
clear what indexes to create. Being left retroactively optimizing for popular
usages, which might be ok for an internal API used by a limited number of
colleagues who can warn of popular usages, but is a nightmare for teams
maintaining public APIs where an API could be brought down by a single user
launching a new product.

Rate limiting can help, but it's worth questioning: what is the purpose of this
API?

Generally an API is not meant to be a database-over-HTTP, so if the API design
feels like it is starting to recreate SQL or some other query language, it might
be going down the wrong path. There are databases which can be used over HTTP
that do not require creation of a database, like FaunaDB, Firebase, or DynamoDB,
which might be a better fit.

### URL Design

Sometimes a filter could or should have been a different endpoint, a different
parameter, or a different way of structuring the data.

If the clients have asked for the ability to show off some "Hot Bargains",
instead of telling clients to pick numbers based on price with `GET
/products?price_lt=20&sort=price`, why not make it quicker and easier for
everyone by creating `GET /products/bargains`.

Cachability is improved, because you can set a 24 hour [network
cache](/api-design/caching) on that
which will be shared by all clients.

Consistency is improved, because the web and iOS versions of the same
application aren't going to pick slightly different numbers for what is
considered a bargain.

## Summary

Filtering is a powerful tool for API designers, allowing users to request only
the data they need. By using query parameters, operators, and standard query
languages, you can create a flexible and intuitive filtering system that meets
the needs of your users, without going overboard and confusing everyone or
making the API wildlife inefficient and unstable.

When in doubt, start simple, and add things later. It's always easier to add new
parameters, endpoints, and additional ways of doing things, than it is to take
them away later.