# Why API producers should care about JSONL
Source: https://speakeasy.com/blog/why-api-producers-should-care-about-jsonl

When end users expect near-instant responses (as they do), API producers reach for streaming responses.

![An image macro shows Pablo Escobar (from the series, Narcos) waiting in three different locations. The meme text reads, "ME EXACTLY 100MS AFTER CLICKING SEND IN YOUR APP," implying the meme represents users' frustration with app waiting periods.](/assets/blog/why-api-producers-should-care-about-jsonl/jsonl/pablo-waiting.jpg)

Yet, many APIs still force users to wait for complete responses before displaying anything, creating that dreaded spinner experience that kills engagement and conversion rates.

We believe that streaming responses will become the norm for most APIs, especially those that wrap or extend LLMs.

In this post, we'll directly address the common misconception that streaming is a complex and time-consuming feature to implement. If you're struggling to implement streaming, the problem is likely that you're ignoring the simplest tool in your arsenal: JSONL.

We'll compare traditional request-response implementations with streaming alternatives, demonstrating not just the user experience benefits of streaming but also the surprisingly minimal code changes it requires. If your team has been putting off streaming support because it seems like a major project, we hope this guide will show you a faster path forward.

## What is JSONL?

JSON Lines (JSONL) is a text format that represents a sequence of JSON objects, with one object per line.

It looks like this:

```jsonl
{"name": "Labrador", "size": "large"}
{"name": "Wire Fox Terrier", "size": "medium"}
{"name": "Schnauzer", "size": "medium"}
```

## OK, but _why_ is JSONL even a thing?

As we'll discuss in more detail below, JSONL's many benefits boil down to one main point:

JSONL data can be processed or emitted record by record while it is read from a file or received over a network without needing to load the entire dataset into memory first.

This makes JSONL inherently streamable.

Let's break that down a bit:

### JSONL can be parsed line by line

There is no need to parse the entire dataset before you begin processing JSONL. This means you can start sending data to your users as soon as you have it, rather than waiting for the entire response to be ready. And users can start consuming data the moment they receive it, rather than waiting for the entire response to be sent. Each chunk you send is a complete response, after all.

### JSONL uses less memory on the server and client

Because only one line (representing a single JSON object or value) needs to be in memory for parsing at any given time, JSONL has a low memory footprint compared to the footprint for parsing a large standard JSON array. This makes it feasible to work with terabyte-scale datasets on machines with limited memory.

### We don't need to know the size of the dataset beforehand

Adding more records to a JSONL file is straightforward and efficient. You can append new JSON entries as new lines to the end of the file without needing to parse or modify the existing data. This is ideal for logging systems or any application where data is continuously generated.

### JSONL can be parallelized

Because each line in a JSONL file is independent, processing is easily parallelized. This means you can take advantage of multicore processors to process multiple lines simultaneously, significantly speeding up data processing tasks.

### JSONL plays well with Unix tools

The line-based nature of JSONL makes it compatible with a wide range of standard Unix command-line utilities like `grep`, `sed`, `awk`, `head`, and `tail`. This makes it easy to explore and filter data without needing to load it into a specialized tool or library. For example, you can use `grep` to quickly find all records that match a certain pattern or `head` to view the first few lines of a large dataset.

### JSONL is human-readable

Subjectively, JSONL is easier to read than a large JSON array. Each line represents a single record, so you can scan through the data without being overwhelmed by the sheer volume of information. This is especially helpful when debugging or exploring data. Copy a line, and you know you have a complete record - there's no risk of missing a closing bracket or comma.

### Error handling is easier

When processing a JSONL file line by line, an error encountered while parsing one line (for example, a malformed JSON object) does not necessarily invalidate the remaining lines in the file. Parsers can be designed to skip or log problematic lines and continue processing the rest of the data.

### It takes your dog for a walk

Just checking whether you're still with us. But seriously, it does make your code cleaner, which can lead to more relaxing time off. Who knows what else it might do for you?

## Why is JSONL important _now_?

![An image macro shows a screenshot from Toy Story. Buzz Lightyear gestures widely, with his other arm around a despondent Woody, and the text overlay reads "AI" at the top and "AI EVERYWHERE" at the bottom, suggesting the ubiquity of AI technology.](/assets/blog/why-api-producers-should-care-about-jsonl/jsonl/buzz-lightyear-ai.jpg)

Large Language Models (LLMs) generate text in chunks. This means that as LLMs produce output, they can stream text to users in real time, rather than waiting for the entire response to be generated.

Users expect this kind of responsiveness, and APIs that don't provide it risk losing users to competitors that do.

Streaming allows AI agents to respond to or act on data as it arrives, rather than requiring them to wait for long and expensive text completions to finish. This makes AI agents far more efficient and agile.

## This seems too fast and loose, where's the spec?

JSONL is a simple format, and as such, it doesn't have a formal specification. However, the [JSON Lines website](https://jsonlines.org/) provides a good overview of the format.

Here's the TL;DR:

- JSONL is always UTF-8 encoded.
- Each line is a valid JSON value.
- Lines are separated by a newline (`\n`) character.

While not prescriptive, the JSON Lines website also adds that the recommended file extension is `.jsonl` and that the MIME type may be `application/jsonl` (though this is not formalized yet, and `application/x-ndjson` is also used).

So, it seems more like a convention than a specification. But the fact that it is so simple and widely used means that you can rely on it being supported by most programming languages and libraries.

## Streaming vs traditional request-response

We'll ask an OpenAI model to generate some text, then do the same thing with streaming, and see what is sent over the wire.

### Requirements

This example assumes you have a working OpenAI API key set as an environment variable called `OPENAI_API_KEY`.

If you don't have one, you can sign up for a free [OpenAI account](https://platform.openai.com/signup), [get a new API key](https://platform.openai.com/api-keys), then run the following command to set your API key:

```bash