Last update: 2025-05-22T22:35:06.000Z
Content:
---
title: "Build a lambda architecture in Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn a useful alternative processing pattern for when the typical Tinybird flow doesn't fit."
inkeep:version: "forward"
---




# Build a lambda architecture in Tinybird [¶](https://www.tinybird.co/docs/forward/work-with-data/optimize/guides/lambda-architecture#build-a-lambda-architecture-in-tinybird)

Copy as MD In this guide, you'll learn a useful alternative processing pattern for when the typical Tinybird flow doesn't fit.

This page introduces a useful data processing pattern for when the typical Tinybird flow (Data Source --> incremental transformation through Materialized Views --> and API Endpoint publication) doesn't fit. Sometimes, the way Materialized Views work means you need to use **Copy Pipes** to create the intermediate Data Sources that will keep your API Endpoints performant.

## The ideal Tinybird flow [¶](https://www.tinybird.co/docs/forward/work-with-data/optimize/guides/lambda-architecture#the-ideal-tinybird-flow)

You ingest data (usually streamed in, but can also be in batch), transform it using SQL, and serve the results of the queries via [parameterizable](/docs/forward/work-with-data/query-parameters) API Endpoints. Tinybird provides freshness, low latency, and high concurrency: Your data is ready to be queried as soon as it arrives.



<-figure->
![Data flow with Data Source and API Endpoint](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-1.png&w=3840&q=75)

<-figcaption->
Data flow with Data Source and API Endpoint

</-figcaption->


</-figure->
Sometimes, transforming the data at query time isn't ideal. Some operations - doing aggregations, or extracting fields from JSON - are better if done at ingest time, then you can query that prepared data. [Materialized Views](/docs/forward/work-with-data/optimize/materialized-views) are perfect for this kind of situation. They're triggered at ingest time and create intermediate tables (data sources in Tinybird lingo) to keep your API Endpoints performance super efficient.



<-figure->
![Data flow with Data Source, MV, and API Endpoint](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-2.png&w=3840&q=75)

<-figcaption->
Data flow with Data Source, MV, and API Endpoint

</-figcaption->


</-figure->
The best practice for this approach is usually having a Materialized View (MV) per use case:



<-figure->
![Materialized Views for different use cases](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-3.png&w=3840&q=75)

<-figcaption->
Materialized Views for different use cases

</-figcaption->


</-figure->
If your use case fits in these first two paragraphs, stop reading. No need to over-engineer it.

## When the ideal flow isn't enough [¶](https://www.tinybird.co/docs/forward/work-with-data/optimize/guides/lambda-architecture#when-the-ideal-flow-isnt-enough)

There are some cases where you may need intermediate Data Sources (tables) and Materialized Views don't fit.

- Most common: Things like Window Functions where you need to check the whole table to make calculations.
- Fairly common: Needing an Aggregation MV over a deduplication table (ReplacingMergeTree).
- Scenarios where Materialized Views fit but aren't super efficient (hey `uniqState`   ).
- And lastly, one of the hardest problems in syncing OLTP and OLAP databases: Change data capture (CDC).

Want to know more about *why* Materialized Views don't work in these cases? [Read the docs.](/docs/forward/work-with-data/optimize/materialized-views#limitations)

As an example, let's look at the *Aggregation Materialized Views over deduplication DS* scenario.

Deduplication in Tinybird happens asynchronously, during merges, which you can't force in Tinybird. That's why you always have to add `FINAL` or the `-Merge` combinator when querying.

Plus, Materialized Views only see the block of data that is being processed at the time, so when materializing an aggregation, it will process any new row, no matter if it was a new id or a duplicated id. That's why this pattern fails.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-4.png&w=3840&q=75)

<-figcaption->
Aggregating MV over deduplication DS will not work as expected

</-figcaption->


</-figure->
## Solution: Use an alternative architecture with Copy Pipes [¶](https://www.tinybird.co/docs/forward/work-with-data/optimize/guides/lambda-architecture#solution-use-an-alternative-architecture-with-copy-pipes)

Tinybird has another kind of Pipe that will help here: [Copy Pipes](/docs/forward/work-with-data/optimize/copy-pipes).

At a high level, they're a helpful `INSERT INTO SELECT` , and they can be set to execute following a cron expression. You write your query, and (either on a recurring basis or on demand), the Copy Pipe appends the result in a different table.

So, in this example, you can have a clean, deduplicated snapshot of your data, with the correct Sorting Keys, and can use it to materialize:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-5.png&w=3840&q=75)

<-figcaption->
Copy Pipes to the rescue

</-figcaption->


</-figure->
## Avoid loss of freshness [¶](https://www.tinybird.co/docs/forward/work-with-data/optimize/guides/lambda-architecture#avoid-loss-of-freshness)

*"But if you recreate the snapshot every hour/day/whatever... Aren't you losing freshness?"* Yes - you're right. That's when the lambda architecture comes into play:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-6.png&w=3840&q=75)

<-figcaption->
Lambda/Kappa Architecture

</-figcaption->


</-figure->
You'll be combining the already-prepared data with the same operations over the fresh data being ingested at that moment. This means you end up with higher performance despite quite complex logic over both fresh and old data.

## Next steps [¶](https://www.tinybird.co/docs/forward/work-with-data/optimize/guides/lambda-architecture#next-steps)

- Read more about[  Copy Pipes](/docs/forward/work-with-data/optimize/copy-pipes)  .
- Read more about[  Materialized Views](/docs/forward/work-with-data/optimize/materialized-views)  .


