Event-driven data pipeline

Intent

This pattern assumes a platform that takes in data as files, and stores it in tiers, as in the serverless data lake layout. It treats every file as an event of its own. A queue receives each file, and a workflow first converts it from whatever format it arrived in to one common format, then runs it through a series of small, shared stages that each do one job, in parallel wherever they don't depend on each other. A tracking record follows the file from arrival to completion, so the platform and the people using it always know where it stands. This enables three outcomes:

1. A bad file fails on its own, without holding up or rerunning the rest.
2. Anyone can see where each file has got to, and why it stopped, without reading logs.
3. Supporting a new file type needs only a stage to convert it, because every stage after the conversion is shared.

Motivation

There are two common ways to process incoming files, and both fail as a platform grows:

1. Functions that call each other directly. When one fails partway through, nobody can say for certain where the file stopped, or whether it's safe to process it again.
2. One large batch job on a schedule. A single bad file holds up every other file in the batch, and recovering from it means running the whole batch again.

This pattern gives every file its own run, so a failure stays with the file that caused it. A queue such as Amazon SQS takes in files as they arrive, absorbs bursts and retries delivery, and sets aside any file that keeps failing in a dead-letter queue for someone to investigate. A workflow engine such as AWS Step Functions decides the order of the stages, retries them and handles branching, so each stage only has to do its own job, and can be tested on its own. The first stage converts the file from whatever format it arrived in to one common format, such as Parquet, so every stage after it is shared. Each stage records its outcome on the file's tracking record, which makes that record the single source of truth for where the file stands. Work that doesn't depend on other stages runs as parallel tasks, and each task catches and records its own failures, so one that fails never stops the others.

Applicability

Use this pattern when:

1. Files arrive throughout the day, one at a time, rather than in one scheduled batch, such as when a user uploads a file through your app, or another system sends one over.
2. Each file needs several processing steps, which must run automatically from the moment it arrives.
3. People need to see the progress of every file, and the reason any file failed.

Do not use this pattern when:

1. The data is a high-volume stream of small events, which a stream processor handles better.
2. All the data arrives in one nightly batch, which a scheduled job handles well.

Structure

Participants

Constraints and trade-offs

1. There are more moving parts than in a single job: a queue, a workflow and a function for each stage, each with its own permissions and deployment.
2. Progress is recorded in two places. The workflow's history holds the order in which stages ran, and the tracking record holds the file's progress, so each has to keep to its own purpose, or the two will disagree.
3. Retries and redeliveries run stages more than once, so every stage has to be idempotent, giving the same result however many times it runs.
4. It's designed for files, not streams. Each file is sent as its own message and processed in its own workflow execution, both charged per use, so a high volume of small events quickly becomes expensive.

Part of how I deliver

When a platform takes in files from many sources, this is the backbone I build to process them. It's one of many patterns I bring through contract delivery, and each new file type adds one stage rather than a new pipeline.

Related