Serverless query layer

Intent

This pattern assumes data held in a data lake, as in the serverless data lake layout, and an API that verifies who each caller is. It divides responsibility for querying by who owns each resource. The lake decides where queries run and where their results are written, the job that writes each dataset describes it in a catalogue, and everyone else queries through an API that accepts structured requests rather than SQL. This enables three outcomes:

1. Query results, which are copies of the data they return, are written only where the platform decides.
2. No caller can build a query that reaches data outside what they're allowed to see.
3. A new type of dataset can be queried as soon as its data is written, with no new code.

Motivation

When every service runs its own SQL with its own settings, querying fails in three ways:

1. Results end up anywhere. Every query result is a copy of the data it returns, and when each service chooses where to write it, copies build up in storage nobody governs.
2. Nobody can see who is querying what, or what it costs, because every service queries in its own way.
3. One badly built query string can reach data it shouldn't, because SQL assembled from a request is open to SQL injection.

This pattern gives each decision to whoever owns the resource it affects. The lake owns the query engine's settings, such as an Amazon Athena workgroup for each consumer, which fixes where results are written and how they're encrypted, and which no caller can override. The job that writes a dataset also registers it in a catalogue such as the AWS Glue Data Catalog, with its columns taken from the dataset's agreed schema rather than worked out from the data, so the catalogue always matches the data. Callers send a structured request, naming the columns, filters and grouping they want, which the API checks against that schema before turning it into SQL. Because callers never write SQL themselves, there's nothing for them to inject, and the API decides which data a query may read from the caller's verified identity, such as the client they belong to, so no request can reach anyone else's data. Small queries return their results straight away, and large ones run in the background, with the caller collecting the results when they're ready.

Applicability

Use this pattern when:

1. People, services or AI agents need to query a data lake.
2. You need control over where queries run, and where their results are written.
3. Each caller may reach only some of the data.

Do not use this pattern when:

1. Callers need low-latency lookups or transactional reads, which belong in a database, since a query over a lake takes seconds.
2. Analysts need to write their own SQL across the lake. Give them direct access to the query engine instead, under settings of their own that control where results are written, such as a separate Athena workgroup.

Structure

Participants

Constraints and trade-offs

1. Structured queries are less expressive than SQL. Callers can ask only what the query language supports, and each new kind of question means extending it.
2. Large queries take an extra step, because the caller has to come back for the results rather than getting them in one request.
3. Every writing job has to register its own tables, which is more work than a tool that scans the data and works out each table's schema for itself, such as an AWS Glue crawler. That kind of tool needs no code, but it can only describe the data it finds, so a column that no row has filled yet is left out of the table.

Part of how I deliver

This is how I let people, services and AI agents ask questions of a data lake without handing any of them raw SQL. It's one of many patterns I bring through contract delivery, and it answers an AI agent as safely as it answers your own services.

Related