How to build AI Data Pipelines on AWS

The problem it answers

Most businesses want a chatbot that knows their own data, not just the public internet. A model on its own is a prompt wrapper: it can't see your data, and handing it broad access is a risk nobody should sign off. The answer isn't a new AI platform. It's the event-driven pipelines you already run, with an agent as one more step, and tools that decide exactly what it can reach.

What you'll take away

By the end, you'll be able to:

• Plan an agentic pipeline from serverless patterns your platform team already runs, with the agent as one more step rather than a new platform to build.
• Map the pipeline onto AWS, GCP, Azure or on-premises services, so the design stays portable and the choice of cloud stays yours.
• Design the toolsets that give an agent exactly the access it needs to reason over your data, turning a prompt wrapper into an assistant the business can rely on.
• Justify an agent framework such as pydantic-ai, which enforces structured output and keeps your application loosely coupled to the model provider.
• Apply vector embeddings in the pipeline, storing similarity alongside the data, so related records can be found with no extra database.
• Select open formats at every layer, so no step locks you into one vendor.

See it working

Each pattern is shown working in a live demo: a bookshelf system where an agent reads book covers in conversation, a pipeline computes the similarities between books, and a chat agent answers questions about the whole shelf. I then open the code that defines the agent and its tools.

Previous and upcoming events

Events where I've run this, or will, with links to each event and its recording where there is one.