You'll leave with: A clear picture of what you're working with and what the real failure modes are.
You shipped an AI feature. It summarized support tickets, suggested replies, extracted structured data from unstructured input. It worked in testing. It looked clean in the demo.
Then a user submitted something slightly unusual. The model ignored your JSON schema and returned markdown. Another user asked a question outside the scope of your system prompt. The model answered anyway — confidently, incorrectly. You tweaked the prompt. It got better for that case and broke for three others.
You have no stack trace. No error log. Nothing to diff. The code didn't change. The model did something you didn't ask it to do, and you have no idea how to reason about it.
This is not a bug in the traditional sense. It's a failure mode that has no equivalent in deterministic software — and most developers walk into it with no mental model for what's actually happening.
That's what this chapter builds. Not a tutorial on API calls. A framework for understanding what you're working with and why it behaves the way it does.
There's a common misconception worth clearing up first.
Many developers think AI engineering means training models: datasets, GPUs, gradient descent, loss functions. That's ML engineering — the discipline of building and training models from data. It's a completely different skill set.
Many others think AI engineering is just calling an LLM API with a good prompt and wiring it into an app. That undersells it dramatically.
AI engineering is the discipline of building reliable applications on top of pre-trained foundation models. You don't train the model. You adapt it — using prompts, context design, retrieval, evaluation, and occasionally finetuning — to do something useful and consistent for a specific use case.
Chip Huyen, in AI Engineering (O'Reilly, 2025), defines the distinction precisely. ML engineering bottleneck: getting a model to learn the right thing from data. AI engineering bottleneck: getting a pre-trained model to behave reliably in your specific application.
Those are genuinely different problems. They require different tools, different instincts, and different mental models.
Where you already have an advantage. If you've spent your career in software engineering, you're closer to AI engineering than you think. API integration, systems design, working with unreliable third-party dependencies, production debugging, performance optimization — all of it transfers. The skills that don't transfer directly: the assumption that your software behaves deterministically.
To build AI applications well, you need a clear picture of what you control and what you don't. The AI stack has three layers:
┌─────────────────────────────────────────┐
│ APPLICATION LAYER │
│ Your product. UI, business logic, │
│ integrations. Code you've written │
│ your whole career. │
├─────────────────────────────────────────┤
│ INTERFACE LAYER │
│ Prompt architecture, context design, │
│ retrieval, sampling parameters, │
│ structured outputs, guardrails. │
│ Your primary engineering surface. │
├─────────────────────────────────────────┤
│ MODEL LAYER │
│ Foundation model weights, inference │
│ infrastructure, post-training │
│ decisions. Mostly not yours to touch. │
└─────────────────────────────────────────┘
The model layer is what foundation model companies (OpenAI, Anthropic, Google, Meta) build. It includes the weights, the hardware to run them, and the post-training decisions that shape how the model responds to instructions. As an AI engineer, you choose between models. You don't modify them.
The interface layer is your primary engineering surface. Everything you control about how you communicate with the model lives here: what's in the system prompt, how you structure context, what documents you retrieve and how you format them, whether you constrain outputs to JSON. Most quality wins happen here. Most production failures happen here too.
The application layer is what you've always built: UI, APIs, integrations, authentication, data pipelines. The difference is that this layer now contains a probabilistic component. One of your dependencies — the model — doesn't behave deterministically.
Most "build an AI app" tutorials focus on the application layer (wiring the API into your code) while skipping the interface layer almost entirely. That's the gap this course fills.
The easiest way to fail at AI engineering is to build the wrong thing. Foundation models are impressive enough that it's easy to get a demo working for almost any use case. A demo that works 80% of the time in controlled conditions can fail catastrophically in production.
Before you commit to an AI feature, ask four questions.
For some applications, occasional wrong answers are acceptable: autocomplete suggestions, content recommendations, internal productivity tools. A bad suggestion is annoying. Nothing breaks.
For others, a bad answer is a liability: medical triage, legal document review, financial decisions, security-sensitive operations. If your failure mode has real-world consequences, you need to define and measure those failure modes before you ship — not after a user encounter triggers the crisis.
Knowing your failure tolerance determines your entire evaluation strategy.
LLMs are strong at tasks where the correct answer lives in their training data: writing assistance, code generation, summarization, general knowledge Q&A, translation, document classification.
They're unreliable for tasks requiring:
A customer support bot that doesn't know your current pricing is worse than no bot — it confidently gives wrong answers.
"It seems to work" is not an acceptance criterion. You need to define correct before you start building — not because you'll always hit it perfectly, but because without a definition you can't evaluate whether you've improved, and you can't detect when a model update breaks something that was working.
For classification tasks this is straightforward: label a test set, measure accuracy. For generative tasks it's harder — there's no single correct answer for a summary or a reply suggestion. We'll spend an entire chapter on evaluation methodology. For now, the key principle: if you can't define what correct looks like, you can't engineer toward it.
LLM API calls are not free, and they're not fast.
At the time of writing (early 2025):
A single call processing a 10,000-token context with GPT-4o costs $0.025. At 1,000 calls per day that's $750/month. At 100,000 calls per day it's $75,000/month.
Model selection is an engineering decision. The most capable model is rarely the right default — and the performance gap between frontier models and faster, cheaper alternatives is smaller than benchmarks suggest for most real-world tasks. We'll cover model selection in Chapter 3.
Here's the concept that changes how you build everything.
A function takes inputs, processes them deterministically, returns outputs. Given the same inputs you always get the same outputs. You can write tests. You can mock dependencies. You can trace bugs to specific lines of code.
LLMs are not functions.
The same prompt sent twice returns different outputs. Model behavior changes between API versions — often silently, without a version bump. A prompt that produced clean JSON yesterday might produce JSON wrapped in a markdown code block today after an unannounced update. The model will confidently assert things that are false. It will sometimes refuse instructions it previously followed.
None of this is a bug. It's the architecture.
LLMs are autoregressive probability distributions. At each step, the model generates the next token by sampling from a probability distribution over its entire vocabulary — roughly 100,000 possible tokens. That distribution is shaped by everything the model has seen: the system prompt, the conversation history, the retrieved documents, the current message. Temperature controls how peaked or flat the distribution is:
The practical consequence: AI systems require a different approach to reliability engineering.
You cannot unit test your way to a reliable AI feature. You need:
We'll build all of this in Chapter 3. For now, internalize the principle: treat the model as a probabilistic dependency, not a deterministic function. This reframe changes your architecture, your testing strategy, and your definition of "done."
With that mental model in place, here's what the actual work looks like:
1. Define the task precisely. Not "AI-powered summaries" — that's a feature. A task has a defined input, output, and definition of correct. "Given a support ticket (input), generate a one-paragraph summary (output) that includes the reported issue, the user's emotional state, and the resolution status, without inventing any details not in the original ticket (definition of correct)."
2. Build an evaluation set first. Before you write a single prompt, collect 50–200 representative examples with known-good outputs. This is the ground truth you'll evaluate against. Building it forces you to sharpen the task definition. If you can't label 50 examples, you haven't defined the task precisely enough.
3. Design the interface layer. System prompt architecture, context structure, output format constraints. We cover this in Chapter 4.
4. Choose a model by evaluating candidates. Run your evaluation set against 2–3 models. Pick based on performance on your data, not on public benchmarks like MMLU or HumanEval. Those benchmarks were not designed for your task.
5. Iterate on the interface layer. Improve prompt, context, and structure until your eval pass rate reaches acceptable. "Acceptable" was defined in step 1.
6. Ship with monitoring. Track output quality in production — not just uptime and latency. We cover this in Chapter 6.
7. Close the loop. Route production failures back into your evaluation set. The system improves over time.
Notice what's not in this workflow: "pick the most capable model and start building." The developers who ship the best AI systems run evaluations first, cheapest model last — and almost always land on something faster and cheaper than their initial instinct.
The thing that makes AI engineering genuinely hard is also what makes it tractable.
LLMs are hard to make reliable because they're probabilistic, opaque, and sensitive to small changes in context. You can't read the stack trace.
But they're tractable because the interface is natural language. You don't need to change the model to change the behavior. You change what you put in the context. That's a much faster iteration loop than training or fine-tuning. A prompt change takes seconds. A training run takes hours or days and costs thousands of dollars.
The developers who are best at AI engineering have figured out how to iterate rapidly at the interface layer — with evaluation infrastructure that tells them immediately whether a change made things better or worse.
That's the skill this course builds.
You now have the frame. The next five chapters fill it in.
Chapter 2 goes one layer down: what foundation models actually are, how post-training shapes their behavior, and what implications this has for what you can reliably ask of them.
Chapter 3 — evaluation — is where most of the professional leverage is. If you skip ahead, skip to that one.
Primary source: Chip Huyen, AI Engineering: Building Applications with Foundation Models (O'Reilly, 2025), Chapters 1–2.
This course is the mental model. The community adds production code repos, AI in a Shell (structured learning app + AI tutor), weekly engineering calls, and monthly 1:1 sessions.