AI, for real

Does your AI actually work? Start here: the AI, for real series

Everyone is building with AI. Very few teams can prove it works. Here's the one loop this series teaches: trace it, test it, guard it, improve it.

This is the companion article to the AI, for real Short. Watch it on YouTube (@ai4real-chelvas).

Everyone is building with AI right now. Chatbots, agents, copilots, RAG apps. But ask a simple question, "do you actually know if your AI works?", and most teams go quiet. They've seen it work in a demo. They've tried a few prompts. That's not the same as knowing.

This series is about the one skill nobody talks about but everybody needs: observability and evals. Observability lets you see what your AI actually did. Evals tell you whether what it did was any good.

Why "it worked in the demo" isn't enough

AI systems fail quietly. They don't crash with a stack trace; they give a confident, wrong answer. A few real examples we cover in the series:

A chatbot that made up a policy. Air Canada's website chatbot told a grieving customer he could claim a bereavement refund after booking. The real policy said no. A tribunal made the airline pay (EP 1).
A model that quietly changed. Researchers measured the same GPT-4 service a few months apart and saw its accuracy on one task collapse from 97.6% to 2.4% (EP 2).
A bot talked into a $1 car. A dealership chatbot agreed to sell a brand-new SUV for one dollar after a user rewrote its instructions (EP 4).

None of these were model-quality problems you'd catch by trying a handful of prompts. They're process problems, and that's good news, because process problems have process fixes.

The loop: trace it, test it, guard it, improve it

The whole series builds one simple loop. Each episode is one step:

Test it (evals). Stop shipping on vibes. Read real outputs and write down what "good" means (EP 1). Then run those tests on a schedule, so a silent change can't slip past you (EP 2).
Trace it (observability). Record every step your AI takes, with the timing, tokens and cost, so when it fails you can see why (EP 3).
Guard it (guardrails). Check what goes into and comes out of the model before it reaches a user (EP 4).
Improve it (close the loop). Turn every failure you catch into a permanent test (EP 5).

Four moves, one loop: trace, test, guard, improve.

No boring theory, real tools

Everything in this series uses tools you can run yourself. I self-host the stack I talk about: open-source tracing (Langfuse) and open-source guardrails (Guardrails AI), on my own server. You don't need a big budget to do this properly; you need a habit.

By the end of the series, you'll know how to trust your AI, and more importantly, how to prove it does what you think it does.

Next: EP 1: You don't have evals. You have vibes.

The videos use my AI-cloned voice and an AI avatar of me. The research, scripts and opinions are mine, and every number here links to its source.