LLMOps and AI evaluation engineer
Learn to test, monitor and cost-control AI features in production, using AI coding tools from week 1 and building the evaluation and monitoring around an AI app yourself, in 20 weeks.
Coming soon · planned, not written yet
- 20
- weeks planned
- ~30 h
- a week
- 5
- milestone projects
- 1
- project, grown every week
These are planned figures. The weeks are written after we check the tools against current job postings and official documentation, so the final route may differ.
Built for the AI era
How AI fits this roadmap
Teams now ship features that call language models, and those features fail in ways ordinary tests do not catch: wrong answers, drift after a prompt change, slow calls and surprise bills. This roadmap teaches you to do the job with an AI coding assistant every week, and to check its code with tests and review. It also teaches the job itself: writing evaluations, tracing every call, and putting limits on cost and quality.
The main project
EvalDesk
A small question-answering AI service that you wrap with test sets, tracing, a cost gateway, release checks and alerts, until a bad change is caught before users see it.
What you will be able to do
- 01
Build a small AI service in Python that calls a language model, working with an AI coding assistant while you read and test every change it writes.
- 02
Write a test set of real cases for the service and score its answers automatically, including with a model acting as judge, and check the judge against your own labels.
- 03
Trace every model call with Langfuse and OpenTelemetry so you can see the prompt, the answer, the delay and the cost of any request.
- 04
Route calls through a LiteLLM gateway with budgets, fallbacks and a spend report per key.
- 05
Block a bad prompt or model change in a CI pipeline using promptfoo or Ragas results, and write up one incident review from a real regression.
01The route
Five months, one month at a time.
Each month ends in a milestone project, so you always have something to show for the work.
Month 1 · Weeks 1–4
A working AI service and AI-assisted coding
Python services, calling a model API, prompts, structured output and Git. You set up an AI coding assistant, learn to prompt it, read its diffs and test what it writes.
MilestoneEvalDesk answering questions from a small document set, with tests and a log of AI suggestions you accepted and rejected
Month 2 · Weeks 5–8
Evaluation
Build test sets from real cases, write scoring rules, use a model as judge and check it against human labels. You use promptfoo and Ragas, and ask an AI assistant to draft cases that you then review.
MilestoneA versioned test set and a one-command report that scores every EvalDesk answer
Month 3 · Weeks 9–12
Tracing and monitoring
Trace each call with Langfuse and OpenTelemetry, record latency, errors and feedback, and sample live traffic into new test cases. You learn to find a bad answer from a trace.
MilestoneA trace dashboard and alerts for EvalDesk, with live answers scored against your rules
Month 4 · Weeks 13–16
Cost, speed and safety
A LiteLLM gateway with budgets, fallbacks and caching, plus red-team tests for prompt injection and unsafe output with promptfoo. You compare models on quality against cost and delay.
MilestoneEvalDesk behind a gateway with spend limits, a model comparison and a safety test set
Month 5 · Weeks 17–20
Release gates, capstone and job search
Run evaluations in CI so a weak prompt or model change cannot ship, practise an incident review, then build the portfolio, interview preparation and applications.
MilestoneA CI gate that blocks a regression, one incident review and a portfolio write-up
03What is inside
Everything a week comes with.
Every week is built the same way, so once you have done one you know how all of them work.
- 01
One project, EvalDesk
You grow one AI service and its testing and monitoring, so every tool lands in something real.
- 02
A day-by-day plan
Each week has a Learn part and a Do part, with a checklist so you know when you are done. Most weeks include a task to do with an AI assistant and a step to check its work.
- 03
A project in every milestone
Five milestones, each ending in something you can show and link to.
- 04
Reference topics
Short topic pages with check questions you can come back to.
- 05
A job track
Interview practice, portfolio and resume help, and an application plan.
- 06
Progress in your Study plan
Add weeks to your Study plan, give them dates, and finished days are counted for you.
04Is it for you
Who this is, and is not, for.
Better to know before you start than six weeks in.
It is for you if
- You can write Python and have built or used a web service, and you want to work on AI features in production.
- You are a backend engineer, tester, DevOps or data engineer who wants to be the person who proves an AI feature works.
- You can give about 30 hours a week for five months. This is a part-time pace.
It is not for you if
- You have never written code. Do a beginner course first.
- You want a job guarantee. Nobody can promise that, and this roadmap does not.
- You want to train models from scratch. This roadmap is about running and checking AI features, not research.
- You want to paste AI output into production without reading it. This roadmap trains you to check it.
?Questions
Before you start
- Is this roadmap ready?
- Not yet. This is the plan. The weeks are being written and checked against current job postings and official documentation. Leave your email and we will write to you when it opens.
- What will it cost?
- The price is not set. Roadmaps are bought once, with no subscription. We will announce it when this one opens.
- Will the plan change?
- It may. Before the weeks are written we check the tools against current job postings, so the tools, lengths and months here are planned, not final.
- Will I pay for model API calls?
- Probably a small amount. The roadmap will aim to keep calls cheap with small test sets and local or low-cost models where it works, and will show how to set a spend limit first. The details come when the weeks are written.
- Will it get me a job?
- It will give you skills and proof of work. It cannot promise a job on a fixed date, because hiring depends on the market, your location and your past work.
Not open yet
Leave your email and we will write once, when it opens.