Data engineer
Learn to build, test and run the data pipelines behind every dashboard and AI feature, working with AI tools from week 1, in 24 weeks.
Coming soon · planned, not written yet
Its own role in the Stack Overflow survey since 2019Boot.dev analysis of Stack Overflow surveys
- 24
- weeks planned
- ~35 h
- a week
- 6
- milestone projects
- 1
- project, grown every week
These are planned figures. The weeks are written after we check the tools against current job postings and official documentation, so the final route may differ.
Built for the AI era
How AI fits this roadmap
AI coding assistants now write a lot of first-draft SQL and pipeline code, so the skill that matters is reviewing, testing and fixing what they produce. AI features also depend on clean, fresh, well-documented data, which is the data engineer's job. This roadmap teaches you to build pipelines with AI help from the first weeks, and to prepare data for search and retrieval over documents.
The main project
DataLine
A pipeline that pulls raw data from several sources into a warehouse, tests and documents it, and finally feeds a searchable AI assistant over that data.
What you will be able to do
- 01
Write and tune SQL queries over joins, windows and large tables, and check AI-written SQL against known answers.
- 02
Build Python pipelines that extract, clean and load data, with tests and clear failure messages.
- 03
Model a warehouse with dbt, with data tests and generated documentation.
- 04
Schedule and monitor pipelines with Apache Airflow, and fix a failed run from its logs.
- 05
Prepare documents for AI search by chunking them, storing embeddings in Postgres with pgvector, and measuring whether retrieval finds the right text.
01The route
Six months, one month at a time.
Each month ends in a milestone project, so you always have something to show for the work.
Month 1 · Weeks 1–4
SQL, Python and AI-assisted coding
Solid SQL and Python for data, Git, and the habit of using an AI assistant then testing what it wrote.
MilestoneA script that loads raw files into Postgres, with tests that catch wrong AI-written queries
Month 2 · Weeks 5–8
Databases and data modelling
How Postgres stores and indexes data, normal forms, star schemas, and reading query plans. You ask an AI assistant to design a schema, then judge it against the plan.
MilestoneA modelled schema with fast, explained queries
Month 3 · Weeks 9–12
Pipelines and orchestration
Batch pipelines in Python, retries and idempotency, and scheduling with Apache Airflow in Docker. You use an AI agent to draft tasks, then review and test every one.
MilestoneA scheduled pipeline that recovers cleanly from a failed run
Month 4 · Weeks 13–16
Warehouse and transformations
A warehouse layer built with dbt, data tests, documentation, and local analytics with DuckDB. AI drafts models and docs; your data tests decide what is kept.
MilestoneA tested, documented warehouse that a dashboard could sit on
Month 5 · Weeks 17–20
Streams, scale and data quality
Event streams with Apache Kafka, larger-than-memory data, monitoring freshness, and handling bad data. You use AI to help read logs and failures, and check its diagnosis.
MilestoneA live feed into the warehouse with freshness and quality alerts
Month 6 · Weeks 21–24
Data for AI and capstone
Chunking, embeddings and pgvector search over your data, checking AI answers against sources, then a portfolio and interview practice.
MilestoneA searchable AI assistant over DataLine, with a retrieval test set and a portfolio write-up
03What is inside
Everything a week comes with.
Every week is built the same way, so once you have done one you know how all of them work.
- 01
One project, DataLine
You grow one pipeline from a script to a scheduled, tested warehouse with an AI search layer, so every tool lands in something real.
- 02
A day-by-day plan
Each week has a Learn part and a Do part, with a checklist so you know when you are done.
- 03
A project in every milestone
Six milestones, each ending in something you can show and link to.
- 04
Reference topics
Short topic pages with check questions you can come back to.
- 05
A job track
Interview practice, portfolio and resume help, and an application plan.
- 06
Progress in your Study plan
Add weeks to your Study plan, give them dates, and finished days are counted for you.
04Is it for you
Who this is, and is not, for.
Better to know before you start than six weeks in.
It is for you if
- You can write small programs in some language and want to work on data, not websites.
- You are a developer, analyst or tester moving into data engineering.
- You can give about 35 hours a week for six months. This is a part-time-plus pace, not a full-time bootcamp.
It is not for you if
- You have never written code. Do a beginner course first.
- You want to train machine learning models. A separate roadmap will cover that.
- You want a job guarantee. Nobody can promise that, and this roadmap does not.
?Questions
Before you start
- Is this roadmap ready?
- Not yet. This is the plan. The weeks are being written and checked against current job postings and official documentation. Leave your email and we will write to you when it opens.
- What will it cost?
- The price is not set. Roadmaps are bought once, with no subscription. We will announce it when this one opens.
- Will the plan change?
- It may. Before the weeks are written we check the tools against current job postings, so the tools, lengths and months here are planned, not final.
- Will it get me a job?
- It will give you skills and proof of work. It cannot promise a job on a fixed date, because hiring depends on the market, your location and your past work.
- Do I need a paid cloud warehouse?
- No. The core work runs on your own machine with Postgres, Docker and DuckDB. Cloud warehouses may be added later, using free tiers where possible.
Not open yet
Leave your email and we will write once, when it opens.