Data Engineer in 2027: The One-Page Plan

If someone told you to "just get DP-203", that exam can't be booked anymore. This page is the replacement plan. For the full certification landscape across every Azure track, see the Azure Certification Roadmap 2027.

🚨 What Changed

☕ My take: I moved 500 TB of healthcare data into Azure. The lesson that carries into 2027 is that the platform names change faster than the work. Ingest it, clean it, shape it, secure it, schedule it. Both new exams test that same work on different tools.

🗺️ The Plan at a Glance

Timings assume 6 to 8 hours a week next to a full-time job.

🧱 Stage 1: The Five Shared Skills

Same five skills, two platforms. Learn them platform-neutral first and a later switch costs you a tool, not a career.

The medallion layers in one line each

• 🥉 Bronze: raw data, kept exactly as it arrived, so you can always rebuild.
• 🥈 Silver: cleaned data. This is where your data quality rules live.
• 🥇 Gold: shaped for the people who query it.

Microsoft calls this the recommended design approach for Fabric, and Databricks uses the same pattern.

Where to learn each skill for free

⚖️ Stage 2: Pick Fabric or Databricks

You need one platform properly, not two halfway.

The job-postings method

1. Open the data engineering job postings in your city.
2. Count how many name Fabric and how many name Databricks.
3. Pick the bigger number. If it's a tie, pick the one your current employer uses.
4. Start that platform's free option the same week.

Picked the "wrong" one? The five skills carry across. Use the translation table below and you're mostly renaming things.

Fabric to Databricks: term by term

Closest equivalents, not exact twins. Each row links to the Microsoft Learn page behind it.

🏗️ Stage 3: The Medallion Project

One public dataset, bronze to gold. Pick something that updates often and is a little messy: city open data, transit trips or weather records.

The 7-step checklist

• 1. Dataset. A public dataset that updates regularly. You can say what's in it and what's wrong with it.
• 2. Bronze. Raw files landed exactly as they arrive. A bronze table nobody edits by hand.
• 3. Silver. Types fixed, duplicates removed, nulls handled. Quality rules you can point to.
• 4. Gold. One or two aggregated tables that answer real questions an analyst would ask.
• 5. Schedule. The whole flow runs on a schedule without you.
• 6. Git. Notebooks and pipeline definitions in a repository someone else could read.
• 7. Decisions. A short write-up of why each layer is shaped the way it is. This is your interview answer.

The core code (runs in Fabric or Databricks)

from pyspark.sql import functions as F
 
# Bronze: land the raw file as-is, with when and where it came from
raw = (spark.read.option("header", True).csv("Files/raw/trips/*.csv")
       .withColumn("_ingested_at", F.current_timestamp())
       .withColumn("_source_file", F.col("_metadata.file_path")))
raw.write.format("delta").mode("append").saveAsTable("bronze_trips")
 
# Silver: fix types, drop duplicates and rows that fail basic checks
silver = (spark.table("bronze_trips")
          .withColumn("trip_date", F.to_date("pickup_datetime"))
          .withColumn("fare", F.col("fare").cast("decimal(10,2)"))
          .dropDuplicates(["trip_id"])
          .filter(F.col("fare") >= 0)
          .select("trip_id", "trip_date", "fare"))
silver.write.format("delta").mode("overwrite").saveAsTable("silver_trips")
 
# Gold: shape it for the people who use it
gold = (spark.table("silver_trips")
        .groupBy("trip_date")
        .agg(F.count("*").alias("trips"), F.sum("fare").alias("revenue")))
gold.write.format("delta").mode("overwrite").saveAsTable("gold_daily_revenue")

Once the first full load works, switch silver to an incremental MERGE in Spark SQL so a rerun updates changed rows instead of rewriting the table. That one change also checks off the SQL "ready when" from Stage 1.

🎓 Stage 4: The Cert

The cert goes last so it confirms something you already built.

Both exams let you open Microsoft Learn during the exam for the occasional lookup. Don't plan on it for every question.

Certs that look related but aren't the pipeline cert

🧪 Free Ways to Practise

Fabric and Databricks both sit on top of Azure storage, databases, networking and secrets. CloudLearn has no Fabric or Databricks lab yet, so use the trial or Free Edition for the platform and these labs for the Azure pieces underneath.

Start with storage. Every bronze layer on either platform ends up as files in a storage account, and knowing how that account is secured is half of the governance skill.

Hands-on LabCloudlearn.io

Getting Started with Azure Storage Account

Create the storage account where raw data lands, in a real Azure environment, with nothing left running afterwards.

labs.cloudlearn.ioStart the lab

🧭 The Order, in One Line

Five skills, one platform, one project, then the cert. Same work as the DP-203 days, new tools, and a project that proves it before the exam does.