Why I Started Writing

I’ve been in data engineering long enough to have Googled the same obscure Databricks error three separate times across three different contracts, each time spending an hour rediscovering the same fix. This blog exists partly to stop doing that.

The other reason: most data engineering content online is either vendor marketing dressed up as a tutorial, or a “hello world” pipeline so stripped of real-world concern that it teaches you nothing. I’ve processed 172 million events per day. I’ve built production Databricks platforms with IaC from scratch. There are things I wish someone had written down when I was figuring it out.

So I will write them down.

What to expect

Posts here will be about things I actually deal with at work:

• Infrastructure as Code for Databricks — Bicep and Terraform, the parts the docs gloss over
• Delta Live Tables — when they’re worth the complexity and when they’re not
• Python patterns for data pipelines — the kind that don’t fall over when the data is messy, the volumes are high, or both
• Azure data services — Event Hubs, ADLS2, Key Vault, Unity Catalog, and how they fit together in a real architecture
• Occasionally: LLMs in production — applied NLP and transformer models in non-demo settings

No beginner tutorials for things that already have official documentation. No SEO padding. Short posts that say what they mean.

Background

I’m a Senior Data & Cloud Engineer based in Groningen, currently contracting at RIVM. Previously Shell (cyber threat detection pipelines), IKEA (Databricks platform rebuild), and others. MSc in Information Science from the University of Groningen, where I specialized in NLP and machine learning.

If any of this sounds useful to you, stick around.