Delta Live Tables: When to Use Them, When to Skip Them

Delta Live Tables is polarising in the Databricks community. Some teams swear by it; others have migrated away. Having used it at Shell and RIVM in different contexts, I have a fairly concrete view of where it earns its keep and where it gets in the way.

What DLT actually gives you

The core value proposition is declarative pipeline orchestration with built-in quality enforcement:

import dlt
from pyspark.sql.functions import col, current_timestamp

@dlt.table(
    comment="Raw ingested events from Event Hub",
    table_properties={"quality": "bronze"},
)
def events_raw():
    return (
        spark.readStream
        .format("eventhubs")
        .options(**event_hub_conf)
        .load()
    )

@dlt.table(comment="Parsed and validated events")
@dlt.expect_or_drop("valid_event_id", "event_id IS NOT NULL")
@dlt.expect_or_drop("valid_timestamp", "event_timestamp > '2020-01-01'")
def events_silver():
    return (
        dlt.read_stream("events_raw")
        .select(
            col("body").cast("string").alias("raw_json"),
            col("enqueuedTime").alias("event_timestamp"),
            col("systemProperties.x-opt-sequence-number").alias("event_id"),
        )
    )

The @dlt.expect_or_drop decorators enforce data quality at the row level and surface failures in the DLT UI without crashing your pipeline. That’s genuinely useful when you’re ingesting from sources you don’t control.

What you actually get with DLT that’s hard to replicate

Automatic lineage. DLT tracks table dependencies across your pipeline and shows them visually. When something breaks, you can see exactly which upstream table caused it.

Incremental processing without foreachBatch gymnastics. DLT handles checkpointing and upserts via APPLY CHANGES INTO without you needing to manage merge logic yourself:

dlt.create_streaming_table("events_gold")

dlt.apply_changes(
    target="events_gold",
    source="events_silver",
    keys=["event_id"],
    sequence_by="event_timestamp",
    stored_as_scd_type=1,  # or 2 for history
)

Retry and recovery. Failed pipelines restart from the last good checkpoint automatically. With vanilla Structured Streaming you implement this yourself.

Where DLT gets in the way

Testing is awkward. You can’t easily unit-test DLT functions because they depend on the DLT runtime context. There are workarounds (abstracting the transformation logic out of the decorator), but they add boilerplate:

# Extract logic to a testable function
def parse_events(df):
    return df.select(
        col("body").cast("string").alias("raw_json"),
        col("enqueuedTime").alias("event_timestamp"),
    )

@dlt.table()
def events_silver():
    return parse_events(dlt.read_stream("events_raw"))

# In your test suite:
def test_parse_events(spark):
    input_df = spark.createDataFrame([...], schema)
    result = parse_events(input_df)
    assert result.columns == ["raw_json", "event_timestamp"]

Cost. DLT pipelines run on dedicated compute that you pay for even at low utilization. For simple pipelines on a tight budget, a scheduled notebook with explicit Delta merge logic is cheaper.

The abstraction leaks under pressure. When you need behaviour that DLT doesn’t expose — custom metrics, non-standard checkpointing, complex branching logic — you end up fighting the framework. At Shell we had one pipeline where DLT limited us more than it helped, and we eventually rewrote it as vanilla PySpark.

💡DLT vs Structured Streaming

DLT runs Structured Streaming under the hood. If you understand checkpoints, trigger intervals, and watermarks in Structured Streaming, DLT will make sense quickly. If you don’t, DLT will hide those concepts until something goes wrong — and then they’ll all surface at once.

My actual decision criteria

Use DLT when:

• You have a multi-hop medallion architecture (bronze → silver → gold) with meaningful quality rules at each layer
• Your source data quality is inconsistent and you want visibility into row-level failures without pipeline crashes
• Your team will benefit from the visual DAG and lineage tracking
• You’re on Unity Catalog — DLT integrates neatly with UC permissions

Skip DLT when:

• Your pipeline is simple (one or two transformations, predictable input)
• You need fine-grained control over retry logic or streaming triggers
• You’re cost-constrained and your pipeline can share compute with other jobs
• You need proper unit test coverage without fighting the framework

A pattern worth stealing

If you do use DLT, put your transformation logic in pure functions in a separate module and import them. Keep the DLT decorators as thin wrappers. Your pipeline becomes testable, your logic is reusable, and when you eventually need to move off DLT, you don’t have to rewrite the actual business logic.

pipelines/
├── dlt/
│   └── events_pipeline.py      # @dlt.table decorators only
├── transforms/
│   └── events.py               # pure transformation logic
└── tests/
    └── test_events_transforms.py

It’s a small structural decision that pays off every time someone asks “can we test this?”

Tailwind CSS has a reputation for making HTML look messy. After years of using it on production projects, I’ve landed on a set of patterns that keep things clean and maintainable. Here’s what actually works.

1. Use @layer components for repeated patterns

When a pattern appears more than twice, pull it into a component class using @layer components. This keeps your HTML clean while still using Tailwind’s utility values.

/* src/styles/global.css */
@layer components {
  .btn {
    @apply inline-flex items-center justify-center gap-2 px-5 py-2.5
           rounded-xl font-semibold text-sm transition-colors focus-visible:outline-none
           focus-visible:ring-2 focus-visible:ring-offset-2 disabled:opacity-50;
  }

  .btn-primary {
    @apply btn bg-blue-600 text-white hover:bg-blue-500
           focus-visible:ring-blue-500;
  }

  .btn-ghost {
    @apply btn bg-transparent hover:bg-slate-100 dark:hover:bg-slate-800
           text-slate-700 dark:text-slate-300;
  }
}

Now your JSX reads clearly:

<button className="btn-primary">Save changes</button>
<button className="btn-ghost">Cancel</button>

⚠️Don't overdo it

Only extract components for genuinely repeated patterns. If a class combination appears once, keep it inline. Over-extracting defeats the purpose of utility CSS.

2. Consistent spacing scale

Instead of using arbitrary spacing values like mt-[13px], stick to Tailwind’s default 4px scale. This creates visual rhythm automatically.

<!-- ❌ Inconsistent, hard to maintain -->
<div class="mt-[13px] mb-[22px] px-[18px]">

<!-- ✅ Uses the scale — predictable and consistent -->
<div class="mt-3 mb-6 px-4">

If you need values outside the scale (brand-specific sizing, for example), define them in tailwind.config under theme.extend:

// tailwind.config.mjs
export default {
  theme: {
    extend: {
      spacing: {
        18: '4.5rem',  // 72px
        88: '22rem',   // 352px
      },
    },
  },
};

3. Dark mode with dark: variants

Set darkMode: 'class' in your config and toggle it by adding/removing the dark class on <html>. Then write your dark styles right alongside the light ones:

<div class="
  bg-white dark:bg-slate-900
  text-slate-900 dark:text-slate-100
  border border-slate-200 dark:border-slate-800
">

Having light and dark variants together makes it easy to ensure you’ve covered both states.

4. Responsive design mobile-first

Tailwind is mobile-first. Unprefixed classes apply to all breakpoints, and prefixed classes (sm:, md:, lg:) apply from that breakpoint up.

<div class="
  grid
  grid-cols-1       <!-- mobile: 1 column -->
  sm:grid-cols-2    <!-- ≥640px: 2 columns -->
  lg:grid-cols-3    <!-- ≥1024px: 3 columns -->
  gap-6
">

Always start from mobile and work up. Never use max-* breakpoints unless you have a very specific reason.

5. Group and peer modifiers

group and peer unlock powerful stateful styling without any JavaScript.

<!-- Hover the card → animate the arrow inside -->
<a href="/blog/post" class="group block rounded-xl p-6 hover:bg-slate-50 dark:hover:bg-slate-800">
  <h2 class="font-bold text-slate-900 dark:text-white">Post title</h2>
  <span class="mt-2 flex items-center gap-1 text-blue-600 text-sm font-medium">
    Read more
    <!-- Moves right when parent (group) is hovered -->
    <svg class="w-4 h-4 group-hover:translate-x-1 transition-transform">...</svg>
  </span>
</a>

peer works similarly but between sibling elements — great for styling labels based on checkbox or input state.

6. Prose for rich text

When rendering Markdown or MDX content, wrap it in prose from @tailwindcss/typography. Customise it through tailwind.config:

// tailwind.config.mjs
export default {
  plugins: [require('@tailwindcss/typography')],
  theme: {
    extend: {
      typography: (theme) => ({
        DEFAULT: {
          css: {
            'code::before': { content: '""' },
            'code::after': { content: '""' },
            code: {
              backgroundColor: theme('colors.slate.100'),
              padding: '0.2em 0.4em',
              borderRadius: '0.25rem',
            },
          },
        },
      }),
    },
  },
};

Then in your layout:

<div class="prose prose-slate dark:prose-invert prose-lg max-w-none">
  <slot />
</div>

✅prose-invert for dark mode

Always pair prose with dark:prose-invert. Out of the box, prose-invert handles dark mode typography beautifully — light text, lighter headings, adjusted link colors.

Summary

These six patterns cover 90% of what I reach for on any Tailwind project:

1. @layer components for repeated UI patterns
2. Stick to the spacing scale — extend, don’t break it
3. Write light dark: classes side-by-side
4. Mobile-first responsive design
5. group and peer for stateful styling without JavaScript
6. @tailwindcss/typography for prose content

Tailwind isn’t magic — it’s a tool. Use it deliberately and it’ll reward you with a codebase that’s easy to change.