I’ve noticed something while learning and working with data systems.
Building a pipeline is one thing. Figuring out why it’s behaving differently in production is a completely different skill.
For example, imagine you’ve built a PySpark pipeline. It works fine with sample data, your transformations are correct, and everything looks good.
Then a few things start happening:
A job that used to finish in 20 minutes now takes 2 hours.
An Airflow task retries after a failure and processes the same records again.
The pipeline succeeds, but the dashboard shows incorrect numbers.
A source API changes its response, and downstream tasks start failing.
I’ve been thinking about how I’d approach these situations rather than immediately searching for another tool or rewriting the code.
Here’s how I’d start debugging them.
1. If a Spark job suddenly becomes slow
I’d open the Spark UI first.
I’d compare stage duration, task distribution, shuffle read/write, and spill metrics against a previous successful run.
If most tasks finish quickly but a few take much longer, I’d investigate data skew. If the input volume has increased significantly, I’d check whether the workload itself has changed.
Adding executors without understanding the bottleneck can just increase the cost.
2. If a pipeline processes the same records twice
I’d check how the pipeline handles retries and writes data.
Is it appending everything on every run? Is there a reliable business key? Is the write operation designed to be idempotent?
For example, a carefully designed MERGE operation can make repeated processing safer, provided the matching logic and transaction behavior are correct.
Retries aren’t the problem by themselves. Retrying a task without accounting for what it already wrote can be.
3. If the pipeline succeeds but the data is wrong
I’d stop treating a successful task status as proof that the output is correct.
I’d compare source and target record counts, check duplicates and nulls in important fields, validate business rules, and look for missing or late-arriving records.
A job can execute exactly as written and still produce the wrong business result.
4. If an API change breaks the pipeline
I’d inspect the actual response, status code, authentication errors, rate limits, and schema differences.
The Python or Airflow code might be fine. The assumptions it makes about the source system might no longer be valid.
The common thread here is that each problem needs a different investigation.
You don’t need to memorize every failure scenario. You need to know where to look, what evidence to collect, and how to narrow down the cause before changing things.
This is also why I think rebuilding the same project with failure scenarios is often more useful than starting another project from scratch.
Take a pipeline you’ve already built and deliberately test what happens when:
The input volume increases.
A task fails after partially writing data.
The same batch arrives twice.
A required field disappears from the source.
The downstream database becomes unavailable.
You’ll learn a lot about your design from the results.
For people working with Spark, Airflow, or other data pipelines: what’s the first thing you usually investigate when a pipeline that worked yesterday suddenly breaks today?
I’d be interested in hearing actual debugging experiences, especially cases where the initial assumption turned out to be wrong.
