I maintain Scriptella, an open-source Java ETL tool, and wanted to establish a reproducible performance baseline for CSV imports.

I compared 11 import configurations using PostgreSQL and MariaDB, with 1 million rows per test.

Some results from the latest CI run:

Native COPY was fastest for PostgreSQL, as expected. Handwritten JDBC also outperformed Scriptella, while both were substantially faster than the tested csvsql workflow.

These are end-to-end timings, not isolated framework overhead measurements. The comparison uses one workload and one CI run, with different import mechanisms.

I've published the source code, Docker Compose setup, raw measurements, charts, and reproduction instructions:

https://github.com/scriptella/scriptella-benchmarks

I'd be interested in feedback from people working with ETL pipelines, particularly on which additional workloads would make this comparison more useful.

Got a slow Spark pipeline with two very believable explanations. The code has grown for years and has joins/transforms that probably deserve a hard look. But data volume has also gone up a lot while the underlying setup has basically stayed put.

A rewrite might pay off, but it is risky and eats engineering time we do not really have. Changing the compute path is quicker to test, but it will not magically save a bad data model. Both arguments sound sensible on their own, which is annoying.

How do you call this? What tells you the job design is the problem versus the infrastructure just being tapped out..