Saw this new post on the Databricks blog about loading huge amounts of data into Lakebase Postgres and thought the approach was interesting enough to share.

The usual problem: bulk loading into Postgres goes through COPY. Spark can read source partitions in parallel, but every row still ends up funneled through a single Postgres writer, and that same primary is serving your live app traffic. So a big load either takes forever or hurts your OLTP latency.

Their fix (LTAP Direct Writes, currently in beta) flips it around. Spark builds valid Postgres heap pages and primary key indexes itself, in parallel, and writes them straight to storage. The pageservers validate checksums when they ingest the files to catch corruption. The primary only publishes the final result through a small WAL record, so live queries aren't affected.

Numbers they claim:

- Up to 147x faster loads

- 1 TB in under 5 minutes in internal benchmarks (that's heap page build time)

- Throughput scales almost linearly as you add Spark capacity

- Multiple large loads into the same destination can run concurrently

The motivating case was a beta customer syncing roughly 1 billion rows a day via Synced Tables, who had to heavily overprovision their OLTP compute just to survive the sync, even with autoscaling.

They also say it's built with Postgres extensions and a table access method, no changes to Postgres core, so it should still behave like regular Postgres.

Curious if anyone here has tried it yet. How does this compare to what you're doing now for reverse ETL or serving lakehouse data to apps?

In this article we explore a new application pattern - offloading more of your Postgres data to ClickHouse. Leading to an order of magnitude better unit economics for your applications, as the data size increases.

pgloupe is a command-line tool that reads PostgreSQL logs and pg_stat_statements snapshots, joins them, and writes one HTML report with what to look at first and the numbers behind it.

I made it because going through logs and stats by hand takes too long, and I didn't want to paste hundreds of thousands of log rows into a hosted tool or an AI agent. It runs entirely on your machine and sends nothing anywhere.

• Reads stderr, csvlog and jsonlog formats
• Takes pg_stat_statements snapshots as CSV, so you can compare points in time. It notices stats resets and entries evicted from pg_stat_statements
• Works with Postgres 14 to 18, including managed ones like RDS
• One static binary: no server, no dependencies, no parser to configure
• Fast and light: on 11 GB of generated test logs it read about 141 MB/s on one thread, using around 44 MiB of memory

Try it:

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/vtrbraga/pgloupe/releases/latest/download/pgloupe-installer.sh | sh
pgloupe --help

Also available through Homebrew, cargo install pgloupe and Docker.

Repo: https://github.com/vtrbraga/pgloupe

Docs: https://vtrbraga.github.io/pgloupe/

This is the first stable release (0.1.0) and my first open source project. I'd like to hear what's missing or wrong: log formats or settings it misreads, findings that aren't useful, anything confusing in the report. Issues and comments are both welcome!

I’m pleased to announce the beta release of pglifecycle 2, an open source schema management tool for PostgreSQL.

pglifecycle keeps your database schema in version control as structured YAML, with one file for each table, view, function, role, and other objects. This makes the current schema easy to browse, review, and change without reconstructing it from years of migration files.

With pglifecycle, you can:

• Pull an existing database into a schema project
• Build that project into a custom format pg_dump archive (-Fc)
• Compare it with a live database and generate the DDL needed to reconcile them
• Deployment is plan first. Destructive changes are withheld unless explicitly allowed, making the generated SQL suitable for review in CI.

The companion pglifecycle GitHub Action can detect schema drift and post deployment plans directly to pull requests without applying them.

Version 2 is a Rust rewrite of the original tool, which has managed AWeber’s production schema for more than seven years. The beta release expands PostgreSQL object coverage and improves deployment convergence and safety.

• Getting Started
• Documentation
• GitHub Repo
• GitHub Action

Feedback and real world test cases are welcome as pglifecycle moves toward a stable 2.0 release.

Mod hat off for a moment:

I'm one of the organizers of Postgres Conference, and the Call for Proposals for 2027 is now open.

Next year is the conference's 20th, so we want the program to reflect what the community has actually learned running Postgres in the real world.

When/where: April 21-23, 2027, San Jose Marriott, San Jose, CA

This year’s program brings back our core technical tracks: Essentials, Dev, Ops, and Extensions, now joined by a new AI track.

We’re also continuing Professional Development & Wellness and Life & Fun. These newer tracks have been so well received that they’ve become an essential part of the conference experience.

We're especially interested in talks from people running Postgres in production, such as:

- Lessons from production incidents: what broke, how you found it, and what you changed afterward
- Autovacuum and bloat tuning on large or high-churn tables
- HA and replication setups, including failovers that didn't go to plan
- Building extensions, or running them at scale
- Benchmarks with methodology you're willing to share and defend

You don't need to be a known speaker. A clear story about a real problem is exactly what we're looking for, and first-time speakers are welcome.

CFP and details: https://postgresconf.org/conferences/2027

Happy to answer any questions about submitting here. 

What's a Postgres problem you wish someone would give a talk about?