Databricks runtime 19 brings the new Spark function trim_array. trim_array(array(‘A’, ‘B’, ‘C’, ‘D’), 2) returns [‘A’, ‘B’]. A count of 0 keeps the array; a count equal to its length returns an empty array.

https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

In Spark Structured Streaming when the stream writes to a output table (sink) you will typically use writeStream (other option is foreachBatch)

writeStream has many benefits (fault tolerant...checkpoint per sink etc) but one issue is that it will process the source N times where N is the amount of sinks/ output tables you have.

So lets say for a bronze -> silver app, where it reads bronze, transforms the data (eg. explode) and then writes to 4 silver tables. In theory this will be inefficient as the source read, deserialize, explode will happen 4 times (once per sink)

This happens as it is visible in the logs, 4x logs of source read etc

This is the theory... has anyone experienced this in practice? is there actually a 4x slowdown or inefficency?

Would introducing an intermediate table help as this would remain 1-1 with only one sink thus no issue.... but introduce extra write cost + read + extra storage layer

What is best practice here? (foreachBatch is not an option due to any possible data loss)

I recently posted about using the new AI Decide feature from Databricks to decide strategy within a virtual transport world using OpenTTD. To my surprise it worked pretty damn well, made it to profit, however, it did over-index on bus stops.

Over the past few days I have been trying to push it further with something a little more realtime and much more risky.

Step aside NASA, Databricks AI Decide can now put rockets in orbit... well in Kerbal Space Program 😄 🚀

Managing everything from lift off, throttle, staging, and orbital mechanics. I was blown away, it did it in ONE SHOT. We are doomed.

It wasn't flawless, some shim, some interesting angles of attack ... but it got the job done.

As of yesterday, the latency improvements have been most welcome.

Check out the AI Decide announcement here: https://www.databricks.com/blog/introducing-aidecide-make-fast-decisions-your-governed-data

Hey folks, I'm a PM working on platform at Databricks. We just shipped Private Network Gateway (PNG) to public preview on Azure (PrPr on AWS, coming soon on GCP).

The problem we were solving: Connecting serverless compute to private resources like databases, Kafka streams, internal APIs, MCP servers, etc. meant creating a separate private endpoint / proxy VM for each one, plus coordination with your network and security teams. That seemed manageable for a few endpoints, but quickly got painful once you needed to reach many resources across peered VNets or on-prem.

What PNG does: You delegate one subnet in your VNet/VPC to Databricks, and serverless traffic reaches that subnet through a managed, encrypted gateway. From there your existing routes, DNS, and firewall rules decide what's reachable, the same way they would for classic compute. You can route to specific destinations or send all egress through your own firewall with your own source IPs. There are no gateway VMs to deploy, patch, or scale; we manage them on our side.

Would love this community's take on:

• Is networking a friction you've run into with serverless compute?
• What are the most important resources for you to connect to?
• If you've reviewed the docs or tried the feature, is there anything missing you'd like to see supported?

Here are the Azure docs for reference.

And a quick visual of the problem we're solving:

Just recently publish my whole thesis on Databricks on this subreddit. Got really great feedback so figured I would write a summary for those too busy for a 20 minute read.

The Core Thesis:

Databricks is one of the largest and most influential companies mostly lying behind the scenes. Databricks is a platform that allows organizations to store, process and analyze massive amounts of data. Built on their open-source platform Apache Spark, Databricks created software that they released publicly. This software eventually gained extremely widespread adoption and built much of Databricks’ moat. Led by Ali Ghodsi, the 47-year-old Iranian-Swedish-American computer scientist, Databricks has leapfrogged competition in the last decade and is now ahead of its main competitor Snowflake in valuation, revenue, and revenue growth rate.

The Numbers to Know:

• $190 billion valuation.
• $5.2 billion in 2025 revenue.
• 71% - 2025 revenue growth.
• 27x - valuation to revenue multiple using Databricks’ estimated July 2026 ARR.
• 140% net revenue retention in 2024–2025.
• 74% - last disclosed gross margin cited, down from 80%.
• $5 billion raised in the last funding round despite $15–20 billion in investor interest.
• 80% of Fortune 500 companies using Apache Spark.

The Bear Thesis:

Databricks trades at 27x annualized revenue which assumes that their fast growth will continue to be sustainable. In addition, while AI is certainly beneficial for Databricks in accelerating revenue, their gross margins are starting to compress, down to a (still healthy) 74% from a previously elite 80%. The company has already acknowledged further compression as compute costs rise. If gross margins compress significantly, even if Databricks’ revenue continues growing at it’s current pace, the company may not be profitable enough to justify its valuation.

The Verdict:

Databricks was one of the most impressive companies I have researched to date. A few years ago, Databricks was behind its main competitor Snowflake in revenue and valuation. Today, that script has flipped on its head with Databricks being ahead of Snowflake in valuation and revenue. This is largely due to Ali Ghodsi and the incredible team at Databricks and it is partly for this reason that Databricks trades at a slight premium in comparison to Snowflake. Their sticky moat, strong leadership and impressive revenue growth are all strong reasons to invest in Databricks. However, I think due to the premium valuation perhaps the right call with Databricks at this moment is to have patience. I will particularly be watching the revenue growth to see if the company is able to maintain the revenue growth they currently have and watch to see if the gross margins continue to compress. This is certainly a company I want to add to my portfolio, at the right time and right price.

How are you using AI in your Databricks workflow?

Are you using AI for things like CI/CD, code reviews, testing, debugging, performance optimization, or automating repetitive tasks?

Hey everyone! I’m attending the Databricks Dev Connect event today at Sheraton Whitefield, and I’m going solo. If you’re going too, let’s meet up, sit together for the sessions, and network over coffee or dinner. Drop a comment or DM me if you’re in!

Liquid clustering is great because getting table layout wrong on Databricks is now cheap. Changing your clustering keys is a metadata change instead of a full table rewrite.

the thing is that the cost didn't go away, it just moved. Clustering happens through OPTIMIZE, which runs on compute you pay for. Databricks recommends running it every hour or two on busy tables, so that's 12 to 24 runs a day per table. And predictive optimization is on by default for accounts created since November 2024 and billed under a serverless jobs SKU, so a lot of teams are paying for automated OPTIMIZE, VACUUM and ANALYZE without ever having decided to.

Full write-up: https://medium.com/@yanivleven/liquid-clustering-isnt-free-it-s-just-cheaper-to-get-wrong-fe51bbf57a2e

... Now with a native Python API!

If you like using PySpark on serverless GC, you can now ingest data faster with:

from pyspark import pipelines as dp

@dp.table(name="main.default.bookings_raw")
def bookings_raw():
  return (
     spark.readStream.format('cloudFiles')
     .option('cloudFiles.format', '<file-format>')
     .load(f'{<path-to-source-data>}')
 )

If you want faster and more performant partition overwrites, try using incremental replace where!

from pyspark import pipelines as dp
from pyspark.sql import functions as F

u/dp.table(
  name="main.default.booking_revenue_rw",
  replace_where=F.col("check_in") >= F.to_date(F.lit("2025-07-01")),
)
def booking_revenue_rw():
  return (
    spark.read.table("samples.wanderbricks.bookings")
    .groupBy("check_in")
    .agg(F.sum("total_amount").alias("total_revenue"))
  )

If you like Materialized Views for faster BI reporting, it's as simple as

from pyspark import pipelines as dp
from pyspark.sql import functions as F

@dp.materialized_view(
  name="main.default.daily_booking_revenue",
  comment="Daily booking revenue.",
  table_properties={"quality": "gold"},
  cluster_by=["check_in"],
)
def daily_booking_revenue():
  return (
    spark.read.table("samples.wanderbricks.bookings")
    .groupBy("check_in")
    .agg(F.sum("total_amount").alias("total_revenue"))
  )

Mine used to be Structured Streaming checkpoints. I’d used them for months before I really understood what gets stored in there, and why deleting or reusing one quietly breaks a stream.

I’m a trainer and I spend most of my week explaining this stuff, so post yours. I’ll try to answer every one: how it actually works underneath, no hand-waving.

Bonus points if you say what specifically feels fuzzy: what it does, how it does it, or when to pick it over something else.

I'm using the default Databricks spark settings for a cluster which has nodes with each 64GB and 8 cores.
Surprisingly the JVM heap limit shows up as around 11.5GB which seems far too low to be total heap size (unless there is a bad default Databricks config).
So what am I seeing here/what is missing?

I have a UDF heavy job which I want to optimize by shifting the memory towards the python side. Thus I adjust the executor, overhead and pyspark memory while keeping the sum of all three well below the limit of the worker (50GB vs. 64GB of the worker).

The issue I'm currently facing is, that it seems pretty random how far you can push the memory settings, while staying well below the node limit. Especially as error about exceeding certain memory limits will only surface at runtime.

More importantly the limits are VM specific, so a configuration might work on for one 64GB node but not for another 64GB node while every else remains the same.

So in the end I wonder if such memory setting requires a whole lot of VM specific trial and error to get into stable territory?

Hey r/databricks, In order to keep the main feed clean, we are implementing a weekly megathread for self promotion for companies who do lots of work with databricks. Please direct all self promotion posts here and keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

Hey r/databricks, we're noticing a lot of repeated interviewing and hiring posts that tend not to get much engagement. We're going to combine them into a monthly thread so that you're more likely to get answers, plus we can ask our recruiting team to keep an eye on them if there are any general questions.

I'm thinking about how AI could be used in Databricks to detect pipeline problems before they cause production failures.

For example, a job usually processes 10 million records in 15 minutes, but one day it processes only 4 million records and takes 40 minutes. The job still succeeds, so normal failure alerts don't trigger.

Could we use AI to analyze job history, data volumes, execution times, and Spark metrics to detect unusual behavior and warn us before the issue affects downstream reports?

Would you use a machine learning model, Databricks Lakehouse Monitoring, or a custom solution for this?

Has anyone implemented something similar in a real project?

I can see how zerobus is used for streaming data into a lakehouse table. I can generally see how you would consume the data as well.

A couple things are still not clear. What is the pattern for removing old rows out of the UC table? Should we do a delete once a year, on messages older than 3 years? Is this a necessary maintenance that we are responsible for, or is there something in the software which will help with housekeeping?

And how would I perform scheduled tasks against this data? Let's say that data represents a news feed and I just need to send an summarized overview once an hour, and then flag the rows that have been summarized. I'm assuming it would be disruptive to try to delete the rows that were processed or even update them (ie. it may interfere with zerobus itself, and create concurrency conflicts). Should I build a DIFFERENT table that tracks the surrogate IDs of rows already processed/summarized?

Sorry for the basic questions, but I wasn't able to find the answer to these yet. I'm sure I could find my own solution, but it would be interesting to hear if there is a common practice

I have a use case where Lakebase CDF (transparent syncing of tables from PostgreSQL to Unity Catalog) would be super convenient.

However I found that currently the CDF doesn't support storage accounts behind a firewall, accessible through a private endpoint, etc. They have to be publicly accessible, which is a no-no for most corporate data.

Can anyone from DataBricks share whether it's on your roadmap to add support for this and when?

Hey r/databricks, please direct all certification and training posts here.

The Databricks Advanced Learning Festival runs from September 16 to October 14, 2026.

Databricks Advanced Learning Festival: September 1... - Databricks Community - 166157

Good luck to everyone on your certification journey!

On-demand state repartitioning lets you change streaming state partitions while the stream is still running: set spark.sql.streaming.stateStore.partitions and restart the streaming query, or set it in a Lakeflow table’s spark_conf before the next update. The ability to finally handle complicated scenarios liek that one in Declarative Pipelines is also a big improvement. And checkpoints survive repartitioning.

more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

​

We’re considering Databricks as the main analytics platform across a large insurer: controlling, accounting, risk, etc. Looking at the whole ecosystem—dashboards, Genie, Metric Views and the Excel add-in.

The gaps I still see are hierarchical filters, bookmarks and finer layout/styling control. Drill-through and cross-filtering have improved a lot.

People often cite Office integration as a dealbreaker. Excel integration is there. Power BI’s live PowerPoint add-in is a difference, but how useful is sharing a presentation when recipients may see different data—or nothing—depending on permissions? What Office workflows am I missing?

What concrete requirements still make Power BI essential, beyond familiarity or better-looking reports? Anyone tried this at enterprise scale and hit a hard wall?

We are kind of fortunate to be late to the party in the sense that we do not have a big Power legacy landscape to migrate.

Hi All,

I work on Zingg - open source Spark based entity resolution. Many of our users are interested in running Zingg on Databricks Serverless. Our codebase contains Spark ML and Graphframes, which are not supported through Spark Connect client. Hence I started looking at server libraries, with a relation or command plugin to expose the complex parts.

I am trying to understand if there is a roadmap to allow adding server libraries to Databricks Serverless? The open source Spark framework does allow that, but I am not sure if its coming to Databricks. https://spark.apache.org/docs/latest/app-dev-spark-connect.html#spark-server-libraries

Would love to hear from the community on their thoughts.