Just recently publish my whole thesis on Databricks on this subreddit. Got really great feedback so figured I would write a summary for those too busy for a 20 minute read.

The Core Thesis:

Databricks is one of the largest and most influential companies mostly lying behind the scenes. Databricks is a platform that allows organizations to store, process and analyze massive amounts of data. Built on their open-source platform Apache Spark, Databricks created software that they released publicly. This software eventually gained extremely widespread adoption and built much of Databricks’ moat. Led by Ali Ghodsi, the 47-year-old Iranian-Swedish-American computer scientist, Databricks has leapfrogged competition in the last decade and is now ahead of its main competitor Snowflake in valuation, revenue, and revenue growth rate.

The Numbers to Know:

• $190 billion valuation.
• $5.2 billion in 2025 revenue.
• 71% - 2025 revenue growth.
• 27x - valuation to revenue multiple using Databricks’ estimated July 2026 ARR.
• 140% net revenue retention in 2024–2025.
• 74% - last disclosed gross margin cited, down from 80%.
• $5 billion raised in the last funding round despite $15–20 billion in investor interest.
• 80% of Fortune 500 companies using Apache Spark.

The Bear Thesis:

Databricks trades at 27x annualized revenue which assumes that their fast growth will continue to be sustainable. In addition, while AI is certainly beneficial for Databricks in accelerating revenue, their gross margins are starting to compress, down to a (still healthy) 74% from a previously elite 80%. The company has already acknowledged further compression as compute costs rise. If gross margins compress significantly, even if Databricks’ revenue continues growing at it’s current pace, the company may not be profitable enough to justify its valuation.

The Verdict:

Databricks was one of the most impressive companies I have researched to date. A few years ago, Databricks was behind its main competitor Snowflake in revenue and valuation. Today, that script has flipped on its head with Databricks being ahead of Snowflake in valuation and revenue. This is largely due to Ali Ghodsi and the incredible team at Databricks and it is partly for this reason that Databricks trades at a slight premium in comparison to Snowflake. Their sticky moat, strong leadership and impressive revenue growth are all strong reasons to invest in Databricks. However, I think due to the premium valuation perhaps the right call with Databricks at this moment is to have patience. I will particularly be watching the revenue growth to see if the company is able to maintain the revenue growth they currently have and watch to see if the gross margins continue to compress. This is certainly a company I want to add to my portfolio, at the right time and right price.

How are you using AI in your Databricks workflow?

Are you using AI for things like CI/CD, code reviews, testing, debugging, performance optimization, or automating repetitive tasks?

In Spark Structured Streaming when the stream writes to a output table (sink) you will typically use writeStream (other option is foreachBatch)

writeStream has many benefits (fault tolerant...checkpoint per sink etc) but one issue is that it will process the source N times where N is the amount of sinks/ output tables you have.

So lets say for a bronze -> silver app, where it reads bronze, transforms the data (eg. explode) and then writes to 4 silver tables. In theory this will be inefficient as the source read, deserialize, explode will happen 4 times (once per sink)

This happens as it is visible in the logs, 4x logs of source read etc

This is the theory... has anyone experienced this in practice? is there actually a 4x slowdown or inefficency?

Would introducing an intermediate table help as this would remain 1-1 with only one sink thus no issue.... but introduce extra write cost + read + extra storage layer

What is best practice here? (foreachBatch is not an option due to any possible data loss)

Databricks runtime 19 brings the new Spark function trim_array. trim_array(array(‘A’, ‘B’, ‘C’, ‘D’), 2) returns [‘A’, ‘B’]. A count of 0 keeps the array; a count equal to its length returns an empty array.

https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a