r/databricks • u/hubert-dudek • 16h ago
Databricks runtime 19 brings the new Spark function trim_array. trim_array(array(‘A’, ‘B’, ‘C’, ‘D’), 2) returns [‘A’, ‘B’]. A count of 0 keeps the array; a count equal to its length returns an empty array.
r/databricks • u/BigDaddyData187 • 15h ago
In Spark Structured Streaming when the stream writes to a output table (sink) you will typically use writeStream (other option is foreachBatch)
writeStream has many benefits (fault tolerant...checkpoint per sink etc) but one issue is that it will process the source N times where N is the amount of sinks/ output tables you have.
So lets say for a bronze -> silver app, where it reads bronze, transforms the data (eg. explode) and then writes to 4 silver tables. In theory this will be inefficient as the source read, deserialize, explode will happen 4 times (once per sink)
This happens as it is visible in the logs, 4x logs of source read etc
This is the theory... has anyone experienced this in practice? is there actually a 4x slowdown or inefficency?
Would introducing an intermediate table help as this would remain 1-1 with only one sink thus no issue.... but introduce extra write cost + read + extra storage layer
What is best practice here? (foreachBatch is not an option due to any possible data loss)
r/databricks • u/Roadtochessmaster • 21h ago
Just recently publish my whole thesis on Databricks on this subreddit. Got really great feedback so figured I would write a summary for those too busy for a 20 minute read.
The Core Thesis:
Databricks is one of the largest and most influential companies mostly lying behind the scenes. Databricks is a platform that allows organizations to store, process and analyze massive amounts of data. Built on their open-source platform Apache Spark, Databricks created software that they released publicly. This software eventually gained extremely widespread adoption and built much of Databricks’ moat. Led by Ali Ghodsi, the 47-year-old Iranian-Swedish-American computer scientist, Databricks has leapfrogged competition in the last decade and is now ahead of its main competitor Snowflake in valuation, revenue, and revenue growth rate.
The Numbers to Know:
The Bear Thesis:
Databricks trades at 27x annualized revenue which assumes that their fast growth will continue to be sustainable. In addition, while AI is certainly beneficial for Databricks in accelerating revenue, their gross margins are starting to compress, down to a (still healthy) 74% from a previously elite 80%. The company has already acknowledged further compression as compute costs rise. If gross margins compress significantly, even if Databricks’ revenue continues growing at it’s current pace, the company may not be profitable enough to justify its valuation.
The Verdict:
Databricks was one of the most impressive companies I have researched to date. A few years ago, Databricks was behind its main competitor Snowflake in revenue and valuation. Today, that script has flipped on its head with Databricks being ahead of Snowflake in valuation and revenue. This is largely due to Ali Ghodsi and the incredible team at Databricks and it is partly for this reason that Databricks trades at a slight premium in comparison to Snowflake. Their sticky moat, strong leadership and impressive revenue growth are all strong reasons to invest in Databricks. However, I think due to the premium valuation perhaps the right call with Databricks at this moment is to have patience. I will particularly be watching the revenue growth to see if the company is able to maintain the revenue growth they currently have and watch to see if the gross margins continue to compress. This is certainly a company I want to add to my portfolio, at the right time and right price.
r/databricks • u/Krishgarikipati • 6h ago
Migrating legacy data platforms to Databricks is more than moving data. It also involves understanding existing workloads, converting SQL and ETL logic, and validating results.
This is where Databricks Lakebridge can help!
What is Lakebridge?
Lakebridge is a migration toolkit designed to accelerate the modernization of legacy data platforms and workloads on Databricks.
How does it help?
1. Assess
Analyze existing SQL workloads and code to understand complexity, compatibility, dependencies, and migration effort.
2. Convert
Help translate legacy SQL and supported ETL workloads into Databricks-compatible SQL, PySpark, or notebooks.
3. Reconcile
Compare source and target datasets to identify discrepancies in schemas, rows, and column values after migration.
A practical example
Imagine migrating an Oracle-based data warehouse to Databricks.
Instead of manually reviewing every SQL script and validating every migrated table from scratch, Lakebridge can help:
➡️ Analyze existing Oracle SQL workloads
➡️ Identify migration complexity
➡️ Accelerate SQL conversion
➡️ Validate migrated data against the source
The result? A more structured, repeatable migration process with less manual effort.
🔹 Why should data engineers care?
As organizations modernize their data platforms, migration is not just about changing technology. It is about preserving business logic, maintaining data quality, reducing risk, and making the most of the lakehouse architecture.
Lakebridge can support this journey by automating key migration activities, while engineering review and testing remain essential.
Remember: Automated code conversion does not guarantee business-equivalent results. Always validate converted logic and reconcile the data before production cutover.
🔗 Official documentation: https://databrickslabs.github.io/lakebridge/docs/overview/
Have you explored Lakebridge for your Databricks migration projects? What has your experience been with automated SQL conversion and data reconciliation?
#Databricks #Lakebridge #DataEngineering #DataMigration #Lakehouse #ETL #DataModernization #BigData
