r/SQL • • 15h ago

Discussion How would you compare two huge tables when there is no primary key and duplicate rows matter?

9 Upvotes

I'm comparing a source table with a target table after a data migration.

Both tables have hundreds of millions of rows, no reliable primary key, and some columns contain NULL values. Duplicate rows are also possible.

A simple EXCEPT query doesn't reveal differences in duplicate counts, and hashing every row may introduce collision risks or performance issues.

For example:

Source: A,A,A,B,C
Target: A,A,B,B,C

Both tables contain the same distinct values, but the actual data is different.

How would you identify missing, extra, and changed records efficiently without a unique key?

Would you use grouped row comparisons, partitioning, hashes, or a completely different strategy? How would you make it scale to hundreds of millions of rows?


r/SQL • • 1d ago

Discussion How much advanced SQL do you truly use in your day-to-day job?

77 Upvotes

Hi! So I’m a student (not business or tech background) I’m learning SQL kind of on my own and I have the basics pretty much down like SELECT, FROM, WHERE, BETWEEN, ORDER BY, ASC, DESC. I have some knowledge when it comes to JOINs and CASE WHEN function but I was wondering if I need to know CTEs and window functions? I’m not trying to be a database administrator or something like that, just using it for analysis and excel as well.


r/SQL • • 19h ago

SQL Server Statistics on columns of a table

4 Upvotes

Limitations of table-valued parameters:

SQL Server does not maintain statistics on columns of table-valued parameters.

statistics of what ? why those statistics even exist for non table-valued parameters ?


r/SQL • • 14h ago

Discussion Better Autocomplete than DataGrip, one familiar language youll never need to write in. For any database or file, as a TUI, with built-in AI round trips, query history... for all languages and Databases

Thumbnail
github.com
1 Upvotes

Hi,

I have been working on a project: A unified workspace for all datatypes/frameworks supported by "ibis" (sort of like sqlglot but includes dataframes too).

To name a few i use on a day to day basis: Parquet, CSV, DuckDB, Postgres, MS SQL, and SQLite/D1, polars, pandas.

This project was built because spinning up Jupyter just to view a Parquet file is overkill, modern CSV editors are either ancient or freemium traps, and standard SQL IDEs struggle with live schema IntelliSense and quick parameter binding, and practically ignore your history.

This project adds a layer to solve those issues and adds more functionality, such as the ability to make parameter driven functions to kick off canned queries (without actually writing the functions to your DB).

Bring your own DB, bring your own AI (optional, but i would definitely recommend), and load any file. (Currently very early. LOTS of work to be done ).

The goal of this is to keep us heavy vibe coders who appreciate TUI experience (opencodecli, pi, oh my pi) and use multiplexers like herdr inside what we love -- terminal user interfaces.

No more bloated Datagrip and DBeaver-- all in one simplified language with shortcuts. Same syntax for all languages.

Some examples:

### 1. Filter without the boilerplate

\from orders

\where status = 'paid' and amount > 100

→

SELECT * FROM orders WHERE status = 'paid' AND amount > 100;

### 2. Top 10 biggest orders

\from orders

\select id, amount

\order amount desc

\limit 10

→

SELECT id, amount
FROM orders
ORDER BY amount DESC
LIMIT 10;

### 3. Automatic grouping — select a field and an aggregate. Done.

\from orders

\select customer_id, sum(amount) as revenue

→

SELECT customer_id, SUM(amount) AS revenue

FROM orders

GROUP BY customer_id;

No \group required. D8R derives the grouping from your selection.

### 4. Multiple dimensions, multiple aggregates — still auto-grouped

\from orders

\select region, status, count(*) as orders, sum(amount) as revenue, avg(amount) as average_order

→

SELECT region, status,
COUNT(*) AS orders,
SUM(amount) AS revenue,
AVG(amount) AS average_order
FROM orders
GROUP BY region, status;

### 5. Join shorthand when both tables share a key

\from orders
\join customers on customer_id
\select orders.id, customers.name, orders.amount

→

SELECT orders.id, customers.name, orders.amount
FROM orders
JOIN customers
ON orders.customer_id = customers.customer_id;

### 6. Join + aggregate + auto-group + ranking

\from orders o

\join customers c on o.customer_id = c.id

\select c.region, sum(o.amount) as revenue

\order revenue desc

\limit 5

→

SELECT c.region, SUM(o.amount) AS revenue

FROM orders AS o

JOIN customers AS c ON o.customer_id = c.id

GROUP BY c.region

ORDER BY revenue DESC

LIMIT 5;

### 7. Filter rows AND aggregate results with the same \where

\from orders

\select customer_id, sum(amount) as revenue

\where status = 'paid' and revenue > 1000

→

SELECT customer_id, SUM(amount) AS revenue
FROM orders
WHERE status = 'paid'
GROUP BY customer_id
HAVING SUM(amount) > 1000;

D8R separates the input filter from the aggregate-result filter.

### 8. Different filters inside different aggregates

\from orders

\select customer_id, sum(amount \where status = 'paid') as paid_total, count(* \where status = 'pending') as pending_orders

→

SELECT customer_id,

SUM(amount) FILTER (WHERE status = 'paid') AS paid_total,

COUNT(*) FILTER (WHERE status = 'pending') AS pending_orders

FROM orders

GROUP BY customer_id;

One query, independently filtered metrics, automatic grouping.

### 9. Calculate directly on aggregates

\from orders

\select customer_id, sum(amount_cents) / 100 as revenue_dollars

→

SELECT customer_id,
SUM(amount_cents) / 100.0 AS revenue_dollars
FROM orders
GROUP BY customer_id;

Numeric conversion, not formatting—and integer inputs don’t truncate the fraction.


r/SQL • • 8h ago

Discussion Before writing SQL in an interview, what would you clarify?

0 Upvotes

Suppose an interviewer asks you something like "find the top three products in each category by sales last quarter." Before you write the query, what would you clarify?

For me, I’d want to know what “sales” means -  units, revenue, or number of orders Then, I'd want to know things like how ties should work, which dates count, and what one row in the output should represent.

I’ve been thinking about how SQL interviews test more than whether you can produce a query. They also give you a chance to show how you frame the problem, explain your choices, and check your work.

So, I put together a guide on what to expect and how to prepare: What to Expect in a SQL Coding Interview

If you’ve recently done a SQL interview, what’s one thing you wish you’d known or asked beforehand?


r/SQL • • 1d ago

Discussion Help with ERD

Thumbnail
gallery
9 Upvotes

Hello I'm a newbie college student studying databases and I'm currently making a database for a small inventory system that's solely focused on monitoring stocks and its shelf life to avoid running out of stock and pull outs for my project. The store I'm basing it off on sells seasoning and other powdered flavoring (beverages, etc.), they also explained that they get bulk products from the supplier and then they repack them into smaller quantities for selling. What would be the proper entities and attributes for this system? I've tried searching other inventory systems but i don't know how the repacking would be translated to the database/ERD.


r/SQL • • 17h ago

Discussion what is Strata sql?

0 Upvotes

Today I accidentally saw "Strata SQL" in a list of extensions for VS Code. It's a somewhat dubious website, but the stated functionality is intriguing. Judging by the browser demo, it's not just a visual DDL editor, it's a full-fledged utility for data documentation and reporting.
Does anyone know what this is? The project looks pretty polished, but it still looks suspicious.


r/SQL • • 1d ago

Discussion What happens when we join two tables with NULL values??

24 Upvotes

 
If both tables have NULL values in the join column,, will those records match each other?


r/SQL • • 22h ago

SQL Server SQl e a teia de aranha mental

0 Upvotes

Bom dia

Estou a aprender SQL desde meio de Setembro em um curso que estou a tirar. Na verdade começamos com uma introdução a SQL e saltamos para diagramas, relacionamrntos e cardinalidades. Agora voltamos para SQL( select,from,join, where, group by, order by, coalesce, null,...) Sinto me super confuso. Basicamente entendo os conceitos mas na parte prático ao tentar reproduzir os mesmos, sinto um espaço em branco. Parece que preciso das tabelas em excel ao lado para conseguir decifrar a solução. Sei que tenho ainda muito que praticar ( e só assim vou chegar lá). Mas parece que é algo que nao consigo ter um fio condutor. Parece um bicho dificil de domar minimamente.


r/SQL • • 1d ago

PostgreSQL Planning to take the PostgreSQL Open Certification (DBA-1) exam at PGConf?

9 Upvotes

I created a free study guide to help you prepare:

https://thesev1database.com/postgresql-certification

I’d really value your feedback.


r/SQL • • 2d ago

PostgreSQL Struggling to wrap my head around writing SQL queries—how do you guys break them down?

33 Upvotes

Hey everyone,

I’m really struggling to get the hang of writing database queries. Whenever I look at a problem or a schema, my brain just stalls out and I can't figure out the right logic or structure to pull what I need.

I see people throwing together JOINs, subqueries, and window functions like it's nothing, and I feel like I'm missing a core fundamental step in how to translate a question into SQL.

  • What is your mental process when you start writing a query from scratch?
  • How do you map out the tables and relationships in your head before you type?
  • Are there any good beginner-to-intermediate tips, tricks, or exercises that finally made things "click" for you?

Any advice, recommended resources or tips .


r/SQL • • 1d ago

Discussion The AI exploit gap is engineering’s problem now

Thumbnail
leaddev.com
9 Upvotes

Security sees the risk. Engineering owns the fix.


r/SQL • • 1d ago

PostgreSQL Is anyone actually using Databricks Lakebase in production?

2 Upvotes

My team is thinking of trying Lakebase instead of running a separate Postgres/ RDS db for our app.
The idea of auto sync to Delta and scaling to zero sound great, but we are worried about latency issu. If u are usin it, how noticeable are the cold start delays when it wake up backup, and does it feel fast enough for live user traffics?


r/SQL • • 1d ago

PostgreSQL EXPLAIN ANALYZE is a sample-size of One

Thumbnail
exobench.ai
0 Upvotes

Everyone knows that you need to look at stats when doing query optimization...

Right? Right!??

Well then how come when we look at EXPLAIN ANALYZE trees we assume the whole thing is deterministic and inviolate?

It's like someone somewhere told us "we'll it's just too hard to do here so we just skip that part".

What if we don't? What if we embrace the pain and actually group the plan trees and try to get statistical info? Here's the first thing that I'm seeing.


r/SQL • • 1d ago

PostgreSQL Review/roast my feature: secrets in migrations

3 Upvotes

Hi!

Around two years ago I started working on a migration tool called spawn, focused on postgres for now (with plans for other db's). Around 9 months ago, I started dog-fooding it for a serious project that has regular daily commits, with frequent database changes.

I'm keen to get feedback/opinions from people who use migration tools, so I thought I'd try posting here a thread here to get feedback. Recently I had a need to get a secret into a database, so I added a feature that tries to do this in a secure way. That means:

  • Don't render secrets by default
  • Allow different secrets for prod vs dev
  • Support many different options for providing the secret value

Would love your feedback on what I ended up creating (which worked well for my use case). Whether there are particular ways you use secrets that this wouldn't cover. I looked at how secrets are handled in some other migration tools, and I think what I've landed on here is a good approach.

Without further ado, the feature...

Secrets are included with a function:

CREATE ROLE app_user WITH LOGIN
  PASSWORD {{ secret("app_password") }};

GRANT CONNECT ON DATABASE postgres TO app_user;

That function will fetch the password at build or apply time, using whatever is defined in your config (there are a few supported options):

[secrets.app_password.default]
source = "host_file"
path = "/run/secrets/app-password"

[secrets.app_password.environments.dev]
source = "literal"
value = "dev-only-password"
insecure = true

Note that there's a default password set, and an override for dev environment specifically. This is to help prevent you from using your dev password inside a prod migration.

When you build a migration (build is used to view the output), it masks secrets by default:

$ spawn migration build 20261014103000-create-role
CREATE ROLE app_user WITH LOGIN
  PASSWORD '***MASKED:app_password***';

GRANT CONNECT ON DATABASE postgres TO app_user;

Or you can run with --reveal-secrets to see the actual password printed. When you apply the migration, it will use the secret directly, and does a best effort attempt to ensure it's hidden from any logs output.

For production, I had passwords stored in pass. When specified similar to the below, when the migration is being applied a password prompt shows up to unlock the password store, so the migration can apply:

[secrets.app_password.default]
source = "command"
command = ["pass", "show", "spawn/app_password"]

As mentioned, I'd love your feedback, particularly if there's use cases you've hit that what I built wouldn't cover.


r/SQL • • 1d ago

MySQL I have doubts in mysql version 8.0.46 can anyone help me ?

Thumbnail
0 Upvotes

r/SQL • • 1d ago

Discussion SQL向けにスキーマを理解する静的解析ツール(アナライザー)を作った。

Thumbnail
gallery
1 Upvotes

​

I built a schema-aware static analyzer for SQL. It reads your schema (CREATE TABLE statements, migrations or a dump) and checks your queries against it, without connecting to a database.

Besides missing tables and columns (with "did you mean" suggestions) and type mismatches, it catches queries that run fine but are almost certainly wrong, like `NOT IN` over a nullable column (returns nothing as soon as one NULL shows up) or a `WHERE` that turns a `LEFT JOIN` into an inner join.

It supports PostgreSQL, MySQL and SQLite. You can try it in the browser without installing anything: paste a schema and a query and see what it reports. For projects there's a CLI, a GitHub Action and a VS Code extension.

I always felt SQL was the one part of my code without a good static analysis tool, so I made one. I've tested it on open-source projects and fixed it until it reported zero false positives there, but I'd really like feedback from real use: false positives, mistakes it misses, or syntax it can't parse.

Playground: https://sqlsift.dev/

GitHub: https://github.com/yukikotani231/sqlsift


r/SQL • • 1d ago

SQL Server My SQL/set‑theory book is on a 99p countdown deal — feedback welcome

Post image
3 Upvotes

I’ve put my book on a 99p KDP countdown deal and wanted to share it here in case it’s useful to anyone working with SQL, set logic, or bitwise operations.

The book focuses on the foundations behind SQL’s behaviour — sets, multisets, binary representation, bitwise operators, power sets, and bit‑set encoding — with lots of worked examples and diagrams.

If you’re interested, the link is below.
Happy to answer questions or get feedback from people who work with this stuff daily.

https://www.amazon.com/dp/B0H2JXP2WQ


r/SQL • • 2d ago

Discussion My SQL query returns the correct row count, but some records are still wrong. How do I find them?

20 Upvotes

I'm comparing a source table with a target table after an ETL job.

Both tables have the same row count, but I suspect some records are missing while others are duplicated or incorrectly updated.

The tables have millions of rows, no reliable primary key, and some columns contain NULL values.

A simple COUNT(*) matches ,but I can't trust the data

How would you identify missing, extra, duplicated, or changed records without comparing every row manually?

Would you use hashing, GROUP BY, window functions, or another approach?


r/SQL • • 1d ago

Discussion When companies say we need expertise in SQL

0 Upvotes

what does this mean to you?


r/SQL • • 2d ago

MySQL Data import question

0 Upvotes

Hey so Im quite new to mysql and I've been using it on mac, and recently I've been trying to import a csv file that contain data but Im not sure how to do it. Ive seen youtube videos but the mysql version that they use is way different than mine. Can anybody help with this please and thank you.!!


r/SQL • • 3d ago

Discussion Survey of 300 engineering leaders: 81% had a production incident linked to AI-coding tools in six months.

Thumbnail
leaddev.com
111 Upvotes

Faster code, slower debugging.


r/SQL • • 4d ago

Discussion Is this SQL learning roadmap enough for real-world data analyst internships in 2026, or am I missing important skills?

16 Upvotes

im building my SQL skills for data analyst, business analyst, BI and eventually business/revenue operations roles. I don't want to stop at basic queries or just finish a bootcamp. I want to be able to solve unfamiliar business problems, work with messy data, validate results and explain findings.

I'm learning MySQL and using a SQL bootcamp that covers topics such as:

  • SELECT, WHERE, filtering and sorting
  • CASE statements and string functions
  • Joins, including self joins
  • UNION and UNION ALL
  • Subqueries and CTEs
  • Window functions, including ROW_NUMBER()
  • Stored procedures, parameters, triggers and events

My proposed practice roadmap is:

  1. Core querying: filtering, NULL handling, dates, conditional logic and sorting.
  2. Aggregation: GROUP BY, HAVING, aggregate functions and conditional aggregation.
  3. Joins and relational reasoning: multiple tables, missing matches, duplicate matches and incorrect totals.
  4. Advanced querying: subqueries, EXISTS/NOT EXISTS, CTEs and UNION.
  5. Analytical SQL: window functions, ranking, running totals, period comparisons and customer activity.
  6. Data quality: duplicate detection, missing values, inconsistent categories, invalid data and validation.
  7. Business investigations: customer retention, revenue changes, sales conversion, marketing performance, payment failures and CRM pipeline analysis.
  8. Interview preparation: unfamiliar schemas, mixed-topic questions, debugging and solving problems without hints.
  9. Database development: transactions, indexes, query optimization, stored procedures and triggers, after building a strong analytical foundation.

I also want to practise with different datasets instead of repeatedly using a single table: e-commerce orders, CRM leads, marketing campaigns, finance/payments, inventory and SaaS subscriptions.

For each problem, I want to define the business question first, inspect the data, write SQL, validate the output, interpret the findings and recommend an action. I also plan to use AI to review my work, while still attempting and explaining queries independently.

My concern is that AI can already generate SQL and dashboards. I don't want to spend months memorizing syntax that doesn't demonstrate real capability. I want to develop skills that are useful in actual analyst work.

I'd appreciate feedback on these specific questions:

  1. Are these SQL topics and their order appropriate for entry-level data analyst or BI internships?
  2. Which topics should I master deeply, and which can wait until later?
  3. What realistic SQL tasks do junior analysts actually struggle with beyond standard interview questions?
  4. Is practising across several business datasets a good approach, or should I focus deeply on one domain first?
  5. What evidence would convince you that a beginner can use SQL reliably at work?
  6. Given AI-assisted analytics, what complementary skills should I prioritize after SQL: statistics, Excel, Power BI, Python, data engineering fundamentals or something else?

Please distinguish your own work experience from general predictions about the future of analytics. I'm looking for practical criticism and missing pieces, not just reassurance that SQL is still useful.

Thank you!


r/SQL • • 3d ago

Discussion I spent months building an app to database schema comparison and migration tools

0 Upvotes

Hey folks,

I've been spending my spare time building **FoxSchema** App, a free and open-source database schema comparison tool.

The goal is simple: compare schemas across different SQL databases and generate migration scripts without paying for expensive commercial software.

It's still a work in progress

Repo: https://github.com/tedious-code/foxschema

Npm: foxSchema install

Download : npm install -g foxschema

Demo:


r/SQL • • 3d ago

SQL Server When do you decide a query hint is justified?

Thumbnail
1 Upvotes