I’m designing a self-service AWS sandbox environment for my organization. Employees should be able to request temporary, isolated AWS accounts for learning, testing, and prototyping.

Each sandbox should have:

-Strict spending and time limits

-Controlled IAM access

-Restrictions preventing access to production or other AWS accounts

-Protection against disabling logging and security controls

-Automated cleanup and account recycling

-Centralized monitoring and cost alerts

I’m considering using separate AWS Organizations member accounts and evaluating Innovation Sandbox on AWS. Has anyone deployed it in production, or used another open-source/project-based solution? I’d appreciate advice about security pitfalls, SCP design, budgeting limitations, account provisioning, and cleanup automation.

I’ve programmed casually for about 10 years in a variety of languages, but I’ve only been doing it professionally (DE/AE) for around 3. I love this career. I used to enjoy the problem-solving of writing code myself, but what I really love is building: smart pipelines, better ways to collect metadata, simple apps that let the business interact with its data.

So naturally, AI has me feeling a sense of dread. The most senior dev on my team (mostly an app developer, 20+ years of experience) now believes AI can out-code him 99% of the time. He’s smart and I trust his judgment, which makes it more disappointing. He doesn’t even think we need to review code anymore, since GitHub Copilot will flag anything problematic.

On the learning side, I’ve read Fundamentals of Data Engineering, I’m partway through Designing Data-Intensive Applications, and I’ve learned Kimball’s dimensional modeling through video courses. Now I’m wondering:

• Is it still worth continuing this kind of deep learning?
• Does system-level and architectural knowledge still set you apart, or is that being automated too?
• Should I just learn alongside AI, questioning why it makes the choices it does?

What level of learning is actually worth it these days, and how are you staying ahead?

me irl

One of the things that got me into DuckDB was a series of technical articles posted to their site. This one was particularly eye opening:

Faster Dashboards with Multi-Column Approximate Sorting

I hadn't fully appreciated how important row order was to OLAP performance. In it they cover space filling curves as a method for mapping multiple columns (dimensions) onto a single "order dimension" to maintain proximity from the higher dimensions.

This works very well, but it means threading a curve though a set of potential values, at equal density, rather than what actually exists in the dataset.

Alternatively, by first approximating the rank of each value within its column (quantiles), and space-filling on that, the curve threads through the set of existing values. This handles skew a lot better.

This is a two-column ordering over the TPC-DS SF100 store_sales table, where ss_net_paid has reasonable skew. The rank-based methods manage to read far less of the table for a 1% range query.

I wrote up a more detailed post here:

https://www.duckstring.com/blog/clustering-for-pipelines

I'll probably be using this to set the row order whenever I overwrite a table and I have a reasonable idea as to what might be queried from it.

Hello All

I am a data engineer I work with databricks mainly but to develop any stuff I need to do from databricks ui and can't develop from local just like python notebook or jupiter notebook

So I buit an vs code extension \[spark cell runner\](https://marketplace.visualstudio.com/items?itemName=PrashanthReddyMunagala.spark-cell-runner) that runs pyspark locally or you can connect to remote databricks cluster either serverless or dedicated cluster

You can sync some data from databricks unity catalog to local catalog so you can test everything from local pool

Try it see if it helps

Happy to answer or you can contribute it's in github

Senior analytics engineer - spent last 5 years in FAANG mostly doing BI/DE work across small and large teams, and since last year i'm mostly doing fullstack for small internal multiservices app (data/stream intensive). WLB is ok but no career progression in current role.

Using AI tools extensively - both for internal purposes as well as for minor system integrations here and there, as well as maintaining knowledge repos enabling better collaboration using AI. Got cloud certifications, but no tool-specific certifications. Spent 2 years doing internships/ or unrelated corporate jobs before that.

I assume no career growth in my current role, but WLB is good and now I mostly work on software stuff (though i am still much much stronger in DE part), the only concerning thing is stagnation and comfort zone. Been contemplating a lot between:

- Rest & vest? ( I am fine with that)

- Sr. DE in scale-ups/startups?

- Attempt downgrading to some AI-company for a mid-DE role?

- Try to pivot internally to PMT / TPM job families? (Exciting, but concerned about the skill gap)

- Try to pivot internally into higher scope but more stress role? (No excited about that)

- Try to pivot into a more AI-parallel role? (Exciting, but I assume knowledge gap)

I want to get a better understanding of the market conditions? How realistic/unrealistic are these assuming some time to prepare? How hard is it to get interviews these days? How code-intensive vs design/architectural vs behavioral are they?

I had this conversation with one of our account manager but they seem to give me so many random answers on how it can be used.

Here are my doubts,

1. How do you define one semantic view, is it based on multiple ideas coming together into a unified view or multiple ideas into multiple views
   
2. After the change in orchestration, Agent tends to write its own SQL definition for the questions asked, how is it utilising the semantic view in this case?
   
3. Following up on the second question, how do verified queries work together. I have noticed it recognises the verified query but doesn’t want to use the definition which is correct.
   
4. Which model is being used for the agent to answer business-related questions?
   
5. Is it better to bring your own LLM and use cortex analysts ability to write code and let the LLM answer?

Let me know if you have found anything useful or good practice while building these views and agents. I want to understand it better.

Hello All

I am a data engineer I work with databricks mainly but to develop any stuff I need to do from databricks ui and can't develop from local just like python notebook or jupiter notebook

So I buit an vs code extension spark cell runner that runs pyspark locally or you can connect to remote databricks cluster either serverless or dedicated

You can sync some data from databricks unity catalog to local catalog so you can test everything from local pool

Try it see if it helps

Happy to answer or you can contribute it's in github

What's your take on the new Claude Dashboard by Anthropic? I still think open-source BI-as-Code will win, and Claude Dashboards are the Jupyter Notebooks of the age of agents. Meaning they are easy to set up quickly, but impossible to manage or maintain in the long run (if needed). A decade ago we discussed no-code vs. code, now it's in-built in frontier models/apps vs. open source or code-first.

Hi Folks - PM from Databricks here and just wanted to share a useful technical blog on how to conduct unit tests with SDP Pipelines today! Here's the use case:

Nightmare scenario - Your new data pipeline updates look good before going to production - every table builds, every expectation passes, and the gold tables have plausible values in them. Looks good right?

But if you push to production, the dashboards would still be showing inflated numbers and an unhappy business partner will be knocking on your DM's

For cases like this you need to utilize Unit Tests to ensure your transformation logic is sound. Expectations help ensure your data looks right, but it still misses what your transformations are doing.

With unit tests, you directly validate your transformations on data inputs you control before the real tables are built. Testing edge cases and finding those annoying "silent" errors

Checkout the blog for a deep dive tutorial on how to use unit testing with Databricks SDP Pipelines!

I'm testing various semantic layers and would love to test a canonical example from MetricFlow. Looking for something that is more than a toy. Any suggestions?

There isn't much openings for any DE roles let alone internships. So, I decided to up my skills and thought maybe whatever Jr. DE does in a company, I should do it too but for my personal projects. Anything might be helpful.

I’ve worked on a few BI migrations and thought some of the things I learned might be helpful if you’re planning one.

Full post is on the Metabase blog if you're interested: https://www.metabase.com/blog/making-bi-migration-suck-less

Weird flair but for real.
We work with dbt using stg, int, and dm layers. We also have 3 GCP projects where each layer's models get materialized (dev, uat, prd).
The semantic layer only reads from the marts layer.
So now imagine everytime we get a change in a KPI logic for example:
Dev goes and checks looker. oh let’s say it’s a sum of a column. Then they check marts, see it’s an aggregate of a calculated column in int.
Then they go to int, make the changes, commit, wait for cicd, verify in dev, move it to uat, validate with the stakeholder, and finally push to prd (all of this waiting for cicd and running dags every step of the way).

All of that shit and wait time just to fix one mini mistake. Do you guys have similar workflows? Is this way too much?

I’m honestly kinda used to it by now, but I can see the struggle with new devs like juniors or interns.
And this is assuming all the data is already available... if the request involves pulling in new raw data, it’s even more painful.

Hi all, I'm a trainee data engineer working under a data architect in a medium sized organisation. The role is a mix between a bootcamp and full time job. 

There's currently no senior data engineer and I'm being asked to create the data engineering platform in Fabric, however, this is a massive ask and I'm just trying to do the best I can.

We have multiple reporting-softwares that I'm now redirecting or connecting with to deposit data in the Onelake. The majority of data is obtained by accessing the SQL database of the reporting-software and making a copy into OneLake.

I need to come up with a plan to manage and standardise our approach to collecting this data and how we work going ahead.

My bare minimum plan atm is to ensure each reporting-software has a bronze, silver and gold workspace (a container effectively) and these can be managed via pipelines. I've been working out who needs access to what.

My larger issue is regarding things like metadata. I've very little experience in that field and I'm not sure how to go about it and could really use advice on how to implement a scalable platform that includes some kind of metadata management.

Context: I am setting up some infrastructure in AWS. In my previous experience at other companies, anything involving new instances must be done by DevOps or service desk. So I asked them to do it and they said I’m on my own.

Got RDS and OSS set up; hit a snag with EKS. I am currently one week into multiple conversations with service desk, management, and C-level to figure out who should be doing this. We deploy via Terraform. So I’ve got to add to that.

My primary concern here is that I’m messing around in places that affect other areas without any guidance on those areas. I’ve had to open so many tickets and bother so many people. Today I asked for Docker to do some local testing and I thoroughly confused the help desk.

We have an insane amount of security and I have multiple logins, MFA, biometric, etc. The layers inside AWS are similar.

Am I being too paranoid or is this somewhat reckless on the DevOps/SD folks? I’ll gladly accept being too paranoid and carry on, but for now, this just doesn’t sit right with me.

So many companies does a setup without much thought to security and they just hope no one finds out. Out of sight out of mind. I understand security is one of those things that aren't visible and one doesn't get rewarded for doing a proper job (and might even be penalized for time spent there). But I hate the common mindset of throwing shit at the wall and seeing what sticks.

Hey guys! Some of you may remember me, some may not, but SQLBuild has come a LONG way since I last posted.

First, SQLBuild has now been in production at my workplace for over a month! Huge milestone, and I'm lucky to have a boss that let me do it.

Second, I now have a much clearer idea of what this tool is and isn't. The goal is to make warehouses refactorable, such that they are easy and safe to change + kept that way over time.

Catch mistakes before anything runs

• A compiler that understands SQL. Types flow through every model, so unknown columns, type mismatches, bad GROUP BYs, invalid function arguments etc... fail at compile-time (offline). Fully OSS and free
• A Rules API for your own compile-time rules, e.g. marts can't reference sources directly, naming patterns, SQL shape. Basically, your team conventions, enforced as code
  • This is a standout feature for those wanting to cut review time

Prove it works

• Parameterised, macro and UDF tests, plus multi-model/E2E tests across whole chains of models
• Native data diffs between dev and prod, or against arbitrary queries

Change it without rebuilding everything

• Replay-on-change policies - When a model's SQL (or something upstream) changes, choose how far back to replay (1h, 14d, 1mo…), not just full rebuild or ignore
• Rename or move a model with sqb rename/ sqb mv. These update every reference (e.g. macros, audits etc...) and keep the existing table's historical data. Plain file renames of models are detected automatically too
  • The old name keeps working through a compatibility view for 30 days (configurable), so that dashboards and other teams' queries don't break straight away
  • sqb plan --as prod previews the migration against prod before you run it
  • If you have tried to do any large renames in your warehouse, you will know how risky and annoying this normally is
• Column renames are also supported. sqb rename stg_orders.amount_revenue renames it in the model and every downstream model that reads it (based on compiled lineage, not find-and-replace)
  • --cascade carries the new name through downstream models that pass the column straight through
• The janitor runs on demand and *archives* before deleting, so tidying up isn't scary

Macros don't have to be global anymore

• In most tools every macro is project-wide, so changing one can break models anywhere. In SQLBuild you can keep macros, enums and constants next to the models that use them, and only that folder can see them:

​

models/
├── staging/
│   └── stg_orders.sql                # can't see anything in marts/_sqlbuild
└── marts/
    ├── _sqlbuild/
    │   ├── _enums/payment_status.sql # only visible to models in marts/
    │   └── _macros/currency.py       # cents_to_dollars, same
    └── daily_revenue.sql             # uses payment_status and cents_to_dollars

• Change one and you know exactly what it can affect. Before moving a model, sqb scope model:x --as-path new/path.sql tells you what it would lose.

What's been removed

• The big one is virtual environments. They were a lot of complexity for something I wasn't using, and most of the things I wanted from them, I can achieve via other means. (feel free to ask below)

What's up next

• A fully fledged deployable UI from which you can run your models etc... There will be SSO, audit logs, fine-grained permissions etc... all baked into OSS at no extra cost. (ETA Jan 2027)
  • If this sounds too good to be true, you are welcome to ask why in the comments
• Compiler likely fully in rust. Compile times are already good, but the only way to let it scale for larger projects (e.g. 10k+ models) is if it's 100% in rust. Planner and executor etc... will stay in Python (dominated by warehouse time)

Try it:

pip install sqlbuild
sqb playground waffle-shop
cd waffle-shop
sqb build

sqlbuild.com · Github

Hey all! We just released Polars 2.0.

We are really happy with the results. It removed many of our "legacy decisions" (mistakes), makes the streaming engine our default and promotes SQL to a first class citizen within Polars. We initially said it would be a small ("boring") release, but we managed to sneak in a lot more than anticipated.

It comes with our initial out-of-core (spill to disk) support, a new Map data type and a lot of performance improvements. In fact, we think Polars is now one of the fastest analytical SQL engines on a single node. See benchmarks in the post: https://pola.rs/posts/release-polars-2/

To help you upgrade, we also posted a migration guide: https://docs.pola.rs/releases/upgrade/2/

Disclosure: I am involved with Polars

For reference, I am the Data Scientist in this scenario. Since we are not on the stage where we need to do ML/DS, I delegated myself to the Data Engineering/Architecture side of the project.

I have been with the team the longest; we needed to add more people in to accommodate more projects.

We are currently working on a system transition for a big company (think millions of new data a day) and everything is being transitioned to DataBricks. The current Data Warehouse is being kept under MSSQL and we are moving away from it due to processing time concerns.

Most of the ingestion process are done through python; which is optimal in this case since the framework is similar between all data sources and it relies heavily on proper automation. Data Engineer designed the ingestion process and he’s done excellent work migrating our DataLake into DB.

However, I am hitting a wall with him on our silver layer transformations. We have allocated the workload between different KPIs. Most of the silver layer transformations here are your simple filtering, joins, dedup, or conversions (rarely).

The analysts I am working with are happy with my Silver Transformations (mostly SQL); the updates are in real time and it’s easier to validate since the codes are written as views so the transformations are transparent to everyone.

Meanwhile, he has more back and forth with his analysts since his transformations are run through notebooks (which none of the analysts have access to).

This is where I am hitting a conflict with him: he is suggesting I should convert every silver tables (i made) from SQL to Python so that the transformations are standardized. I made an argument on how 90% of the time, the transformations we do is more practical in SQL (processing and validation wise) and there’s really no need to implement only one approach. We will use python if we need to integrate more complex transformations or when the need occurs.

He said python has less issues overall (which sounds dismissive for me).

I am posting here to gain more perspective. He had worked with the Data Engineering side for twice as long as I have; but I have relevant experience with the data process (from start to finish) which allows me to relate which makes things easier and convenient for the analysts. I am open to converting everything to what he needs if given the right argument.

I'm a data scientist and work with python/pandas data structures quite a bit, but work in a company where excel is still commonly used. I kept running into the same problem when translating spreadsheet logic: the obvious pandas equivalent often isn't semantically identical to excel. So, I built a small browser tool that translates common excel and pandas patterns client-side without the need for an llm.

I've tried to develop the tool to handle as many edge cases as possible (duplicate lookup keys, approximate lookups, IFERROR-wrapped lookups, etc.), but it's still fairly new, so there might be edge cases I'm still missing, which I'm working on improving.

Tool: https://dataframe.tools/excel-to-pandas/

One of the things that surprised me while building it was how many seemingly simple excel formulas need extra handling in pandas to actually preserve the same behaviour.

A lot of software engnieers I work wth now spend 99% of their time tellng agents how to write code.

I am curious to know how much we delegate work to agents as Data People - given so much of our job is not just writng code

Please include a score out of 10! 10 = agents do all my stuff, 0 = still doing everything by hand

I am a data engineer in Finance. Recently I am suffering from understanding financial terms and its accounting treatment. One big issue is, as a frequently used term, Repo and Rev Repo I understand them several times, but after a period not touching the terms and related data I will forget almost all of the details. And then have to revisit the definitions and relink them to financial activities.

For example, recently I am involved in a meeting talking about Repo Fails transactions. User wants to know what’s the exact accounting policy we applied in our data logic. My first reaction is, how to identify the Repo in our Fails data, however, Fails will only record as Fail to deliver or Fail to receive. The only way to find a Repo is from the trading account. We maintain certain accounts for Repo trade only. Question is I even can’t have a clear filter to scope which accounts are Repo or not.

It takes me almost a whole day to look into firm’s internal documentation to understand what the user means and how our data pipeline process it. Then, with these background, I joined the meeting. In the meeting, I listened carefully but didn’t fully comprehend what the user means as lot’s of non Tech terms. Fortunately I got something like below

For a repo trade how it is recorded in accounting

On Trade date (T0)
Dr (i.e., Assets in BS): Repo
Cr (i.e. Liabilities in BS): Rev Repo

On T1 if settled successfully
Dr: Receivables ???
Cr: Cash ???

On T1 if settlement failed
Dr: Failed to Deliver
Cr: Payables

On TX(usually no longer than 5 calendar days) if settlement success
Dr: Receivables
Cr: Cash

This time my understanding for memory: Repo means repurchase, we should provide securities to the client and client give us money. And by someday (possibly overnight), we need to return the money + interest to client. So the money measured assets is still on our side on trade date. We recorded an assets repo and liabilities Rev repo. If not failed, user doesn’t mentioned. If failed, we failed to deliver the securities to client and thus we record FTD in ledger, by this time , it already lose the ability to track whether the trade is related to Repo or not, we just know it’s fails and needs trading account level information to know whether it is a repo trade. Once this fails finally got settled, we record receivables in Assets and we got cash from client but record payables in future.

Additionally, we have repo control accounts, customer control accounts that store all these misc temporary trade activities and will be cleared to 0 finally.

Ok, now I got a very detailed explanation this time. But after a period, I will forget and have to repick these details when user reaches months later.

How I can have these details memorized in my mind so next time it can be automatically presented in my eyes when user reaches months later and asked for another related question?