I have JSON files in an external stage (S3) and I'm trying to load them into Snowflake first in a landing table and then eventually into a dimensional model in a target schema.

I just learned that I can't run COPY INTO statements with fancy SELECT queries that include a lateral function, since that's what I need to parse out the JSON file. I started to just build a basic query that SELECTs the data from the JSON file and just insert it into a table by doing essentially this:

SELECT 
    mycolumns
FROM
@mydata.my_stage_name/ALL/
, LATERAL FLATTEN(input => $1, outer => true) as data

Are there any better options that I could use or at least try out before I fully commit to that option?

My company currently has JSON data landing in a "staging" table in Snowflake nightly at 11 pm. There's dynamic tables that transform the data from a variant column into a structured table. Currently we have the dynamic tables set up on an hourly lag even though data only comes in once a day. The reason being is there are multiple scheduled tasks that refers to these dynamic tables after 11 pm and we do not want the lag to be such a wide range incase the dynamic tables have not refreshed, but these tasks run and will not capture the latest data in the "staging" tables.

Should I change the set up? Is it costing the company Snowflake credits by having the dynamic table checked hourly even though we know data will only come in once a day? Is there no way to schedule a refresh on a specific time for a dynamic table?

PM here from Snowflake Postgres.

I was the chief product office of the Crunchy Data team that was acquired just before Summit over a year ago. The top ask from most early customers was easing getting their Postgres data into Snowflake. We were already hard at work on it from day one, and even some of the foundations of it with pg_lake date back to over 3 years go.

Today that capability to mirror your data from Postgres to Snowflake is going GA. The blog post has more details, but happy to field any questions folks may have about how it works, the engineering behind it, or just Postgres in general.

Blog post with more details at: https://www.snowflake.com/en/blog/data-mirroring-ga-snowflake-postgres/

My client is investing heavily in dbt cloud even though we were extremely happy with dbt Projects in Snowflake but more of a client's management decision . Our challenge is that in dbt cloud we cannot install most of the dbt packages like Elementary etc . At this stage security has only allowed us to use dbt_utils . We do a need to a lot of Data Quality which can be done using Snowflake DMF especially in the silver and gold layers for regulatory reporting .

We need to store the results of the failed results just not the "number of records which are null " . We are now looking at hybrid data quality framework, any pointers ?

We have well defined DQ tests in dbt .yml files which we would hate to loose and I can use the DQ in dbt for inline DQ testing ( we have approx 17K Not Null Tests defined in dbt ) and the DMF for more of a regulatory reporting framework . Any pointers .

An important write up about Snowflake's largest competitor.

I'm running a query in Snowflake, and it works fine when I test it directly. But when I run the same query through a data pipeline, it fails with a SQL compilation or permission error.

I'm using the same Snowflake account, so I'm not sure what is different.

Could this be related to the active role, warehouse, session settings, or how the pipeline connects to Snowflake?

How would you troubleshoot this?

Hi all, we have recently migrated from RDS to Snowflake Postgres, mainly to store our data within the Snowflake databases using Data Mirroring feature.

In general there were a lot of nuances such as secure network connectivity, adjusting default encoding to UTF-8 and making sure to keep our auto vacuum configurations intact.

I have documented my experience in this Gist.

Furthermore, we observed index bloat, and I was surprised to see that after the restore, the data volume dropped from 139 GB to 30 GB of "clean" data.

I’m curious how you approach migrating large volumes of data to Snowflake Postgres and what tools/practices do you use?

Is there anyone who will attend the world tour on October 15th? If so, let’s connect :)

As the title says I was able to pass the snowpro core exam today. Plenty of questions on copy into, RBAC, data replication, accelerating, and what will happen once a query is executed relating to views that a user or app is supposed to see. Anyone passed the advanced admin exam recently? I have employer paying for the exam. Trying to decide between data engineer vs advanced admin since I am devops and not a pure data engineer. Any insight would be good. I plan to pivot to a snowflake admin role but from the job posting I am seeing are asking for more than what I consider to be admin level tasks. Any insight on interviews for the admin level roles. I also have not located any admin level courses or paths for this cert.

I'm building an application that ingests MatrixCare data directly from Snowflake. I'm currently working with the Orders and Progress Notes tables, but I've run into a couple of documentation gaps that I couldn't resolve via Google or Al search:

1. Orders . OrderStatus**** What do the numeric values in this column represent? Is there an official lookup table or mapping list for these status codes?

2.

User Roles / Permissions for ProgressNotes**** I need to determine the role/permission of the note creator (e.g., Nurse, Provider, etc.). I noticed DisciplineID in the ProgressNote table, but I'm unsure if or how it maps to creator roles or permissions.

Has anyone integrated MatrixCare data from Snowflake into an external app before and knows how to translate these IDs/numeric statuses?

Thanks in advance for any insights or reference docs!

Hey, we have been on snowflake and there have been times where every now and then a datatype from the source will be larger than the target data type which at some point will lead to a break. For example, lets say source is varchar 125 and target is varchar 60.

Going through each ETL and reviewing for this would be very time consuming and could still happen for future builds. Is there any out of the box functionality in Snowflake that would help for this? What have you all done to prevent this issues and have proactive alerting before the failure?

We updated our dbt pricing guide because the way dbt is packaged and priced has gotten a bit more nuanced.

One thing we think gets overlooked is the difference between the cost of the dbt engine itself and the cost of running dbt as part of a production data platform.

The engine is free. But once you factor in things like consistent developer environments, usage-based pricing, orchestration, infrastructure, and the engineering time needed to maintain everything around it, the picture changes quite a bit.

We also added a breakdown of the current dbt Platform plans, the usage-based components, and some of the other costs teams should think about when comparing options.

Curious how other teams are looking at this. Are you mostly evaluating the software/license cost, or are you also factoring in the engineering and infrastructure costs around dbt?

https://datacoves.com/post/dbt-pricing

A while ago, I forked a Snowflake emulator and started extending it as a learning project using Go + DuckDB.

The idea is to experiment with Snowflake concepts locally without needing a cloud account. It supports a subset of Snowflake’s behavior, so there are still limitations.

This week I released v0.5.0. The web console now lets you collapse the results panel and highlights the line being executed in real time when running multiple SQL statements in sequence.

You can try it with Docker:

docker pull ghcr.io/oswaldonisango/snowflake-emulator:0.5.0

docker run --rm \
  --name snowflake-emulator \
  -p 8080:8080 \
  -v snowflake-data:/data \
  -e DB_PATH=/data/snowflake.db \
  -e STAGE_DIR=/data/stages \
  ghcr.io/oswaldonisango/snowflake-emulator:0.5.0

Then open http://localhost:8080.

I’m building this to learn and experiment. If you work with Snowflake or are interested in SQL, Go, or DuckDB, I’d love to hear your thoughts. Bug reports and contributions are welcome too.

Repository: https://github.com/OswaldOniSango/snowflake-emulator

Release v0.5.0: https://github.com/OswaldOniSango/snowflake-emulator/releases/tag/v0.5.0

Hi All,

I just passed my exam this morning, and if you are like me you were nervous and looking on this forum for any guidance.

I will say the test was very hard. I did all of Tom Bailey's practice exams as well as some additional resources. I really didn't think the test was similar to any of the resources publicly available. I answered most of them through process of elimination and through deductive reasoning.

I know everyone has a random subset of questions so I don't know how helpful this will be but some of the frequent topics I saw were:

- Data Staging and Loading (Snowpipe, COPY INTO formatting, file formatting, JSON files, how to call a staged file)
- Data Sharing (Private access, secure access, how to receive a data share, permissions needed etc,)

- RBAC and permissions (lots of questions on inherited permissions on databases / tables / schemas)

- a few questions on query optimization / processing - good to know about QAS , Search, Micro partitioning, Pruning

There were some questions on info I hadn't even seen before (could have been the tester questions)

I was told to know a lot of SQL so I studied a lot of the Tom Bailey questions for those and tried to memorize commands - I don't think any of them really appeared in my exam.

Just wanted to brain dump in case this helps anyone. Best of luck!

It all started with "workspace".

Then AI agents and all.

Then Coco.

Now the whole object explorer is changed, making us to navigate a labyrinth of buttons just to get the definition of an object.

Previously the query history and result panes were used to be side by side, looked so beautiful and elegant. Then some genius decided to put one of them under the other and occupy the whole screen. This is by far the worst and most frustrating UI downgrade for me personally.

UI team of snowflake, please change your design philosophy.

What's up, Snowflake enthusiasts! I tested and built a way to systematically build Snowflake Streamlit apps with Claude Code and other AI tools. I developed this open-source system that I call Streamsnow, which walks you through the process of getting a GitHub repo scaffolded and structured and connected with your Snowflake, so you can utilize Claude Code to build all of your Streamlit applications in a standardized Way, while also providing you the SQL to check to make sure all of the queries being utilized within the app check out as correct.

In my time building dashboards with AI, I always found it confusing how to quality-check them and how to build them in a scalable way across data teams, and so I wanted to build a super standardized system that made building, deploying, reviewing, and coordinating across teammates is super easy, and I spent weeks working on this to achieve that. I think one of the best features is that you can actually go into an auto-review cycle and just have Claude code iterate over the dashboard, making it better without your input. You can also standardize the design for the dashboards as well.

It is completely secure, so AI will not see any of your secrets that you use to connect GitHub to Snowflake. By default, it is set to put these Streamlit apps in a staging area on Snowflake, so you don't have to connect a GitHub repository, and all of the deployment is automatically handled in the CI. To get started, all you need to do is tell your coding agent to download the plugin and then run the onboard command. It will walk you through it all in a process that I've timed down to about 5 minutes of actual work. You will need to be a Snowflake admin or be coordinating with a Snowflake admin to get it set up too.

By the end of my time at my job, we had basically replaced external business intelligence tools with this system, so it saved us literally tens of thousands of dollars. Hopefully you all can extract the same value out of this!

I have an upcoming project that uses Airflow, dbt, and Snowflake. I’m comfortable with Airflow and Snowflake, but I have no experience with dbt.

I’m looking for advice from people who have worked with this stack. Can you recommend any good courses or learning resources to get started?

I’ll be working as a team lead, so I want to get up to speed as quickly as possible. Honestly, I’m a bit nervous about whether I’ll be able to handle the role well.

For those who have learned dbt, how long does it usually take to become comfortable with it and work confidently on real projects?

Hey, for a few years, we have had end users query views instead of table. This was especially true when we were on Redshift until we migrated to Snowflake. While being in snowflake we have heard and seen the execution of queries against views be slower than tables. For example, most of our views are same as the table ddl just without a few audit columns. I can't seem to understand this. Interested to get your rule on which ones you have end users (analyst, applications, reports) using and if you have also noticed the performance difference and know why it occurs

I'm working with a Snowflake process where the following session variables are set:

SET PII_HMAC_PAYLOAD = 'null|null|null';
SET PII_PROCESS_NAME = 'PPC_SNOWSIGHT';

I'm trying to understand how these properties are used for PII encryption/decryption.

For example:

• Are these variables used by a stored procedure or UDF?
• Where is the actual encryption/decryption logic defined?
• How is the encryption key or secret managed?
• Can the encrypted value be decrypted back to the original value in Snowflake?
• Is there a recommended Snowflake-native approach for this use case?

Has anyone worked with PII_HMAC_PAYLOAD and PII_PROCESS_NAME before and can explain the expected flow?

We have started to experiment with Cortex Analyst. I made a set of prompt instructions that are supposed to limit use to topics that are against EU AI Act and similar no-go use cases. Then I made a set of evals to test the alignment and it seems to be really easy to bypass the instructions with user prompt.

What are the best practices and learnings from the crowd here?

My company is looking into deploying more “sophisticated” and flexible internal apps directly around our Snowflake data.
We’ve tried using Streamlit, but we’ve run into some limitations and performance issues. We recently tested Snowflake Runtime Apps, and my team really liked the flexibility and overall experience.
The main concern is that we haven’t found many real-world examples of companies using Runtime Apps at an enterprise scale.

I’d love to hear from anyone who has actually deployed them in production:

How has your experience been with security? Any concerns around data exposure, permissions, or potential vulnerabilities?

How does the app actually query Snowflake data compared with a more traditional Streamlit app?

Have you noticed meaningful differences in query performance or compute consumption?

How has cost behaved once you have multiple users/apps running?

Would you consider Runtime Apps production-ready, or would you still recommend keeping traditional dashboards?

I’ve been exploring how AI agents should access enterprise data in Snowflake.

Connecting an agent to Snowflake and letting it generate SQL is relatively straight forward.

The harder questions are:

• What data should the agent actually access?
• Which definition of “revenue” or “customer” should it use?
• How do you prevent sensitive data exposure?
• How do you stop agents from bypassing governance?
• How do you audit what the agent actually did?

I’m experimenting with an architecture like:

AI → Identity → MCP → Cortex Agent → Semantic Layer → Governed Data

The idea is to avoid giving an agent unrestricted access to raw tables and instead put identity, governance, semantic definitions, and auditing between the agent and the data.

This made me think:

Should enterprise AI agents ever get direct SQL access to raw Snowflake tables?

Curious how others are approaching this in production.

Our Snowflake account team has been saying we could potentially deprecate our Tableau instance by going all in on Snowflake Cowork, along with using CoCo to build Streamlit or JS apps on SPCS.

We do have a number of business users, including execs, using Cowork, but questions seem to quickly fall outside the scope of our semantic views, or the agent struggles to generate accurate SQL. This puts pressure on our engineering team to build/maintain more semantic views or verified queries.

I haven’t tested CoCo much for building Streamlit or JS apps, but I’m sure it can do a decent job. I could see it being great for our engineers, but I’m hesitant to have business users who don’t know SQL, let alone Python or JS, vibe code business facing or critical applications.

Has anyone actually been successful deploying this new tech to less technical users? Have you been able to deprecate your BI tool, or at least reduce seat count, as they’ve suggested?

Curious how others are thinking about these tools fitting into their overall tech stack.

I've been Snowflake certified for 5+ years now but only ever renewed my Core cert. With the now updated COF-C03 I'm wondering if I should just shoot for the Architect level cert instead since I'll need to again either renew my current cert by next March or level up with a pro cert.
As such, for those who have passed the Architect cert do you have any good resources that could help with studying or prep?

Has anyone successfully gotten a VS Code extension whitelisted for Snowflake CoCo Desktop?

I opened a support ticket and was told to work through my account manager, but so far I haven't gotten anywhere. I'm specifically looking at the new dbt Charts extension, but there a handful of others that would be nice to use.

We've been loving CoCo desktop, but the blocking of most extensions has been a pain.