Is Trunk Based Development possible if an integration test takes too long to run?

[u/Data-Panda]

I'm proposing my team move from long-lived dev/stg/prd branches to trunk-based development, with PRs straight to main.

I'm fairly new to this, so forgive me if I'm missing something obvious.

The sticking point is testing before merge.

Our current integration test runs the entire DAG end-to-end. This is fine as a single check before merging our test branch (which will contain all commits from developers over the sprint) to main. But it's slow and costly if it were to run as part of every PR. Today we run it manually, check the outputs, then merge.

I am wondering if it’s bad practice to leave this integration test until the end of the sprint. As in, developers merge to main (with some automated checks running as part of each PR before merge), and then when are team want to deploy the sprint changes, first run a single integration test before creating a new release tag In GitHub.

This does mean that main can’t be considered prod ready at all times.

Edit: we’d be using release tags, so for rollbacks we’d rollback to last stable tag. For hotfixes, we would cut from last release, the merge changes to main and create a new release with target set to hotfix branch.

[u/razzledazzled]

I think it’s a bad idea to leave main in an undeployable state. What if you have to roll back? What if you need to hot fix?

The testing suite may need some improvements, there should be unit testing for branches that is fairly quick and then integration test once the PR is up for review. This keeps things in a reasonably certain state of “it works” throughout your development process

[u/Data-Panda → u/razzledazzled]

We would be using release tags. So rolling back would involve rolling back to last stable tag. For hotfix, essentially cutting from last release, making changes, merge to main, and then finally a new release with target set to hotfix branch instead of main. 

Integration test once PR is up for review is part of the problem. It takes too long to run for every developers PR (more than an hour), and there also concerns around cost if there’s multiple PRs submitted daily 

[u/IN-DI-SKU-TA-BELT → u/razzledazzled]

In my experience this rarely happens. Enforce people’s short lived branches to be rebased before a merge, and make sure tests are passing.

People will be quite quick to fix the main branch, and worst case you can git revert and they can fix it and try again later.

[u/Pretend-Clock8313]

main not being prod-ready isn't the real risk, it's not knowing which commit broke the dag until sprint end

[u/Data-Panda → u/Pretend-Clock8313]

So we would actually be running the latest code on main into staging each evening on its own schedule - and would have alerting setup. So technically our integration “test” would be running everyday, and we could identify issues throughout the sprint 

[u/Own_Candidate9553]

You don't say how long the rest run takes, which changes things.

Regardless, if it takes a long time, it could be run nightly instead as a compromise, if everyone commits to fixing main if the suite fails. This can be hard to do. If you have support for it you could automatically block merges to main of the last test run failed to force it.

[u/Data-Panda → u/Own_Candidate9553]

The integration test we basically run the full DAG which can take over an hour. 

We will be running it nightly. And yes having main fixed as a priority if the staging run fails overnight was my initial plan. 

[u/WilliamMButtlickerIV → u/Data-Panda]

Coming from the world I'm in, an hour doesn't sound so bad 😅

Can you pick a handful of tests that cover the broad cases, and use them as a pre-submit gate? Then run the full suite nightly?

[u/Data-Panda → u/WilliamMButtlickerIV]

Yes I think that’s what I’ll go for. By pre-submit gate, are you referring to tests that need to pass before merging can happen?

[u/WilliamMButtlickerIV → u/Data-Panda]

Yes

[u/nullpotato → u/WilliamMButtlickerIV]

Same, we have to test on prototype hardware so anything doing a regression suite under 6 hours is "fast". We still block PR pending results of those depending how potentially inpactful the changes are. Config change for one teams tests? Easy review. Changes to scripts that must work for every test company wide? Integration overnight suite and checked manually too.

[u/Data-Panda → u/nullpotato]

For the high-risk changes, how do you decide what counts as critical? Do you flag certain scripts so a PR touching them automatically blocks until the full suite passes? And does the overnight run batch up all the day's PRs into one run, or does each PR get its own?

Just curious how this works. 

[u/Own_Candidate9553 → u/Data-Panda]

Depends, what are you using to run the tests?

In various ci/cd systems, you can usually configure a job to run when a particular file is changed. So you could move critical scripts into a special folder or just make a list of them, and run the "critical scripts full test job" when any of them are updated in the PR or branch.

[u/Wooden_Jelly_5295]

Could you run a smaller critical-path check per PR and the full DAG nightly? End-of-sprint failures come with a much larger suspect list.

[u/Data-Panda → u/Wooden_Jelly_5295]

Yes that’s pretty much what I’m planning. So we would have the latest code from main being run into staging every night, and we would have some basic alerting in place if anything critical breaks. 

[u/taleodor]

Main being not prod-ready at all times is fine as long as you keep track of what exactly is prod-ready - like mapping specific commits to releases. If you have that, you're able to cut a release branch at any point from that state.

[u/Data-Panda → u/taleodor]

We will use release tags, which I think covers it? Once a release tag is created, we’ll have an automation that updates our release config (in Dataform) with this latest tag. 

Although I haven’t come up with a hotfix approach yet 

[u/Crimzx → u/Data-Panda]

Why would a hotfix be any different?

[u/Data-Panda → u/Crimzx]

I mentioned in an edit on my post what we’d do for a hotfix. 

Concern was that I wanted to create a release with just that hotfix (ignore all other commits that have been added to main since the last release). I think my suggestion in my post edit covers this

[u/Crimzx → u/Data-Panda]

You can also cherry pick the commit into main which may be cleaner.
So create a branch from the release tag. Put hotfix there. Tag a release from that branch with the hotfix, then cherry pick the hotfix commit back to main.

Maybe you already meant this and it's just semantics though.

I know your question was about tests but just wanted to back ya up that you're thinking about it right.

[u/Data-Panda → u/Crimzx]

Thanks - yes I think that's the same thing. I guess I'm not sure if there's a difference between "merging the hotfix branch into main" and "cherry picking it back to main". I think it's the same thing here?

[u/Crimzx → u/Data-Panda]

Cherry pick keeps the main line linear where as a merge adds a merge commit, so it just keeps the git history a bit cleaner.
So like if you have version 1.1.0 on main and you are currently running 1.0.0 and 1.1.0 isn't released yet you probably have a version file or something like that. You create the hotfix branch from 1.0 tag, add your 1.0.1 and bump the version in the file.
If you merge this back you now have a merge conflict on the version file to resolve.
Instead you can just cherrypick the hotfix change commit back to main and main stays at 1.1 just now has the 1.0.1 hotfix in it, and your 1.0.1 branch is able to be released as 1.0.1 until 1.1 is released with the hotfix in it as well.

Ultimately though it's a team preference. I've worked places that didn't care about git history and places that REALLY cared about it.

[u/amarao_san]

I hate when staging is living in separate branch.

For infra repos, master is the current production. Dev and staging are productions too, you just care about them less.

Each environment is a separate directory (tf vars, inventory, whatever), code is shared. You can deploy staging from non-produciton branch (you have this semantics), but master is the reference state for both three (or more).

All other models are broken and diverging.

[u/Adventurous-Hyena220]

Trunk-based development can still work here, but Id look at splitting fast PR checks from slower end-to-end tests and keeping releases tied to a verified commit.

[u/Data-Panda → u/Adventurous-Hyena220]

So my current plan so far is we have the fast PR checks, then every evening we have an end-to-end test in the staging project, and we'll keep an eye on alerts for that each day.

And for releases, we'll use Github release tags. On a dedicated day at the end of the sprint, no new PRs will be approved, a final end-to-end test run, and a release created once it's all signed off.

Something like that.

[u/qa_tinkerer]

You can do trunk-based without running the full DAG on every PR. The part I'd push back on is accepting that main won't always be releasable. That's what bites you: by release day you have two weeks of commits and no idea which one broke the pipeline.

What tends to work:

• Fast PR checks that catch most breakage in minutes: unit tests, schema and contract checks, and a smoke run of the DAG on a tiny fixture dataset.
• Full integration run on main after merges, either every merge or batched (every few hours, or every N merges) if it's expensive. When it goes red, the list of suspect commits is short and you revert or fix forward straight away. Treat red main as stop-the-line.
• A merge queue (GitHub merge queue, GitLab merge trains) if you want main to stay green. It runs the heavier checks on a batch of PRs before they land, so you don't pay the cost per PR.
• Feature flags for anything half done, so merging doesn't mean exposing it.
• Tag releases only from a commit that passed the full run, not just whatever is latest.

It's also worth asking why the integration test is that slow. A scaled-down dataset often catches the same kinds of failures (schema, wiring, permissions), and the full-volume run only needs to happen nightly or before a release.

[u/paul_h]

Automate that integration test. Run it nightly as a first goal. Don’t get stuck in the “human reviews results after run”. Have that a machine decision. It should be perfectly correct for that to be a release candidate.

I co-created Selenium the web testing tech before playwright 22yrs ago: at conferences QE folks would find me and ask “what % of test failures is ok for the overnight selenium test suite” and would look shocked when I said 0%. It’s 0% red for your test too .. whatever that means for a single automated test that’s not a suite as such.

I’m also the maintainer of trunkbasedevelopment.com, for what it’s worth.

[u/ShipKitDev]

One option nobody has mentioned yet is GitHub's merge queue. PRs get fast checks (lint, unit tests, maybe a small DAG subset), and the slow integration run is triggered by the merge_group event instead of pull_request. The queue can batch several approved PRs into a single run, so you pay for one hour-long test per batch rather than per PR, and main only ever moves to commits that passed the full suite.

It's also worth checking whether you can run only the affected part of the DAG on each PR. If you can work out which models or nodes changed plus their downstream dependents, that subset is often a fraction of the full run and catches most breakage, leaving the full end-to-end run for the queue or nightly.

Either way you keep main releasable, which makes the hotfix question simple: fix on main and tag, same as any other release.

[u/Data-Panda → u/ShipKitDev]

Thanks - I’ll look into merge queues. And yes have considered running just part of the DAG but having trouble with it.

The problem with fix on main and tag is you end up releasing all changes up until that point along with the hotfix. 

My hotfix approach so far is cut a branch from the latest release commit (what’s actually running in prod) make changes, then:

1. Merge that branch into main 
2. Create a new release with the target branch set to the hotfix branch instead of  main

[u/givebackmac]

The rolling back argument always seemed strange to me. We keep all packages deployed to production environments for a significant period of time. Rollback doesn't involve source control, it involves redeploying the last good known package/artifacts.

That said, the time it takes to run tests matters a lot and depending on your situation it may make sense to do integration tests on a schedule if they are long running. Honestly though, long running integration tests are a problem and should be investigated. Is it too heavy/bloated? So many potential causes of long running tests to list but that should be a priority to at least understand.

[u/Data-Panda → u/givebackmac]

I should say that integration “test” right now is just someone running the full DAG in staging and manually checking the outputs. The DAG just has many steps and phases that make it long running and slow. 

For rolling back, I think because I’ll be suggesting using Git release tags, basically our rollback process will just be changing out prod release config with the last stable release tag - so should hopefully be ok.

[u/0x4ddd]

Really depends on your testing strategy.

You could run some subset of tests on every PR and setup cron to run full test suite once per day on main.

[u/Data-Panda → u/0x4ddd]

Yes I think this is what I’ll propose to the team. Not perfect, but maybe the best solution for us

[u/relicx74]

I don't know how many people you have, but that doesn't sound great. A big part of running the tests every time is that when they fail, you can point to the commit that broke them and have a much easier time fixing the code or the test if requirements changed.

The way your proposing you know you're going to end up in trouble.

Git feature branch workflow was always my favorite when there are multiple developers.

[u/Data-Panda → u/relicx74]

It’s a small team, so easy enough to see what commit was the problem. We would still be running tests, just not necessarily the full end-to-end test. That would be done once a day or at the end of the sprint. 

And we’ll still be using feature branches that get PR into main. That’ll trigger the tests

[u/relicx74 → u/Data-Panda]

If people there like it, that's great then. I don't have to live with it, so doesn't make much difference to me.

[u/Data-Panda → u/relicx74]

I mean, I’m just trying to go with whatever the best solution is. So far this seems to be it despite the flaws.

[u/Immediate_Wheel1953]

We ran exactly this: fast PR checks, full e2e on main every night. The bit that actually hurt us was nobody owning the nightly run, so a red build would sit for days and then nobody could tell which commit broke the DAG. What fixed it was making the on-call owner of any red main and bisecting back to the culprit, which usually took minutes, not hours. Your plan sounds solid as long as fixing main beats new work when it goes red.

[u/ekydfejj]

We do two levels of "trunk" branches. RC and develop. Everyone branches from develop for normal daily work. RC can be branched from for hotfixes. Allowing develop to lag until you're actually ready to cut a new realease. CI runs against both develop and RC, but many steps omitted from develop (building containers, deploys etc) and the base CI steps run, i.e. lint, compile, test, integration test etc.

This keeps CI quick when merging into develop, RC is always long as it will run all the develop steps, plus actual build and deploy images to ECR.

We have long term "feature" branches when they are required that will auto release to a siloed dev server, but that is for an extra testing stop before they reach develop. And don't have to be part of the flow.

[u/MasterOfTheWind1]

Trunk based is good for replacing a dev env.

But an env between that and production is desirable so you can test how branches integrates

[u/Used-Alternative-01]

Pretty common setup actually, you don't need every merge to be prod ready, you need every release tag to be. What helped us was splitting the suite: a cheap subset of the DAG (or the same DAG on a small sample dataset) runs on every PR, and the full end to end run happens nightly on main plus once more before you cut the tag. The nightly run is the important bit, because if you only run it at sprint end you find out about a break two weeks after it landed and have to bisect through a pile of commits. Feature flags help too, so half done work can sit on main without being live.

[u/divestblank]

Yeah, it's possible, but you must risk some level of regressiona going into trunk. Some features are more critical than others.

[u/Gabe_Isko]

It's not optimal, but it is still doable. This is what continuous integration is supposed to mitigate - a new integration test is performed on every commit through integration, so when it is time to merge in a PR, you already know what the test results are.

The issue with this approach is that it scales horizontally, and the larger and more complicated your app is the harder it is to test end to end, and it becomes more expensive as well as you spin up multiple instances per commit. Maintaining the automation to make this happen can become an extreme overhead to - but it is doable. My team at work is perenially working on this.

I would imagine on your system, there is something preventing you from running a full integration test at that kind of volume (per commit) which is pretty common at the scale of some larger apps that aren't architected as microservices for various reasons. Micro-services come with their own issues as well. Distangling this the modern devops conundrum, because the feasibility of these approaches can depend a lot on the structure of the application itself. And then you are making development decisions around operations, not what is best for the applications functionality and development.

My case for the DevOps profession in general is that it is a genuine engineering discipline to make all of these competing interests work in harmony, per application, at scale. But you should at least be able to broach the subject of per commit integration testing if you want to continue with trunk based development.

[u/usually_guilty99]

I wouldn't make every PR pay for the entire DAG.

The useful question is which downstream nodes and data contracts each change can affect.

Run those checks before merge and keep the full DAG run as a separate release gate.

The important part is knowing what wasn't tested, rather than treating a handful of passing checks as proof that the whole pipeline is safe.