Hey gophers,

I’ve been working on FlowChan, a zero-dependency concurrency toolkit focused on composable channel-based primitives: heterogeneous worker pools, typed pipelines, ordered/unordered maps, batching (fixed + sliding window), retry with jitter + DLQ, and explicit termination.

The library is intentionally opinionated about a few things:

- Every operation is of the form `<-chan T → <-chan Result[T]`

- Context cancellation is checked at every blocking point

- WaitGroups track workers, never individual tasks

- User functions are always wrapped in recover so a panic becomes an error

- Metrics and rate limiting are opt-in, never global

I’m looking for feedback from people who have shipped similar infrastructure (CI runners, stream processors, job systems, or internal concurrency libraries).

Specific design questions I’m still wrestling with

1. Backpressure vs. deadlock risk in the batcher

The current `Batcher` (and RealtimeBatcher) uses a capacity-1 output channel and a blocking send. Under a slow consumer the batcher goroutine itself blocks, which means it can no longer react to `ctx.Done()` or a manual Flush.

I’ve considered:

- non-blocking send + drop/error

- configurable buffer size

- a secondary “force flush with timeout” path

What patterns have you found least surprising for library consumers when the producer and consumer speeds diverge significantly?

1. Ordered concurrent processing and memory growth

`OrderedMap` tags every item with a sequence number, processes concurrently, and releases results in order. Under high variance (some items take 10ms, others 2s) the pending map can grow without bound.

Have you found a practical upper bound / circuit-breaker strategy that doesn’t destroy throughput, or do you simply document “use OrderedMap only when the variance is bounded”?

1. Worker pool lifecycle for long-running services

The pool is reusable (`Start -> Submit -> Drain -> Stop`). Workers are tracked by a WaitGroup so cancellation never hangs. Rate limiting happens at Submit time (not inside workers) to avoid multiple workers firing on the same ticker tick.

Is the Start/Submit/Drain/Stop surface area the right one for a library, or have you preferred a more “set and forget” model with automatic restart of failed workers?

1. Error enrichment vs. caller ergonomics

We wrap errors in `StageError` / `PoolError` that carry stage name and attempt count, while remaining compatible with `errors.Is` / `errors.As`. This has been useful for observability, but it does add a small amount of allocation and indirection on the hot path.

At what point did you decide the extra context was worth the cost, or did you keep errors completely opaque?

1. Sliding-window batcher timer hygiene

The realtime batcher resets a `time.Timer` on every item. We currently ignore the return value of `Stop()` and do not drain the channel. Under continuous load this can occasionally deliver a stale tick.

Is the classic “Stop + conditional drain” pattern still the best practice in 2026, or have people moved to other approaches (e.g. a single long-lived ticker + last-activity timestamp)?

What I’m deliberately not doing (and why)

- No global registries or singleton pools

- No reflection

- No external dependencies (goleak is test-only)

- No built-in persistence or logging

I’d be particularly interested in war stories around:

- places where pure channel composition started to fall apart at scale

- how you handled “zombie” goroutines that escaped context cancellation

- decisions around ordered vs unordered processing when latency SLAs were involved

- any patterns that turned out to be more robust than the classic select + ctx.Done() + WaitGroup combination

I am undergrad trying to work in this space so any feedback regarding project structure or anything casual will also work for me. Happy to go deeper on any of the packages (pool, pipeline, batch, stream, retry, termination) if people want to critique specific implementations.

Thanks.
Repo : (https://github.com/Atul-Koundal/FlowChan) Glad of anyone wants to refer to this more!

Good morning,

I am making a star craft 2 replay analyzer in go, sc2prot reads the replay and gives you a high level api to be able to get the values out,

Here is some Code that I have as a example

type Sample struct {
Loop                  int64
Time                  time.Duration
PlayerID              int64
Minerals              int64
Gas                   int64
MineralCollectionRate int64
GasCollectionRate     int64
ActiveWorkers         int64
CurrentSupply         float64
SupplyCapacity        float64
}

type Economy []Sample

func NewFromReplay(r *rep.Rep, playerId int64) Economy {
var economy Economy

for _, event := range r.TrackerEvents.Events {

matchTime := time.Duration(event.Loop() * 44_642_857 // blizz why?

sample := Sample{

Loop:                  event.Loop(),

Time:                  matchTime,

PlayerID:              event.Int("playerId"),

Minerals:              event.Int("stats", "scoreValueMineralsCurrent"),

MineralCollectionRate:event.Int("stats","scoreValueMineralsCollectionRate",

GasCollectionRate:    event.Int("stats","scoreValueVespeneCollectionRate"),

Gas:                   event.Int("stats", "scoreValueVespeneCurrent"),

ActiveWorkers:         event.Int("stats", "scoreValueWorkersActiveCount"),

CurrentSupply:         float64(event.Int("stats", "scoreValueFoodUsed")) / 4096,

SupplyCapacity:        float64(event.Int("stats", "scoreValueFoodMade")) / 4096,

}

economy = append(economy, sample)

}

return economy
}

// MaxMinerals returns the highest recorded mineral balance.
// It returns 0 for an empty economy.
func (samples Economy) MaxMinerals() int64 {
if len(samples) == 0 {
return 0
}
var maxMinerals int64
for _, sample := range samples {
maxMinerals = max(maxMinerals, sample.Minerals)
}
return maxMinerals
}

then in main.go 

func main() {

// open the replay file
r, err := rep.NewFromFile(replay.SC2Replay) // Hardcoded for now
if err != nil {
panic(err)
}
defer r.Close()

fmt.Println("Replay file opened successfully")
fmt.Println("[Map]:", r.Details.Title())

for _, player := range r.Details.Players() {
fmt.Println(player.Name, player.RaceString(), player.Result())
}

economy := e.NewFromReplay(r, 1)

fmt.Println("Max Minerals", economy.MaxMinerals())



}

this is the direction I have been taking as of recent is just methods, but when you think about something like polars or pandas it's really tedious,

I am sure someone is going to say why not just use python? Honestly I don't enjoy python at all, I never feel like I am getting a grip on the language, every new package you install requires a relearning all over again. I want to be the programmer and solve problems with my tool kit, not solve my toolkit over and over again,

My job is also doing data analytics so this ties to that as well.

I would love to hear your opinion on this

A Go service can burn CPU collecting garbage without having a memory leak.

That is the counterintuitive part of Zomato's Go garbage collection story.

The reported symptoms were striking: garbage collection consuming 25–60% of CPU in some services, alongside containers getting OOM-killed without an obvious memory leak.

These look like separate problems. But they can share a common cause: the garbage collector's pacing decisions don't automatically account for the container's actual memory budget.

The problem with doubling the live heap

Go's garbage collector uses GOGC to control when the next collection should happen.

With the default GOGC=100, the GC target is roughly based on the live heap growing by 100% relative to the previous cycle, with additional accounting for GC roots and other implementation details.

In simplified terms:

Next GC target ≈ live heap × (1 + GOGC/100)

At GOGC=100:

Live heap:       500 MiB
Approx. target: 1,000 MiB

That target is reasonable as a starting point when the process has enough memory available.

But containers introduce another constraint: the process has a memory budget imposed from outside the Go runtime.

The GC's heap-growth target and the container's memory ceiling are two different things.

When they aren't coordinated, two failure modes become possible.

Failure mode 1: OOM kills without a leak

Suppose a container has a memory limit of 1 GiB.

The Go process has a live heap of 600 MiB. A simplified GOGC=100 calculation puts the next target around 1,200 MiB.

But the container cannot safely accommodate that much memory. The process also needs room for goroutine stacks, runtime metadata, native allocations, and other memory outside the live heap.

The container can hit its limit before the collector's normal heap target becomes appropriate.

The result can be an OOM kill even when reachable memory isn't growing uncontrollably.

That doesn't mean every OOM without an obvious leak is a GC pacing issue. Transient allocation spikes, native memory, and other causes still need investigation.

The important point is that a heap-growth target alone isn't a container memory budget.

Failure mode 2: Excessive GC CPU

Now consider the opposite situation.

A service has a relatively small live heap and plenty of memory available in its container.

The GC keeps using a target derived from that live heap. Depending on the allocation rate and workload, it may collect frequently even though the container has significant unused headroom.

Every collection consumes CPU.

At fleet scale, unnecessary GC work can become a meaningful compute cost.

This is why the following combination deserves attention:

• High GC CPU.
• A relatively flat live-heap graph.
• No convincing evidence of sustained heap growth.

It is a diagnostic clue, not proof. A service can allocate and discard large volumes of temporary objects while maintaining a flat live heap. Allocation rate, GC frequency, and allocation profiles matter too.

But before treating every high-GC-CPU incident as a memory leak, inspect the collector's pacing and the container's actual memory constraints.

The fix: Give Go a realistic memory limit

Go provides GOMEMLIMIT to let applications communicate a soft memory limit to the runtime.

Unlike GOGC, which controls GC aggressiveness in relation to heap growth, GOMEMLIMIT helps the runtime account for its overall memory footprint when deciding how aggressively to collect.

Conceptually, the runtime must balance two competing costs:

• Collect too late, and the process risks exhausting its container memory budget.
• Collect too aggressively, and the application wastes CPU reclaiming memory it could have used safely.

A memory limit gives the runtime another constraint to consider.

In the Zomato case, the reported approach used container metadata to derive a per-container limit, refreshed it periodically, and combined memory tuning with CPU scheduling adjustments.

A limit around 90% of the container ceiling can be a starting point for investigation, not a universal safe value. The correct margin depends on the application's non-heap memory, traffic spikes, allocation behavior, and operational requirements.

The goal is to give the runtime a realistic budget while preserving enough headroom for the rest of the process.

Why GOMEMLIMIT isn't a hard wall

This distinction matters in production.

GOMEMLIMIT is a soft limit, not a strict process-memory cap. The runtime can exceed it when necessary, and it does not guarantee that a container will avoid OOM kills.

The runtime also has to balance memory pressure against GC CPU overhead.

If the limit is too high, it may offer little protection against container memory pressure. If it is too low, the runtime can spend excessive CPU collecting garbage in an attempt to stay within the target.

That is why blindly setting GOMEMLIMIT equal to the container limit is not a complete solution.

A production rollout should account for:

1. The container's actual memory limit.
2. Heap and non-heap memory usage.
3. Peak allocation rates and traffic bursts.
4. GC CPU and pause behavior.
5. OOM events and memory headroom after deployment.

Start with representative workloads, observe the results, and tune based on measurements rather than assuming a single multiplier works everywhere.