Hi folks! I wanted to share a project I've been doing off and on for the better part of 2 years, and in recent months I've gotten the time to make more significant progress.

I was inspired by PureJaxRL a while ago to build my own massively-parallel environment for a task that interested me personally, which is Rocket League. RocketSim, a highly performant CPU-resident simulator, already exists and is baked into RLGym for the purposes of RL training, but I figured it would be valuable to port the physics logic to run in a parallel fashion on-device via CUDA. I learned a lot about GPU programming and optimization, and I still have a lot more to learn so I'm all ears for any advice!

The environment I created is called CARL - CUDA-Accelerated Rocket League. It keeps all simulation state information on the GPU, which during training entirely eliminates host-device transfer overhead. Many thousands of games can be run and controlled in parallel, leading to significant reductions in training wall clock time (as was observed in the original PureJaxRL post).

I've been doing some training of Rocket League bots with it and they transfer seamlessly to the actual game, so the physics appear to be at parity. More work can be done on parity assessments, benchmarking, etc. but I figured now that it's "feature complete" that it would be cool to open it up to the community and see what people think!

As a bit of proof so you know I'm not a liar and a scoundrel, here's a plot from a recent training run that shows marked improvement in aerial control over ~20B decision steps using 8192 parallel environments on a single RTX 4090 (frameskip=4, so 80B ticks total, ~21 years), taking just over 21 hours in wall time.

The method of training as hinted at by the run name is a generative imitation learning approach that (ideally) transfers behavior style from expert demonstrations, which I've obtained from ballchasing.com

Here is an example of the capability it was able to reach in that amount of time. Pretty impressive, although I would have totally saved it if I were trying even a single iota...

https://reddit.com/link/1x77aan/video/ynwh0bbhgrvh1/player

I plan to post a follow up with demonstrations as I round out the functionality. On my TODOs are:

• Built-in training pack API for targeted training
• Benchmarks for speed and parity comparisons
• Tests and other best practices
• Eventual jax & tf integration(?)

Thanks for reading if you did!

P.S. I've been having trouble with the image & video in this post (hence the multiple attempts so sorry if you've already seen this post) so let me know if they're broken please and thank you :)

I made a short explainer of this preprint. Summary for people who don't want to watch:

Agents that keep notes in a memory system usually get improved from a score or a short summary of failures. A score alone can't say which part of the memory code failed, so earlier approaches often rewrite large parts of it.

MemScope adds a Patch Agent that reads the step-by-step trace of a failed question and makes a small in-place edit to the memory code. In one recorded case the trace showed zero steps and no notes, and the paper's proposed cause was one early-return line that may stop the search too soon.

Results from the authors' own small GAIA test (public research questions; one number per setting, no spread reported):

- Patch Agent given only a short summary: 63%

- Patch Agent given the traces plus the saved notes: 87% (best setting tried)

- Full MemScope with patch history: 83%

In the paper's recorded code changes, the median change touched one function, and the median change ratio was 4.4%.

Paper: https://arxiv.org/abs/2610.21170

Curious whether anyone has tried trace-driven patching on their own agent memory, and whether small patches hold up over many rounds.

"Please publish the benchmarks" is the right demand for anything that claims to learn during a run. What I'd argue about first is the shape of the answer, because a single score hides the two things you'd want to know.

reef-eval defines two shapes. An autoresearch task, where the optimal score is unknown, so the result is the best score reached and how long it took to get there, or how many evaluations. And a task stream, where one agent works through an ordered list of tasks, each in its own fresh container and scored on its own, with the agent's memory directory carried from task to task and a snapshot kept at every step. In both, the question is whether the state the agent picked up during the run changed what it scores.

What one number loses: on an open problem, two runs can end on the same best score, one after a handful of evaluations and one after hundreds, and a leaderboard that reports the score alone calls them equal. On a stream, average the per-task scores and you lose whether a late task went better because of what the agent carried forward from the early ones, which is the only thing the stream was built to measure.

Its task sets are the public ones plus six of its own, and it publishes no results, so before the numbers exist the argument is the two shapes: best score and time for open problems, per-task scores with carried state for streams, and an A-vs-A control beside both.

Training pre build

HarnessDev (this month) is the first paper I've seen that measures the loop of letting an agent revise its own harness, keeping the revision when the feedback says so, and repeating. Across 64 adjacent version switches, the feedback score and the held-out score moved in the same direction 34 times, 53.1%. Only 2 of 9 lineages ended on their held-out-best version. Mean gain across the evolution stage was +3.11 points. Executor dependence was severe: one harness scored 69.3 under one model and 33.0 under another.

So a gate that decides on feedback alone is, by their measurement, a coin flip on whether the gain generalises, and that's a selection-policy problem more than a proposer problem.

Two things I'd do differently from the paper's loop, and can set in reef, which I run. Selection: by default a candidate publishes when it wins more task comparisons than it loses, past a noise floor you set, and the Pareto policy in the method guide publishes only when no task regressed and at least one improved. That rule can only refuse on the list the gate runs, and the moment you select on a list it is your feedback score, not a held-out one, so it doesn't recover the paper's 30 by itself; what it does is stop a candidate that lost anywhere on the list from shipping on a net win. The held-out check has to be a list the proposer never sees, and that's on you. Memory of rejections: the proposer can receive the recent rejected proposals with their verdict reasons and the previous step's failure manifest, so it can stop re-proposing the switch that lost.

What none of it fixes is the task list. It's short because it's the cost of every step, and the whole paper is a warning that a short in-distribution list gets gamed by the feedback loop. Whether no-regression on held-out tasks is the right acceptance rule or too conservative to ever move is the open question; the paper's 30 bad switches make me lean towards conservative.

https://reddit.com/link/1x6i6v5/video/h22absbdylvh1/player

I wanted to get my head around RL so I thought of learning it by teaching an AI how to play one of my favorite childhood games : TempleRun.

The agent reads game state directly from memory via a Frida gadget and acts through the game’s own methods. Training happens in a headless simulator I built with Claude from the game’s internals and calibrated on measurements from the real phone, with a C++ version that does 10M steps in about 6 minutes.

Results on the device: median 4,736 m, best 16,782 m over 14 runs.

👨‍💻 Github: https://github.com/matthieucs/TempleRun-RL/tree/main

Happy to answer any question.

And if you want to try it, the score to beat is 16,782 m.

Hey everyone,

I've open-sourced PCEI (Physically Constrained Emergent Intelligence), an experimental architecture designed to run on consumer hardware (Mac Apple Silicon and Linux/CUDA) without the memory explosion of standard transformers:

• BitNet 1.58b Quantization: 24 layers of ternary weights {-1, 0, +1} reducing parameter memory by ~4x compared to FP16.
• Mamba State Spaces: Linear-time sequence modeling with constant O(1)O(1) inference memory (no ballooning KV cache).
• FastKAN Spline Manifolds: Kolmogorov-Arnold B-splines applied to continuous motor control across 4 streams in MuJoCo physics.
• Empirical Telemetry: 11 companion research papers, including an evaluation of continuous latent buffers proving an irrecoverable Shannon information loss factor of LL.

The full framework, training pipelines, and evaluation suites are open under MIT-NC: https://github.com/MtnMCG/PCEI

Feedback and critiques are very welcome!

Hi, I’d like to announce that I have released Metis-Core v0.4.0-alpha

What’s new? I have implemented an “Asymmetric Multi-Head AlphaZero” algorithm in Metis-Core, along with an example in C++ (Visual Studio 2022): KamikazeDroneSwarm.exe.

https://github.com/felixromo314/METIS-Core/

KamikazeDroneSwarm uses the asymmetric multi-head AlphaZero implemented in Metis-Core:

An Imperial Shuttle freighter loaded with beskar, armed with a rotating laser turret, must reach the Death Star.

Two Rebel kamikaze drones try to intercept it before it gets there.

Without being programmed to do so, the drones learn a coordinated pincer attack on their own to split the turret’s attention.

You can play by controlling the turret yourself (human in the loop) against the trained drones.

The framework allows you to create two teams, Blue and Red, and add to each team as many agents as your problem requires.

I’m sure there will be bugs and a lot of things to fix, so just let me know and I will try to fix them.

I hope you find this new feature useful.

Thanks!

You can find more information in this tutorial:

https://github.com/felixromo314/METIS-Core/blob/main/tutorial/Tutorial-AlphaGoZeroAsymmetric.md

Metis-Core is distributed as binaries (.dll / .lib / .h), but the full source code of the example is included, so you can see exactly how the framework is used and build your own environments on top of it.

Note on training: the drones are not scripted, but I do use a small material heuristic to speed up training (setMaterialHeuristicWeight(0.3)). It blends a hand-crafted position evaluation (approach, separation between drones, turret exposure) with the network’s value head during MCTS. The terminal reward is still +1/-1/0, and the pincer manoeuvre itself is not explicitly encoded. But you can set via setMaterialHeuristicWeight the value you wanted, 0.2, 0.1, or if you don not call to this function the heuristic is not activate.

Hi everyone,

I wanted to share a small reinforcement learning project I've been working on in Unreal Engine 5, inspired by the vault chase scene in Fast & Furious 5. I'd really appreciate any feedback.

**Setup*\*

- Two cars on a looping track with many sharp turns, defined by a spline

- A police car trained with PPO to patrol along the road (then frozen as a fixed opponent). It was never taught to dodge, so it simply plows into whatever is in its way

- A race car towing a one-ton vault, trained with PPO to flip police cars

- Both agents use the same action space: throttle, brake, steering

- Training: neural network on GPU, physics simulation on CPU (the CPU is the bottleneck)

**Reward function for the race car*\*

- Large reward: flipping a police car, or steering the vault into one

- Small reward: staying on the road and driving in the right direction

- Small penalty: getting hit by the police car, or driving the wrong way

- Large penalty: leaving the track, or being stuck for more than 3 seconds

**What happened*\*

1. **Bullied.** Early on, the car just lay still and took the hits. Being hit was cheap, while moving the vault usually threw the car off the track, which was expensive. This looks like a classic local optimum.
2. **Trap.** After a while, it started dragging the vault onto the police patrol route, waiting, and swerving away at the last moment so the police car hit the vault. This technically satisfies "vault hits police car", so it's a nice example of reward hacking / specification gaming. It wasn't what I had in mind.
3. **Hero.** I increased the penalty for standing still, moving slowly, or getting stuck (reward shaping). The trap disappeared, and the car started charging at the police car, timing its speed and angle to launch it, and sometimes dodging at the last second so the police car hits the vault behind it.

I turned this into a video that explains the basics (observation/action/reward, PPO, local optima, reward hacking) in a light-hearted way, mainly for people who are new to RL:

https://youtu.be/UUoUOKSQ8FQ

I'd love to hear your thoughts, especially:

- Is there a better way to discourage the "wait and trap" strategy than penalizing idle time?

- Any suggestions for making training with a towed object more stable?

Thanks for reading, and I'm happy to answer questions about the setup.

Hello, community!

I recently launched a website to experiment with and better understand reinforcement learning, particularly Markov decision processes.

It’s completely free, public, hosted on Vercel, and the repository is also available.

Hosted at: https://markov-kingdom.vercel.app/

Repo: https://github.com/xavendano/markov-kingdom

I’d appreciate any comments and/or feedback.
Thanks!!!

Hey all,

I wasn’t very familiar with PPO when I implemented it and I mainly used the original paper as a reference. However, after comparing my code with some common implementations, I noticed an important difference.

I run a large number of environments in parallel. For each PPO iteration, I reset every environment once and let each one run until it either finishes or reaches the maximum horizon (tasks are episodic and rewards are only assigned to the goal state). Environments that finish early are masked while the others continue. Once all environments have finished, I perform the PPO updates.

Therefore, I always collect the same number of episodes, but not the same number of transitions. As the policy improves and finishes episodes faster, the rollout buffer becomes smaller. In contrast, the SpinningUp implementation from OpenAi seems to fix the buffer size and collect a fixed number of transitions.

I compute and normalize the advantages over all transitions, shuffle them, and then perform several PPO epochs using a fixed (relatively high) minibatch size. This means that the number of optimizer steps per rollout changes also depending on the size of the buffer. Since i can not really control the number of transitions, the final minibatch can also be smaller than the others while still receiving a normal optimizer step.

It seems to work fine on some simple custom problems, but i wonder if this could have any consequences that i might be missing (e.g. on long horizon problems).

Can the changing number of optimizer steps lead to problems ? Is the (potentially) small final minibatch a concern like noisy gradients ?

In rl how can someone do research , mine most of the time is getting to know which problem to work on and in which method there is research gap and all

Can you tell me what are the methods to come up with a research gap

Like by doing/ implementing the research paper from scratch and then see what are the limitations of the paper and then we get the another research paper that explores that specific idea and this go on

And we can't find any problem to work on.

Any researchers or students here who produced research papers can tell what worked for you when you were searching for a problem to work on

I am doing rl for robotics and now exploring world models and searching for a research gap in this field...

I've been training 4-agent squads to play survev (an open-source surviv.io clone) on a local, unmodified game server.

every player in it is a bot. yotube clip: https://www.youtube.com/shorts/PPAVNHRAb7k
you can find more videos on my channel.

https://reddit.com/link/1x55w7l/video/3erybs5vaavh1/player

Setup
- MAPPO with a centralized critic. One actor network is shared by the four squad members (agent id as input). The actor only sees what a real game client receives; privileged state goes to the critic only, so the same network can run from a browser extension.

- Actor input: ~1.4k features, two 24x24 local grids and a minimap through small CNNs, a set encoder over up to 12 visible enemies, then a GRU (256).

- Hierarchical actions: the policy picks an intent (loot, explore, take cover, peek, retreat, regroup, push, go to zone...), a target, and what to use (gun, melee, grenade, smoke, heal).

- One team reward shared by the squad: damage dealt and taken, knocks and kills, placement, plus small teamwork terms (focus fire, crossfire, revive).

I'm not releasing the code or the full details: a bot like this could be abused as a cheat on public servers. Happy to answer high-level questions.

Hello scientists! Is there a formula for adjusting hyperparameters for PPO based on the number of agents you're training simultaneously?

I'm using ML-Agents and Unity engine to train robots.

After extensive tuning, I finally got my quadruped robot to walk really well. However, training is extremely slow because I'm only training 10 robots simultaneously. If I increase the number to 100 or 1,000, the training performance deteriorates and the training graphs look terrible.

How can I overcome this? My hardware is capable of handling up to 10,000 robots training simultaneously, but I don't understand which parameters I need to adjust to make this work.

Is there some kind of formula for scaling hyperparameters based on the number of parallel agents?

Most of my initial hyperparameters were taken from the quadruped robot examples provided with ML-Agents.

Hyperparameters for 10 agents:

max steps per agent life: 5000

Observation space: 123

Action space: 12

        trainer_type:   ppo
        hyperparameters:
          batch_size:   2048
          buffer_size:  20480
          learning_rate:        0.0003
          beta: 0.005
          epsilon:      0.2
          lambd:        0.95
          num_epoch:    3
          shared_critic:        False
          learning_rate_schedule:       linear
          beta_schedule:        linear
          epsilon_schedule:     linear
        checkpoint_interval:    500000
        network_settings:
          normalize:    True
          hidden_units: 512
          num_layers:   3
          vis_encode_type:      simple
          memory:       None
          goal_conditioning_type:       hyper
          deterministic:        False
        reward_signals:
          extrinsic:
            gamma:      0.995
            strength:   1.0
            network_settings:
              normalize:        False
              hidden_units:     128
              num_layers:       2
              vis_encode_type:  simple
              memory:   None
              goal_conditioning_type:   hyper
              deterministic:    False
        init_path:      None
        keep_checkpoints:       10
        even_checkpoints:       False
        max_steps:      30000000
        time_horizon:   1000
        summary_freq:   30000
        threaded:       False
        self_play:      None
        behavioral_cloning:     None

Follow-up to an earlier write-up on the REINFORCE → PPO → GRPO lineage. That one ended with GRPO as "PPO minus the critic, plus a group-mean baseline", which raises an obvious question: if it's cheaper and good enough for DeepSeek's reasoning models, why does nobody use it for control?

Two reasons, and they're different kinds of problem - presence of bias and a common feedback applied uniformally to each step. The latter is REINFORCE's original credit-assignment problem, back because the critic is gone. An LLM gets away with it (one terminal reward, quiet environment, replayable start state, pretrained prior). A robot doesn't.

De Oliveira et al. (2511.03527) ran critic-free methods vs PPO on five classic control tasks. PPO won everywhere except CartPole, and GRPO's group weighting did no better than a plain batch mean.

Write-up with interactive figures: https://sreejithb.com/grpo_dissect/single Disclosure: the argument and the script are mine; Claude helped with polish and the visuals.

I've been studying model-based RL and wanted to really understand how Dreamer learns behavior in "imagination," so I wrote up a walkthrough of Dreamer V1 and built a PyTorch implementation alongside it.

The article explains the full Dreamer V1 architecture and includes detailed math derivations of the world model loss and other formulas, with each piece linked to the corresponding code.

Article: https://goltsman.dev/blog/2026/dreamer1/
Code: https://github.com/golkir/dreamer-v1-pytorch

I'd appreciate any feedback.

The world economy around us depends on complex financial work. How well can AI do it?

We(a team of ex-EY employees) built FAB - Finance Agents Benchmark, testing AI Agents on the work behind financial due diligence. Agents can find the relevant facts, but still struggle to carry them through to a complete, reliable analysis.

We’ll keep expanding FAB to more companies and testing more models. Building and running this benchmark isn’t cheap, so we’re scaling it in stages.

The benchmark is public.
GitHub: Hugging Face: https://github.com/SecondState-ai/finance-agents-benchmark
data room and tasks: https://huggingface.co/datasets/secondstate/finance-agents-benchmark-traces

I trained in Genesis, then validated in MuJoCo (mjlab) before deploying.
It can recover from pushes and tracks forward, lateral and yaw velocity commands.
Happy to answer any questions!!