Gemini 4 Argon became Pokémon Champion using less than half the tokens Gemini 3.8 Flash spent earning 3 badges

[u/waylaidwanderer]

https://x.com/TheCodeOfJoel/status/2110203222979641826

[u/Throwawayforyoink1]

Pretty cool. I think the new pokemon benchmark should be models battling it out in pokemon showdown

[u/MeOneThanks → u/Throwawayforyoink1]

I think that would be much less impressive than playing the actual games. For an entire game the model needs to understand more than "More damage good"

[u/Super-Award-2244 → u/MeOneThanks]

Vgc is way more complicated than "more damage good". It requires some long term decision making 

[u/AnswerGrand1878 → u/Super-Award-2244]

Vgc is much harder than normal gameplay.

[u/SSUPII → u/MeOneThanks]

You can beat all Pokemon games by just spamming high damaging moves. They are not hard games and can be beatd by a 7 year old. It can only be impressive with a competitive setting, where "big damage" gets you crushed

[u/MeOneThanks → u/SSUPII]

that's what I mean, the fighting is not the hard part. it's the routing, walking, understanding where your next objective is that is impressive for a machine to do. If you just want to make an AI do battles, you could probably just train a random forest

[u/6969its_a_great_time → u/MeOneThanks]

Another part that people neglect is the team building aspect. Would be nice to see some combination of RL with an LLM to create a team that was good enough to win in a vgc setting

[u/Fit-Coconut-3699 → u/MeOneThanks]

This betrays a fundamental misunderstanding of competitive Pokémon. The skill ceiling on VGC or 6v6 singles is orders of magnitude higher than on the single player games and approaches the classics (chess, poker, go) in terms of strategy and theory crafting. An llm reaching the top of any popular ladder on showdown would be much more impressive and difficult than one completing a play through of the single player games.

This video is long but does a nice job explaining the nuances of competitive strategy
https://youtu.be/wrR6MquGU6A?is=wV1oVTC8QaEXtm26

[u/MeOneThanks → u/Fit-Coconut-3699]

We've had superhuman chess, poker and go machines for decade(s). Sure it is different for an limited to do it but I don't think it's a very impressive feat for machines to beat humans in these games with a clear goal and rules

[u/Fit-Coconut-3699 → u/MeOneThanks]

I see your point but I think we’re mixing generalized and specific solutions now. Yes, a generalized llm completing Pokémon emerald may be a “greater” achievement than a specific competitive game engine (like stockfish for chess, or some new model for showdown), but my point is that a generalized llm hitting number 1 on a showdown ladder or beating stockfish would be a level above both (certainly a level above beating emerald).

[u/trandanggiabao0203 → u/Throwawayforyoink1]

knowing what The Pokémon Company did with hosting PTCGABC earlier this year, they should make it a showmatch at Worlds 2027

[u/barbozas_obliques → u/Throwawayforyoink1]

Fuck it have them play yugioh

[u/Gotisdabest]

I'd say it should be running against sol 6.1 and opus 5.5, at least. It's obviously a huge step over 3.8 but this is a decent test for comparing the frontier. Not all valuable by any means but an interesting data point.

[u/Long_comment_san]

But did it have fun along the way?

[u/waylaidwanderer → u/Long_comment_san]

Gemini always says it's having a lot of fun when asked!

[u/AIGODSEND]

champion on less than half tokens is the real headline, not the badges. tokens-per-badge is secretly a reasoning efficiency benchmark, flash brute-forces, argon plans. showdown idea is elite, live ladder with elo would end all benchmark theater overnight. my money is on the model that switches least, indecision burns tokens. which starter would an llm even pick?

[u/Ok_Dot525]

Against who? If it was random scrubs online, that not that impressive. But if there were regular tournament players who have years of experience under their belt, then that kinda impressive.

[u/Keeltoodeep → u/Ok_Dot525]

Pokebench is a meme leaderboard against other LLM models playing the game

[u/DarthFader4 → u/Ok_Dot525]

Not that kind of pokemon champ (not PvP). This was beating Pokemon Blue in a record low number of turns and tokens for the Gemini family.