paper | demo | model | project page
Since I’ve been using large language models since 2023 the obvious cadence is that all AI models got smarter and can do longer tasks. Especially the models you can download and run on your mac have much more capability and capacity to do mundane tasks, and I put it to the test on a real-world problem: selling metro tickets and providing accurate travel data. I chose this because in my personal experiences riding the metro in Amsterdam and Beijing, a lot can go wrong with my itinerary if anything in the system breaks, and requires me to re-plan the trip or the day at all.
Similar to a operating a metro system many other, many bounded tasks exist that are programmed by engineers and instructed by an operator, which have huge upside potential to be assisted by AI if any random event appears which the engineers didn’t code for or the operator had to cater for.
Now it is no great surprise that new code can be made at lightning speed, high intelligence with great tools such as claude and codex. I use both extensively myself aswell, as I did when beating Claude’s challenge in my previous post. But do we eventually want to connect every traffic light on the planet to a hyperscaler?
That’s why I built the interactive benchmark MetroLLM-Bench, testing AI LLM models, including small ones that you can run locally without internet, to simulate six real metro systems each with their own complexity, rules, customs and (natural) events. 955 unique cases put the model at the test to see if it is able to plan, route, and sell a (virtual) fare, complete with information the traveler needs to know about the itinerary, such as closures, inspections, natural events, and other cultural items.
This creates an unique, new benchmark that not tests only the large hyperscaler-size models at their capability, but one that also tests the smaller ones, the ones that fit on your mac or pc, decreasing in size to gauge how good the smallest models do.
And sure they do.
Muse Glimmer 30B from Meta Superintelligence Labs, and Qwen 3.5-27B from Alibaba Cloud both outscore GPT-5.6-sol, a frontier model from OpenAI, out of the box, without any changes or training to the models. These smaller models can be run on consumer hardware that its purchasable today, no datacentre-grade hardware or environment is required. I did it at my home.
Moving to even smaller models, they can be finetuned to make them behave better at tasks, without adding more information to the prompt or context. A finetuned Qwen3.5-4B model, 2.6GB in size, matches GPT-5.4 at maximum reasoning effort. This specific fine-tuned model you can run on any macbook purchasable today.
For the operator, changing behaviour is just a line of text. “No service this weekend on Line 5, free bus ride instead” is all it takes for the AI to understand, and informs and advises accordingly.
This changes the landscape for many bounded tasks today, I estimate by 2031 many similar small models do all kinds of simple, everyday work that requires mild reasoning by AI to perform, millions of times per day. And to do that no cloud is needed, no hyperscaler is needed. The box it runs on is something you own, it keeps working when the network doesn't, the passenger's data never leaves it, there is no meter running, and nobody deprecates your model on a Tuesday. The LLM inside it is as unremarkable as firmware. Nobody will remember that your ticket purchase used to go to a cloud.
So obtaining the intelligence turned out to be the easy part, what took months to get to this was using the right tools, obtaining metro system station, line and fare data, creating the station graphs, the AI guardrails, the 955 test cases that grill the model and catches it hallucinating. This is my day job, and I am comfortable that by 2031 there will be even more engineers doing this on a box we own, rather than on someone else’s meter.
You can play around with the demo today here (sign up for HF for more gpu usage), or download and run the model yourself from here, go break a metro line in plain English and watch it reroute, keep going until it gives up.