Factorio as an Agent Benchmark: What FLE 0.3.0 Reveals About World Modeling

The Factorio Learning Environment (FLE) puts language model agents inside Factorio, a game about building automated factories, and scores them on whether the factory actually produces things. The 0.3.0 release, assembled by Neel Kant, Jack Hopkins, Harshit Sharma and Kian Kyars, drops the dependency on the game client, conforms the environment to the OpenAI Gym interface, and ships a CLI plus Weights & Biases logging so a full evaluation sweep runs from one shell command. The underlying benchmark is described in arXiv:2503.09617.

An agent-built iron gear wheel factory at the end of a successful trajectory. Image from the FLE 0.3.0 release (MIT licensed).

This is not a forecasting benchmark, and we are not covering it as one. We are covering it because of what the failure analysis shows. The release states its headline finding without hedging: "In human terms, frontier models are shockingly bad at playing Factorio. They find it difficult to represent and model dynamic environments, and rarely develop formal abstractions they can use as tools in future."

That first clause is a prediction problem. An agent operating a factory, a substation, or a fulfillment network has to answer the same question a forecaster answers: given the current state of a dynamic system and an action I am about to take, what happens next? FLE 0.3.0 isolates that capability better than most agent evaluations, and the result is worth reading if you are building anything that acts on a physical process.

#What lab-play measures

The benchmark evaluated in 0.3.0 is the lab-play setting. The agent starts with a fixed inventory on a prepared map and has a production quota: "16 per minute for solid items and 250 per minute for fluids." It gets a maximum of 64 steps, with early stopping when the quota is met. Scoring is pass@8.

The agent acts by writing Python against the FLE API. Each step returns a structured observation: the raw stdout and stderr of the last program annotated with line numbers, the full entity list with positions and orientations and inventories and warnings, the agent's own inventory, the technology tree state, production flow statistics, task verification results, any helper functions it previously defined and kept in its namespace, and optionally a base64 PNG render of the factory for visual agents.

The harness is deliberately minimal. Environment interactions append to a single conversational history, and when the token budget runs low the agent summarizes its own older history. The release is explicit that this is a floor rather than a ceiling: "We do not evaluate agents with backtracking and/or reflection logic as we did in FLE 0.2.0, and instead we encourage the community to experiment with more advanced agent designs." So these numbers describe base model capability under a thin scaffold, not the best achievable agent.

#The results

Lab-play results, models by rows and tasks by columns, sorted by ingredient dependency depth. Image from the FLE 0.3.0 release (MIT licensed).

The results grid is more informative than a scalar score, because the tasks are ordered by how many ingredient dependencies they pull in. Reading left to right is reading up a difficulty gradient, and every model's row terminates somewhere along it.

Claude Opus 4.1 clears the most tasks at 16 of 24, including advanced circuit and military science pack. GPT-5 clears 15, running unbroken through logistics science pack. Gemini 2.5 Pro clears 11 and Grok 4 clears 9. The frontier ordering, Claude then GPT then Gemini then Grok, holds.

Two things are more interesting than the ordering. First, the right-hand side of the grid is empty: engine unit, battery, low density structure, processing unit, chemical science pack, production science pack and utility science pack were solved by none of the eleven models. The benchmark has substantial headroom left. Second, the open source rows are no longer far behind. DeepSeek Chat v3.1 and Qwen3 235B clear more tasks than GPT-4o and sit alongside Claude 3.7 Sonnet, which is what the release means when it says open source models "have caught up to the SoTA performance observed in v0.2.0 (May 2025)."

The release also notes that this ordering tracks GDPVal, which measures automation of economically valuable work, more closely than it tracks static exam-style benchmarks where weaker models sometimes place higher. That is a useful signal about which evaluations generalize, and it echoes a point we have made about benchmarking challenges in forecasting: an aggregate score on a fixed test set measures something narrower than most people read into it.

#The error breakdown is the actual finding

Error category distribution by model. Image from the FLE 0.3.0 release (MIT licensed).

FLE describes four failure categories in prose: syntactic (invalid Python), semantic (misusing the API), pragmatic (the code runs and the call is legal, but it is wrong given the actual state of the world), and planning or control. The chart groups them into three, and the name it gives the dominant class is the finding in four words: state tracking failures.

For Claude Opus 4.1, state tracking accounts for 97.7% of all errors. Invalid Python is essentially absent, and API misunderstanding is about 2%. The strongest model in the evaluation had the syntax, had the API, had the recipes and pattern guides in its system prompt, and still failed almost exclusively on the question of what state the world was in. The release names the mechanism directly: these errors dominate "as the game state is dynamic and challenging to model implicitly." The canonical example is trying to insert an item the agent no longer has.

One caveat on reading the chart: these are shares of each model's own errors, not rates. GPT-5's lower state tracking share (63.2%) does not mean it tracks state better than Gemini 2.5 Pro (81.1%); it means more of GPT-5's error budget is spent on invalid Python, at 21.2%. Rank by overall error rate instead and the picture is the familiar one. Mean error rates across all trajectories are 22.99% for Claude Opus 4.1, 25.05% for GPT-5, 27.29% for Gemini 2.5 Pro, and 40.89% for Grok 4. The best model has both the lowest error rate and the purest concentration of those errors in state tracking.

Mean error rate by step number. Image from the FLE 0.3.0 release (MIT licensed).

Error rate against step number shows how this plays out over a trajectory. Every model is close to clean on step one, when the world is exactly what the prompt said it was. The rates climb as the agent's own actions accumulate and the gap between its mental model and the actual factory widens. Grok 4 saturates near the top of the scale from around step nine and never recovers, which is the same deficit expressed as thrash rather than a single wrong call. The release describes it falling into "degenerate debug loops" instead of recovering gracefully.

Anyone who has evaluated a forecast over a long horizon will recognize the shape of that chart.

The second half of the headline quote is the companion failure. Gemini 2.5 Pro was the only model that regularly defined helper functions and abstractions, despite the environment explicitly persisting them in the agent's namespace across steps and handing them back in the observation. Given a free mechanism for building tools, most models did not build tools.

The two failures compound. The release notes that misplacement of an entity "often cascades into larger failures, since the agent is unable to recover efficiently." Without a maintained model of the layout, a small error is not detectable as an error, and without abstractions there is nothing to re-run to check.

#The evaluation design deserves attention on its own

Three choices in this benchmark are worth stealing regardless of what you are evaluating.

The holdout period. An agent could hit a throughput quota by pre-filling buffers rather than building working automation. FLE closes this: "We mitigate this by enforcing a holdout period during eval, in which an agent must leave their factory alone for 60 seconds before we test whether quotas are met." This is the same class of control as a proper information cutoff in a forecast backtest. If the system under test can stage the answer in advance, you are measuring staging, not capability.

A throughput target instead of a similarity score. The metric is whether the factory produces at rate, not whether the agent's plan resembles a reference plan. That is a decision-grade metric, and it is the same argument we make in from forecasting to decision support: a score only creates value once it crosses a decision boundary.

Naming the remaining exploit. The roadmap flags reward hacking as an open problem, since agents "rely on semi-manual strategies rather than building robust automation for more complex tasks," shuttling resources by hand and using chests as buffers. Higher throughput targets and METR-style task scaling are the proposed fix, along with establishing a human baseline. A benchmark that publishes the shortcuts its own agents found is more useful than one that publishes a cleaner number.

The roadmap beyond that is open-play on procedurally generated maps with no starting inventory and sparser resources, megabases with thousands of interconnected machines, continuous rather than turn-based operation under latency constraints, multi-agent cooperative and competitive dynamics, out-of-distribution environments via mods, and a native computer-use interface rather than an API.

#Why this matters if you are putting forecasts in an agent loop

Factorio is a toy, and a well-chosen one. It is a dynamic system with feedback, throughput constraints, propagation delays, and states that persist and evolve while the agent is reasoning about them. Those are the properties that make an industrial process hard, and they are the properties that FLE 0.3.0 shows frontier models handle worst.

The lesson generalizes cleanly. General reasoning capability does not deliver a calibrated model of a dynamic system as a byproduct. The agents in this evaluation had a complete, structured, machine-readable observation of world state handed to them every single step, and the dominant failure was still misjudging that state. An agent given a noisy, partially observed industrial process has a strictly harder version of the same problem.

This is the boundary we described in TSFMs versus world models. Predicting how a system evolves is a distinct capability from reasoning in language about it, and it is the capability time series foundation models are trained for. An agent that has to decide when to charge a battery, when to pre-cool a building, or when to reorder stock does not need to infer the dynamics from first principles inside its context window. It needs a component that predicts the trajectory with calibrated uncertainty and returns it as an observation, the same way FLE returns production flow statistics.

That division of labor is already how the better agent systems are built, and it is why we expose forecasting through an MCP server: the agent handles goals, tool use, and recovery, and a model built for the job handles the forward prediction. FLE 0.3.0 is a clean measurement of what happens when you ask the agent to do both. Pair that reading with our notes on reasoning evaluation for forecasting and predictive maintenance in manufacturing, where the same loop runs against a real plant instead of a simulated one.

FLE is open source and worth running. Install with uv add factorio-learning-environment, start a cluster with fle cluster start, and evaluate with fle eval. The code, documentation and position paper are all public. To try the forecasting half of the loop, browse the model catalog or run a series through the playground.