Over the past couple of months I have been bootstrapping my personal software factory. Because this is what you do these days. I quite like it at this point and as the saying goes:
“There are many others like it, but this one is mine!”
The exact inner workings of the factory are of little importance. As is usual with these things, it is the author’s (in this case my) personal preferences: the engineering choices I have made, the cloud providers I use, the level of autonomy I want my agentic coders to have and mostly to the way I think about tasks and projects. So we won’t go into much detail there. The short version is this:
I write tasks as GitHub tickets, a loop picks them up and dispatches them to agentic handlers. The loop itself is dull and deterministic. All the smart stuff happens in the agentic handlers. Development tasks would hence end up in a terminal window with an agentic builder banging at them.
Originally my factory had exactly one kind of worker: Claude. Then it started to annoy me. A LOT! The thing that I found most bothersome is Claude’s refusal to be autonomous in cases where authentication is necessary. Manipulating API keys, logging into websites. Claude very annoyingly resists helping with them. It usually takes this grandstanding voice: “Emil, this web page requires a password!” “Emil, I am NOT ENTERING PASSWORDS ON WEB PAGES, EMIL!” If I insist it retorts: “Emil, I AM NOT DOING THAT!!!”. While I have managed to alleviate the issue with various combinations of purpose built chrome extensions and vaults, it was still annoying enough.
So I wanted to try out Codex and Grok and see how they compared. Once all three start churning out work, the very obvious question comes up: how good are they? Is one better than the others?
So I set off to measure it. (I mean … I set off giving orders to agents to measure it)
The setup
I took ten real tickets from actual products (a conversation recording platform called Chui and a sports coach called Jaksa). Every one of these tickets had already been built and shipped once.
Then I had each builder build them again. They all had the same starting point in the code and the same ticket and instructions. There was no peeking at the version that actually shipped. Every rebuild went through its own code review before it was accepted.
Once a build completed, I had it blindly evaluated by the three builders: Claude, Codex and Grok. They each read every version without knowing who built which. As the agents were building the system they noticed that the judges would often form a preference for the first solution they saw so we went and added multiple evals with the order swapped. At the end of it all every version gets a score out of 10 for correctness, tests and clarity.
The matrix
Each number is the average score from all three judges. “Shipped” marks the version that actually went to production; everything else is a rebuild made for this test.
The judges are not neutral
Before we get to who won, something jumped out pretty quickly: the judges are biased. Every one of them quite likes the work of their own family.
Codex gave its own work an 8.1. The other two judges gave the very same work a 6.5 and a 6.7. Grok thinks Grok is better than Codex. Claude thinks Claude is best. I suppose this is to be expected.
That’s the whole point of having multiple judges after all. With all three counting equally, the biases mostly cancel out. When I only had two judges, that alone was enough to put Codex on top.
So who won?
Nobody, really.
All three land within about a fifth of a point of each other, out of ten. Every one of them won some and every one of them lost some. Claude’s average is a little higher, but flip one or two pairs and the ranking looks different.
Where they do differ: time and money
So quality is a wash. Time is not.
A typical Claude build took 10 minutes. Codex took 34. Grok took 54.
A lot of this may be somehow related to my setup. Or not. I haven’t looked into it that much. These things run asynchronously and in the background 10 minutes or 50 doesn’t really make that much of a difference.
Money is where it gets really interesting. All three run on flat monthly subscriptions, so a build doesn’t really cost anything … until you run out of your weekly allowance. So rather than dollars, the agents measured what share of the week each build eats up, and then turned that into dollars of a $200 a month plan (about $46 a week):
That makes Claude about four times cheaper than Codex and about sixteen times cheaper than Grok, per build. Again, that’s what I am seeing. YMMV … and I’d love to know if it does!
So which one will I use?
Clearly all of them. Or more accurately for my factory: whichever one has tokens left this week … which I suppose means Claude will be doing most of the building because it stretches tokens the furthest.

