// HACKER NEWS — CYBERSECURITY
44% on ARC-AGI-1 in 67 cents
I trained a small transformer from scratch in 1.5hrs on a 5090 Beats many LLMs, and scores the same as TRM/HRM
This is an upgrade to my previous model Faster, better, cheaper and still open source.
This is the 3rd blog in a series of works on ARC-AGI. Prev: Blog 2, Blog 1.
Many ppl thought the prev result was impossible. It got attention from top researchers and went viral on X. Eg: Discussions by Lucas Beyer, Jeremy Howard, Rohan Anil, and comments by many others.
I think sample efficiency is the most important problem in AI today and I want to solve it.
The intention behind this work is to (1) find the limits of sample efficiency when restricted to transformers / today’s deep learning methods and (2) reduce costs so iteration is much faster and cheaper.
Next, I’ll work on new research ideas to break these limits. I’ll try to keep costs low so that anyone in the world can work on this.
The overall approach is similar to last time (full technical details here), but I added a bunch of upgrades. Here’s a quick summary of the approach:
The main goal was to find improvements to the architecture / algorithm that improve the sample efficiency of the model.
A major change is that I don’t train on input tokens anymore. This means the loss function only includes output tokens (which makes the approach supervised). This. performs slightly better 40% $\to$ 44% but I don’t understand why. Perhaps finite model capacity