it wasn't obsolete. it was abandoned.

scale worked. that's the uncomfortable truth. the jumps in capability we've seen in the last five years alone that have left most in awe came from compute, more data, more parameters, and i'm not going to pretend otherwise.

yet the demand for gpus keeps climbing because the current obsession with scale assumes that the more compute we throw at the problem, the closer we get to agi. compute posing as a substitute for good engineering. well i disagree. throwing infinite vram at a problem just hides bad architecture. the next era of true breakthroughs won't come from building more massive data centers. that path hasn't produced agi, and each increment now costs more in compute and watts than the last.

that is the part i care about. when scaling gets expensive enough, the engineering has to get better. the next gains will come from squeezing constrained hardware, relying on smart code, new ideas, and approaching ai from first principles through bare-metal engineering and architectures that do more with less.

modern ai infra is suffocating under stacks of abstraction. someone who just wants to build, integrate, or understand a local model is forced through a learning curve that has nothing to do with the actual science. massive python wrappers, gigabytes of dependencies, heavy frameworks built on top of other frameworks. i'm not arguing against abstraction itself, i'm arguing that the ladder is missing rungs. you should be able to climb down when you need to, especially now that the weights are open.

the card

the hardware is a gtx 1050. 4gb vram, 112 gb/s, sm_61, pcie x16, in a laptop with 16gb of system ram.

the industry calls pascal obsolete for ai. it isn't obsolete. it was abandoned.

pascal, compute capability 6.1, is the last consumer architecture nvidia built before tensor cores. and the software simply stopped speaking to it. triton refuses to compile below sm_70, so it has to be volta and up. flashattention 2 needs ampere, nvidia’s microarchitecture turing is supported by a separate fork, flashattention 3 is exclusively designed for hopper gpus, and here is the thing, pascal is excluded from every one of them. moreover, cuda 13 outrightly removed compilation and library support for maxwell, pascal and volta.

well the card still works. it draws power, it runs kernels, it does math. in fact raw cuda still compiles for it on the 12.x toolkit. what is gone is everything built on top, this includes the kernel dsl, the attention implementations, the libraries everyone actually uses. you can still write to the metal. you just can't use anything anyone has written in the last four years.

here is the part that actually worries me. that abandoned card has the fastest memory in this entire machine. which is around 112 gb/s. the system ram it is bolted next to… does around 25 gb/s. the sata ssd where a 70b actually has to live does about 0.43 gb/s. the fastest tier in this laptop is the one the ecosystem walked away from, and i can't use it for attention because triton won't compile for it.

that isn't a physics problem. that's a decision someone made.

i have the key to a door that has been bricked over.

[note: i measured the ram with sysbench, disk with fio at 24gb and direct i/o so nothing actually ends up sitting in page cache. my first three attempts at this pointed at /tmp (system directory), which is a ram disk on this machine, and it cheerfully reported 7 gb/s. the tools will lie to you if you let them. so being extra careful is mandatory.]

so the first phase runs on the cpu

avx2 kernels, hand written, from scratch. i know the cpu isn't the faster path. it's the one that isn't locked.

the gpu work happens later in the year, and it happens on rented ampere. that's me being honest. i'm not going to pretend a card the toolchain refuses to compile for is secretly enough, and writing triton means renting hardware that can run triton. equally, what i won't do is let the rental become the default, because the constraint is where the understanding comes from. you learn what a memory hierarchy really costs by living inside a small one, not by having enough vram that the question never comes up.

the 16gb ceiling isn't a limitation i have to suffer through. it's a tight environment where only real effort down at the metal pays. there is nowhere to hide. you find out what the silicon is actually doing, usually the hard way, and you bleed for every byte.

and in economically constrained countries, including the one i'm based in, and across the globe, there are billions of constrained machines. for a lot of people who can't keep buying their way up against the spiraling cost of ram, going down to the metal is the only door left.

the hierarchy, in one laptop

tier              bandwidth     what lives there
gtx 1050 vram     112 gb/s      nothing yet, the stack won't talk to it
system ram        25 gb/s       an 8b at q4, comfortably
sata ssd          0.43 gb/s     anything bigger, at 0.01 tok/s

the fastest tier in this machine is 263 times faster than the slowest, and i'm locked out of it.

a 70b at q4_k_m is 42.5 gb of weights, and every single token reads all of them. divide that by the disk row and you hit the absolute limit, about a hundred seconds per token. i haven't run a 70b to tell you that, it's just basic maths and division doesn't need a demo.

that spread is the entire thesis. you don't out engineer bandwidth. you either fit in memory or you pay the disk tax, and knowing exactly what the disk costs is where the real work starts. most of the arguments i read about model quality are being made by people who have never measured the number that decides whether the model runs at all.

what success looks like

datacenter power draw is now one of the hardest limits the big labs face, and joules per token is a number the field already measures. ml.energy does it on datacenter gpus. hugging face rates models on it. there's a good raspberry pi study that does it with a proper hardware meter.

what i haven't found is whole-system energy, wall-metered, including the disk, across a spread of cheap heterogeneous machines that people actually own. that gap is what i'm going after this year.

the benchmarks will run on llama.cpp, not on my engine, because a number measured on software nobody else runs can't be compared to anything. my engine gets one row in that table like everyone else, and it has to earn it.