Need to let loose a primal scream without collecting footnotes first? Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid: Welcome to the Stubsack, your first port of call for learning fresh Awful youāll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cutānāpaste it into its own post ā thereās no quota for posting and the bar really isnāt that high.
The post Xitter web has spawned soo many āesotericā right wing freaks, but thereās no appropriate sneer-space for them. Iām talking redscare-ish, reality challenged āculture criticsā who write about everything but understand nothing. Iām talking about reply-guys who make the same 6 tweets about the same 3 subjects. Theyāre inescapable at this point, yet I donāt see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldnāt be surgeons because they didnāt believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I canāt escape them, I would love to sneer at them.
(Semi-obligatory thanks to @dgerard for starting this.)
so openai is claimed to be doing great on the FrontierMath dataset. Iāve already seen the usual sort of dipshits using this to pump ai on reddit, and hereās a post that went to the frontpage on HN:
https://xenaproject.wordpress.com/2024/12/22/can-ai-do-maths-yet-thoughts-from-a-mathematician/
(tl;dr only a few problems from the dataset are public but if representative the problems are about 25% survivable by an undergrad; coincidentally this is the % openai says their models are completing.)
this post is by kevin buzzard. he has a letās say not easily beloved personality, but I donāt think of him as credulous or grifty, and people in his area regard him as an excellent mathematician.
he points out but I think does not focus enough on how discrediting the secretive nature of the dataset is. the fact that you canāt make it public is necessary to run such experiments in a scientifically reasonable way, but also makes it totally impossible to run the experiment in a scientifically reasonable way. an experiment which cannot be examined or reproduced is actually the opposite of science. itās pure grift fuel
If you release the test set, all models magically jump to 87.3% accuracy.
The ARC scores donāt matter too much to me at 3k a problem. Like the original goal of the prize had a compute limit. You canāt break that rule and then claim victory ( I mean I guess you can, but like not everyone is gonna be as wowed as xitter randos, ensemble methods were already hitting 80% + acc to francois )
And unfortunately, with Frontier math, the lack of transparency w.r.t. which problems were solved and how they were solved makes it frustrating as hell to me, as someone who actually would like to see a super math robot. According to the senior math advisor to the people who created the data set, iirc 40% solved problems were in the easiest category / 50% in the second tier category and 10% were in the āhardā tier, but he said that he looked at the solutions and that they looked like mostly being solved āheuristicallyā instead of plopping out any ānewā insights.
Again, none of this is good science, just pure shock and awe. Iāve heard rumors that OAI is hiring strong competition style mathematicians to supervise the reinforcement learning for these types of problems and if they are letting O3 take the test, then how the hell does that not leak the problem set? Like now the whole test is compromised now right? Since this behemoth uses enough electricity to power a city block, theres no way they would be able to run it locally. Now OAI can literally pay their peeps to solve the rest and surprise surprise O3++ will hit 80%
OTOH, with code forces scores and math scores this high, I can now put on my LW cap and say this model has 2 trillion IQ, so why hasnāt it exterminated me and my family yet like big Yud promised? Itās almost as if there is no little creature inside trying to take over the world or something.