Evaluating 35 open-weight models across three context lengths (32K, 128K, 200K), four temperatures, and three hardware platforms—consuming 172 billion tokens across more than 4,000 runs—we find that the answer is “substantially, and unavoidably.” Even under optimal conditions—best model, best temperature, temperature chosen specifically to minimize fabrication—the floor is non-zero and rises steeply with context length. At 32K, the best model (GLM 4.5) fabricates 1.19% of answers, top-tier models fabricate 5–7%, and the median model fabricates roughly 25%.
deleted by creator
I wouldnt read too much into the lower scores, they include some absolutely tiny models. The one 70% lower than the top score at 24% correct is a 1B model from 2024. Honestly that it can do any information retrival from a 32k context is impressive.
I understood a few of those words.
Basically you’ve validated the study that LLMs make shit up, right?
deleted by creator
is “potato frontier” an auto-correct fail for Pareto or a real term? Because if it’s not a real term, I’m 100% going to make it one!
deleted by creator
Obligatory reference since you mention AI and no win scenarios: https://www.msn.com/en-in/news/India/claude-discovers-the-kobayashi-maru-test-what-is-the-benchmark-safety-test-the-ai-chatbot-outsmarted/ar-AA1YqyEY
Are all outputs hallucinations? It’s just some happen to be correct and some aren’t. It doesn’t know and can’t tell unless it’s specifically told (hence the guard rails).
But if I’ve gotta build so many hand rails (instructions) then is it really “AI”?
deleted by creator
I refuse to call it AI
It’s a LM… Pure and simple. Anyway none of the LMs can come up with theory of relatively (if you gave them all of the known physics up to 1915).
Nor can they play paper scissors rock (they don’t realise it’s pointless).
As far as I can tell they’re wrong more times then they’re right and the only use I have for them is as a glorified search engine (and even then they’re still fricking wrong.
They’re only useful if you already know the answer because if you don’t know the answer you don’t know if they’ve given you the wrong answer.