this post was submitted on 05 Nov 2025
380 points (99.2% liked)
Technology
76609 readers
3491 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
They can ALL be run on RAM, theoretically. I bought 128GB so I can run GLM 4.5 with the experts offloaded to CPU, with a custom trellis/K quant mix; but this is a 'personal use' tinkerer setup basically no one but hobbyists will touch.
Qwen Next is good at that because its very low active parameter.
...But they aren't actually deployed that way. They're basically always deployed on cloud GPU boxes that serve dozens/hundreds of people at once, in parallel.
AFAIK the only major model actually developed for CPU inference is one of the esoteric Gemma releases, aimed at mobile. And the bitnet experiments, which aren't very big so far.
(In case it's not obvious, this is my special interest, and I'm happy to ramble on about how to set up 'niche gaming rig hybrid models' for anyone interested).
I for one would enjoy triggering your unskippable cutscenes in setting up local CPU based AI if it can work on Linux with an older amd card.
Don't have funds for anything fancy, but would be interesting in playing around with it. Been wanting to get something like that setup for home assistant.
You can use Vulkan fairly easily as long as you have 8G vram
https://blog.linux-ng.de/2025/09/27/running-llms-with-llama-cpp-using-vulkan/
The key is which model, and how.
For the really sparse MoEs, you might be better off trying ik_llama.cpp, especially if you are targeting a 'small' quant. But the dense Gemma models (as good as they are) are probably not the best choice for 8G RAM these days.