this post was submitted on 05 Nov 2025
380 points (99.2% liked)
Technology
76609 readers
3491 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
There was a recentish model, qwen next that was advertised as smth that can be run entirely on RAM.
They can ALL be run on RAM, theoretically. I bought 128GB so I can run GLM 4.5 with the experts offloaded to CPU, with a custom trellis/K quant mix; but this is a 'personal use' tinkerer setup basically no one but hobbyists will touch.
Qwen Next is good at that because its very low active parameter.
...But they aren't actually deployed that way. They're basically always deployed on cloud GPU boxes that serve dozens/hundreds of people at once, in parallel.
AFAIK the only major model actually developed for CPU inference is one of the esoteric Gemma releases, aimed at mobile. And the bitnet experiments, which aren't very big so far.
(In case it's not obvious, this is my special interest, and I'm happy to ramble on about how to set up 'niche gaming rig hybrid models' for anyone interested).
I for one would enjoy triggering your unskippable cutscenes in setting up local CPU based AI if it can work on Linux with an older amd card.
Don't have funds for anything fancy, but would be interesting in playing around with it. Been wanting to get something like that setup for home assistant.
Plenty of folks do AMD. A popular homelabsetup is 32GB AMD MI50 GPUs, which are quite cheap on eBay. Even Intel is fine these days!
But what's your setup, precisely? CPU, RAM, and GPU.
I have a MI50/7900xtx gaming/ai setup at homr which in i use for learning and to test out different models. Happy to answer questions