Selfhosted

52803 readers

661 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago

MODERATORS

HybridSarcasm@lemmy.world

HybridSarcasm@lemmy.hybridsarcasm.xyz

What budget friendly GPU for local AI workloads should I aim for? (lemmy.dbzer0.com)

submitted 1 day ago by pjusk@lemmy.dbzer0.com to c/selfhosted@lemmy.world

30 comments fedilink hide all child comments

Hiya,

Recently upgraded my server to an i5-12400 CPU, and have neen wanting to push my server a bit. Been looking to host my own LLM tasks and workloads, such as building pipelines to scan open-source projects for vulnerabilities and insecure code, to mention one of the things I want to start doing. Inspiration for this started after reading the recent scannings of the Curl project.

Sidenote: I have no intention of swamping devs with AI bugreports, i will simply want to scan projects that i personally use to be aware of its current state and future changes, before i blindly update apps i host.

What budget friendly GPU should i be looking for? Afaik VRAM is quite important, higher the better. What other features do i need to be on the look out for?

you are viewing a single comment's thread
view the rest of the comments

[–] drkt@scribe.disroot.org 10 points 1 day ago (2 children)

It's all VRAM, that's the bottleneck for even the best GPUs. AMD support is spotty so you should stay in Nvidia's claws unless you know what you're doing. Figure out what kind of money you're willing to part with, and then get whatever Nvidia GPU gets you the most VRAM.

[–] meldrik@lemmy.wtf 5 points 1 day ago (1 children)

The size of the LLM should be less than the amount of VRAM available.

[–] Jakeroxs@sh.itjust.works 4 points 1 day ago

Learning about quant level was helpful as well

[–] anamethatisnt@sopuli.xyz 4 points 1 day ago

Yeah, for a budget friendly AI GPU I would look for a 5060 Ti 16GB.