Why not get a 0.5 or 1 tb nvme ssd and set it all as swap?
It will run probably 10 times slower, but it's cheap and doable.
this post was submitted on 01 Jul 2025
25 points (90.3% liked)
Selfhosted
46671 readers
1349 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (donβt cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 2 years ago
MODERATORS
This is the way.
Depending on the nature of the sim, it could probably even be done with ~80 GB or less of existing SSD space using zram w/ zstd.
That's kinda an insane amount of ram for most simulations. Is this like a machine learning thing? Is his python code just super unoptimized? Is it possible he's making a bunch of big objects and then not freeing the references when he's done with them so they're never garbage collected?
Needing that much RAM is usually a red flag that the algo is not optimized.
Researchers always make some of the worst coders unfortunately.
Scientists, pair up with an engineer to implement your code. You'll thank yourself later.
*looking at the 14TB cluster I had running for 18 hours
Yep, nobody would ever need that much memory
Wow, yea I think you win that contest lol.
To be honest, it was a very paralell process. I could do a fraction of the compute, needing a fraction of the RAM, but taking a shit ton more time.
Also, theres no perfect machine for this use. I can have 3.5 times more RAM than needed, or start swapping and waste time.
Nope. Some algorithms are fastest when a whole data set is held into memory. You could design it to page data in from disk as needed, but it would be slower.
OpenTripPlanner as an example will hold the entire road network of the US in memory for example for fast driving directions, and it uses the amount of RAM in that ballpark.
Sure, that is why I said usually. The fact that 2 people replied with the same OpenStreetMap data set is kinda proving my point.
Also, do you need the entire US road system in memory if you are going somewhere 10 minutes away? Seems inefficient, but I am not an expert here. I guess it is one giant graph, if you slice it up, suddenly there are a bunch of loose ends that break the navigation.
I host routing for customers across the US, so yes I need it all. There are ways to solve the problem with less memory but the point is that some problems really do require a huge amount of memory because of data scale and performance requirements.
First, define what you are asking for.
Do you want someone to send you a cardboard box full of RAM? Then forget it. Nobody would be stupid enough to lend that much expensive hardware to someone on the internet.
Or are you asking for someone to let you run random code on their PC for a few hours? Then forget it. Nobody would be stupid enough to open "a single SSH port" to someone on the internet to run potential malware on their PC.
That's exactly what cloud platforms are there for, and if you don't like google, get any other cloud provider.
Seconded. If they can't optimize their code (which, I have never seen applications require 256 gigs of ram even in FAANG so I find that doubtful), then they need to rent a machine. The cloud is where you rent it. If not Google, then AWS, Azure, Digital Ocean, any number of places let you rent compute
Yeah, it's an economics student running something on python. I can guarantee that it's horribly unoptimized.
that's probably way too much for any sane Python algorithm. if they can't run it, how do they even know how much is needed?
Probably they should only make a prototype in Python, and then reimplement it in a compiled language. it should reduce the resource usage massively
Tell your friend to open source the algorithm.. Somebody will surely point at a easy optimization. 100 others will just shit on your friend
AWS has an r4.8xlarge 244gb ram with 32 vcores for $2.13 an hour If they can handle Linux. $2.81 an hour for windows.
The computer I'm typing on has 96 gb ram. Most of my equipment is ancient in terms of PCs. This one I build about 14 years ago, and I fully stocked it with the cutting edge tech of the day. My intent was to build a LTS PC, as it were. LOL Back then, SLI was the thing, but I've upgraded the GPU. I have some old stuff in the parts bin tho, but it's ancient as well.
I've got 512GB of RAM in my server, and 128GB of RAM on my desktop cause you can never have too much.
I do not have any RAM to share, sorry.
An economics simulation in Python needing 200+GB of RAM sounds preventable.
In your friend's shoes, I might start asking for pointers over on the programming.dev Lemmy.
As others have said, a rewrite in a faster language like C or goLang could help - but my guess is there's also ways to cut that memory need way down, while still using Python.
Borrow it from NewEgg, then return it
i have 8 gigs thats been living on my desk for the last 4 years
As an economist I can confidentally say that he should go a different route. You/ he can show me/us more if he wants to. Maybe we can tell where the problem is.
If not, swap as others have stated is the way to go
All my extra RAM was super old and I let it get offed when I hired a junk hauling company clear out my last place when I moved (I'd been there for like 15 years, so there was a lot of worn out furniture and stuff).
Apply for compute time at a university cluster. It is free and usually easy.
As a hardware guy there is so little info here
DDR2, 3, 4, or 5? Clock speed? ECC? Registered?
Yeah I have boxes of older memory. But there needs to be a lot more specifics. Most of my home lab machines have at least 384gb (VMs need a lot of memory).
I don't think OP wants you to lend them physical RAM modules but asks about letting his friend run random code on your high-RAM machine.
Maybe? After rereading it Iβm really not sure β¦
That's at least what I got from the comment with the SSH port.
Yeah I can definitely see your point.
First of all, he should drop Python for anything resource intensive as such a simulation. And then think about how to optimize the algorithm.
Not anymore... Had a box of old/junk parts lying around before I moved but I didn't bring it with me because it was essentially just garbage I never got around to taking to the electronic recycling thing because it was just a once a year thing that cost me money.
I'd have no problem giving away stuff like that. I wouldn't take parts out of an active machine to let someone "borrow" tho.
Define buttload
put your butt on a scale, convert the result to RAM, duh
Yup, thatβs some random ass memory.