this post was submitted on 16 Mar 2025
23 points (96.0% liked)

Programmer Humor

22155 readers
2324 users here now

Welcome to Programmer Humor!

This is a place where you can post jokes, memes, humor, etc. related to programming!

For sharing awful code theres also Programming Horror.

Rules

founded 2 years ago
MODERATORS
anzo@programming.dev
Feyter@programming.dev
BurningTurtle@programming.dev
pylapp@programming.dev
23
muskrat's data eng expert's hard drive overheats while processing 60k rows (lemmy.dbzer0.com)
submitted 2 weeks ago by SchwertImStein@lemmy.dbzer0.com to c/programmer_humor@programming.dev
100 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] wise_pancake@lemmy.ca 1 points 2 weeks ago

60k rows is generally very usable with even wide tables in row formats.

I’ve had pandas work with 1M plus rows with 100 columns in memory just fine.

After 1M rows move on to something better like Dask, polars, spark, or literally any DB.

The first thing I’d do with whatever data they’re running into issues with is rewrite it as partitioned and sorted parquet.