this post was submitted on 29 May 2025
25 points (96.3% liked)

Linux

7659 readers
288 users here now

A community for everything relating to the GNU/Linux operating system

Also check out:

Original icon base courtesy of lewing@isc.tamu.edu and The GIMP

founded 2 years ago
MODERATORS
 

I want to have a mirror of my local music collection on my server, and a script that periodically updates the server to, well, mirror my local collection.

But crucially, I want to convert all lossless files to lossy, preferably before uploading them.

That's the one reason why I can't just use git - or so I believe.

I also want locally deleted files to be deleted on the server.

Sometimes I even move files around (I believe in directory structure) and again, git deals with this perfectly. If it weren't for the lossless-to-lossy caveat.

It would be perfect if my script could recognize that just like git does, instead of deleting and reuploading the same file to a different location.

My head is spinning round and round and before I continue messing around with find and scp it's time to ask the community.

I am writing in bash but if some python module could help with it I'm sure I could find my way around it.

TIA


additional info:

  • Not all files in the local collection are lossless. A variety of formats.
  • The purpose of the remote is for listening/streaming with various applications
  • The lossy version is for both reducing upload and download (streaming) bandwidth. On mobile broadband FLAC tends to buffer a lot.
  • The home of the collection (and its origin) is my local machine.
  • The local machine cannot act as a server
you are viewing a single comment's thread
view the rest of the comments
[–] dgdft@lemmy.world 18 points 1 week ago* (last edited 1 week ago) (1 children)

Speaking as a data engineer, you’re having trouble because git is the wrong tool for the job. You can make it work if you use git-lfs + custom hooks — but if you choose to go that route, be aware you’re making things unnecessarily hard for yourself.

If you want to make this easy, separate out your concerns:

  1. Versioning: take periodic snapshots of your unconverted files with a binary-friendly diffing tool like restic or borg. Alternatively, ZFS/btrfs snapshots are an excellent way to handle this.
  2. Conversion: keep your original files in their own directory. Set up a small script that searches your directory of original files recursively, passes the files to lame to encode to V0 or V2, and outputs them to a separate directory of lossy mp3 files.
  3. Syncing: use rsync with the --delete flag to copy your lossy files to the server + clear out files you’ve removed locally.
[–] A_norny_mousse@feddit.org 3 points 1 week ago* (last edited 1 week ago)

Thank you, this is very comprehensive! Yes, after some thought git is no good here.

I am coming round to a similar concept now: one job for lossy files - relatively easy with rsync I guess - and another for the lossless.