sunaurus

joined 2 years ago
[–] sunaurus@lemm.ee 9 points 1 month ago (3 children)

Hard to say exactly what the cause was, as I don't know what username they used on lemm.ee (doesn't seem to be the same one that was used on Reddit), but it's most likely:

  1. They did not verify their e-mail address
  2. They did not agree to follow lemm.ee rules in the sign-up form
[–] sunaurus@lemm.ee 5 points 1 month ago (2 children)

We'll get some more control over images when 1.0 comes, I will take a look at it again then, but in general the new user cooldown for image uploads has really significantly reduced the amount of image related abuse we have seen.

@Character_Locked@lemm.ee please understand, nobody on our admin team is getting paid to moderate disgusting images, they are volunteering to do it, and it takes a mental toll. Also, image hosting itself is not free. Our donations right now are barely breaking even with server costs. So realistically, our options are not "current situation vs fully unrestricted image uploads", they are "current situation vs completely burned out admin team and costs increasing beyond donations".

1
submitted 2 months ago* (last edited 2 months ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hey folks!

I'm writing this because funding for the Lemmy project has dropped to critical levels, which could seriously impact its future development.

Thanks to the generous support of our lemm.ee community, our server infrastructure costs are covered, and we even have a few months of runway. I'm deeply grateful to everyone who has contributed - lemm.ee wouldn't exist without your help.

However, infrastructure alone isn’t enough. Our servers run Lemmy software, and without ongoing development, the platform cannot grow or even be maintained.

Lemmy is an open-source project with many contributors, but the vast majority of development work has been carried out by a small group of core maintainers. A few maintainers work full-time on the project, relying solely on donations and occasional grants to support themselves.

I've seen Lemmy development up close, and the maintainers have consistently gone above and beyond what I consider the standard for small open-source teams - they are constantly writing code, mentoring contributors, and keeping everything running. Their work is essential, and without continued support, it cannot be sustained.

If you value Lemmy, please consider supporting its maintainers directly. Every bit helps.

Please check out this post for more details about how to support the maintainers: https://lemm.ee/post/63034576

Thank you for reading, I hope you have a great weekend!

[–] sunaurus@lemm.ee 2 points 2 months ago (1 children)

Most applications are handled automatically, a smaller percentage is sent to manual approval for admins. This is based on different signals and the aim is to potentially reduce some abuse, or at least make it a bit harder.

Median time for accepting applications in the past month on lemm.ee was 1 minute and 42 seconds. And that includes the time it took for users to manually verify their e-mails first.

[–] sunaurus@lemm.ee 6 points 2 months ago (1 children)

We briefly had some database issues, it should be solved now, but I will continue to monitor.

Sorry for the inconvenience!

[–] sunaurus@lemm.ee 17 points 3 months ago (1 children)

Definitely not 😄

246
submitted 3 months ago* (last edited 3 months ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hey folks!

For the past few hours, lemm.ee has been bombarded with abnormal (almost definitely automated) traffic from a range of different IP addresses. This managed to overwhelm our servers, and we were offline for the past hour or so.

I was in the middle of celebrating my birthday, so response was a bit slow, but I believe we are recovering now, with mitigations in place to try and prevent further issues. Some of you may be inconvenienced by some bot checks when you browse lemm.ee, I am sorry about that, but it's necessary for now.

Sorry for the issues and I hope you have a nice weekend ahead!

[–] sunaurus@lemm.ee 1 points 3 months ago (1 children)

I’ve just restored it

[–] sunaurus@lemm.ee 8 points 3 months ago (1 children)

Hey, there isn’t any default community right now. There are a few different databases that track graphs of such things, for example, fedidb.org.

[–] sunaurus@lemm.ee 40 points 3 months ago

Thank you very much for the support!

Our infrastructure costs are currently quite stable at around 200€ per month, and considering that the instance is right now quite decently supporting nearly 6000 monthly active users, you could say that you are indeed relatively contributing a ton - you are effectively covering server costs for 60 people!

The fact that it’s a monthly amount is particularly great, because with recurring income, we will get advanced warning if there is danger of funds starting to run low.

[–] sunaurus@lemm.ee 28 points 3 months ago* (last edited 3 months ago)

We had a few really huge days in 2023, but other than those, it seems like the growth so far in March is definitely outpacing our initial wave of new users in 2023.

1318
submitted 3 months ago* (last edited 3 months ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hi folks!

Over the past few months, we have started seeing a significant amount of new user sign-ups. I would like to take this opportunity to welcome all of our new members, and to share some useful resources and info about lemm.ee.

First, some stats

Here is a bar chart of daily new users (this is only counting users which have been approved by our admins):

collapsed inline media

As you can see from the chart, for most of 2024, we were accepting roughly around 10-20 new users every day. Then, from the start of this year, the daily numbers have been constantly growing. Yesterday, we approved a massive 609 new users on lemm.ee.

The increase in sign-ups is significant enough that I have been taking several steps to improve our monitoring & anti-bot measures, but so far, it seems the vast majority of the new users are completely legitimate real humans! (Thank you all for not being bots 😅)

About lemm.ee

This Lemmy instance is turning 2 years old very soon. It was initially created around the time of the Reddit API changes, when existing Lemmy servers were getting overloaded with new users - lemm.ee was intended to help spread the load. We're now the second largest Lemmy server when it comes to monthly active users.

Our core philosophy for this instance has always been to treat it as a generic gateway to the Lemmy network. I want to provide our users a stable and reliable home for their Lemmy account, so that they can have easy access to all of their communities, regardless of what instance the community is actually hosted on.

We run on some decently beefy hardware, and our setup is fairly customized in several ways in order to ensure a smooth experience for our users (most of the time, this has worked out quite well!). Our servers are currently hosted in Finland.

Our infrastructure has been funded by the community almost from the start through GitHub sponsorships and Ko-Fi donations. I am sure I speak on behalf all of our users when I say that I am extremely grateful to all supporters - you are really responsible for the continued existence of this instance!

Lemmy itself is open source software, and while it has improved massively during the time I have been using it, it definitely still has some rough edges. Please be patient when using Lemmy, and remember that it is being built collaboratively by humans (not corporations), without any intent of ever turning it into a business.

Useful resources

Don't forget to participate!

Communities on Lemmy only work if people actively use them. Even upvoting/downvoting based on quality of content is a great start, but I would really like to encourage you all to comment and even write posts, because that's really the best way to build communities.

If you have any questions or thoughts about lemm.ee or Lemmy in general, feel free to post a comment below this post, and myself or one of our veteran users will definitely respond.

I hope you enjoy your time on lemm.ee, and I wish you all a great week!

[–] sunaurus@lemm.ee 0 points 3 months ago (3 children)

Hey, could you make a post in that community?

 

Hey folks

Just a quick heads up, we will be performing some database maintenance today. Expected downtime is ~15 minutes.

Sorry for the inconvenience!


Update: maintenance complete!

[–] sunaurus@lemm.ee 1 points 4 months ago

The e-mail successfully went out from our side - please check your spam filters etc

[–] sunaurus@lemm.ee 2 points 4 months ago (4 children)

That's just Lemmy behavior for when you click the e-mail verification link multiple times. It basically means "e-mail already verified".

2
submitted 6 months ago* (last edited 6 months ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hey Folks

Just a quick note to let you all know about some changes in the lemm.ee admin team. After discussing things with the other admins, we've decided to shuffle around our roles a bit.

Up until now, I’ve been the head admin at lemm.ee - handling infrastructure, maintaining rules and policies, and acting as the main contact person for the admin team.

However, I’ve come to realize that this role has taken a toll on me. While I still love the idea of Lemmy and everything it stands for, being an admin has slowly drained the joy I once had for the platform. The occasional negative experiences have been increasingly difficult for me to shake off. For the past several months, I’ve found myself hesitating to check my DMs or the moderation queue, simply because I’m bracing for some new drama that I no longer have the energy to manage.

After some conversations with the team, we’ve agreed on a plan to ensure my burnout doesn’t negatively impact the instance:
  1. I am stepping down as head admin of lemm.ee.
  2. The new main contact person for the admin team will be @EllaSpiggins@lemm.ee.
  3. I’ll continue to maintain and update the infrastructure behind the scenes.
  4. The rest of the admin team will now handle all moderation issues, managing our policies, and any general admin communications.

It’s been an honor to serve as your head admin, and I’m incredibly grateful for the amazing people I’ve met here. I’m excited to stay involved in a capacity that works better for me and allows me to enjoy this community once again.

See you around!

 

Hey folks!

I'll be updating lemm.ee to 0.19.7 shortly. I'll take it offline & also perform some database maintenance at the same time.

I have been investigating some incoming federation issues, and I haven't been able to find the actual cause. My current theory is that they are related to some server timeouts. I am hoping that the combination of the Lemmy upgrade and database maintenance may help improve the situation, but if not, then I will try other approaches.

Edit: the upgrade has been completed!

 

Hey!

Unfortunately, Hetzner (our hosting provider) is currently experiencing some network issues. They are planning to address this with an emergency maintenance in roughly 13 hours from now, which will cause lemm.ee downtime. Hopefully we'll be fully recovered later tomorrow!


UPDATE: Sorry for the false alarm, I was on the move when I posted this and missed the fact that the Hetzner notice was actually for next month! So it's not as imminent as I originally understood. As we have a whole month to prepare, I will probably be able to come up with some alternative solution to prevent the downtime while they are conducting this maintenance.

 

Hey folks!

I am looking for feedback from active lemm.ee users on what you all value when it comes to images on Lemmy. I'll go into a bit of detail about what our options are, and then I would ask you to voice your opinion about the issue in the comments.

First, some context for those who don't know. Lemmy software can be configured to handle images in three different ways:

  1. Store images locally - whenever an external image is posted somewhere, lemm.ee will download a permanent local copy. When you view posts, you are seeing our local copy of the image.
  2. Proxy all images - similarly to the first option, lemm.ee will download a local copy of external images, however, this copy is temporary. It will be automatically deleted shortly after, and if users open the relevant post/comment again in the future, there will be another attempt to download a temporary copy at that point.
  3. Pass through external images directly - lemm.ee never downloads any external images, users will always connect directly to the source servers to load the images.

There are pros and cons to each configuration.

Storing images locally

Benefits:

  1. Your IP address is never leaked to external image hosts, as you never connect directly to the source server. External image hosts only see the IP address of the lemm.ee server.
  2. External servers don't become bottlenecks for opening lemm.ee posts. If an external server is slow, it won't matter, because the image is always available locally

Downsides:

  1. As time goes on, our storage will fill up with hundreds of gigabytes of useless images, most of which will never be viewed again after the relevant posts fall off the front page.
  2. Many big external image hosts will rate limit bigger Lemmy servers, causing broken images when we fail to make a local copy.
  3. Crucially: some people love to spend their time uploading illegal content to online servers. There are tools to try and filter out such content, but these are not perfect. The end result is that there is a high chance of some content like this inadvertently reaching lemm.ee storage and staying there permanently. This downside is why lemm.ee has not, and will not, use this particular configuration.

Proxying images

Benefits: In addition to the same benefits as exist for the permanent local storage, by only temporarily making local copies for the moment they are requested by our users, we free up a ton of storage & remove the risk of permanently storing illegal content on our servers.

Downsides: The key downside is that external rate limits hit us much harder, as we will be requesting external images far more often. This results in a lot of constant broken images on lemm.ee.

Passing through external images

Benefits:

  1. Images are rarely broken, unless the source server goes down.
  2. The images never touch our servers, removing a lot of risk with illegal content as well as with storage costs.

Downsides:

  1. Our users lose a degree of privacy. Every external image that is loaded on your browser will result in the remote server getting a request directly from your computer to fetch that image - this is pretty much the same as you had visited that external server directly, which lets them log your IP address if they wish.
  2. When remote servers are slow, it can slow down the entire page load in some cases.

Current situation

Initially, lemm.ee was using the third option of passing through images. Ever since support for option 2, image proxying, was implemented in Lemmy code, we immediately switched to that option, mainly for the privacy benefits. However, after many months, and being blocked by more and more external servers, it is clear that image proxying is seriously degrading the user experience on lemm.ee. We often end up with broken images, and our users have to deal with the results.

I still believe image proxying is a really valuable feature, but I am starting to believe it is a better fit for small instances which make much less requests to external servers.

As a result, I am now seriously considering switching back to the previous method of passing through external images.

This is where you come in - I would ask you as users to please let me know which do you value more: the privacy that you get from image proxying, or the better user experience you get from directly passing through images from their source. Please let me know in the comments how you feel. If I get enough feedback about people being against image proxying, then I will be switching it off for lemm.ee soon. Thanks for reading & sharing your thoughs, and I hope you have a great weekend!

8
submitted 1 year ago* (last edited 1 year ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hey folks!

For anybody stumbling on this post from outside lemm.ee: I am the head admin of lemm.ee, a general purpose Lemmy instance, which recently turned 1 year old. I am writing this post to elaborate on how we approach defederation on lemm.ee.

Anybody who has been on Lemmy for a while has most likely seen several public defederation drama posts (most recently regarding lemmy.ml, but there have been many many others previously). As an admin, I have probably seen far more than what is visible publicly, as I regularly receive private messages on the topic, ranging from polite questions about federation, to outright demands that I immediately defederate, and even to threats and personal attacks over the fact that I have not defederated some particular instance. It is definitely a topic that will keep coming up for as long as Lemmy exists, which is why I feel it would be useful to condense my current thoughts about it in a single place.

Note that while I strongly believe everything this post contains, it is definitely a subjective topic, and there is no single right answer here. Other instances have completely different approaches to federation compared to lemm.ee, and that’s of course totally fine. The beauty of Lemmy is that everybody can choose their home instance, and in fact, everybody is free to spin up their own instance and run it however they feel is best. For an absurd example, if you want to create an instance which defederates any instance with an “L” in their name, then nobody can stop you!

Quick intro to the lemm.ee federation policy

Very shortly after creating lemm.ee, I wrote down a federation policy, which basically boils down to “we treat defederation as an absolute last resort, and we do not use it as a generic way to curate content for lemm.ee users”. This policy can always be found in the sidebar of the lemm.ee front page.

In practice, this has meant that we have had extremely few defederations, and that we mostly solve problems with other means. I am very happy with the results, as it means that lemm.ee has become a great entry point into the Lemmy network, with very few artifical limitations on who our users are allowed to interact with.

The benefits of federation

I hope that this part of the post is very uncontroversial, but I firmly believe that federation is the absolute strongest feature of Lemmy. While we all know that the concept of federation can cause confusion for new users, this is usually overcome extremely quickly (for example, using the common e-mail providers analogy to explain Lemmy instances). To me, it’s completely clear that the benefits of federation far outweigh the downsides.

For example, by splitting the Lemmy network between thousands of independent nodes, we ensure that:

  1. Any single entity is not a single point of failure for the whole network. Even if the biggest instance goes down tomorrow, their content will still be accessible through all the other federated instances.
  2. The maximum impact of admins is limited to their own instance. As a lemm.ee admin, I can ban a remote user from posting on lemm.ee, but I can’t completely ban them from the entire network.
  3. Private user data (such as ip addresses, e-mails, etc) are never shared between instances. No single malicious instance can harvest user data for the entire network, and extremely privacy sensitive users can always spin up their own instance if they don’t want to put their trust in any existing admins.

One thing which is probably important to note here is that I tend to view Lemmy instances as infrastructure, rather than as communities. I know that there are alternative approaches, as quite a few large instances are in fact run as mega-communities, but that’s not the approach I take with lemm.ee, because I feel like such an approach encourages centralization and negates some of the benefits of federation (if all communities related to one topic condense on a single instance, then that instance does effectively become a single point of failure for a large number of users).

In general, I feel like it should be a goal to encourage and cultivate decentralizing the network through federation as much as is practical, in order to maximize the above benefits.

The downsides of dedeferation

Conversely, defederation has a lot of downsides.

  1. It obviously negates all the benefits of federation mentioned above. Every time two instances defederate, the Lemmy network becomes less redundant, some communities become a bit more centralized, and the danger of malicious admins for those communities becomes much greater.
  2. There is a lot of collateral damage. The most common reason I have personally seen for defederation demands is related to moderation of either a single user, or a handful of users. For example, a lemm.ee user gets into some heated arguments with people from an instance with hundreds of active users, and then links this heated thread to me as proof that the instance should be immediately defederated. However, in this situation, there are hundreds of other users who were not even involved (or even aware of) the thread in question. By defederating, I would be making a decision to cut off every single lemm.ee user from every single one of those hundreds of innocent remote users.
  3. Ironically, defederation actually makes moderation more difficult. It was recently pointed out to me by a user on another instance that they are afraid they can’t effectively moderate communities on lemm.ee, because their instance has defederated several other instances, which means they would not be able to see posts from those instances on lemm.ee communities.
  4. It is extremely easy for malicious actors to abuse. In the year I’ve been on Lemmy, I have already seen two separate cases of users creating accounts on another instance and posting garbage, and then going back to their home instance and demanding their admins defederate over the content they themselves created. Basically, if an instance is known to use defederation as a tool to punish misbehaving users on other instances, then it’s actually quite easy for users to manipulate the situation to a place where admins have no alternative except to defederate.

It seems to me that a lot of users don’t think of such downsides when demanding defederation, or they just don’t consider them as important enough. In my opinion, these are all significant issues. I do not want to end up in a fragmented Lemmy network, where users are required to have accounts on 5 different instances in order to be able to access all their communities.

What’s the alternative to defederation? Should Lemmy become some kind of unmoderated free speech abolutism platform?

I want to be very clear that I do NOT believe in unmoderated social networks. Communities should always be free to set and enforce rules which foster healthy discussions. On top of that, instances should always be free to set and enforce rules for all of their users and communities.

In the case of lemm.ee, we have some instance-wide rules, and we will enforce them on all lemm.ee users, as well as all remote users participating in communities hosted on lemm.ee. For example, we never want to offer a platform for bigotry, so we regularly issue permanent bans for users who want to abuse lemm.ee to spread such hate. In practice, site bans have been extremely effective at getting rid of awful users, whether they are remote or local.

On top of site bans, Lemmy admins also have the option of removing entire remote communities. There are certainly cases where a community might be allowed on instance A, but not instance B - rather than defederating (and potentially cutting off a lot of innocent unrelated users), instance A can just “defederate” a single community.

Finally, a lot of issues can be solved through simple communication between instance admins. Often having a discussion with another admin results in pretty clear alignment over whether some user is problematic, and the user will end up being banned on their home instance.

Being one of the most openly federated large instances with such an approach, we have discovered several things:

  1. If we were to defederate over every rule breaking user or community on the Lemmy network, we would not be federated with any of the large instances at this point
  2. In the vast majority of cases, remote users who have broken lemm.ee rules have ended up banned on their home instance anyway - there is very little additional moderation workload for our admins from being widely federated
  3. If a user truly wants to spread some kind of hate, defederation wouldn’t stop them anyway, as they will just create accounts on any instance which they want to “attack”

The longer I run lemm.ee, the more sure I become that in the vast majority of cases of abusive users, the best approach is to simply hand out site bans.

When is defederation the only option?

Having said all of the above, I still believe that there a few cases when defederation is the best option:

  1. When an instance is abusing the Lemmy network - generating spam, advertising, illegal content, etc - either deliberately, or through inactive admins (this has been the most common reason for lemm.ee to defederate any instance in the past)
  2. When an instance is just causing too much moderation workload. So far, we haven’t experienced this yet on lemm.ee, but I can’t rule out that it could happen in the future.

Conclusion

I hope this post helps clarify my stance on defederation. Like I said in the beginning, I realize a lot of this is subjective, and there are no right or wrong answers - this is just the way we have been (and will be) doing things on lemm.ee. I intend to save this post and link it in the future when people bring up defederation requests. If you feel like I didn’t address something important, please feel free to raise it in the comments!

 

Hey all!

Upcoming lemm.ee cakeday

Can you believe that lemm.ee is almost 1 year old? In just a couple of weeks (specifically, on the 9th of June), we will be able to celebrate our first instance cakeday.

I am thinking of compiling some stats about how lemm.ee has been used in its first year, if you have any specific stats in particular you would like to see, feel free to comment below. I will try to accommodate any ideas as I start gathering this info!

Infrastructure updates

A few weeks ago, I posted about plans to make some changes to our infrastructure in order to deal with different intermittent networking issues.. It took a bit longer than I hoped (just did not manage to get enough free time between then and now), but I am happy to report that this work has now been completed! Additionally, I have decommissioned our stand-alone pict-rs server.

With the two changes mentioned above, I believe lemm.ee should now be much more resilient going forwad, and I expect a significantly lower rate of infrastructure-related issues for the rest of the year!

I'll leave a tehcnical overview about the problem & solution below for those interested, but if these details don't interest you, then you can safely skip the rest of this post.


For context, lemm.ee has been hosted on Hetzner servers for most of this year (having migrated from DigitalOcean initially), with everything except our database being hosted on the Hetzner Cloud side, and the database itself living on a powerful dedicated Hetzner server. This mix allows a great amount of flexibility for redeploying and horizontally scaling our application servers, while still allowing a really cost-effective way of hosting a quite resource-hungry database.

In order to facilitate networking between the cloud servers and the dedicated database server (which live in different networks), Hetzner provides a service named "vSwitch". This service basically allows you to connect different servers together in a private network. Unfortunately, I discovered quite quickly that this service is very unreliable. During the short few months that we have been using the vSwitch, we have gone through one extended period of downtime (where the service was just completely broken for several hours), as well as dozens (if not hundreds at this point) intermittent disconnects, where servers randomly lose their connections over the vSwitch. After such a disconnect, the connection never recovers without manual intervetion.

For most lemm.ee users, the majority of these vSwitch issues have been mostly invisible, as we have redundancy in our servers - if one server loses its connection to the database, other servers will take over the load. Additionally, I have generally been able to respond quite quickly to issues by redeploying the broken servers (or deploying other temporary workarounds). However, in addition to a huge amount of these issues which lemm.ee users hopefully haven't ever noticed, there have also been a few short periods of downtime this year so far, as well as a few cases of federation delays. These more extreme cases were generally caused by multiple servers losing their vSwitch connections at the same time.

After several attempts to work around these issues, I decided that we need to migrate away from vSwitch.

As of earlier today, lemm.ee is no longer using Hetzner's vSwitch at all!

I finally found enough time earlier today to focus on this migration, and I was able to successfully complete it. None of our networking is relying on the vSwitch anymore.

In the end, I went with quite a simple solution - I configured a host-level firewall (nftables) on our database dedicated server, which will deny all connections by default. Whenever any cloud servers are added/removed, their corresponding public IP addresses are added/removed in the allowlist of our database firewall. It would have been ideal to do this whole logic in Hetzner's own firewall, but that one unfortunately has a limit of only 10 rules per server, which is just not enough for our setup.

Bonus: our pict-rs server has been decommissioned!

Pict-rs is the software which Lemmy uses for everything related to media (image storage mostly). Initially, pict-rs required a local filesystem to store both files as well as metadata about files. Since the beginning, lemm.ee has used a dedicated server just for pict-rs, in order to ensure we could easily redeploy the rest of our servers without losing any images.

Over the past year, pict-rs has gained the ability to store files in object storage, and metadata in a PostgreSQL database. This meant that the server running pict-rs itself no longer contained any of the important data, so it became possible to redeploy without losing any images. Additionally, this meant that it would be possible to run multiple pict-rs servers in parallel.

While we had already migrated our pict-rs server to use object storage and PostgreSQL several months ago, we still had the single dedicated pict-rs server up until today. I have been planning for a while to decommission this server, and start running pict-rs directly on each one of our Lemmy application servers. Earlier today, I was able to complete this plan. This should hopefully mean that our pict-rs server is less likely to get overloaded, and it also means a tiny reduction in our overall monthly infrastructure bill (due to one less server running).

With the above changes, I think our infrastructure has become more robust, and hopefully, we will experience less issues with images, federation, and general downtime going forward.


That's all from me for now. Feel free to leave any thoughts or questions in the comments, and as always, I hope you're having a great day!

1
submitted 2 years ago* (last edited 2 years ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hey folks!

I made a short post last night explaining why image uploads had been disabled. This was in the middle of the night for me, so I did not have time to go into a lot of detail, but I'm writing a more detailed post now to clear up where we are now and where we plan to go.

What's the problem?

As shared by the lemmy.world team, over the past few days, some people have been spamming one of their communities with CSAM images. Lemmy has been attacked in various ways before, but this is clearly on a whole new level of depravity, as it's first and foremost an attack on actual victims of child abuse, in addition to being an attack on the users and admins on Lemmy.

What's the solution?

I am putting together a plan, both for the short term and for the longer term, to combat and prevent such content from ever reaching lemm.ee servers.

For the immediate future, I am taking the following steps:

1) Image uploads are completely disabled for all users

This is a drastic measure, and I am aware that it's the opposite of what many of our users have been hoping, but at the moment, we simply don't have the necessary tools to safely handle uploaded images.

2) All images which have federated in from other instances will be deleted from our servers, without any exception

At this point, we have millions of such images, and I am planning to just indiscriminately purge all of them. Posts from other instances will not be broken after the deletion, the deleted images will simply be loaded directly from other instances.

3) I will apply a small patch to the Lemmy backend running on lemm.ee to prevent images from other instances from being downloaded to our servers

Lemmy has always loaded some images directly from other servers, while saving other images locally to serve directly. I am eliminating the second option for the time being, forcing all images uploaded on external instances to always be loaded from those servers. This will somewhat increase the amount of servers which users will fetch images from when opening lemm.ee, which certainly has downsides, but I believe this is preferable to opening up our servers to potentially illegal content.

For the longer term, I have some further ideas:

4) Invite-based registrations

I believe that one of the best ways to effectively combat spam and malicious users is to implement an invite system on Lemmy. I have wanted to work on such a system ever since I first set up this instance, but real life and other things have been getting in the way, so I haven't had a chance. However, with the current situation, I believe this feature is more important then ever, and I'm very hopeful I will be able to make time to work on it very soon.

My idea would be to grant our users a few invites, which would replenish every month if used. An invite will be required to sign up on lemm.ee after that point. The system will keep track of the invite hierarchy, and in extreme cases (such as spambot sign-ups), inviters may be held responsible for rule breaking users they have invited.

While this will certainly create a barrier of entry to signing up on lemm.ee, we are already one of the biggest instances, and I think at this point, such a barrier will do more good than harm.

5) Account requirements for specific activities

This is something that many admins and mods have been discussing for a while now, and I believe it would be an important feature for lemm.ee as well. Essentially, I would like to limit certain activities to users which meet specific requirements (maybe account age, amount of comments, etc). These activities might include things like image uploads, community creation, perhaps even private messages.

This could in theory limit creation of new accounts just to break rules (or laws).

6) Automated ML based NSFW scanning for all uploaded images

I think it makes sense to apply automatic scanning on all images before we save them on our servers, and if it's flagged as NSFW, then we don't accept the upload. While machine learning is not 100% accurate and will produce false positives, I believe this is a trade-off that we simply need to accept at this point. Not only will this help against any potential CSAM, it will also help us better enforce our "no pornography" rule.

This would potentially also allow us to resume caching images from other instances, which will improve both performance and privacy on lemm.ee.


With all of the above in place, I believe we will be able to re-enable image uploads with a much higher degree of safety. Of course, most of these ideas come with some significant downsides, but please keep in mind that users posting CSAM present an existential threat to Lemmy (in addition to just being absolutely morally disgusting and actively harmful to the victims of the abuse). If the choice is between having a Lemmy instance with some restrictions, or not having a Lemmy instance at all, then I think the restrictions are the better option.

I also would appreciate your patience in this matter, as all of the long term plans require additional development, and while this is currently a high priority issue for all Lemmy admins, we are all still volunteers and do not have the freedom to dedicate huge amounts of hours to working on new features.


As always, your feedback and thoughts are appreciated, so please feel free to leave a comment if you disagree with any of the plans or if you have any suggestions on how to improve them.

 

Hey folks

I have been receiving a lot of messages every single day about federation with hexbear. Some of our users are vehemently against it, others are in full support. The conversation does not seem to be dying down, rather, the volume of messages I receive about it seems to be increasing, so I am opening this public space where we can openly discuss the topic.

I am going to write a wall of text about my own thoughts on the situation, I’m sorry, but no tl;dr this time, and I ask anybody participating in this thread to first read through this post before commenting.

Before I go any further, I want to be clear that for anybody who participates here, it is required to focus on the quality of your posts. That means:

  • Be kind to each other, even if you disagree
  • Use arguments rather than calling people names
  • Realize that this is a divisive topic, so your comments should be even more thoughtful than usual

With that out of the way, there are a few things I want to cover.

On defederation in general

First of all, I am a firm believer that defederation must be reserved only for cases where all other methods have failed. If defederation is used liberally, then a small group of malicious users can effectively completely shut down the federated network, by simply creating the type of drama between instances which would inevitably result in defederation. In my view, federation is the biggest strength of Lemmy compared to any centralized discussion forum, so naturally I think maintaining federation by default is an important goal in general.

I am also a believer in the value of deplatforming hateful content, but I think defederation is not the best way to do this. Banning individual users, banning communities and establishing a culture of mutual support between mods and admins of different instances should be the first line of defense against such content. There are some further steps that can be taken before defederation as well, but these are not really documented anywhere (in order to prevent circumvention). The point is: for myself, defederation is the absolute last resort, only to be used when it is completely clear that other methods are ineffective.

Finally, I am wary of creating a false expectation among lemm.ee users that lemm.ee admins endorse all users and communities and content on instances we are federated with. Here at lemm.ee, we use a blocklist for federation, which means our default apporach is to federate with all new instances. We do not have the resources (manpower, skills and knowledge) necessary to pass judgement on all instances which exist out there, as a result, users on lemm.ee are expected to curate their own content to quite a high degree. In addition to downvoting and/or reporting as necessary, individual lemm.ee users are also able to block specific users and communities, and the ability to block entire instances is coming very soon as well.

Having said all that, in a situation where all other methods do indeed fail, defederation is not out of the question. Making such a call is up to the discretion of lemm.ee admins, and doing it as a last resort is completely in line with our federation policy.

Regarding hexbear

Hexbear is an established Lemmy instance, focused on many flavors of leftism. They have quite a large userbase who are very active on Lemmy (often so active that they leave the impression brigading all popular Lemmy posts). One important thing to note is that while some forms of bigotry seem to be quite accepted by many hexbear users (but seemingly not by mods - more on that below), they at least are very protective of LGBT rights (and yes, I am quite certain that they are not just pretending to do this, as many users seem to believe). Additionally, while I have noticed quite high quality posts from hexbear users, there are also several users there who seem to really enjoy trolling and baiting (very reminiscent of 4chan-type “for the lulz” posting), and it’s important to note that this kind of posting is in general allowed on hexbear itself.

The reason this whole topic is important to so many people right now (despite hexbear being a relatively old instance), is that hexbear only recently enabled federation. A combination of their volume of posts, their strong convictions, the excitement about federation, and the aforementioned trolling has made them very visible to almost all Lemmy users, and this has sparked discussions about the value of federation with hexbear on a lot of Lemmy instances.

My own experience with hexbear

I want to write down my own experience with interacting with hexbear users, mods, and admins over the past few days. I believe this experience will highlight why I am hesitant to advocate for immediate full defederation from hexbear at this point in time, and am for now still more in favor of taking action on a more individual user basis. Please read and see how you feel about the situation afterwards.

Background

My first real contact with hexbear users was in the comments section of a post in this meta community requesting defederation from hexbear by @glimpythegoblin@lemm.ee. That post is now locked, because several hexbear users very quickly started doing the aforementioned “for the lulz” type spamming of meme images in the comments (these are actually just emojis, but they are rendered as full-size images on all instances other than the source instance, due to a current Lemmy bug).

I did not want to take further actions in that thread in general (for archival purposes), but I did take one action, which in retrospect was a mistake: I removed a comment which contained the hammer and sickle symbol. I ignorantly associated this symbolism with Kremlin propaganda, and the atrocities my own people suffered at the hands of the soviet union during the previous century. Many users (including hexbear users) correctly (and politely) pointed out to me in DMs that the symbol has a much broader use than just as the symbol of the USSR, and people elsewhere in the world may not associate it with the USSR at all. I am grateful for users who pointed this out to me without resorting to personal attacks.

Let me be clear here: while I do not have anything against leftism or communist ideas in general (in fact in today’s world, I think discussion of such ideas is quite necessary), Kremlin propaganda has no place on lemm.ee. Any dehumanizing talking points of the Kremlin on lemm.ee are treated as any other bigotry, and if communist symbolism is used in context of Kremlin propaganda (that is the context in which I have been exposed to it throughout my whole life), then it will still be removed. But there is no blanket ban on communist symbolism in general on lemm.ee, and discussing and advocating for leftist and communist topics (as distinct from the imperialist and dehumanizing policies of the Kremlin) is certainly allowed on lemm.ee.

Hexbear user response

Coming back to the events of the past few days: soon after my removal of the comment containing the symbol from the meta thread, two posts popped up on hexbear. One was focused on insulting and spreading lies about me personally. Another was focused on diminishing the horrors of the soviet occupation in my country. In the comments under both of these posts (and in a few other threads on hexbear), I noticed some seriously disturbing bigotry against my people. There were comments which reflected the anti-Estonian propaganda of the current Russian state, things like:

  • Suggesting that my people has no right to exist
  • Stating that my people (and other Baltic nations) are subhuman
  • Claiming that anybody critical of both nazi and soviet occupations is themselves a nazi and a holocaust denier

I expect to hear such statements from the Russian state - here in Estonia, we are subjected to this and other kinds of bigotry constantly from Russian media - but to see it spread openly in non-Russian channels is extremely disturbing. Such bigotry is completely against lemm.ee rules in general. Additionally, my identity is public information, because I feel it’s important for the integrity of lemm.ee that I don’t hide behind anonymity. Considering this, I’m sure you can understand why I am very worried about my own safety when people leave comments in many unrelated threads (where my original posts are not even visible), baselessly calling me a nazi and a holocaust denier.

Note that the goal of this post is not to start a new debate in the comments about the the repressions of the soviet union in Estonia or other occupied territories, but if the topic interests any users, I can recommend the 2006 documentary The Singing Revolution (imdb). The trailer is a bit cheesy, but the actual film contains lots of historical footage from the soviet occupation, and also many interviews with people who experienced it, who share stories which are deeply familiar to all Estonians. If anybody is interested in further discussion, then I suggest making a post about it in the Estonian community here: !eesti@lemm.ee.

Hexbear admin response

After the above events had played out, I reached out to hexbear admins for clarification on their moderation policies and how they handle such cases. I was actually very happy with their response:

  1. They immediately removed the personal attacks and dehumanizing comments containing Kremlin propaganda from Hexbear, and assured me that such content is always handled by mods
  2. They told me that while there are all kinds of leftists on hexbear, Russian disinformation is generally either refuted in comments or removed by mods
  3. They implemented some additional rules on hexbear to try and reduce the trolling experienced by many other instances, including ours: https://hexbear.net/post/352119
My personal take-aways

Let me play the devil’s advocate here and employ some “self-whataboutism”: among all users that have been banned on lemm.ee for bigotry, the majority were actually not users from other instances, and in fact people with lemm.ee accounts. If we judge any larger instance only by bigoted posts that some of its users make, then we might as well declare all instances as cesspools and close down Lemmy completely. I believe it’s far more useful to judge instances based on moderation in response to such content. Just as we remove bigoted content from lemm.ee, I have also witnessed bigoted content being removed from hexbear.

At the same time, I am aware of some internal conflict between hexbear users over the more strict moderation they are now starting to employ, and I am definitely keeping an eye on that situation and how admins handle it.

I am also still quite worried about the amount of distinct users on hexbear who have posted Kremlin propaganda. I so far don't have reason to believe that these users are employed by the Russian state, but the fact that they are spreading the same hateful content which can be seen on Russian television seems problematic to say the least, and it remains to be seen if moderators can truly keep up with such content.

Where thing stand right now

I am not convinced that we are currently at a point where the “last resort” of defederation is necessary. This is based on the presumption that our moderation workload at lemm.ee will not get out of hand just due to users from that particular instance. My current expectation is that as the excitement of federation calms down (and as new rules on hexbear go into effect), the currently relatively high volume of low effort trolling will be replaced by more thoughtful posts. If this is not the case then we will certainly need to re-evaluate things.

Additionally, nothing is changing about our own rules regarding bigotry. Especially relevant in the context of Kremlin propaganda, I want to say that dehumanizing anybody is not allowed on lemm.ee (hopefully I do not have to spell it out, but this of course includes Ukrainians, LGBT folks, and others that the Kremlin despises), and action will be taken against any users who do this, regardless of what instance they are posting from.

Finally, I am very interested to hear thoughts and responses from our own users. I am super grateful to anybody who actually took the time to read through this massive dump of my own thoughts, and I am very interested to get a proper understanding of how our users feel about what I’ve written here. Please share any thoughts in the comments.

view more: next ›