I spent several hours tracing in production (updating the code a dozen times with extra logging) to identify the actual path the lemmy_server code uses for outbound federation of votes to subscribed servers.

Major popular servers, Beehaw, Leemy.world, Lemmy.ml - have a large number of instance servers subscribing to their communities to get copies of every post/comment. Comment votes/likes are the most common activity, and it is proposed that during the PERFORMANCE CRISIS that outbound vote/like sharing be turned off by these overwhelmed servers.

pull request for draft:

https://github.com/LemmyNet/lemmy/compare/main...RocketDerp:lemmy_comment_votes_nofed1:no_federation_of_votes_outbound0

EDIT: LEMMY_SKIP_FEDERATE_VOTES environment variable

  • King@vlemmy.net
    2·
    1 year ago

    Thanks for doing all this.

    Do we have any real numbers from a real server? How many votes are trying to be federated to how many servers?

    Just ballparking some approximate numbers:

    15000 * 4000 * 10 = 600,000,000 federated actions. That is around 7,000 per second 24/7 for one community.

    IMO, this real time federation just doesn’t scale. We need to start planning the specs for federation batching.