Update and Roadmap

TL;DR: The server died! The new one is racked up. FSE will be back shortly, but it will be Revolver rather than Pleroma. If you had an FSE account, the same username and password will get you into fsebugoutzone.org, but you may or may not want to use that one, for reasons that will be explained shortly.

What happened to FSE?

We were struck down by God for our hubris. (The computer blew up. It took a minute to get a new one ready.)

I decided (https://shitposter.club/objects/19a96a25-67f3-466f-bbed-aab2fba66226 and https://shitposter.club/objects/abfe062f-6f43-4d15-93ab-903f823ccf2b, but you can't click those; see the aside below) to take the opportunity to just focus on Revolver.

Aside about links on fedi: it appears that most people don't know this, but those /objects/ links you get out of Pleroma? You don't have to click on them and load the other server's identical webapp to view the posts they point to: you paste them into your instance's search box. If the post has already federated, then you'll see it right there. If the post has not federated, your server will attempt to fetch it. If it can be found or fetched, then you'll just see it right there. Since SPC is down, and since their API was locked down anyway, to avoid scrapers (more about those guys in a while, fascinating stuff), some of which are state actors (seriously) nowadays, you won't be able to fetch those. But you can still see the threads and interact with the posts thanks to the decentralized nature of the fediverse.

After the smoke got out, I got some very generous offers to help fund the operation, including people that don't use FSE directly. I received an offer from a friend to rack up a dedicated machine in a datacenter in Idaho! I am very grateful to everyone that offered to help, whether the offer was big or small. It was genuinely heartwarming, to the extent that I felt almost guilty for declining to take people up their offers: if it meant enough to people that they were willing to front the cost. On the other hand, FSE has, from the beginning, taken no donations. CofeSpace and Revolver have, but for FSE itself, it's been funded out of pocket since the beginning, and I don't want to change that. If you read those posts, you already know I had a better idea, and that running FSE, especially on that flaky box, was going to eat a little time from the development schedule.

Okay, then what happened?

Next, SPC was struck down for their hubris! Since that was where I declared my alt if FSE went down, this was a problem. So I hopped over to Bae Street. Then SPC came up, then it went down again, then Baest went down temporarily for maintenance, then SPW arrived (and started federating around March 30) and accounts were automatically migrated, and that was cool. (Moon's the best.) I do not remember the specific order of what happened where, but it got difficult to reliably tell people where I was going to be, and besides that, for hacking on Revolver, it is very useful to have a live server, so I decided that once I got the new hardware in line, I'd roll out the FSE bugout zone.

FSE Bugout Zone

I set up FSE with a few vague goals: I thought people that could run a server should run one, I didn't want to make anyone else accountable for something I said, I wanted to push back against censorship and the bootlickers that enforce it, and I wanted to make a place for people that would be banned elsewhere. I had not ever listened to Alex Jones, but I enjoyed all the meme songs based on his rants, things like that, and then suddenly he was getting banned from everywhere at once by the San Francisco Data Cartel, and I left Twitter permanently.

(As an aside, despite the San Francisco Data Cartel being a confederacy of the most ruthless billionaires in the US, and despite having a cozy relationship with the government and being inextricably intertwined with the military-industrial complex and the so-called "surveillance capitalism" state, despite its extremely cozy relationship with law enforcement, has an inexplicably good relationship with the alleged far left. It boggles the mind that these companies are so adored by people that allegedly dislike billionaires and the US government and cops. Even more absurd, some of them insist that the phrase "big tech" is actually a dogwhistle for evil fascists. It makes no sense to me how they could be that thoroughly coopted, or that none of them would notice, but the habit they have of shouting down people that question the orthodoxy is almost calculated to perpetuate a situation where people run around shouting slogans without stopping to think.)

Those are still the goals for FSE, though the vehicle in the longer term will be Revolver. The bugout zone has some different goals, being a temporary place while FSE is down.

So, fsebugoutzone.org is desiged to function with these goals:

• A live Pleroma server that I can poke at, which is useful for developing Revolver: I can view the logs, I can compare Revolver's responses to Pleroma's, etc.
• A testbed for some changes to make Revolver and Pleroma understand each other. (To ease load on heavier server; you'll be able to see the changes on git.fse when it's back up.)
• A stable place to be, which I needed to have so people could find me.
• A stable place for anyone else that had an account on FSE.

It's very important, if you decide to stop by there, that I cover a couple of things that you may not like so that you are operating with all of the information. I do not want to give you the impression that it will work the same way as FSE, because you will be disappointed. Full disclosure, no false advertising, etc. First, the bugout zone is going to break a lot. Active development. I'm testing changes live. So I am trying to keep it stable enough to be on, and you feel free to be there too, but if I am hacking on something and attachments don't work, maybe it's like that for a day or two. Maybe something gets ridiculously slow. I plan to keep it up and running and be there, but there will be dumb stuff going on. The other thing is the bugout zone is under martial law. Anything that makes me think for more than a minute or two about a non-technical issue will be shot on sight: I hate moderating grown-ass adults and I never wanted to do it, but FSE is FSE, the bunker is a bunker. My idiot hacks will create enough problems without interpersonal problems getting in the way, and we're behind schedule, so I am going to go with the fastest way to solve any problem that isn't code-related. robots.txt will be extremely locked down. Hopefully it will be a reasonable place to be and talk to your friends and exchange jokes and interesting information and drunken lies, but I would really rather under-sell it than over-sell it, so fair warning: it will be broken a lot and I am going to be lazy about moderation by just shooting motherfuckers and instead of general policies intended to produce the sort of place I'd like to be, all administrative action will be subject to random whims. I don't have time to let administering another FSE get in the way of Revolver, so while the "FSE Autonomous Zone" was me essentially announcing that I don't care and I was going to abandon accountability, I am going to be somewhat more aggressive about not caring about letting a temporary bugout zone get in the way of hacking on the thing that finally achieves the goals of FSE without requiring admins.

The bugout zone is going to get instance-blocked by people that think FSE is part of some nefarious secret plan (a plan which I have apparently taken nearly six years to get started; it's almost like the plan was exactly the plan I laid out from the beginning, and every time anyone asked me, and a couple of paragraphs above), or by instances that do not like adult images or adult humor or working-class language, or people that don't like me or don't like other people on FSE. That's fine: I don't like it (instance-blocking individuals and reaching out to the admin is a better idea), but the nice thing about decentralized systems like fedi is the high degree of individual autonomy, so they're all free to do things I don't like. They're free to do things nobody likes! It's less fine that people have lied about FSE, or how FSE works, or about me personally, or instance-blocked FSE based on suspicions, and I expect anyone that participated in creating or spreading those lies to go ahead and block this instance, too. There are also plenty of people that just assume bad faith from the start, and they will assume that this is a block-avoidance maneuver, even if they see this post or they read the instance description, and that I have some evil plan and this is a critical part of it and if we're not blocked right away, this is immensely dangerous, and I don't like that, either, but that's how some people are, and I don't intend to fight human nature. So if you come to the the bugout zone, please do not go out of your way to get us instance-blocked again; you don't have to be nice or self-censor, but there's a vast gulf between hiding your beliefs and opinions and spamming gamer-words at TWKN because you clicked on the "Show me everything" button and saw some things you didn't like. There are servers where the people running them like getting instance-blocked, but especially given the nature of this temporary server as a testbed for development, I'd like to maintain broad federation and a way to communicate with the people that are

It is also a somewhat older backup: the machine died on February 1st, but the backup I loaded and tweaked was from January 11th. The import process (see the roadmap) is using a copy of the actual database that was on the disk when FSE went down, but the this machine is

The Current Roadmap

As you may be able to tell from the SPC posts (if you were able to dig them up), the hardware failure has redirected the efforts somewhat. The initial plan was to hack on Revolver, then release when it's ready, then get FSE moved. That plan has changed as of the death of the old box. The new machine does not work so great as a Pleroma server at FSE's scale, but the hardware procured works much better for a series of Revolver nodes, which is why I got what I got.

So, here is the current roadmap and the state of the progress:

Done:

• Capacity planning for new hardware.
• Procure new hardware, install OSs, configure, etc.
• Rack up the new hardware at the datacenter.

In progress/testing:

• Pleroma-to-Revolver ingestor. This is working well (after some tweaks to cope with servers that sent weird objects), but there are some performance bottlenecks in bits of Revolver, so I'm fixing some of those, some pieces need to be re-designed. This is more or less as expected: I did some very simple things with the intention that I'd just leave them extra simple for as long as I could get away with it,
• A simple static timeline, mostly for debugging. TWKN and user timelines and hashtags are working fine. There is no more "local", but the domain timeline is working fine so maybe the local timeline just uses the timeline for whatever domain you're logged into. It is largely for debugging, but it is way, way faster than expected and is a thing I have wanted (FSE had to disable the static FE in Pleroma because attempting to view a hellthread in it would tie up Postgres for a few minutes.)
• Initial support for Pleroma FE. I am targeting the version on FSE, but the newer FE should work; I haven't tested it yet (and don't want to get side-tracked by the moving target, so I don't even intend to assess how fast the target moves). Some stuff works, most stuff does not. It does enough for the UI to come up and, if you move a couple of the endpoints around, it will show you timelines.
• Enough AP support to converse with the rest of fedi in a reasonable way.

Near future timeline:

Some of these have initial work done or architecture designed or sketched out, but they are not actively in progress. In particular, the P2P stuff was the main focus of development work before FSE exploded.

• Do a full import of FSE's database into Revolver from a blank slate to ensure that it works and to give rough performance estimates on doing this kind of thing from scratch.
• FSE resumes accepting posts. This is basically an nginx tweak and will happen once I have reworked the inbox a little, which is the same chunk of code as the ingestor.
• Once the FE works for people that are not logged in, start serving that on FSE. Logging in and registering won't work, but you can poke around and read posts and threads.
• Once the FE supports basic interactions, start letting people log in, possibly register. At that point, I will probably be back on FSE, so at minimum, it will have to support long-ass threads and excessive amounts of smashing that like.
• Ensure compatibility with newer versions of Pleroma FE and bloat.
• Do an excessive amount of testing and refining the P2P stuff. The sheer amount of ways that this can screw up nearly ensures that there will be things I failed to account for, but I would like the most solid base I can produce.
• Deliver the code to the people that I said I'd send early releases to. This would effectively be a limited beta.
• Document the P2P protocol. This will allow other people to start building clients that talk to Revolver; I don't expect anyone to do it before the big release, but it doesn't hurt to open the door up and to be ready, and maybe someone notices a problem with the protocol and either pings me or exploits it, and then it can be corrected without worrying about code that is already deployed and in the wild.
• Once the code meets the minimum functionality and doesn't accidentally DoS itself periodically, do a release. It'll be on git.fse and probably mirrored to other places, development discussions will probably happen on a mailing list and there should be an IRC channel.
• Retire the bugout zone or convert it to Revolver.
• I am no longer maintaining a fedi instance but nodes in a fully P2P network and I can just chill and code. Nobody complains that I banned someone or failed to ban them.

The Server

The new server is not a server, but an ARM cluster. I made a joke post on FSE a long time ago about taking one of my old UltraSPARC machines, gutting it, and filling it up with ARM SBCs, taking it down to the datacenter, and telling them it was a computer. In the intervening period, this actually became possible, though it required less time spent with a dremel: there are ARM and even RISC-V cluster boards. I picked up a Turing Pi 2, expecting to use it as a testbed, but it's been pretty reliable. Way more reliable than the Dell R820 that powered FSE. And, although having 384GB of RAM was very useful for handling about 30 reqs/second for a service that used a Postgres server that was about 500GB on-disk and growing without bound, the RK1 with 32GB of RAM actually seems over-powered for Revolver, which has been tested and stress-tested on much smaller machines. media.fse is actually just the screamshitter.club Revolver test node and, once I added the streaming interface for block storage, the crowded little test node has been handling FSE's traffic about as efficiently as nginx streaming files from disk. I have been running Revolver on Plan 9 VMs and little ARM/RISC-V machines (mostly the DevTerm, but I have an ODROID XU-4 that has been doing some of the work) and dinky VPSs with poor I/O and I can't manage to get the thing to slow down.

So, we're running on an ARM cluster! Three RK-1s and a CM-4. One of the RK-1s is serving up the bugout zone, blog, git, etc., and they're all running Revolver in addition to whatever other stuff they do. If one of them falls over, I should be able to bounce it remotely rather than filing a ticket with the datacenter. It's newer (and for some tasks, faster). They're all running a build of CRUX. Once FSE has completely moved to Revolver, any of them should be able to fall over without affecting operations (except, of course, operations that were in progress when it fell over).

THANK YOU FOR READING MY BLOG POST

UPDATE:

Actually, forget the part where I said it was working. (Also: I hate Elixir so much.)

Currently downgrading OTP to 23.3. I should probably just downgrade it to the lowest version supported by Elixir 1.11.

Actually, forget the part where I said I was downgrading OTP to 23.3. It doesn't support build with current versions of OpenSSL, and if I have to patch something, it's probably easier to just patch the code and dependencies in-tree. Hilarious! (Elixir is garbage.)

...So help me, if I end up having to create a chroot just to run this shit, I will start slicing up motherfuckers.

Also, now hackney is the problem. (Elixir sucks.)

Also, now hackney is still the problem. As far as I can tell from the code, Pleroma doesn't bother with any of this, so I'm not actually sure how this was working before. I strace'd it and it's not trying to open it up. The hackney project apparently changed this around, but the git commit for hackney is hard-coded so I have the same version I was using before. I'm fuckin' hungry, also fuck Elixir.

ALSO UPDATE

It's up, but also some stuff is broken. After some of it is fixed, I'll get bloat.fse and git.fse back up.