Ice Rock @IceRock@glaceon.social

Hiking to the center of the region's coldest tundra, you finally find it: The Ice Rock. It radiates with an unknowable energy.

You reach forward and touch it, wiping away a layer of snow. The ice is crystal clear, so it's not difficult to see that something is frozen inside it.

A laptop computer is running, despite no obvious source of power, and messages are occasionally printed to a terminal window. If you squint, you can just about make out what they say...

Quick heads up that I've switched g.s into "manually approve new users" mode. The number of spambots targeting fedi is slowly rising and having completely open registration was starting to be a problem.

I have every intention of approving any account that looks like it's a real person as soon as I can get to it.

There we go! That wasn't bad at all. Exactly as smooth as promised. I took the instance down as a precaution but honestly if I had just updated normally, I'd have been done in less time than it took to make sure everything was properly stopped.

Hello, I will also make an announcement

There's a critical mastodon security update happening some time today that I'll be applying as soon as possible. It shouldn't take me too long; the most recent update was actually released specifically with the intent of making today's update as smooth as possible.

In about two-ish to four-ish hours from this post, the update will be released. I'm going to bring the instance down right away when it's released. And when I say "right away" I really do mean it, nobody currently knows what the nature of the security issue is but the amount of preparation they're doing for this update implies to me that it's pretty important! So I'm not even going to be making a "hey we're going down now" post. Sorry.

We shouldn't be down for long. Again, the most recent update was intended to make this one easy, so I don't anticipate problems. We'll see!

TL;DR Prepare for downtime 2 to 4 hours from now

So! That actually went really smoothly. You didn't hear from me as quickly as I would have liked you too because there's a new security feature I was unaware of.

Here we have a feature where any subdomain of glaceon.social points to the same place. You can access the instance from [anything].glaceon.social. This is useful because your browser will remember different cookies for each subdomain, so you can be logged into different accounts. For example, I use my main account from glaceon.social and this account from admin.glaceon.social.

The new security feature prevents this from working! So I had to log in to my admin account for the first time in a while, which meant remembering which email address it uses, etc etc.

Other than that, as far as I know, everything is good! If you run into any other troubles, please give me a shout at @monorail where I'm more likely to actually see it.

Currently working on getting the subdomain thing fixed, but I'm not sure anyone other than me ever uses it so it shouldn't be too much of a disruption :P

Here we go.

Hey everyone! Just a heads up that I'm going to be updating the instance. I think it'll be pretty smooth (no merge conflicts 🎉) and I'm anticipating a few seconds of downtime. But you know how these things go. If you can't reach the site for more than a minute or so in the near future, my backup account is at @monorail, where you'll probably find me swearing into the void while trying to fix everything.

Heads up. I'm going to be defederating from .social/.online. There's just too much shit to put up with for the sake of being welcoming to users who are new to the fediverse.

I don't know exactly when yet but if you're there, please consider moving. Registrations are open here if you want a new place to stay, I might be able to help you troubleshoot problems spinning up a single-user instance or something, etc. The flagship instances are a blight on the fediverse and I can't force users here to be exposed to that for much longer.

Okay one person says they restarted their PC and the errors seem to have stopped so if you're getting intermittent cert errors, try closing your browser all the way? And if that doesn't help try doing a reboot maybe?

If you're still having issues with intermittent cert errors, maybe try this?

https://www.a2hosting.com/kb/getting-started-guide/internet-and-networking/clearing-a-web-browsers-ssl-state​

No idea if it will help but it's an idea. Haven't been able to reproduce any problems yet.

post incident report

what happened?

i woke up and g.s was down

how was it fixed

sudo reboot

/shrug

Hello everyone!

You may notice that the instance just went down for a little while. Normally, I would have loved to make this announcement first, but this wasn't exactly planned downtime. The instance just... went down. In fact, I started typing up an announcement on my backup account (@monorail) in case anyone remembers that, but I was partway through it when g.s came back up.

But, while it was down, I did something I've been putting off for a while: I've upgraded the VM that g.s runs on!

This should hopefully, hopefully, be enough. I don't think it'll be going down randomly in the near future.

Knocking on wood, though.

Okay, we're back!

We're currently on a basically completely vanilla install of glitch-soc, and I'll be slowly adding customizations back in. Here's hoping things continue to actually work!

Hello everyone,

As I've been saying recently, things are broken and I'm planning on basically rebuilding the whole instance from scratch (no data will be lost, though, we're keeping the database)

There is one last thing I've realized I need to do before I can really get working on that and it will mean probably about an hour of downtime. Maybe more, maybe less.

In about 5 hours from this post, 10 PM EST, the instance will go down for a little while. That seems late enough to avoid the most active hours.

My apologies for the inconvenience. This should be the last significant downtime before at some point in the next couple days, things are actually sort of fixed around here.

Okay so things are broken and I don't really understand why, and they're even more broken on the back end. My current plan is actually to spin up a new instance and then restore a database backup from this one onto that one and then everything will be good

Please ignore things like the huge avatars on user pages, I know, I can't fix the individual things so a huge fix for everything all at once is coming

Hey everyone! It's been... actually too long since I updated this instance, so I'm planning on working on that today. I'm going to get started on that fairly soon but it means doing some merge conflict resolution and stuff that always takes longer than I expect it to.

Expect a little bit (~a few minutes) of downtime when I actually apply the new version, but, uh, prepare for more than that if something goes wrong and I have to work out how to fix it. The Mastodon codebase is, well, it's something.

As always, if anything does break, you can find me at @monorail where I'll probably be swearing into the void as I try to put the pieces back together.

Hi, sorry for the recent instability, I didn't think it was going to last long enough for it to be worth making an announcement. Just an in and out thing. But no, everything is harder than it has to be.

As a consolation prize, a feature that has been broken for ages and ages now works again! glaceon.social is now available from any subdomain!

That means that if you go to https://alt.glaceon.social, it'll be exactly the same as going to https://glaceon.social

The reason this is nice is that your cookies won't carry over. You can be logged on to multiple accounts and choose which one to use based on your subdomain! For example, I'm logged into my main on glaceon.social, but I'm typing this from admin.glaceon.social. No need to juggle incognito mode, or anything like that.

Again, sorry for the instability, but things should be doing okay for now!

hello

i rebooted g.s

i don't know if it helped but. i dunno. maybe?

This is a jortage issue apparently

Back to bed

Hi everyone!

You may have noticed that remote statuses with images aren't federating here. Don't panic!

...I mean, I don't know why it's happening either, but I'm panicking enough for the both of us. So don't worry about it.

I'll make another post with more info when I have it!

Here's the commit for the change btw

Figuring out what number of backslashes to put in the replacement string for that first gsub was a nightmare of trial and error and I still don't know why this is the correct answer.