Please show @Kjun some love for making this, and make your own pet Mastodon

https://toot.garden/@Kjun/110768779465567603

I love this so much, I had to post it on the admin account -AJ

#mastodon #coloring #pet #art

Hello and good morning or night!

After having issues last week with media/files bogging down the server (x12 increase in requests!) I've moved media files to cdn.toot.garden, which runs on Backblaze B2, recommended by a number of fellow admins and cool folks.

This also addressed another issue, the site media files loading slowly regardless of load because of its location in Germany. Now media files are replicated across the globe, and should load extremely quickly no matter where you are. It also should help speed things up if you are on a slower connection.

If anything seems to be missing, let me know, there may have been a few files that got missed in the upload process, but from what I can tell all is well.

Hope you all are well, and thanks for all your kind words over the last two weeks while I've handled some random issues as they arrose, love yall! - AJ

First and foremost I'd like to apologize for the random blips (1-2min) of Web UI downtime in the past 24h.

A lot of server maintenance, and updates, but the biggest issue causing some blips on the front-end has been moving to using Cloudflares caching system. The site was beginning to struggle a bit with the increased media requests, which meant I needed to switch some things up pretty quickly!

The good news is, the site should load miles faster now, and it is no longer getting slammed with media queries in random spikes.

The bad news is, a few folks had issues with their Browser cache not expiring as it's supposed to.

If you are for some reason still unable to view the site via the Web UI, clearing your cache for toot.garden will/should fix the issue. I would write out the whole process but everyone's device is different!

Any apps, including the Official Mastodon app have been working as expected though.

Here's to another many months of smooth sailing from here on out! -AJ

Updated to the latest changes on the Mastodon main branch in prep for the security update, and I updated the custom themes with the UI changes as well.

Now that I've said that, I just wanted to give another shoutout to @monaant for being an awesome moderator here, and a cool person overall. 😎

That's about it for now! See you all Thursday 🫡

Toot.garden is now on v4.1.2, with Garden Tweaks!

Garden Tweaks below:

- Added 2 new themes, with some more to come later (Cafe, and Vibrant)

- Increased the API limits for 3rd party apps, no more ratelimit issues!

- Properly handles FHD and 4k videos with less compression

- Updated Grafana, later this week there will be a public dashboard to view live server stats

- Fixed theming gaps that came with 4.1

- Tweaked Sidekiq and Postgresql tuning, currently seeing a 20% performance increase :)

- Email sending now routes to outlook addresses without going to spam (Thanks Microsoft support!)

For a full list of changes, please see the commits and info from the mastodon/main branch on github.

https://github.com/Toot-Garden/mastodon

2/2

- For those who don't check the status page every now and then (most people don't, hence me posting it here! :]) there will likely be about an hour of downtime on the 13th while I upgrade the server to Mastodon v4.1.2, I'll also be adding some new themes and tweaks to the server that will be documented that day in another post!

Hope you all are doing well, and have a great weekend!

- AJ

Hey all!

Two things:

- I have silenced mastodon.social for the time being anyway. Lately after they switched to pushing people onto their server on the official app there has been a large spike in their active people count. Not all of these accounts are kind, follow the rules, etc. Unfortunately their moderation cannot keep up, and has not kept up for a long time. They generally are able to resolve reports within a few hours of getting them, but when there are tons of accounts that need to be moderated, it begins to lag behind. I, and the moderators here are not paid by Mastodon.social, and cannot do their job for them. As such, they have been "limited". You can still follow people from mastodon.social and see their posts, but they won't show up on the timelines.

1/2

Hey all, I will be upgrading the server to v4.1.2, to fix a few things that v4.1.0 had wrong with it. Additionally, to align with most other instances with a higher character count, I will be raising ours from 850 to the "standard" 1000. Even I have noticed it is always slightly too small for a larger toot.

I aim to roll this out over the weekend, or tonight at some point. Finding time is hard lol. I'll post an announcement 30min before I start 👌

- AJ

As a reminder, today is #transdayofvisibility - I'd like to share this article from the Trevor Project, it's a good reminder and article to share around.

https://www.thetrevorproject.org/resources/guide/a-guide-to-being-an-ally-to-transgender-and-nonbinary-youth/

Additionally, I have seen some posts and content that is very much harmful, and I have been monitoring the federated feed for toot.garden today and suspending accounts to prevent our instance from relaying the posts to others.

If you are a member of toot.garden, please utilize the report feature! It sends me an immediate email, and text message.

And, if you are an admin or moderator for another instance, please keep an eye out for content that is, or could be harmful.

#mastodon #lgbtq #mastoadmin

5/5
Lastly, I did an immediate check on other system log files and was able to confirm there was no privilege escalation, or other suspicious activity on the machine, or containers.

I value transparency highly, and while some may consider this a trivial thing, that did not need to be posted about since it had no confidential or private data, I am.

Below I have added screenshots of the notice I obtained from hetzner. I have emails from their domain set to pass through any DND filters, specifically for reasons like this. I appreciate Hetzner's quick action to alert me to the situation.

If there are any other instance admins seeing this, I recommend adding the same additional protections I have, I also welcome everyone's feedback on me, my response to this, and the incident as a whole.

- AJ

4/5
Measures, cont.

- Any, and all, networking configuration changes will now be run by a friend of mine, this friend is also the same one who holds powers to the server in the event I am unable to fix an immediate issue myself, or am ill, etc.

- Specific changes to the docker daemon have been made to explicitly disallow iptables changes, should an update to the daemon attempt to override this configuration, rules that explicitly block the docker network interfaces the mastodon services use will prevent future prepends, or appends from doing harm.

- Shodan monitoring *will* be added to both the IPv4, and IPv6 IPs the instance uses.

3/5
Now, you are likely curious what the mastodon elasticsearch contained. Thankfully, and by design, it contains nothing confidential/nothing that is not publicly available via the web, or apis that clients use. It is purely used for the full-text search of hashtags, and public toots. It also contains some quick metrics on hashtag popularity. No IPs, location data, logins, passwords, media data, etc- are included within this. I am not downplaying the severity of this, it is not something I am happy with, and as such some additional measures have now been implemented, or are being implemented at the time of writing.

The measures that have been taken:

- I have configured an automatic port scanner to detect any unintentionally open ports.

- Critical services have been blocked by hetzner's physical firewall.

2/5
The ports, were indeed blocked, but- at a system level the docker daemon is able to modify iptables rules, and in this case, said rules were being prepended to any rules set by ufw. This means that regardless of what was assumed to be blocked and what was set to be blocked, the port was actually being exposed/allowed because the the firewall goes off a "first match" system- the first rule to match, is the one it uses. The bad news, is that this was quickly exploited by a bot. Upon investigation, an index containing a readme with a ransom message had been created. This message contained a bitcoin wallet address, and threatened to dump the database. Slightly positive thing is, that wouldn't have mattered, the search is re-indexed from scratch every so often, daily- because of a mastodon bug.

1/5
Hello,

I was notified at approx. 3/8/2023 10:43 PM (EST), by Hetzner, that the ElasticSearch container used for the instance had inadvertently been exposed to the public web. This surprised me, as the firewall rules have not changed as of late, and it was previously locked down tightly, with multiple tests and checks done prior.

The good news, the system/os itself was not breached. The issue lies with an update to the docker-compose.yml config file used for the instance. While updating to match new v7 recommendations I unintentionally added specific ports to be exposed. Usually what I did would not have been an issue, except that for certain containers I needed the ports to be exposed (aka, monitoring services that are intentionally exposed) meaning that docker was not as a whole blocked. Why does this matter?

Update completed!

This change includes several things from Mastodon's main branch, the biggest being markdown support - so it plays friendly with other federated services/applications.

Additionally, I did change some wording on account suspensions and limits to better describe what an account being limited means and does, and why, etc.

Additionally, I fixed an annoying issue with FLAC/Ogg files displaying "Infinity:Infinity" on the frontend if you tried to play audio.

Lastly, https://blog.toot.garden exists now! I'll be posting some tutorials, guides, etc. for both mastodon users, and admins, along with other fediverse "things"

If anything seems broken, ping @linkeddev or email alienknight@toot.garden!

#mastodon #mastoadmin #update #markdown

Aiming to update the instance with some changes from the main mastodon github, and some additional information note tweaks I've added, later today at some point. My last update script caused 0 downtime, so in theory there shouldn't be any this time, but I'll aim to run the script around 2pm EST, or about 5h from now!

Additionally, there will be a new blog created specifically for transparency, and general musings related to administration and moderation, for both users and other admins to checkout. I'll be posting some tutorials and maybe videos on there as well, if I can figure out automatic CC.

- AJ

#mastodon #update #iloveyall

Hello all, I've updated the instance to v4.1.0, along with some various tweaks.

- Fixed some theming readability issues

- Timelines now explicitly state if a post is public, private, etc

- The toot character limit is now 850, as opposed to the default 500

- Backend API Ratelimits were increased - please see https://github.com/TheAlienKnight/toot.garden/pull/16/commits/4728313953822a189e2b8939aecf3e43ad093577

- All Mastodon main changes - please see https://github.com/mastodon/mastodon/releases/tag/v4.1.0

#mastodon #update #mastoadmin #tootlimit #api #instance #github

👋 In the past 24h, it seems that Sidekiq decided to only "sometimes" add things to the ingress queue, while things at first looked fine server side, and showed as green, etc. I elected to restart the container just in-case something was amiss... there were ~21k jobs that previously didn't show up before a restart. Things should be catching up now. Thank you to @westmeadow and @Labrodorite for reporting the issue!

An update to this, all should be smooth sailing from here on!

For any #mastodon #mastoadmin #sysadmin or related, I am *very* much open to hosting provider suggestions, toss me an affiliate link if you'd like!

- AJ