:bc: Attention Beige Party-goers! :bc:
State of The Instance - February 2023
-- Server Issues --
As you have probably noticed we've had some growing pains over the past couple of days. It's been about 24 hours since something has broken so I am moderately confident in saying that the issue has been fixed and the Helsinki Toot Cloud is light and fluffy and benign once again.
A couple of lessons learned from this experience:
1. The rate limits set in rack_attack.rb are your friend. They are what keep your Sidekiq queues from getting overwhelmed. Don't change them unless you know exactly what you're doing or you're interested in going for a wild ride.
2. An article I read about scaling up your Mastodon instance suggested running multiple SideKiq processes against a single queue to process it faster. DO NOT DO THIS. At least don't do it with the push and pull queues. Eventually two processes will try to access the same file at the same time and then all your images will be broken until you restart everything. Instead, you can run different processes for different queues and increase the thread count to handle increased demand as needed.
This was painful to figure out in real time with everything breaking all around me, but the experience has at least served to make me marginally less incompetent, so hopefully this specific issue will not happen again in the future.
-- Instance Improvements --
As a result of our issues with Sidekiq queues getting overloaded, I've upgraded the memory on our server so we can have more processes working through the job queues. This also means the web server has more memory to process http requests and I can allocate more memory to elastic search, so in general things should be running faster.
I have also added a hot standby Postgres database server in a different physical location from our main Helsinki Toot Cloud server. This server constantly replicates the production database where all your toots live so that in the event of a database failure, we can quickly switch over to the standby with no loss of data. We all love Helsinki around here, but if (god forbid) something catastrophic were to happen to the Toot Cloud, rest assured that our toots will go on.
-- Wafflec.one Cross-Poster --
@probgoblin over at wafflec.one has graciously made his Twitter cross-poster available to Beige Party-goers. Unfortunately it looks like Elon is going to be turning off the API soon, so feel free to use it while you still can!
https://xpost.wafflec.one/
And a big thanks to Johnny Normality for providing help and advice when I was struggling to get this instance off the ground.
-- Welcome New Party-Goers! --
In the month of January, we gained 9 new party-goers. In addition to the 3 new party-goers that joined in the first few days of February, we are now 93 strong. In terms of active users (using the completely arbitrary metric of people who have logged in within the last week) we are around 56.
Welcome to all the new party-goers! If you have any questions or concerns, please feel free to reach out to me at any time.
-- Finances --
In the month of January we received $150.00 in donations. After PayPal fees, that comes out to $143.11. We had $24.98 left over from December, so our available funds for paying the January hosting bills were $168.09.
Our total hosting fees for this month break down as follows:
DMCA Agent Registration: $6.00
VPS Hosting: $19.98
S3 Storage: $1.97
Total Hosting Cost: $27.95
The DMCA registration fee is once per year, but I'm still expecting S3 storage to cost around $5.00 a month (I still had some credits on my account this month), and the recent server upgrade is going to run around 3 extra euros a month, so I think hosting fees are going to remain around $30.00 a month once everything settles.
After paying the hosting bill for January, we have $140.14 left in donation funds, plus an additional $26.98 (after PayPal fees) donated in the first few days of February, for a total of $167.12. That means we have enough money in the bank right now to keep this instance running for 5 months, which is great news!
Thank you to all who have donated, I really appreciate it! At the moment, we are in pretty good shape, but if anyone would like to make a donation, you can do so here:
https://ko-fi.com/drinky
Once again, thank you to all the party-goers for making this instance what it is. I've done a lot of navel-gazing this past month, and I think our secret sauce is that we have a great community that is invested in making this a nice place for absurdities of all stripes. Thank you all for being who you are, and I appreciate all of you.
And as always, Beige-bless :bb: