Status
● The ToO is online! We’re monitoring the server for further issues.
After a weekend of investigation, troubleshooting, headaches, mistakes, technical documentation, late nights, weird sleep, coffee, brain fog, other activities, annoying cats, and so on and so forth... a long-term solution to the Town of Odyssey Message Boards’ increased malicious traffic woes has been implemented. Along with other maintenance, this strengthens our site’s security and performance, as well as provides more flexible tools to manage the server infrastructure.
Updates
Update #18 [May 13 @ 02:19 PDT]
At long last... It. Is. Ready. Thank you to everyone for your patience with this process. It’s been quite the weekend for me with the other activities/challenges going on. In the coming days I’ll be catching up with responding to communications as well as writing an incident “postmortem" post to share details about what happened, what steps were taken, and where we’ll be going from here. (Hint: It’s pretty much gonna be business as usual, except with new tools to manage the server, an overall better security posture, and proactive, automated monitoring for malicious bots/traffic that will help prevent future slowdowns and outages.) —Eugene
---
Update #17 [May 12 @ 18:27 PDT]
We’re very, very close to going back online! There’s just an issue for how images are loaded between phpBB and Cloudflare. Once that’s figured out, I think we’re all ready! There’s still a bunch of junk traffic targeting the website, but most of it is being blocked, and we’re better equipped to handle it now. (Almost certainly without pesky downtime.) ;P I’ll be monitoring and tweaking the rules to get better results as time goes on. —Eugene
---
Update #16 [May 11 @ 23:15 PDT]
Apologies for the extended time without updates. I took a nap earlier and ended up with a bad headache- so I was sleeping on and off while doing self-care. There is an update though! The host has answered all of my outstanding questions and it turns out Cloudflare is still available to us. Yay! That's a big help. So the final pieces of the security plan have fallen into place! I have begun the process of re-adding The ToO to Cloudflare as the primary "shield" against the newer, nastier bots. Now Shadowpaw will need to get involved to connect some of the final pieces, but otherwise things are looking good. Unfortunately, it appears that something happened to the Adventures in Odyssey Wiki recently as well. I just happened to discover that a few minutes ago. -_- I believe that's a completely different issue though. At this point, the priority is The ToO. It won't be long now, friends! Thanks for staying tuned! —Eugene
Update #15 [May 11 @ 11:58 PDT]
Okie dokie. Progress is being made! I won’t promise a timeline, but we’re not looking at days. There are just so many things to consider for addressing the evolved bot traffic. Especially when there isn’t a (figurative) pot of money from which to draw so a specialized service can simply be put in place. *sigh* More information to follow. —Eugene
Update #14 [May 11 @ 01:28 PDT]
I used to enjoy a podcast called It's Been a Long Day. I'm feeling that title very much right now. Except I'm gonna swap 'month' for 'day'. ⁑ϸ Communicating with the host has been slow going, especially with other tasks that I've needed to attend to. They're in the process of deploying a new service that addresses this exact issue, but it's not yet ready. (Booooo...) A few plug-n-play external options I explored aren't viable with our setup or cost a substantial amount of money. The integration our host had with Cloudflare is no longer in place (that is one of the factors that added to this situation) and configuring it without the integration is nearly impossible. I'm in the process of manually implementing a stop-gap solution to restore access for everyone. It's time-consuming, but I'm ensuring things are done securely and in a way that keeps maintenance effort low going into the future. The server will remain offline until we can effectively filter out the high volume of malicious traffic we've been seeing. (It's still been coming even with the server downtime. The super high numbers have dropped though.) Now it's time for bed. —Eugene
---
Update #13 [May 10 @ 16:44 PDT]
I've been working to implement more modern, multi-layered solutions to deal with the malicious traffic that we receive. (It reached a couple of hundred-thousand several times today!) It's just going to take time now to implement and ensure that everything works correctly without causing more problems since we still have older stuff hanging around. But fear not, friends; I am on the job!! —Eugene
Update #12 [May 10 @ 12:18 PDT]
After some sleep, food, and such, I’m back at it. Just gonna get re-oriented with where things are at and how to proceed. —Eugene
Update #11 [May 10 @ 03:08 PDT]
So we’ve been experiencing slowdowns/outages again. -_- I thought that everything was returned to normal... but the firewall and other malicious traffic-mitigating strategies, though now fixed and re-tuned, are just no longer enough to handle what we’re facing. Or something changed in the host’s recent server update. Either way, as a result, I’ve had to explore more drastic measures! In the meantime, it’s quite late, I’m quite tired and need to sleep. I’m leaving the ToO offline until tomorrow when I can finish things up properly. I also have a ticket in with the web host to confirm my findings and see what they have to say. I apologize for the further extending downtime. Feel free to open a ticket here if you’d like to reach me. —Eugene
---
Update #10 [May 9 @ 19:33 PDT]
Everything that needed to be completed during today’s maintenance & downtime has now been completely completed! Again, thank you for your patience. If you notice issues or have any problems with your account, go ahead and open a ticket here! Also, if you use web push notifications, you’ll have to re-enable those. —Eugene
Update #9 [May 9 @ 19:20 PDT]
I believe that this maintenance period is now complete. Thank you for your patience! —Eugene
Update #8 [May 9 @ 19:13 PDT]
About to put the server and message boards back online! —Eugene
Update #7 [May 9 @ 18:10 PDT]
I forgot to say earlier that I was done all the server stuff. Well, the primary stuff. Now I’m done updating phpBB. Just doing testing and updating a few of the extensions. Should not be long now! —Eugene
Update #6 [May 9 @ 17:31 PDT]
Progress is being made! For one, I’ve swapped this list of updates to be newest to oldest instead! xD
I’ve also discovered outdated software/incompatibilities/problems with my tools and such. So that’s somewhat helpful, if annoying. —Eugene
Update #5 [May 9 @ 16:36 PDT]
Yanno when things are taking a long time and you just have to wait for those things to finish? That’s basically what’s happening here. A lot of “hurry up and wait”! Bleh... I wish that I could speed things up! —Eugene
Update #4 [May 9 @ 16:08 PDT]
Got a few more tasks to complete, but progress is being made! —Eugene
Update #3 [May 9 @ 15:36 PDT]
This will likely take longer than initially expected. I’m sorry about that. —Eugene
Update #2 [May 9 @ 15:12 PDT]
Downtime for the ToO server has begun. —Eugene
Update #1 [May 9 @ 14:34 PDT]
The ToO will remain online for a little longer. Downtime will likely happen closer to 15:00 PDT. —Eugene