On Friday night, a number of players characters were deleted while they were playing. Most game devs would not tell the story I am about to... but that's why you love us, right? This is the story of how we discovered the problem, investigated it, and put a temp fix on it as we began the work to permanently fix it.
If so, don't worry. We can easily restore your character. Just contact player support.
We're about to dive into some deep late Saturday-night story time. There is a reason other game studios and tech companies are not as transparently open as I am about to be with you. Because they know that the extremely vocal and opinionated portion of their player base will not read the whole thing, jump to conclusions, start pointing fingers, and ultimately result in a witch hunt. Also, the ones who read the whole thing will too. It is also possible that by telling this story it may encourage copy cats and more trouble. But then again, after how this story is about to end-- maybe not as much. I think very highly of you.... and would like you to see this issue though my eyes (which are still pink. Curse you pink eye!) as it happened today.
(Still hear that in Kimberly's voice... every time.)
It was late Saturday night and our servers were being pounded with another round of DDOS attacks (distributed denial of service attacks). It had been going on since Thursday afternoon. Well, technically, it goes on all the time. They say you hit it in the big leagues when you're popular enough to have groups of haters out to get you. Apparently, we have been in the big leagues for over a decade. Over the past year we have taken a three pronged approach to mitigating DDOS attacks. Because of this, players do not notice most of them at all. They are literally happening all the time. At the end of Friday, which is always our toughest day of the week, I finally fell into a coma like sleep-- and for good reason. There was a big event coming the next morning.
Saturday morning was the Savage Race. The course was a 6-miles of ninja-warrior style obstacles in the mud. (and by mud, they really mean cow poo... there was *A LOT* of cow poo). Zhoom came by to pick me up. We were meeting nearly the entire AQ3D coding team at the race, and Dage & Thyton too. We have been training for it all 2019. After a while on the road, I finally got a moment to check my phone. O_O OMG. The Red Dragon server had been down since 1:30am and no one had called Zhoom nor I. Worse yet, Oishii & Korin said players were saying their characters were deleted while they were online. A quick peek at Twitter confirmed my horror.
I think the way you react to an emergency is a strong sign of your character. It is important not to panic. To keep a clear head and back up and look at the big picture. Well, OK, honestly, I just let myself feel the full on fear for about 10 seconds while breathing out... then being done with it, I move on to the previously mentioned clear headed thinking. (Did I learn this from one of the Batman movies or Facebook?)
We are nearly at the Savage Race. The only coder not there is Rolith, so I call him. Once I let him know what wass happening, he brings Red Dragon back online and we start going through all the steps to figure out what in the world is going on. First was to take a close look at the deleted characters. We compiled a list of players who messaged us. The first sigh of relief was the characters were still there. The sigh is probably for a different reason then you expected. Because of the way the characters were deleted, we immediately could rule out a lot of the ways it was done. See, we do not instantly delete characters in AQ3D. They get flagged for deletion, and then we purge them in batches at a later date. This is good for both security and account recovery reasons. further more, the accounts had not been accessed by anyone but the actual owner. Which was good. But we still needed to figure out how they had gotten deleted by someone other than the user.
We made a huge list of possible things to check first. We were pulling into the Savage Race-- Zhoom and I has Rolith on my cellphones speaker phone. Based on what we knew so far, the quick temp fix was to disable the server-side code that deletes characters. It was not actually in the game. It was a completely separate page on the website. With this disabled, no one can delete your character-- so we could safely continue our investigation and we could make it just in time to start the race. Rolith continued going through the list we made, and Captain Rhubarb (our database coder) joined him with the help of the other staff that were online.
So there the rest of us were... still Saturday morning, running, jumping, climbing, swinging, crawling, and swimming through the 6-mile obstacle course. Did I mention the sheer volume of cow poo? One way to look at this-- if we intentionally put ourselves, mentally & physically through that much ****, then we are going to be way over prepared to handle the **** a few disgruntled players throw at us. XD Everyone did great on the run by the way. Apologies for not remember the name of the player who gave some tips about mud runs in a previous post, but thank you :D Then we hosed off... and got back online with the team to fix the deletion problem in progress.
We checked way to many things to put in this post. But the obvious blaring discovery was a 1,000,000+ hits to the character delete page. The previous day, it was only 40. By doing some cross-referencing we were able to identify at least one individual (with certainty) who was involved in this. You could say his finger prints were all over the place. (This was not the first time.)
At this point we had combed through the code logic, combed through the network for old pages that might somehow be problematic, and... well, went overboard. Even though we were pretty certain we had zeroed in on the actual problem, I think it is better to assume it is worse and broaden the search. So security testers and white hats (ethical hackers who often help us investigate problems) like Root gave their careful advice and reached out to the community to see if anyone had anything to add. We used to have official white hats, but for a very long time now we have been using a simple bug bounty system. It works for the bigger companies, and I think it is a better system. If you have tips, you can always submit the to the Artix bugs tracker. (https://bugs.artix.com)
There is one sacred rule to trouble shooting a problem like this. NEVER assume you fixed it. Also, you probably should never tell exactly the problem was and how you fixed it. This is probably why big companies write such ambiguous posts when handling a major issue. But, if you are still reading-- I think you deserve the gnitty gritty. So here we go. Again, we are working on several other possible fixes. But the most likely is that through a great amount of testing, the attacker discovered that our character delete page did not have the same brute force protection as the other pages do. This page only has one function, to delete characters. By knowing your character Id# (which they could get by being in a room with you in game) and brute forcing that page-- they were able to get lucky and delete your character. There were a total of about 200 characters deleted. But a large number of them was the attacker testing deleting characters they had made. No characters have been deleted since we disabled the feature. We are currently adding several layers of added protection to that page (and others including an overhaul of our token system) before bringing it back online.
Restoring the deleted characters is really easy for player support. Nythera was online tonight and helped everyone who wrote in so far. So if you messaged player support today-- your character should be back. It is technically possible to have multiple characters in AQ3D-- it was designed that way even though the game only allows one. So if you wrote in and still only see a new character you made... that might be what is happening. Just write back so Nythera can delete the extras.
So what about the attacker? I mean, we said we know at least one of them right? Yeah. So-- you know we need to take action at this point. On Monday, we are having a meeting with our attorney to review the options before taking action. It will probably lead to some more short term problems. But the soft way we have handled this in the past has clearly not been effective. Do not get me wrong, we support people who find bugs and report them so we can fix them. However, there is a line-- maliciously attacking servers to delete the data of paying customers, distributing code for others to do the same, and DDOS'ing servers to prevent people from accessing them are all illegal. Plain and simple. I hope you stand behind me in this. Also hope you know this is something I dread. My passion is building video games-- this is a massive distraction from my life's purpose.
Several things I read say to be grateful for the challenging moments in your life. That you need these moments to grow and define your character. Zhoom said, "That is... a very optimistic way of looking at it."
So, here I am past the end of Saturday at 1am. Pretty close to a 13 hour workday with a 6 mile obstacle race at the start of it. XD Probably a good time to wrap up this letter to you. I will post the Screenshot Saturday #X (we are on 10 now) for you after this as a palette cleanser. Hope you appreciated and enjoyed the complete transparency in this post. My apologies to those who had their character temporarily deleted. Thank you for your understanding as we solve it, and looking forward to getting you fixed and back in game ASAP if you have not already. We have some great releases coming your way in AdventureQuest 3D-- and we will keep making the game bigger, better, more secure, more fun, and also a little dark and funny. Battle on!
Greetings all. I was asked to wait until today to post this follow up of good news. After this post went live, we were sent a flurry of useful information including the source code which shows how the exploit was done. Remember above where I said "Rule #1: Never assume you fixed it"? While we had correctly zeroed in on the core problem and the changes we were making would have solved the exploit-- we were wrong about what the actual problem was. What? Might be easier to tell you the details. Which I can safely do now-- because the patch has officially been applied everywhere. This issue also turned out to be the exact same issue that allowed attacker(s) to send messages from AQWorld mod Soki's account a week ago. This is going to be a tad technical. So if you are fine with "Oh, you know exactly how it was done and have 100% fixed it" then great.... you are good to stop reading. Or maybe you should read anyway-- because the sheer amount of misinformation that gets spread when issues like this happen are unbelievable. When you log into our online games, the game generates a special token for you. It is, essentially, an extra and stronger password. It only exists when you re playing. Both of the previous statements were thought to be true-- but the attacker(s) discovered a weird flaw in them. Tokens generated at the exact same "server tick" were the same. We believe they discovered this by accident when flooding the server with bots. The process that seeded our randomizer made it the same for any new tokens created at that moment. The first thing you might be thinking is, "How could you not have caught that?" to which any hacker would say... how would you have any idea that it wasn't working? It sure seemed like it was. Each login token looked different. That was, until someone created thousands at the same moment and noticed otherwise. It was an edge case scenario to say the least. Secondly you might be thinking-- well, that's great but what did the attackers do? Log in every tick of the server? .... O_O ...yes. It was a pretty sophisticated and pre-mediated attack. They collected 1,000-ish tokens which they used for the attack. (Because our token would have taken 2 centuries to crack using regular brute forcing). Fixing this issue only took one line of code (But we did a lot more which has been done across all of our games as of this post). But before we found it, this allowed the attacker(s) to brute force their list of captured tokens against the character delete page along with your character ID. There were certainly other intended uses of this. All which have now been blocked with the fix. We added some additional security to the token on top of it. If a password length and complexity that would take 200 years to brute force is not enough for you, we quadrupled it. Because... why not? We are working on an additional system that will work like a reverse password that will protect you further.
We are really glad that this problem has been fixed. Hope you appreciate our extreme transparency with this. While we still have some follow up actions to take. On the technical side, this fix solved one of the most pressing issues we have had in our gaming network. Which is good. Now we can get back to making more fun content and features. Battle on!
Official Twitter of AQ3D
Pun-slinging Paladin
Undead Legion Ruler
Scribble Scribe
Dawnforger
