Last week, Warframe’s hotly anticipated Old Peace update launched, kicking off a saga that digs into the very foundations of the nearly 13-year-old game’s lore. Of course, like clockwork, servers immediately took a tumble, resulting in crashes, outages, and chat issues. But why has this pattern become so predictable with online games? Especially when developers are well aware that a storm of their own making is on the horizon?
During a Game Awards-adjacent event celebrating The Old Peace’s launch last week, I asked creative director Rebb Ford.
"You've gotta spin up capacity,” she told Aftermath, referring to the practice of paying money to a distribution partner for additional servers ahead of or during moments when many players will be trying to access content. “You're allowing so many connections. We're an always-online game, right, so every time a player does something, there's a server call. There's something that needs to be verified server and client side. … Login, mission complete, anything that needs to talk to us to say 'You did this, you did that’ – it happens to us at a volume level that's very hard to account for."
In The Old Peace’s case, Digital Extremes was ready for a stampede the moment it opened the gates, but not quite ready enough.
"We actually didn't fall over as much as I thought we would,” said Ford. “That's when we realized 'Oh, we didn't think this was gonna be bigger than TennoCon [Warframe’s annual convention that often drives record player numbers].' We spun up IRC servers, we spun up things just to deal with volume. But sometimes you just cannot be prepared enough when you didn't predict it to be the third-best day in the history of the game. That was an error on our part, but it's not so much a tech error; it was an anticipation error. We fixed it very quickly."
The ability to quickly rectify server issues is also the result of preparation – in some cases years of preparation.
"One of our most important things to do is make sure people can get the content as fast as possible,” said Ford. “With Warframe, when we have the build or the update, we release it to our distributing partners, and we do something called a pre-heat of our CDN, or content delivery network – which is basically us saying 'People shouldn't all be fetching the game data from one node.' Because that will take forever. It'll get congested. So we distribute it, and this is through years of network partner shopping, working with really good network partners. We have content servers in 16 or 17 central population hubs."
The pre-heat, Ford explained, ensures that the whole network doesn’t hinge on a single point of potential failure.
“So sometimes you'll be going in the Philippines instead of being routed to our deploying headquarters, which is Ontario,” she said. “We pre-heated our server structure across the globe so that people can fetch [new content] quicker, and that takes a lot of load off.”
But that’s only one stair in what Ford characterized as a winding staircase of individual, overlapping needs.
"So that's the first point of failure: Can you download the game at all?” she said. “Second point of failure is: Can you login at all? When that happens, that all comes to us through login capacity. That one, you just have to spin up more capacity. Then you have the question of 'Can people play missions?' So you can kind of see the staircase: Can you download the game? Can you login to the game? Can you play the game? And each one of those is a slightly different sector of stability."
In Warframe’s case, elements can function independently, but if they’re not all working in conjunction, players quickly begin to see the seams.
“A lot of people can be logged into the game, and that's cool, but if you can't play, [then there's a problem],” said Ford. “It's like Maslow's Hierarchy of Needs. Can you chat? You don't need chat to play the game, but chat servers run independently. Those were hit the hardest [on Old Peace launch day], and we fixed it fairly fast by spinning up more capacity."
Games are complicated, as is the process of distributing them to millions of different computers with as many hardware configurations as there are stars in the sky. You will not be surprised to learn, then, that many other things can also go wrong.
“We have issues where we release new code in this build, and then maybe one piece of code fires every second on a heartbeat,” Ford said. “And sometimes we find these heartbeats, and we're like 'What is pinging the servers every second on the second,' and we're like 'It's the new title system we put in,' for example. ‘It's checking against server-client to issue you a title, but it's doing it in a way where we were unsure because it's checking all this indexed stuff.’"
Warframe has been around for over a decade and regularly pulls in tens – or in Old Peace launch day’s case, hundreds – of thousands of concurrent players. Nonetheless, said Ford, Digital Extremes still frets about The Ramifications as though it were a much smaller company.
"We still feel very young and scrappy, and we're like 'Can we even afford $600 more per month in capacity?'” she said. “That's the kind of question we ask ourselves on launch day. And then we're like 'Just do it! Just do it!'"
Warframe’s servers weren’t quite able to withstand the sheer weight of years’ worth of anticipation on launch day, but Ford was relieved that they didn’t go down for “hours and hours,” which would’ve necessitated a suitably less jubilant speech at the launch event in LA.
"It's exciting. It's thrilling. Everyone did an amazing job,” she said. “We asked our team to do the impossible with this update, so even though those little hiccups happened, we had two speeches prepared – funeral or the celebration – and we undoubtedly got to do the celebration."
Recommended
Luke Plunkett
