The Dark AgesUntil not long ago, NetherGames was using the same infrastructure design we had back in 2020. While that wouldn't sound bad, a lot has happened since then. NetherGames has rapidly grown, and we had to get used to higher demand.
In the old design of the infrastructure, we had several physical nodes. We call them nodes in this blog post. You can imagine a node as a physical server in a data centre. And we had a few of these. (See on the right)
Now, what's the issue, you might ask. Well, we managed all of these servers separately. If we wanted to schedule more Bedwars Squads servers, we would have to manually start them after calculating which node would be the best to start them on.
Imagine the pain of manually restarting servers on all of these nodes separately when pushing an update. You had to essentially open every node's control panel and restart the servers there. That usually took us a couple of minutes.
Changing settings on, say, all Bedwars servers was a huge deal back then. It probably took us 20-30 minutes to update all the individual servers' settings.
All this mess with tons of servers led to us creating beautiful big charts with server locations to know where these servers are.
The Big ChangeNow, that was the dark age. Rolling out updates was slow, updating nodes tedious, and scaling game modes up was a pain.
So what did we change? We moved our entire infrastructure into a Kubernetes cluster in a month-long preparation project.
Now, What Is Kubernetes?That's a question that one of our executives, Callum, has asked me recently. And explaining it in its full feature set to a non-techie would be quite a challenge. So I responded with this:
"Kubernetes is essentially a way to connect all of our nodes into one cluster." While Callum was satisfied with this explanation, it's much deeper than that, and we will explore that depth today!
Kubernetes connects our servers in a single network. As such, nothing from the outside can reach them unless otherwise specified.
You can imagine this like a castle. Our game servers are rooms inside the castle, hidden away from public view. And then you have the entrance, our game proxies. Through our game proxies, you enter the court and can move between rooms under the guidance of the proxy.
Now, that's very useful. It solves a few security challenges. All our game servers and other services can seamlessly communicate with each other.
ScalingAnd yes, there's another issue solved. Kubernetes essentially lets us provide templates for our servers. We say any Bedwars-Solo server should look like this, enabling it to create any amount of replicas of these servers with ease among the entire cluster. We can specify everything in a simple configuration file (per server type).
As such, the scaling part goes from being a considerable undertaking down to this little slider:
SchedulingAnd that's something else that Kubernetes does for us. Say, we want 12 US BW-Squads servers. We give that instruction, and Kubernetes does the rest for us. It calculates which nodes in our cluster fit the resource requirements of the service we want to schedule, it automatically spreads them accordingly, so no node handles all the load at once, and it automatically respects region boundaries, so no US servers are spawned on the AP region.
I Tripped Over a Power CordAnd yes, that has happened before. So what if a node completely goes offline for one of a vast number of possible reasons? Kubernetes will automatically shift all the services on this node to the other available nodes, so they won't be completely offline while services restart. Kubernetes will move them to a node that is indeed online.
The DownsideYes, as with everything, there's a downside. Kubernetes is better suited for stateless services. Those services do not persist any data, for example, to disk. However, looking at Skyblock or Factions, we face the issue of having to persist data. While it is difficult to get to work, there are many solutions to resolve this issue by providing network-synchronized, highly available storage to services that need it.
At the same time, Kubernetes works well on the cloud, so on Google Cloud Platform, Amazon Web Services, IBM Cloud or similar. However, NetherGames brings its servers, so the configuration and integration are more challenging.