Replay System & the adventure we had...

Nov 14, 2019
3
13
3
Lahore, Pakistan
In general, to avoid spamming you and so we always have something to talk about, we decided to limit to one developer blog per week. This is a special occasion, count yourself lucky.

Vision:
It was @TobiasDev who originally had the idea to make a replay system. He envisioned a system where every action the player does, will be noted down into memory (a computer's notepad).

The structure he had in mind would be simple: Player does something -> an entry generated with necessary data to recreate action -> entry saved.

How it went:
When the project got approved for development, I was assigned to it and knew from the first glance that whatever I was doing was something not done in our team, or the community before - at least not with our high requirements in place. What we needed was a system that could record every player action, accurately, without affecting your gameplay. On top of that, we had to think about storage - since we had loads of games running and their replays would be off significant size. For reference, your average 1080p video at 60 FPS which is a minute long amounts to an average of 90MB. That's 5400MB, or 5.4GB. We have an average of 32,000 games occurring every day. If we used that method, we'd be at 172.8 TB in just a day. Storage is cheap, but even that's expensive.

So, we had to record without performance issues and store them efficiently. This was a big task.

The initial plan became to use the notepad system @TobiasDev had in mind and store relevant game data. Instead of recording the player, we'd just note down specific data to replay player actions. What would be 75kb (0.05s of replay) using the 1080p video method, could now be stored in data that'd be less than 0.05kb (not exact data metric, but just an example). That was a big achievement.

On the other hand, recording these actions is just the same as handling them which we already do in almost all minigames you play. This wasn't a performance drawback of any sort.

So that was storage sorted out in memory, the next step was long-term storage. We agreed to use Named Binary Tag (NBT), a format made by Mojang themselves. It came with its stream through which we could convert our entries into NBT and then compress it all into a single string. It was a good plan and meant we had less work on our hands.

Next came the compression. Compression is a compute-intensive task and requires the CPU to kick in the high gears. zLib was our go-to library. However, even with that, it would take a couple of seconds to compress the entire replay so it could be saved in our database. We couldn't do this on the main thread or it'll lag the servers (making gameplay for you, a total mess) and our upcoming blog post about threading will explain why. We had to move this data over to another thread (a good imagination of threads is desks: each desk can only do work step by step, but more desks can do work in parallel) so it could be compressed.

With this plan in motion, development began and we made substantial progress.

During this, we encountered an interesting problem we had to face with movement. We couldn't just record positions every tick. In the difference of 0.05 seconds, the player can move across a corner and the two positions (the past one, and the present one) would have a wall in between making the movement extremely inaccurate. Hence, we had to remake the movement accurately by saving even more parameters such as speed.

In a month, we had been able to make a system that would record most actions accurately, and with a few bug fixes: we'd have a system ready. Then it hit us, the big serialization issue (Serialization is a complex topic, better explained in the threading post, but a quick take on it: to move objective data such as entries over to another thread, they needed to be serialized). Serialization is a performance expensive task and hence would, in the end, need to be done on the main thread. We didn't realize this issue while testing this on the test servers since we only had one game running. This was instead found out when we did a quick production test and realized that after every game ended, the entire server would lag for one or two seconds. This was bad.

..and to add to the fire, I was extremely busy with other work regarding my education that I no longer had time to work on this project. Being the sole developer of this project, and now busy, led to the project being put on hold for the next couple of months.

Coming back, months later, I decided to take this project back on and get it ready. We had to rethink our storage method since if we wanted to do multi-threading without making it too complex, we had to look for an object -> thread-safe data conversion, and it had to be load-balanced across. We developed an in-house system for this and sorted out this problem by having a dedicated "threaded memory". This memory would be shared across the main thread and our compression thread. It was also the time when we switched from zLib to our new compression system using Facebook developed Zstandard.

The idea of creating a dedicated thread just for compression was a bad idea and we couldn't have been any less dumbfounded. So much could go wrong...

..and it did. Multiple segmentation faults and memory leaks.

Multiple hours of trying to debug without the proper tools (lack of tools for threading debugging will be explained in the threading post), we gave up. It was a pointless endeavor.

Then while discussing our new threading system made for Zstandard initially but now used by everything, a spark occurred. Rather than having a dedicated thread, what if we move the compression task to a bunch of workers (the magic behind this will be explained in the threading post)? This would mean we can use our new threading system for replays as well.

After a little bit of reconfiguring, we did just that. It worked. It was a magical moment for the team. Until it didn't...
MemoryGarbage.png

For some reason, it worked fine-ish on some servers but had terrible memory management on others. Luckily, this time around with our new threading system, we had debugging tools. We got on to the task, and found out we were at first not marking memory as unallocated, and then not clearing this unallocated memory. We were missing what is referred to as Garbage Collection. This was half a mistake in our replay framework and a half in our new threading system.

With all this sorted out, we redeployed the framework and to our expectations: it worked perfectly!

Our final step-by-step recording process looks like this:
Replay System.png

Our memory was managed perfectly, no performance issues were experienced by the players and our profilers picked up nothing unusual. It was a win, and the destination of our adventure was reached.
MemoryManaged.png

In almost 2 days, we have amounted to a mind-boggling 50,000 replays making up 5.3GB of data.
ReplayDBSize.png

With this said, the replay system is still in development and very much in the beta phase but we're very proud to say the current version has been stable enough to run production.

We also have decided to help the development community learn & improve by open-sourcing our work: Replay System Framework Source in hopes that everyone learns.
It's not a plug & play as it relies a lot on our closed-source libraries but we hope you can learn from the source code to make your crazy inventions.

To bring an end to this, I'd like to ask everyone to remain quarantined whenever possible and join us as a community on NetherGames. NetherGames <3 Opensource!
 

Ps Danew

Member
Mar 31, 2018
43
10
8
Canada
In general, to avoid spamming you and so we always have something to talk about, we decided to limit to one developer blog per week. This is a special occasion, count yourself lucky.

Vision:
It was @TobiasDev who originally had the idea to make a replay system. He envisioned a system where every action the player does, will be noted down into memory (a computer's notepad).

The structure he had in mind would be simple: Player does something -> an entry generated with necessary data to recreate action -> entry saved.

How it went:
When the project got approved for development, I was assigned to it and knew from the first glance that whatever I was doing was something not done in our team, or the community before - at least not with our high requirements in place. What we needed was a system that could record every player action, accurately, without affecting your gameplay. On top of that, we had to think about storage - since we had loads of games running and their replays would be off significant size. For reference, your average 1080p video at 60 FPS which is a minute long amounts to an average of 90MB. That's 5400MB, or 5.4GB. We have an average of 32,000 games occurring every day. If we used that method, we'd be at 172.8 TB in just a day. Storage is cheap, but even that's expensive.

So, we had to record without performance issues and store them efficiently. This was a big task.

The initial plan became to use the notepad system @TobiasDev had in mind and store relevant game data. Instead of recording the player, we'd just note down specific data to replay player actions. What would be 75kb (0.05s of replay) using the 1080p video method, could now be stored in data that'd be less than 0.05kb (not exact data metric, but just an example). That was a big achievement.

On the other hand, recording these actions is just the same as handling them which we already do in almost all minigames you play. This wasn't a performance drawback of any sort.

So that was storage sorted out in memory, the next step was long-term storage. We agreed to use Named Binary Tag (NBT), a format made by Mojang themselves. It came with its stream through which we could convert our entries into NBT and then compress it all into a single string. It was a good plan and meant we had less work on our hands.

Next came the compression. Compression is a compute-intensive task and requires the CPU to kick in the high gears. zLib was our go-to library. However, even with that, it would take a couple of seconds to compress the entire replay so it could be saved in our database. We couldn't do this on the main thread or it'll lag the servers (making gameplay for you, a total mess) and our upcoming blog post about threading will explain why. We had to move this data over to another thread (a good imagination of threads is desks: each desk can only do work step by step, but more desks can do work in parallel) so it could be compressed.

With this plan in motion, development began and we made substantial progress.

During this, we encountered an interesting problem we had to face with movement. We couldn't just record positions every tick. In the difference of 0.05 seconds, the player can move across a corner and the two positions (the past one, and the present one) would have a wall in between making the movement extremely inaccurate. Hence, we had to remake the movement accurately by saving even more parameters such as speed.

In a month, we had been able to make a system that would record most actions accurately, and with a few bug fixes: we'd have a system ready. Then it hit us, the big serialization issue (Serialization is a complex topic, better explained in the threading post, but a quick take on it: to move objective data such as entries over to another thread, they needed to be serialized). Serialization is a performance expensive task and hence would, in the end, need to be done on the main thread. We didn't realize this issue while testing this on the test servers since we only had one game running. This was instead found out when we did a quick production test and realized that after every game ended, the entire server would lag for one or two seconds. This was bad.

..and to add to the fire, I was extremely busy with other work regarding my education that I no longer had time to work on this project. Being the sole developer of this project, and now busy, led to the project being put on hold for the next couple of months.

Coming back, months later, I decided to take this project back on and get it ready. We had to rethink our storage method since if we wanted to do multi-threading without making it too complex, we had to look for an object -> thread-safe data conversion, and it had to be load-balanced across. We developed an in-house system for this and sorted out this problem by having a dedicated "threaded memory". This memory would be shared across the main thread and our compression thread. It was also the time when we switched from zLib to our new compression system using Facebook developed Zstandard.

The idea of creating a dedicated thread just for compression was a bad idea and we couldn't have been any less dumbfounded. So much could go wrong...

..and it did. Multiple segmentation faults and memory leaks.

Multiple hours of trying to debug without the proper tools (lack of tools for threading debugging will be explained in the threading post), we gave up. It was a pointless endeavor.

Then while discussing our new threading system made for Zstandard initially but now used by everything, a spark occurred. Rather than having a dedicated thread, what if we move the compression task to a bunch of workers (the magic behind this will be explained in the threading post)? This would mean we can use our new threading system for replays as well.

After a little bit of reconfiguring, we did just that. It worked. It was a magical moment for the team. Until it didn't...
View attachment 3589

For some reason, it worked fine-ish on some servers but had terrible memory management on others. Luckily, this time around with our new threading system, we had debugging tools. We got on to the task, and found out we were at first not marking memory as unallocated, and then not clearing this unallocated memory. We were missing what is referred to as Garbage Collection. This was half a mistake in our replay framework and a half in our new threading system.

With all this sorted out, we redeployed the framework and to our expectations: it worked perfectly!

Our final step-by-step recording process looks like this:
View attachment 3587

Our memory was managed perfectly, no performance issues were experienced by the players and our profilers picked up nothing unusual. It was a win, and the destination of our adventure was reached.
View attachment 3590

In almost 2 days, we have amounted to a mind-boggling 50,000 replays making up 5.3GB of data.
View attachment 3588

With this said, the replay system is still in development and very much in the beta phase but we're very proud to say the current version has been stable enough to run production.

We also have decided to help the development community learn & improve by open-sourcing our work: Replay System Framework Source in hopes that everyone learns.
It's not a plug & play as it relies a lot on our closed-source libraries but we hope you can learn from the source code to make your crazy inventions.

To bring an end to this, I'd like to ask everyone to remain quarantined whenever possible and join us as a community on NetherGames. NetherGames <3 Opensource!
New forums, Nice
 


Feb 16, 2019
9
1
3
France
Very good work
But now you need TITAN rank to enjoy it .
It's a shame, you have developed an incredible system to make a replay system for games and for players, but now we need TITAN rank (relatively expensive) to access it, it's frankly disappointing ... I think its need to be free to replay the games...
Please do not make this server a pay to win server ...
 


Oct 6, 2019
43
13
8
Very good work
But now you need TITAN rank to enjoy it .
It's a shame, you have developed an incredible system to make a replay system for games and for players, but now we need TITAN rank (relatively expensive) to access it, it's frankly disappointing... I think its need to be free to replay the games...
Please do not make this server a pay to win server ..
i know ng just wants $$$ even though we arent billionares even if we were we wouldnt use all our money on NetherGames
 

Civiled

Developer
Staff member
Developer

May 20, 2020
1
2
1
i know ng just wants $$$ even though we arent billionares even if we were we wouldnt use all our money on NetherGames
You have to realize the expenses that it takes to run a server at this scale. They have to pay for Hosting, developers, domains, and sometimes advertisement to keep their services running and available to the free playerbase that you have. By adding these cosmetic items and cool perks for their ranks, it allows them to pay these heavy expenses.

The server hosts a constant 400 players and that comes at a great cost. Considering 1 GB of ram is barely enough to host 10 players, NetherGames has to have the maximum capacity of over a thousand players, meaning over 100 GB of ram. With the cheapest host, this costs a minimum of $100 per month for hosting their servers alone. For NetherGames to continue running their server as lag-free as possible, they need to add sales pitches for said ranks and store items.
 

TheRealLargeknome

Mod
Staff member
Mod
Builder
Discord Moderation Division

Aug 2, 2018
135
104
43
15
Creative
github.com
The above is true, and all the money from these ranks goes into providing the servers needed for ng. things like replay are ultimately not essential, but nice to have for these ranked players who contribute a massive amount to the server's income
 

  • Like
Reactions: ClannishBeret25

Oct 6, 2019
43
13
8
You have to realize the expenses that it takes to run a server at this scale. They have to pay for Hosting, developers, domains, and sometimes advertisement to keep their services running and available to the free playerbase that you have. By adding these cosmetic items and cool perks for their ranks, it allows them to pay these heavy expenses.

The server hosts a constant 400 players and that comes at a great cost. Considering 1 GB of ram is barely enough to host 10 players, NetherGames has to have the maximum capacity of over a thousand players, meaning over 100 GB of ram. With the cheapest host, this costs a minimum of $100 per month for hosting their servers alone. For NetherGames to continue running their server as lag-free as possible, they need to add sales pitches for said ranks and store items.
i never though about it that way i thought hypixel didnt take much money at all but seeing that people get low ping and theres not much lag hypixel must cost thousands
 

TobiasDev

Developer
Staff member
Developer

Nov 12, 2019
3
3
3
Germany
Yes it does, but they are of cause earning enough so that they can pay those expenses without going into negative numbers
 

Support Us

NetherGames is a community-based Minecraft server network that operates a large amount of in-game servers and online websites for our users. The majority of our staff members are dedicated volunteers who put their valuable time and effort into the server everyday for free. We rely on displaying advertisements, donations and earnings from purchases to keep our services running.

If you want to keep driving & supporting our team and the network forward, please buy a rank or donate now.

Advertisement

Join our Discord Server

Like us on Facebook

AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features of our website. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock    No Thanks