Latency in MMOGs

This is a reprint of my message on an another forum.

"Ping" has, in the MMOG context, two meanings. The operating system ping command sends and receives ICMP echo packets, while the MMOG might just send an ACK command through a pre-existing TCP stream. When you ICMP echo ping someone, the response you get comes from the operating system network stack. When you pseudo-ping a MMOG server over TCP, you have to wait until the server goes through the TCP streams it's receiving, handles the ACK message, and queues a response on the TCP stream. These two pings have obviously very different characteristics. I'll disregard the UDP and hybrid UDP/TCP options here, because their characteristics aren't too dissimilar from TCP when we're talking about the difference of ICMP ping and app level ping. Only ICMP pings can be compared across board, different games give completely different application level ping results.

In a MMOG, you need to create a virtual environment by keeping track of these things on the server side: player location, map characteristics (which impede the player's movement,) AI-controlled mob locations, player-placed process marker locations (piles, projects, harvesters), event trap locations (portals, traps) and so forth. When the player's client sends requests for, say, movement, we need to make sure that the movement isn't impeded by the map, by a dynamic item on the map, by a flag (dead) or a dynamic process (getting tired.) All of the things I mentioned take up memory space on the server.

When we make the decision to grant a player's request for movement, we need to consider the factors I mentioned, but since considering a factor takes a certain number of nanoseconds, we generally try to only consider those factors which can be easily estimated to be relevant. For example, we can be pretty sure that we don't have to care what someone is doing in city A when we're considering the movements of people in city B. The tricky part about this is we don't want the time that is taken by these calculations to grow either in a linear or exponential fashion when the number of players and items active grow when new players buy the game and log in.

The reason you hear about how MMOG operators buy gajillion servers to run their games is because of the three nasty units which haunt the nightmares of programmers everywhere: milliseconds, microseconds and nanoseconds.

1 second = 1 000 milliseconds = 1 000 000 microseconds = 1 000 000 000 nanoseconds.

It takes about 5 milliseconds to read data from an internal 10KRPM SCSI disk to RAM, and about 50 nanoseconds to read data from RAM.

If a player action causes you to read stuff from the disk, that player action has just taken as much time as 100,000 player actions which only touch the RAM would have taken. Ideally, you'd like to run all of your simulations with just the commands and data which can fit into the processor cache, but people rarely manage that. The UltraSparc T2 line looks interesting from a MMOG latency perspective, though.

So, let's get back on track: when you're making the decision on whether to grant or deny a player's movement request, you only want to consider the minimum amount of factors and you want to be able to locate all of the factors in main memory. Developers have created many different ways to ease the task of deciding which facts you want to account for and how to locate those facts. The most common of these is breaking the virtual environment up to cells and storing pointers to all facts in that cell in a easily accessible manner. Then you just make sure you handle the event where people leave cells and join neighboring cells. Another way is to have two layers of cells, the static ones I just mentioned and dynamic ones to accomodate, say, a larger battle. In reality, things are much more complicated than this, but I don't have time to go into more detail.

In any case, from what I have read, it looks like the XXX developers were, for whatever reason, and I'm not judging them here, unable to realistically test their decision-making trees with simulated traffic before launch. At this point, they must be refining their algorithms and considering, for example, whether their hardware architecture would handle some of the FP math better through integer math simulations. I'm just guessing here, of course.