In our projects, when returning videos from servers, the question of disk subsystem efficiency comes up. Obviously, gigabit network interfaces are more efficient than, say, a 2 disk RAID 0. If video clips shared the same popularity, then hard disks would be a bottleneck when returning content.

We, however, have been lucky and there has always been a small set of clips which make up 80% of the traffic and a long line of less-frequently-accessed clips. This small set subsides in the file system’s cache and is returned practically without any disk activity.

For example, if we have 100 gigabytes of clips on a server, and the server has 24 gigs of memory, then we can draw this graph. Along the horizontal line, we have files sorted by popularity; along the vertical line, we have traffic caused by these files. The total traffic is the surface area under the line.
(more…)