Ever since Ruby 1.9.3’s first stable release (October 31 2011), we started thinking NebuLab required some updates. To keep things lean and simple, we wanted to run some benchmarks to ensure updating was worthwhile. This post presents the results of those benchmarks.
Up until recently, for our web applications, we used Ruby Enterprise (ree if you use rvm). If you never heard of it, Ruby Enterprise is plain old Ruby 1.8.7 patched for better memory consumption and geek friendly garbage collector (GC) tuning. Looking at the benchmark results, we gladly decided to use Ruby 1.9.2 for our production environments and it’s performing great!
The data is generated using ruby-benchmark-suite running on a MacBookPro5,5. We also used rvm to install the following Ruby versions:
As you may have noticed, the versions chosen to run the benchmarks are not the latest. Understanding performance gains should not be a problem since we are looking for big differences to choose a fast and reliable Ruby version.
The chart is drawn using Highcharts a great js library for drawing dynamic charts. You can explore the chart by clicking and dragging a section with your mouse.
Some regions of the chart may look like missing data, this is mainly due to benchmarks which took less than 1 second to run. The same happens when a benchmark gets 0 execution time because of its failure. This means that, if you see an empty space between bars, that particular Ruby version failed that particular benchmark because of an exception.
Looking at the chart, it seems obvious that 1.9.2 and 1.9.3 are the two fastest and stablest Ruby versions. The more “exotic” Ruby versions are, in some benchmarks, extremely fast but often take a lot of time to complete a benchmark (if they don’t fail miserably).
Personally, I was shocked to find out how clearly Ruby Enterprise seems outdated. After these benchmarks, I can say that Ruby Enterprise is only fit for those production environments where you need to keep running an old outdated application which you do not want/do not need to update. That said, Ruby Enterprise also offers other benefits (not strictly related to performance), like memory optimization for multiple apps running on the same server or the fact that the GC can be configured easily.
Anyway, those benefits are not a Ruby Enterprise exclusive anymore. Ruby 1.9’s performance and memory consumption have been greatly improved thanks to the inclusion of YARV and, since 1.9.3, the GC can be configured just like ree.
To conclude with, I know that for some of you, the geekiest ones, the chart will be too approximate (it is for me too :D) so I thought I’d give you the original CSV benchmark results file, so that you can play with it however you see fit.