16:57 sobel: i can't find it now, but arstechnica benched all the popular dynamic languages 16:58 sobel: using C/C++ as the standard (1.0) they ranked other languages as multiples of C/C++ performance 16:58 sobel: java was a 2 16:58 sobel: twice as slow as C. and it was the winner, head and shoulders above the rest 16:58 sobel: i think Erlang placed somewhat favorably 16:59 sobel: python/perl were near the middle, at something like 25-35x slower than C 16:59 sobel: Ruby: a full 72x slower than C
Ruby loses against invisible Ars Technica benchmarks in the sky with unknown URLs! 72x slower than C! Over twice as slow as Python and Perl!
Fortunately, we don't have to rely on some one-off benchmark Ars Technica may or may not have done at some indeterminate point in time whose URL cannot be located offhand, because there's a site with a fairly decent reputation which provides ongoing benchmarks across multiple programming languages using implementations submitted by fans of said language. It's been around for awhile and is relatively well-trusted.
That site is the Programming Languages Shootout, and unlike the alleged Ars Technica benchmark, you can actually visit their web site at shootout.alioth.debian.org. What do they have to say about programming language performance?
According to this benchmark suite, JRuby is 34.5 times slower than (not C, not C++, but) Fortran. Ruby 1.9 (MRI/YARV) is 43.80 times slower than (not C, not C++, but) Fortran. Both JRuby and Ruby 1.9 beat Python, PHP, and Perl by a considerable margin. The nearest competitor is Python 3, at 47.93 times slower than (not C, not C++, but) Fortran. By the way, did I mention that the fastest language on their benchmark is... not C... not C++, but Fortran? (nothing personal sobel, but unsubstantiated hearsay is bad!)
Yes, that's right folks: according to the Programming Languages Shootout, Python, PHP, and Perl are all slower than Ruby. Did you think Ruby was slower than Python? Guess what, you're wrong. Ruby used to be one of the slowest popular languages, but that has changed. Ruby performance has advanced considerably over the years, so if you're still repeating some offhand information you may or may not have gotten from Ars Technica at some point but can't find the link to as your metric of Ruby performance, you may want to try again, and find modern, relevant information you can actually get a link to.
There are many future VM improvements in the pipe for Ruby, Python, and PHP (and I guess Perl users might continue dreaming of a Parrot-powered Perl 6). Rubyists can look forward to the upcoming JRuby 1.7 release which features InvokeDynamic support and allows for Java-speed method dispatch... in Ruby. InvokeDynamic is a game changer for the JVM in general, and it's a game changer for Ruby, because InvokeDynamic makes JRuby dispatch potentially as fast as Java.
Python users can look forward to PyPy, which is posting some incredibly impressive numbers, especially around numerical computing. Python users can also look forward to resumed work on Jython, which is adding InvokeDynamic support which can potentially make Python as fast as Java. Finally, PHP users can look forward to the HipHop VM developed at Facebook, which will provide much improved performance for PHP. These are all great projects, but none of them are really ready for general consumption yet (including JRuby 1.7).
All that said, the Programming Language Shootout doesn't include any of these unreleased development versions in the benchmarks you see when you visit their site. They show the numbers for the latest production releases, and those numbers show Ruby is faster than Python, PHP, and Perl.
The game has changed: you just weren't paying attention.
Last but not least, if you've seen some benchmark somewhere, even if you have an eidetic memory and remember but the numbers were, but can't even dredge up a link to it, please, please, don't quote said benchmark, even if you have an eidetic memory and remember what the numbers were.
For benchmarks to be remotely scientific, they must be both reproducible and falsifiable, and hopefully in addition to both those things they have a good methodology. If you can't even dredge up a link to the benchmark in question, please don't go quoting numbers off the top of your head to people who might be influenced by them.
Let's advance computer science beyond the state of witch doctors telling people to bleed themselves with leeches because at some point someone said they might make you feel better maybe.

38 comments:
Comparing Python to Ruby directly, we see this chart: Python vs. Ruby. You can see that they are on the same order of magnitude, and to say that one is faster than the other is an illusion. They operate at the same speed when averaged over many different types of tasks.
When Ruby is slow it's over 5X slower than it's median time. So the critical question would seem to be, how representative is the median timing of this test of real world use? C++ is always fast. PHP, at it's slowest, is only 2X slower than it's median timings. If the real world performance tends more toward the slower end then Ruby is slow.
JavaScript v2: 2.85
That means, your Browser is at least 12x faster then your webserver.
Let's move to Fortran!!!
Think it's a little unfair that the x86 architecture was cherry-picked of the four results sets to prove your point :/ on x64 quad-core, it was actually the slowest of all.
That's nit-picking, though, in general my opinion on this piece is that amongst the dynamic languages, these results mostly make the point that they all perform pretty similarly, and picking between them should probably be a choice more of team preference and library support for the target tasks. If performance is of the most supreme importance, the right choice between all those listed is none of them :)
I can't comment on the other languages' shootout implementations, but last time I checked some of the Ruby ones were astonishingly inept. I honestly wouldn't trust the aggregate rankings of the Shootout at all.
I mean, I don't even know where to start in figuring out if this is efficiently implemented, but just look at it!
http://shootout.alioth.debian.org/u64q/benchmark.php?test=meteor&lang=jruby
Also, the shootout scripts were overhauled at some point. They used to be more general, and now they're much more focused on computation. That's part of why Fortran (which is optimized for mathematics) gets first place.
I do agree that the conventional wisdom that Ruby is the slowest of the 'big four' (Perl, Python, PHP, Ruby) is unjustified.
@djur I don't even know where to start in figuring out if this is efficiently implemented
In that case we really don't need to bother about what you think of that program ;-)
@djur They used to be more general, and now they're much more focused on computation.
I don't think you know that.
@djur ...why Fortran...
Because a good Fortran programmer wrote some programs, and perhaps because that's a good compiler.
@Jon Eisen -- here's the direct Ruby1.9 :: Python3 comparison
The real story: all of the good functional programming languages (which are way more expressive than scripting languages anyway) are 10-15 times faster than the scripting languages.
How do you like them apples?
@bellyfullofrage -- Well done! (For actually thinking through what those numbers might mean.)
@Will Buck -- You're letting Tony off too easily.
It's very important to acknowledge that program performance is likely to different when we run the same programs on x64 rather than x86, and when we have 4 cores available rather than one core, and ...
@Tony That site is the Programming Languages Shootout...
No, it isn't.
(And it hasn't been for 5 years.)
@Isaac Gouy You want to talk about the multicore performance of Ruby versus Python? The "standard" interpreters for each of these languages have a GIL and thus don't support multicore.
Multicore only really matters for JRuby, Rubinius, and Jython, as these implementations do not have a GIL and can thus leverage multicore with a single virtual machine.
@Tony -- Here's a Python3 program that leverages 4 cores.
@Isaac Gouy I was referring to the fact that the style is a mess and the author refers to himself as a "Ruby novice". I find the Python version of that test easier to read, and I've been using Ruby almost 10 years longer than Python.
If that's your code and I hurt your feelings, I sincerely apologize.
And yes, the shootout was formerly more generalized. I recall many more micro-benchmarks on things like object instantiation, recursion, function calls, data structures, etc.
I'm going to wait for Fivetran.
@djur If that's your code and I hurt your feelings, I sincerely apologize.
You've already refered to the program author, so you know that isn't my name.
@djur And yes, the shootout was formerly more generalized. I recall many more micro-benchmarks on things like object instantiation, recursion, function calls, data structures, etc.
Back in 2005! (Whatever had been inherited from Doug Bagley's abandoned 2002 code base.)
Things like method calls and object instantion are not more general -- they are specific to a particular style of programming.
Things like ackermann and fibonacci and nested loops make the current tiny tiny programs seem like major exercises in software engineering!
Lies. Damned lies. Statistics. And then whatever this is. Kids, if you can write it in Fortran, chances are you don't do it.
One comparison which was done in 2008: http://blog.dhananjaynene.com/2008/07/performance-comparison-c-java-python-ruby-jython-jruby-groovy/
Notice the author clear says that this is only: "Implementing an identical object oriented solution to the Josephus Problem(...) and measuring the performance results thereof."
@Isaac Gouy Here's a Node.js fibonacci server that will light up all your cores: https://gist.github.com/2018811
Spinning up a separate interpreter per CPU core adds an awful lot of complexity though...
@Carter Shanklin
"After all, facts are facts, and although we may quote one to another with a chuckle the words of the Wise Statesman, 'Lies--damned lies--and statistics,' still there are some easy figures the simplest must understand, and the astutest cannot wriggle out of."
Leonard Henry Courtney, 1895
While Ruby 1.9 got increased performance as seen in the benchmarks, I was more interested in the Lua, Go and C# numbers. Is sad C# haven't got much attention in the open source (outside Windows) world while having numbers like those.
Perl, Python, Ruby, PHP, C, C++, Lua, tcl, javascript and Java comparison
See http://raid6.com.au/~onlyjob/posts/arena/
I'd be curious to know how Python 2.x fares in this benchmark. I have no idea whether it's faster or slower than Python 3.x, but 2.x is what I use most of the time.
@dguaraglia -- I'd be curious to know how Python 2.x fares in this benchmark.
Pretty much the same, The Wayback Machine has this snapshot from 2010
@camel7k -- Is that a comparison of single byte ascii strings against double byte Unicode strings?
In all cases if any of these languages were to make a change where my programming time was cut in half in exchange for their execution speed being quadrupled, I'd welcome it!
as someone else mentioned they all run great.,
Yes, Ruby is pretty fast, considering how dynamic and powerful it is, but it may just be too powerful.
From what I've seen, Ruby developers very often tend to value inner library code beauty and nice outer API design over performance.
Look at Cucumber for example, that is a two layers of DSL over regular Ruby code.
People actually construct damn fat classes in runtime based on results of introspection ( that is slow ), nifty DSL's ( they are slow too ) and, of course, then they use string manipulation to build code, and later eval it.
Also, things like long inheritance trees and breaking up code in modules make method lookup more complex, and if that is not bad enough, people use method_missing a lot, and blocks are everywhere.
In the end, this gives us just insane Rails startup times, and people go around "Bundler so slow!".
So while Ruby can be fast ( if you pay a lot of attention to optimizing the code ), most Ruby libraries are slow.
And you will probably end up using them.
@ahtoh: +100
That's a far more realistic analysis than just running over-optimized tiny bits of code.
@anton: oh hell yes. Nicely stated.
Benchmark squabbles tend to sound like the obsessions of medieval scholasticists, discussed with the wit and intellectual discipline of Redditors.
"38 Python programmers can dance on the head of a pin!"
"43 JRuby programmers can dance on the head of a pin!"
"Not fair! Two of them only had one leg, so their footprint was smaller!"
"lol pogo"
"Oh, so what about the PHP chick in a wheelchair, are we going to be ableist and exclude her too? Jerk"
"You know, Haskell is wheelchair-optimized. Any PHP v. Haskell wheelchair pindance benchmarks out there?"
"Hey, the C# people totally cheat on their pindance benchmarks because the Macarena is not real dancing!
"Lua is best for capoeira dance-off"
"What about line dancing? Lisp totally wins line dancing."
"Only if you've got Moebius pinheads, which nobody uses in production."
"Band name! Moebius Pinheads FTW"
...and so on.
More seriously, anton describes a far more realistic approach. I'd bet that well over 90% of real-world web apps would operate acceptably irrespective of the chosen language.
Web App performance bottlenecks ordinarily don't become significant unless the app is scaling to large transactional volumes, or the developers did some seriously faulty designs. In my experience, faulty design is highly portable. :-)
@Bill Blondeau >>I'd bet that well over 90% of real-world web apps...<<
I'd bet that you know nothing at all about well over 90% of real-world apps :-)
@Bill Blondeau >>Web App performance bottlenecks ordinarily don't become significant unless...<<
Tautology -- Performance bottlenecks aren't significant unless they're significant.
@Unknown: "where my programming time was cut in half".
Very true.
I'm a project manager, so on a certain level I think of developers and servers as costs.
When the decision is about to hire Java developers for 2X in money or PHP/Ruby devs for X and if I can buy a HP DL360G7 with 2 CPUs (that is 8 cores at least, but can be 16) from 10 days of a PHP/Ruby developer's cost, what will I choose if the project has about 80 developer days?
The PHP/Ruby one. When slow means "it cannot handle too many requests" then I don't care an inch if it is relatively slow. I'll just buy more servers.
3-5 years later - when I trash the servers - the next ones I buy will have ~24 cores, consume less power, and cost the same.
@Tony >>ongoing benchmarks across multiple programming languages using implementations submitted by fans of said language<<
Tony, please try your hand at writing better Ruby programs and contribute them to the benchmarks game! ;-)
I think I am going with Scala :)
Rubyists can look forward to the upcoming JRuby 1.7 release which features InvokeDynamic support and allows for Java-speed method dispatch... in Ruby.
@Tony -- Here are jruby-1.7.0.preview2 measurements with invokedynamic=true
Ruby is actually right in line with its comparable languages (Python, PHP, Perl...). See http://blog.mnewton.com/post/30542907442/ruby-is-slow-but-the-its-same-speed-as-comparable
Well said Tony! I get so sick of people belching numbers and crowing. I much prefer the evidence, even if it didn't say Ruby was fast.
Sometimes it's like "My Dad would beat your Dad in a fight any day!" playground mentality.
In any case, I don't use my language of choice because it's fast, but because it gets the specific job I want done, done, and does it well. The fact that it is a pleasure to write (and think) in is a BIG bonus though.
Post a Comment