Wednesday, June 6, 2012

Ruby is faster than Python, PHP, and Perl

There's a pervasive myth that Ruby is slow, and moreover, that it's the slowest language in popular use. Everyone knows Ruby is slow. Right? Who would possibly disagree that Ruby is slow? Here's an example IRC discussion on freenode's #postgres which happened just yesterday:

16:57 sobel: i can't find it now, but arstechnica benched all the popular dynamic languages
16:58 sobel: using C/C++ as the standard (1.0) they ranked other languages as multiples of C/C++ performance
16:58 sobel: java was a 2
16:58 sobel: twice as slow as C. and it was the winner, head and shoulders above the rest
16:58 sobel: i think Erlang placed somewhat favorably
16:59 sobel: python/perl were near the middle, at something like 25-35x slower than C
16:59 sobel: Ruby: a full 72x slower than C

Ruby loses against invisible Ars Technica benchmarks in the sky with unknown URLs! 72x slower than C! Over twice as slow as Python and Perl!

Fortunately, we don't have to rely on some one-off benchmark Ars Technica may or may not have done at some indeterminate point in time whose URL cannot be located offhand, because there's a site with a fairly decent reputation which provides ongoing benchmarks across multiple programming languages using implementations submitted by fans of said language. It's been around for awhile and is relatively well-trusted.

That site is the Programming Languages Shootout, and unlike the alleged Ars Technica benchmark, you can actually visit their web site at shootout.alioth.debian.org. What do they have to say about programming language performance?


According to this benchmark suite, JRuby is 34.5 times slower than (not C, not C++, but) Fortran. Ruby 1.9 (MRI/YARV) is 43.80 times slower than (not C, not C++, but) Fortran. Both JRuby and Ruby 1.9 beat Python, PHP, and Perl by a considerable margin. The nearest competitor is Python 3, at 47.93 times slower than (not C, not C++, but) Fortran. By the way, did I mention that the fastest language on their benchmark is... not C... not C++, but Fortran? (nothing personal sobel, but unsubstantiated hearsay is bad!)

Yes, that's right folks: according to the Programming Languages Shootout, Python, PHP, and Perl are all slower than Ruby. Did you think Ruby was slower than Python? Guess what, you're wrong. Ruby used to be one of the slowest popular languages, but that has changed. Ruby performance has advanced considerably over the years, so if you're still repeating some offhand information you may or may not have gotten from Ars Technica at some point but can't find the link to as your metric of Ruby performance, you may want to try again, and find modern, relevant information you can actually get a link to.

There are many future VM improvements in the pipe for Ruby, Python, and PHP (and I guess Perl users might continue dreaming of a Parrot-powered Perl 6). Rubyists can look forward to the upcoming JRuby 1.7 release which features InvokeDynamic support and allows for Java-speed method dispatch... in Ruby. InvokeDynamic is a game changer for the JVM in general, and it's a game changer for Ruby, because InvokeDynamic makes JRuby dispatch potentially as fast as Java.

Python users can look forward to PyPy, which is posting some incredibly impressive numbers, especially around numerical computing. Python users can also look forward to resumed work on Jython, which is adding InvokeDynamic support which can potentially make Python as fast as Java. Finally, PHP users can look forward to the HipHop VM developed at Facebook, which will provide much improved performance for PHP. These are all great projects, but none of them are really ready for general consumption yet (including JRuby 1.7).

All that said, the Programming Language Shootout doesn't include any of these unreleased development versions in the benchmarks you see when you visit their site. They show the numbers for the latest production releases, and those numbers show Ruby is faster than Python, PHP, and Perl.

The game has changed: you just weren't paying attention.

Last but not least, if you've seen some benchmark somewhere, even if you have an eidetic memory and remember but the numbers were, but can't even dredge up a link to it, please, please, don't quote said benchmark, even if you have an eidetic memory and remember what the numbers were.

For benchmarks to be remotely scientific, they must be both reproducible and falsifiable, and hopefully in addition to both those things they have a good methodology. If you can't even dredge up a link to the benchmark in question, please don't go quoting numbers off the top of your head to people who might be influenced by them.

Let's advance computer science beyond the state of witch doctors telling people to bleed themselves with leeches because at some point someone said they might make you feel better maybe.

38 comments:

Jon Eisen said...

Comparing Python to Ruby directly, we see this chart: Python vs. Ruby. You can see that they are on the same order of magnitude, and to say that one is faster than the other is an illusion. They operate at the same speed when averaged over many different types of tasks.

bellyfullofrage said...

When Ruby is slow it's over 5X slower than it's median time. So the critical question would seem to be, how representative is the median timing of this test of real world use? C++ is always fast. PHP, at it's slowest, is only 2X slower than it's median timings. If the real world performance tends more toward the slower end then Ruby is slow.

Unknown said...

JavaScript v2: 2.85
That means, your Browser is at least 12x faster then your webserver.

MareizioBlog said...

Let's move to Fortran!!!

Will Buck said...

Think it's a little unfair that the x86 architecture was cherry-picked of the four results sets to prove your point :/ on x64 quad-core, it was actually the slowest of all.

That's nit-picking, though, in general my opinion on this piece is that amongst the dynamic languages, these results mostly make the point that they all perform pretty similarly, and picking between them should probably be a choice more of team preference and library support for the target tasks. If performance is of the most supreme importance, the right choice between all those listed is none of them :)

djur said...

I can't comment on the other languages' shootout implementations, but last time I checked some of the Ruby ones were astonishingly inept. I honestly wouldn't trust the aggregate rankings of the Shootout at all.

I mean, I don't even know where to start in figuring out if this is efficiently implemented, but just look at it!

http://shootout.alioth.debian.org/u64q/benchmark.php?test=meteor&lang=jruby

Also, the shootout scripts were overhauled at some point. They used to be more general, and now they're much more focused on computation. That's part of why Fortran (which is optimized for mathematics) gets first place.

I do agree that the conventional wisdom that Ruby is the slowest of the 'big four' (Perl, Python, PHP, Ruby) is unjustified.

Isaac Gouy said...

@djur I don't even know where to start in figuring out if this is efficiently implemented
In that case we really don't need to bother about what you think of that program ;-)

@djur They used to be more general, and now they're much more focused on computation.
I don't think you know that.

@djur ...why Fortran...
Because a good Fortran programmer wrote some programs, and perhaps because that's a good compiler.

Isaac Gouy said...

@Jon Eisen -- here's the direct Ruby1.9 :: Python3 comparison

j2kun said...

The real story: all of the good functional programming languages (which are way more expressive than scripting languages anyway) are 10-15 times faster than the scripting languages.

How do you like them apples?

Isaac Gouy said...

@bellyfullofrage -- Well done! (For actually thinking through what those numbers might mean.)

Isaac Gouy said...

@Will Buck -- You're letting Tony off too easily.

It's very important to acknowledge that program performance is likely to different when we run the same programs on x64 rather than x86, and when we have 4 cores available rather than one core, and ...

Isaac Gouy said...

@Tony That site is the Programming Languages Shootout...

No, it isn't.

(And it hasn't been for 5 years.)

Tony said...

@Isaac Gouy You want to talk about the multicore performance of Ruby versus Python? The "standard" interpreters for each of these languages have a GIL and thus don't support multicore.

Multicore only really matters for JRuby, Rubinius, and Jython, as these implementations do not have a GIL and can thus leverage multicore with a single virtual machine.

Isaac Gouy said...

@Tony -- Here's a Python3 program that leverages 4 cores.

djur said...

@Isaac Gouy I was referring to the fact that the style is a mess and the author refers to himself as a "Ruby novice". I find the Python version of that test easier to read, and I've been using Ruby almost 10 years longer than Python.

If that's your code and I hurt your feelings, I sincerely apologize.

And yes, the shootout was formerly more generalized. I recall many more micro-benchmarks on things like object instantiation, recursion, function calls, data structures, etc.

John said...

I'm going to wait for Fivetran.

Isaac Gouy said...

@djur If that's your code and I hurt your feelings, I sincerely apologize.

You've already refered to the program author, so you know that isn't my name.


@djur And yes, the shootout was formerly more generalized. I recall many more micro-benchmarks on things like object instantiation, recursion, function calls, data structures, etc.

Back in 2005! (Whatever had been inherited from Doug Bagley's abandoned 2002 code base.)

Things like method calls and object instantion are not more general -- they are specific to a particular style of programming.

Things like ackermann and fibonacci and nested loops make the current tiny tiny programs seem like major exercises in software engineering!

Carter Shanklin said...

Lies. Damned lies. Statistics. And then whatever this is. Kids, if you can write it in Fortran, chances are you don't do it.

s said...

One comparison which was done in 2008: http://blog.dhananjaynene.com/2008/07/performance-comparison-c-java-python-ruby-jython-jruby-groovy/

Notice the author clear says that this is only: "Implementing an identical object oriented solution to the Josephus Problem(...) and measuring the performance results thereof."

Tony said...

@Isaac Gouy Here's a Node.js fibonacci server that will light up all your cores: https://gist.github.com/2018811

Spinning up a separate interpreter per CPU core adds an awful lot of complexity though...

Isaac Gouy said...

@Carter Shanklin

"After all, facts are facts, and although we may quote one to another with a chuckle the words of the Wise Statesman, 'Lies--damned lies--and statistics,' still there are some easy figures the simplest must understand, and the astutest cannot wriggle out of."

Leonard Henry Courtney, 1895

Brittle Bones said...

While Ruby 1.9 got increased performance as seen in the benchmarks, I was more interested in the Lua, Go and C# numbers. Is sad C# haven't got much attention in the open source (outside Windows) world while having numbers like those.

camel7k said...
This comment has been removed by the author.
camel7k said...

Perl, Python, Ruby, PHP, C, C++, Lua, tcl, javascript and Java comparison

See http://raid6.com.au/~onlyjob/posts/arena/

dguaraglia said...

I'd be curious to know how Python 2.x fares in this benchmark. I have no idea whether it's faster or slower than Python 3.x, but 2.x is what I use most of the time.

Isaac Gouy said...

@dguaraglia -- I'd be curious to know how Python 2.x fares in this benchmark.

Pretty much the same, The Wayback Machine has this snapshot from 2010

Isaac Gouy said...

@camel7k -- Is that a comparison of single byte ascii strings against double byte Unicode strings?

Unknown said...

In all cases if any of these languages were to make a change where my programming time was cut in half in exchange for their execution speed being quadrupled, I'd welcome it!

as someone else mentioned they all run great.,

Антон Устюжанин said...

Yes, Ruby is pretty fast, considering how dynamic and powerful it is, but it may just be too powerful.
From what I've seen, Ruby developers very often tend to value inner library code beauty and nice outer API design over performance.
Look at Cucumber for example, that is a two layers of DSL over regular Ruby code.
People actually construct damn fat classes in runtime based on results of introspection ( that is slow ), nifty DSL's ( they are slow too ) and, of course, then they use string manipulation to build code, and later eval it.
Also, things like long inheritance trees and breaking up code in modules make method lookup more complex, and if that is not bad enough, people use method_missing a lot, and blocks are everywhere.
In the end, this gives us just insane Rails startup times, and people go around "Bundler so slow!".
So while Ruby can be fast ( if you pay a lot of attention to optimizing the code ), most Ruby libraries are slow.
And you will probably end up using them.

Fisa said...

@ahtoh: +100

That's a far more realistic analysis than just running over-optimized tiny bits of code.

Bill Blondeau said...

@anton: oh hell yes. Nicely stated.

Benchmark squabbles tend to sound like the obsessions of medieval scholasticists, discussed with the wit and intellectual discipline of Redditors.

"38 Python programmers can dance on the head of a pin!"

"43 JRuby programmers can dance on the head of a pin!"

"Not fair! Two of them only had one leg, so their footprint was smaller!"

"lol pogo"

"Oh, so what about the PHP chick in a wheelchair, are we going to be ableist and exclude her too? Jerk"

"You know, Haskell is wheelchair-optimized. Any PHP v. Haskell wheelchair pindance benchmarks out there?"

"Hey, the C# people totally cheat on their pindance benchmarks because the Macarena is not real dancing!

"Lua is best for capoeira dance-off"

"What about line dancing? Lisp totally wins line dancing."

"Only if you've got Moebius pinheads, which nobody uses in production."

"Band name! Moebius Pinheads FTW"

...and so on.

More seriously, anton describes a far more realistic approach. I'd bet that well over 90% of real-world web apps would operate acceptably irrespective of the chosen language.

Web App performance bottlenecks ordinarily don't become significant unless the app is scaling to large transactional volumes, or the developers did some seriously faulty designs. In my experience, faulty design is highly portable. :-)

Isaac Gouy said...

@Bill Blondeau >>I'd bet that well over 90% of real-world web apps...<<

I'd bet that you know nothing at all about well over 90% of real-world apps :-)

@Bill Blondeau >>Web App performance bottlenecks ordinarily don't become significant unless...<<

Tautology -- Performance bottlenecks aren't significant unless they're significant.

karatedog said...

@Unknown: "where my programming time was cut in half".
Very true.

I'm a project manager, so on a certain level I think of developers and servers as costs.
When the decision is about to hire Java developers for 2X in money or PHP/Ruby devs for X and if I can buy a HP DL360G7 with 2 CPUs (that is 8 cores at least, but can be 16) from 10 days of a PHP/Ruby developer's cost, what will I choose if the project has about 80 developer days?
The PHP/Ruby one. When slow means "it cannot handle too many requests" then I don't care an inch if it is relatively slow. I'll just buy more servers.

3-5 years later - when I trash the servers - the next ones I buy will have ~24 cores, consume less power, and cost the same.

Isaac Gouy said...

@Tony >>ongoing benchmarks across multiple programming languages using implementations submitted by fans of said language<<

Tony, please try your hand at writing better Ruby programs and contribute them to the benchmarks game! ;-)

Suraj Tamang said...

I think I am going with Scala :)

Isaac Gouy said...

Rubyists can look forward to the upcoming JRuby 1.7 release which features InvokeDynamic support and allows for Java-speed method dispatch... in Ruby.

@Tony -- Here are jruby-1.7.0.preview2 measurements with invokedynamic=true

Matthew Newton said...

Ruby is actually right in line with its comparable languages (Python, PHP, Perl...). See http://blog.mnewton.com/post/30542907442/ruby-is-slow-but-the-its-same-speed-as-comparable

Kevin McCaughey said...

Well said Tony! I get so sick of people belching numbers and crowing. I much prefer the evidence, even if it didn't say Ruby was fast.

Sometimes it's like "My Dad would beat your Dad in a fight any day!" playground mentality.

In any case, I don't use my language of choice because it's fast, but because it gets the specific job I want done, done, and does it well. The fact that it is a pleasure to write (and think) in is a BIG bonus though.