Comments on Unlimited Novelty: Twitter: blaming Ruby for their mistakes?

It's a fact that RoR/Ruby is slower than Scala...

2009-10-08T08:36:06.402-07:00

It's a fact that RoR/Ruby is slower than Scala, why kill the messenger for the message? It's pretty humorous how all the RoR hypesters are now jumping off a sinking ship.

FWIW, I'd like to know how ActiveMQ did not me...

2009-06-11T11:54:02.493-07:00

FWIW, I'd like to know how ActiveMQ did not meet the needs of Twitter so that we can use that information to improve ActiveMQ. There has been no discussion of configurations or topologies or use cases so I'm not even sure where to begin.

FYI, ActiveMQ was born out of the Apache Geronimo project but actually began life at the Codehaus. The first releases of ActiveMQ began appearing back around 2003 or 2004. It was only later in 2005 that we moved it to the Apache Software Foundation.

There are some comments about RabbitMQ from Alexis regarding the page-to-disk feature so I can address that. ActiveMQ already has this feature so that messages (and even references) are not held in memory (see message cursors here: http://bit.ly/10u5WJ).

The additional comments from Alexis are exactly correct. Building a messaging system that will suit any use case, running on any number of systems, using any topology with all the necessary features is a difficult task and certainly takes time to perfect.

There was also a comment from Alex Payne that experts were contacted. Was anyone from the ActiveMQ community contacted?

Since the research has already been done, please let us know why ActiveMQ did not work for Twitter so that we can improve it.

Apache ActiveMQ is highly configurable - you can a...

2009-06-10T10:57:35.867-07:00

Apache ActiveMQ is highly configurable - you can always make it scale or perform better - depending on your use case: see Scaling ActiveMQ

So Twitter is awesome for Taking traffic to your w...

2009-05-01T05:12:00.000-07:00

So Twitter is awesome for Taking traffic to your website . It is very
simple to setup and it's a fun positive way to keep in contact with
people. To get more followers on twitter check out this amazing
tool, Twitter Traffic Machine.

"Any messaging server where IO doesn't dominate to...

2009-04-08T09:32:00.000-07:00

"Any messaging server where IO doesn't dominate to the point where language choice is irrelevant is badly written." -- Vidar

"For every complex problem there is an answer that is clear, simple, and wrong." -- H. L. Mencken

:)

But seriously, building a scalable architecture for something as complex as a messaging server involves more than just "throwing cores at it until it's I/O bound". :) The optimum case, which you will never achieve except "on the average", is for the processor and I/O utilizations to be approximately equal and for the number of "users" to be at a point known in queuing theory as "N*"

This is called "asymptotic bounds analysis" (ABA) and can be found in lots of places, but a good start is

http://www.cs.washington.edu/homes/lazowska/qsp/Images/Chap_05.pdf

HTH :)

Any messaging server where IO doesn't dominate to ...

2009-04-08T09:19:00.000-07:00

Any messaging server where IO doesn't dominate to the point where language choice is irrelevant is badly written.

Writing one in Ruby is fairly simple. I wrote a Ruby Stomp server for inhouse use years ago, and it was trivial to optimize it to the point where the time spent in my Ruby code was less than 10% of the total time spent - the rest of the time was spent in the kernel handling IO syscalls.

Very interesting conversation. I work on ActiveMe...

2009-04-08T07:59:00.000-07:00

Very interesting conversation. I work on ActiveMessaging, and have used most of the brokers mentioned above.

I'll throw out one more - there is the reliable messaging gem that is lovely for development - persists to disk, very lightweight, and trivial to install and use. I don't think I would begin to recommend for high demand production, as I haven't tested it for such, but for smaller projects and development, I am a big fan.

I know from several other large applications that ActiveMQ is troubled at load, and am not surprised it fell over.

Stomp is not a perfect protocol, such as lacking a good way to rollback message receipt, but there are more than a few impls out there in python, ruby, and java via rabbit and activemq. I always found it disappointing that twitter didn't use the protocol if not the impls out there, as it seems a good fit (short text messages are stomp's bread and butter).

I do understand that twitter is a messaging app at core, so it makes sense to invest in building something, I only wish that they had adopted a messaging spec/protocol such as amqp or stomp for their impl.

Personally, I have high hopes for RabbitMQ/amqp, and that is the next broker/protocol I'll be integrating with ActiveMessaging.

-Andrew Kuklewicz

why would twitter want to redistribute Berkeley DB...

2009-04-08T06:03:00.000-07:00

why would twitter want to redistribute Berkeley DB itself?

It's perfectly legal to distribute source code that links against it, and even that doesn't seem to be a real concern for an internal product.

The claim that BDB costs money does not really apply in this situation unless they want to sell a product that is prebuilt with it, the way I understand the licensing issues anwyay.

2009-04-08T05:58:00.000-07:00

This comment has been removed by the author.

Twitter is a successful company, therefore, whatev...

2009-04-07T09:29:00.000-07:00

Twitter is a successful company, therefore, whatever it does, must be 'correct'. If your company is successful then whatever it does is 'correct' even if it is completely different from Twitter.
No tongue in cheek intended. Going toward Scala must be a good thing, otherwise they wouldn't do it.
Results are everything.

One thing I really feel I should say:

For t...

2009-04-06T20:24:00.000-07:00

One thing I really feel I should say:

For the purposes of my job, we do not need a stateful message queue. We run an automation system which keeps all state in a database. Most of our system is effectively stateless, including our message queues. If our message queues crash, our system recovers from the state in the database.

Twitter, who is running a messaging system, apparently needs to persist state across their entire system. I don't know if the system could be architected in a more stateless manner, but I'll give them the benefit of the doubt and assume they have tried to make their system less stateful. This means all their queues need to be disk logged. The performance tests we performed which I anecdotally cited did not ever use a disk logged message queue intentionally, but in the case of Starling we were as this appears to be the only option Starling provides.

I'm afraid this is an apples-to-oranges comparison and I may be unduly sullying Starling.

That said we were testing a persistent configuration of Ruby Stompserver, which despite being written in Ruby was able to outperform Starling (at the level of ~100 readers/writers)

In Twitter's case it sounds as if they were running a rather large "flock of Starlings" to handle the load. This works, but in our system we run one queue per server.

nothingmuch said...
"I don't want to pay for Or...

2009-04-06T19:44:00.000-07:00

nothingmuch said...
"I don't want to pay for Oracle BerkeleyDB either, that's why I use it without giving them money. It is opensource, you know =P"

It's under a copyleft license which prevents its redistribution along with your software unless you distributed your source or pay them a (very large last time I heard) fee.

I don't want to pay for Oracle BerkeleyDB either, ...

2009-04-06T16:36:00.000-07:00

I don't want to pay for Oracle BerkeleyDB either, that's why I use it without giving them money. It is opensource, you know =P

"EventMachine didn't exist or were so immature tha...

2009-04-06T15:53:00.000-07:00

"EventMachine didn't exist or were so immature that they were essentially useless."

That is most certainly not the case. I've been using EventMachine since 2006, and while I am not a fan of the API (hence writing Rev), it was most certainly stable and mature back then.

The EventMachine-based Ruby StompServer was released in late 2006 and outperforms (non-Evented) Starling.

Tony: as the guy who wrote Starling, I'll be nice ...

2009-04-06T11:35:00.000-07:00

Tony: as the guy who wrote Starling, I'll be nice and simply say that you haven't done your research. Your Picard there is hitting his head over the stupidity of the internal combustion engine.

Starling was written over two years ago (it took a long time to release). It was written in about a day and a half. At the time, no memcache-based queue servers existed, RabbitMQ required knowledge of erlang, and rev and EventMachine didn't exist or were so immature that they were essentially useless. ActiveMQ didn't have Ruby client libraries, and Spread, well. Anyhow.

If you do the math, for the number of queue operations Twitter was doing *two years ago*, Starling is plenty fast enough, because each queue action took several orders of magnitude longer than the queuing / de-queueing itself.

Moreover, you might try turning off the fsync() that is present in the default Starling distribution, which makes it easily four times slower than normal operation. And then you might compare apples to apples, and ensure that your queues are doing the same thing.

And to top it all off, in the README for Starling I wrote:

"Starling is "slow" as far as messaging systems are concerned. In practice, it's fast enough."

It was a case of worse-is-better, and frankly, its existence has been an important part of discussions around message queue design from a usability standpoint. Prior to Starling, queuing was HARD for small organizations, and the agenda was set by JMS and "the enterprise" - oh, there's Picard again.

I'm not going to defend Starling as a great example of high performance and resilient message queue or networking software design, and I tend to recommend RabbitMQ or Kestrel to people looking for a simple queue.

In the future, as Alex suggests, please feel free to email developers of software before making many unsubstantiated and incorrect claims. My email address is and has always been in the Starling source code.

Bob: part of research, for us, was considering cos...

2009-04-06T09:56:00.000-07:00

Bob: part of research, for us, was considering cost and integration time. See Nick's comments above for why Kestrel ended up being much easier to test than other complete-rewrite solutions. We did appreciate your suggestion.

Glenn: if you search around for what people make on technical books, you'll see that I don't have much to gain from book sales. I'm putting a lot more time into my half of the book than I'll ever make back. But it's well worth it for what I'm learning in the process.

"Glenn" has to muddy the waters with his statement...

2009-04-06T09:14:00.000-07:00

"Glenn" has to muddy the waters with his statement about the twitter folks not being all that honest since one of them is writing a scala book.

Give me a break dude, and go off with your tabloid comments elsewhere. Just when folks start to have a reasonable discussion ...

-jpb

Maybe I am cynical, but In the interest of transpa...

2009-04-06T08:03:00.000-07:00

Maybe I am cynical, but In the interest of transparency please also note that Alex may also be pimping his own book about Scala which *may* in part help us understand why he is being more vocal in re-opening the debate on 'Ruby v. Scala'.

I am not in any way saying his arguments and experiences are not well thought out (any programmer worth his salt knows there is no silver bullet language), and I won't take a position on Ruby vs. Scala since I have only used one of them.

He does though in fact appear to have a vested interest in this language based religious war (above and beyond his Twitter affiliation) which has proven over several years to provide excellent link-bait...

http://oreilly.com/catalog/9780596157746/

I'm just saying...

@alex> I research open source and commercial so...

2009-04-06T07:25:00.000-07:00

@alex> I research open source and commercial solutions before cutting a new path.

I recommended 1.5 million message/second Tervela box through numerous channels, and I don't have any evidence that it was ever considered.

http://www.tervela.com/tmx

Looks like the folks over at SecondLife did an eva...

2009-04-06T06:45:00.000-07:00

Looks like the folks over at SecondLife did an evaluation of several message queues and posted their results. Check it out here:

http://tinyurl.com/c8x6z5

Hello, Alexis here from the RabbitMQ team. I am t...

2009-04-05T20:53:00.000-07:00

Hello, Alexis here from the RabbitMQ team. I am the guy who visited the Twitter offices, that al3x mentioned in his comment above.

First off I would like to second the statement made by some of the Twitter guys above that they do not have an 'NIH' mindset. While some individuals in their team like to code new stuff, which can become a form of NIH, it is true that *as a team* they approach problems systematically.

Secondly, I know something about what the Twitter team do with messaging, and what they don't use messaging for. And I have just read a large number of blog posts and comments about Scala, messaging, Twitter (and Alex's book!). People are making massive assumptions about how Twitter do and don't use messaging. Most of these assumptions are completely wrong. Give these guys a break. They are trying to make improvements in a running system which is scrutinised every minute of every day, for a single sign of failure. This would drive anyone to drink, madness, or worse yet, functional languages ;-)

The choice of language is secondary to the design of any messaging system. RabbitMQ is written in erlang/OTP which like Scala can use a share-nothing model. But this is no guarantee that your messaging system will work well in every possible scenario that any customer could ever use. This is why writing good messaging systems is hard, and in the case of RabbitMQ it leads to our being careful to add major features slowly, because we don't want the product to be compromised by bad designs that are hard to remove later. We have to do things this way because we have a lot of (mostly happy) users.

Now, quite a few of those users have asked us to add a feature called 'page to disk'. We are adding this feature now. 'Page to disk' means that when messages are persistent, they do *not* get held in memory at the same time. Note that in the current version of RabbitMQ, if a message is persisted, then a copy is held in memory as well. One of the RabbitMQ team recently blogged about this here: http://www.lshift.net/blog/2009/04/02/cranial-surgery-giving-rabbit-more-memory (please note that the messages are few but large in order to test the overflow properties of the system)

The scenario in which page-to-disk is needed is as follows:

1. You have relatively slow consumers with durable subscriptions. E.g. they disappear for days at a time.

2. You have to keep all messages that they have not seen yet and cannot flush them on a timeout basis.

3. You have enough producers and data that this fills up the memory of your broker.

4. You cannot, or don't want to, run the broker on multiple machines, e.g. using RabbitMQ clusters.

Since it is not my place to speak about what Twitter actually does under the hood, I shall leave it to readers to figure out when, if at all, these criteria apply to the several ways that Twitter might or might not use messaging.

It's quite easy to write a messaging system that manages balanced transient flows where ingress and egress are similar. Writing messaging systems that work under any combination of flows, on any number of machines, and in multiple different reliability scenarios ... is a more interesting problem. Page-to-disk is a way to make RabbitMQ better and address more scenarios.

If you are reading this and have other ways to improve the broker, please send us information via the mailing list or privately to info at rabbitmq.com, and please be as detailed and concrete as possible.

If you are in SF and want to know more about how you can build a twitter type system using RabbitMQ, we shall be talking about it, a little bit, this Wednesday evening: http://www.bayfp.org/blog/2009/03/25/next-meeting-rabbitmq-wednesday-april-8th-730pm/ (with beers afterwards). We'll just be talking about *messaging* in various scenarios.

I'd like to finish by asking everyone to check out Harper Reed's most excellent new project: http://www.awesomeupdater.com/

Cheers,

alexis

PS: to one person who shall remain nameless, thanks for a completely fatuous tweet. I'll buy you a beer next week if you come to the talk on Wednesday :-)

So what I get from these comments from the Twitter...

2009-04-05T19:31:00.000-07:00

So what I get from these comments from the Twitter guys is that Starling sucked and was subsequently replaced by a new Scala messaging system.

In other words Ruby wasn't the problem, it was Starling (Which as the Twitter guys said was a good messaging system when written but just couldn't handle the increased load in the long run).

RE: forking or improving an existing open source p...

2009-04-05T17:40:00.000-07:00

RE: forking or improving an existing open source project and integrating it with your systems...

Asked and answered. Thanks for your transparency.

Wow, thanks. That was quite a satisfactory respon...

2009-04-05T17:02:00.000-07:00

Wow, thanks. That was quite a satisfactory response. I'll post a followup blog here in a bit.

Tony,

Hoo boy. First of all, I hope you've ...

2009-04-05T16:54:00.000-07:00

Tony,

Hoo boy. First of all, I hope you've had a chance to read my general reply to the articles about my Web 2.0 Expo talk [1] and this response to a vocal member of the Ruby community [2]. I sound like a pretty unreasonable guy filtered through the tech press and Reddit comments, but I hope less so in my own words.

Secondly, the quote at the top of your post is from my coworker, Steve Jenson, who's been participating in the discussion on this post.

On JRuby: as Steve said, we can't actually boot our main Rails app on JRuby. That's a blocker. Incidentally, if you know of anyone who has a large JRuby deployment, we'd be interested in that first-hand experience. If you don't, it might be a little early to say it would solve all our problems.

It's also incorrect to say that the way JRuby and Scala make use of the JVM is exactly the same. Much like our other decisions haven't been arbitrary, our decision to use Scala over other JVM-hosted languages was based on investigation.

On our culture: if you'd like to know about how we write code, or how our code has evolved over time, just ask us. We're all on Twitter, of course, but most of the engineers also have blogs and publish their email addresses. There's no need to speculate. Just ask. There's not a "raging debate" internally because we make our engineering decisions like engineers: we experiment, and base our decisions on the results of those experiments.

It's definitely true that Starling and Evented Starling are relatively immature queuing systems. I was eager to get them out of our stack. So, as Steve said, we put all the MQ's you think we'd try through their paces not too long ago, and we knocked one after another over in straightforward benchmarks. Some, like RabbitMQ, just up and died. Others chugged on, but slowly. Where we ran into issues, we contacted experts and applied best practices, but in the end, we found that Kestrel fit our particular use cases better and more reliably. This was not the hypothesis we had going into those benchmarks, but it's what the data bore out.

We get a lot of speculation to the tune of "why haven't those idiots tried x, it's so obvious!" Generally, we have tried x, as well as y and z. Funnily enough, I was actually pushing to get us on RabbitMQ, but our benchmarks showed that it just wouldn't work for us, which is a shame, because it advertises some sexy features.

Personally, I'm extremely NIH-averse; I research open source and commercial solutions before cutting a new path. In the case of our MQ, one of our engineers actually wrote Kestrel in his free time, so it was bit more like we adopted an existing open source project than rolled our own. Pretty much the last thing we want to be doing is focusing on problems outside our domain. As it so happens, though, moving messages around quickly is our business. I don't think it's crazy-go-nuts that we've spent some time on an MQ.

I hope my colleagues and I have been able to answer some of your questions. As I said, in the future, please consider emailing us so we can share our experience. Then, we can have a public discussion about facts, not speculation. Perhaps, as commenter sethladd suggested, the onus is on us to produce a whitepaper or presentation about our findings so as to stave off such speculation. Time constraints are the main reason why we haven't done so.

[1] http://al3x.net/2009/04/04/reasoned-technical-discussion.html
[2] http://blog.obiefernandez.com/content/2009/04/my-reasoned-response-about-scala-at-twitter.html#IDComment18212539