Dear Viacom: You're Doing It Wrong

I've been very excited about the upcoming release of The Beatles: Rock Band after hearing about it earlier this week. It's the first PS3 game sold on disc I'm going to snap up since, well, Rock Band 2. Beyond the simple fact that it's Rock Band loaded with Beatles music (and I love the Beatles), I've been rather impressed by the visual style, particularly this intro video directed by Pete Candeland of Gorillaz music video fame:


Hopefully this video hasn't been flagged for a copyright violation by the time you read this post. It rules

However, I was rather surprised to see that a particular copy of the intro video was flagged for removal from YouTube due to a copyright violation. Go ahead, hit play, I dare ya:



Yes, Viacom has decided that they want to forego free advertising for their upcoming video game in order to defend their copyright. WTF? Somebody doesn't get it. Viacom, this isn't someone infringing your copyright. This is someone providing you with a viral marketing campaign for free. You are effectively telling them: "no, don't advertise our product for free. We don't want that"

I forsee a long uphill battle until old media companies finally realize that viral video distribution is actually a good thing. Eventually they'll be drug kicking and screaming to the realization that piracy is good.

Ubuntu's Jackalope not so Jaunty

I'm not one to write reviews of things like desktop Linux systems typically. In fact, any of you who read my blog for Reia should probably just stop now. But I just tried desktop Linux for the first time in two years, and my experience was anything but pleasurable.

My Background (a.k.a. chance to be a blowhard)

For the past several years OS X has been my desktop of choice. I get a beautiful, slickly animated GUI interface, seamless 3D compositing of all UI elements, nifty commercial software, and Unix underpinnings. Sure, it's proprietary, but I don't give a crap.

That said, I am no stranger to desktop Linux. My first desktop Linux experience was using FVWM on a Slackware 2.3 system back in 1995. So yes: I'm one of those Linux users that survived the transition from a.out to ELF and from libc5 to glibc. I'd try a few different distributions, next RedHat and finally Debian before becoming a Debian person. I tried RedHat 5.0, when they made the switch to glibc, and it was such an unmitigated disaster I destroyed the install CD (purchased from a store) out of rage. That is the lowest low I think I've ever seen Linux reach.

I remember trying out an early Enlightenment, which leaked memory so quickly it completely consumed the 16MB of RAM I had installed at the time. Eventually I would discover WindowMaker, which would be my standby window manager for years to come. I flitted about with OS choices after that, running FreeBSD as my primary desktop for quite some time.

Around 2001 I discovered the Synergy software which lets you seamlessly share a keyboard and mouse across two computers. From then on I loved running two computers, typically one with Windows and one with my *IX du jour. This has remained my standard configuration for quite some time.

Around 2006 I was given a new monitor for work, with a strange 1680x1050 resolution. I was running Debian at the time, ripping my hair out hand editing my X config trying to get it to work properly. I could not for the life of me figure out what was wrong, and this was after spending 5 years as a Linux sysadmin. I decided to give Ubuntu a go. I stuck in the install CD, and BEHOLD it booted straight into X and my monitor was automagically configured to the right resolution! I was awestruck.

I'd been against Gnome for years, but by now it seemed almost downright usable. I actually liked having things like desktop icons! It was pretty nifty.

However, shortly thereafter I would buy a MacBook and ditch desktop Linux entirely. I've been running an OS X/Windows Synergy setup ever since (although now I use OS X exclusively at home)

Fast Forward to Today

Amidst many of my coworkers setting up their computers to dual boot Windows and Linux, I figured I'd do the same. I thought it'd be pretty nifty to have OS X one one computer (which would remain my primary development computer) and Linux on the other.

First I installed Windows, which wasn't without its hiccups but when I was done I was left with a 30GB Windows partition and 220GB free for Linux.

I threw in the Jaunty Jackalope CD one of my coworkers had and started up the graphical installer. I missed the good old text-based Debian installer I had used for over a decade, but hey, it's the 21st century, nothing wrong with graphics, right?

I got to the partitioning step. Now, I've dealt with some pretty bad graphical partition managers in the past. Solaris's was particularly atrocious. At first glance Ubuntu's seemed fine... it recognized I had an NTFS volume and offered me the option to "Install Windows and Linux side by side". I figured this was such a common use case it would just naturally know the right thing to do.

So, I click OK and it pops up a modal dialog asking me if I want to resize my NTFS partition. WTF? Resize my NTFS partition? NO! You have 220GB of free space to work with there, why would you resize my NTFS partition? It offered three buttons I could click: one that said "Go Back" which was highlighted (and I guess the one I was supposed to choose), one that said continue/proceed or something, and the traditional "X" in the corner of the modal dialog window to close it. I clicked the latter, which was the wrong decision.

It then popped up another modal dialog window, informing me it was resizing my NTFS partition to fill the entire disk.



Seriously, are you kidding me? Closing a modal dialog window with the "X" button does NOT MEAN I WANT YOU TO PERFORM A DESTRUCTIVE ACTION. And furthermore, who installs Linux and wants it to resize their Windows partition to eat up the entire disk? Frustrated, I opened up a terminal, started gparted, shrank my Windows partition back down to 30GB, and rebooted to try again fresh.

This time I chose to manually partition my disk (although I still longed for cfdisk) and things seemed to go a little more smoothly for awhile.

After I booted into X for the first time I was prompted to install updates. I hit the "Check" button which prompted for a password. It downloaded a list of updates. I hit "Install Updates". Nothing happened. I clicked it again, some 30 times. Nothing. The button depressed, and that was it. I called a Linux-loving coworker over, he looked at it and shrugged. "I don't use the graphical updater". Yes, popping open a terminal and typing apt-get upgrade was seeming like the way to go here. I clicked "Check" again then "Install Updates". Magically it worked this time.

After all was said and done, my display was not at the right resolution. It popped up a little notification prompting me to install restricted drivers. I installed the nVidia drivers, which completed successfully, then tried to open my display preferences.

An error dialog popped up, saying that display preferences couldn't be launched, and I need to run my vendor tool instead. It said I could hit OK to do so, but when I did, it said there was an error launching the vendor tool, and I needed to run a particular command from the command line.

Are you kidding me? At this point I'm seriously confused: who is Ubuntu targeting? When was the last time Windows or OS X asked me from a modal dialog to pop open a terminal and type some shit on the command line? I tried running the command and got yet another error. Frustrated I rebooted.

Now when I try to go to the display preferences, at first I get an error saying I need to run the vendor tool, and then it launches the nVidia preferences. Only... the native resolution of my monitor is not listed. All of them are below the native resolution of my monitor.

I thought: oh well, I'll just edit my xorg.conf by hand. So I did. And I rebooted. I was still at the same resolution, and the changes I made to my xorg.conf were clobbered by something. I don't know. I reopened the file and they were completely gone. What process overwrote it? I don't have a freaking clue. I remember when Linux gave you a sense of control over what you're doing, but now I feel powerless.

Now, insult to injury: this is the exact same monitor which in 2006 I stuck an Ubuntu install CD into my computer and it launched straight into X at the native resolution. I didn't have to install any restricted drivers. I stuck in the CD and it just worked.

3 years ago Ubuntu had me excited that maybe, finally, desktop Linux was reaching a level of general usability. Now here I am, a power user, and I've run into seemingly intractable problems I can't solve.

Pre-Emptive Anti-Zealot Repellant

Did I go onto forums and ask about my problems? Did I post bugs on Ubuntu's trackers? No. Know what I did? I rebooted into Windows. And here I think I will stay. I freshly installed Windows the same day and had it up and running to my satisfaction in a few hours. Windows is working and I am happy, therefore I don't feel the need to try to help Ubuntu sort out their mess.

All I can say is, without a doubt, Ubuntu 9.04 represents a rather severe regression from my previous experience with using Ubuntu on a desktop. We still continue to run it on our servers at my place of employment and there I have few complaints. However, at this point I cannot see myself running it on a desktop, and worse I've lost my sense that desktop Linux is actually getting better over time.

Reia Presentations

I've given a few presentations on Reia lately. The first was at last month's Erlang Factory conference, where my presentation included a short talk on building languages on top of Erlang, followed by a talk on Reia geared towards Erlang programmers. About a month later I gave a talk on Reia at my local Ruby group, focused on Rubyists.

In case anyone is interested in these talks, I've posted the on Google Docs and made PDF versions of them available.

Here's my talk at Erlang Factory, ostensibly geared at an Erlang-focused audience:


PDF version available here

And here's the same talk, slightly modified for Rubyists, and given at the boulder.rb group:


PDF version available here

Happy 1st Birthday, Reia!

Reia turns one year old today, as measured by the first commit to the Reia repository (well, not literally, there was one commit of the leex code before it which isn't mine). Later that day I'd begin fleshing out the scanner (with leex) and the initial grammar (with yecc). I'd been playing around with leex and yecc before but this was the first time I really tried to put it together into a coherent language, one which could at least calculate things with C/Algol-style syntax.

Things have come a long way from then! I never would've thought I'd give a talk about Reia before it was even a year old. I've noticed most language designers like to tinker in obscurity for a few years before releasing their creations to the world. I've instead just thrown it up on github and let anybody who wants poke around with it and send me patches.

In the past year, I've managed to implement the following features in Reia:

• Ruby-like syntax with destructive assignment
  
• Rich set of "builtin" types including integers, floats, strings, atoms, tuples, lists, maps (i.e. dicts), "binaries", regexes, funs (i.e. lambdas), process pids, and constants (i.e. module/class names)
• Pattern matching
  
• Asynchronous object system based on Erlang processes with single inheritence
• Erlang-style processes (i.e. Actors)
• Ruby-style blocks
• Function/method references via Python's receiver.method syntax (implemented as funs)
• List comprehensions
• (Almost) pure expression grammar, allowing modules and classes to be defined on the shell
  
• Self-hosted test system for the language itself
  
• Interpolated strings

There are many future features on the roster. Stay tuned and I'll have a series of follow up blog posts on what I'd like to add.

Erlang Factory: A Retrospective

I recently presented on Reia at Erlang Factory in San Francisco. It was a lot of fun, both attending and presenting. Videos of my talk, and the many others I heard people raving about over IRC and Twitter are forthcoming but will hopefully be available soon. There are certainly many I intend to check out.

The conference certainly exceeded my expectations. I thought I might be something of an outsider in the community, but I was surprised to see a number of Ruby people in the community, which I found pretty interesting. Ezra's talk on Nanite touched on an important point: there's not a lot of crossover in what Ruby and Erlang do well, and for that reason they're relatively compilentary languages. I ended up pitching Reia as what I hoped to be a best-of-both-worlds solution.

There was certainly ample talk about CouchDB, which got me thinking about using Reia as a language for CouchDB views. CouchDB creator Damien Katz gave a pretty cool talk as well, not so much about code as about the personal circumstances which brought him to create CouchDB.

The Powerset guys talk on Katamari, the evolution of Fuzed, was pretty cool to catch as well. I've certainly run into the issue of needing an intelligent proxy in front of a bunch of slow web services, which so far HAProxy has managed to solve, but if I need a more intelligent frontend to the services of our application Katamari is something I certainly intend to check out, at least as soon as it's open sourced.

It was pretty cool to check out Nitrogen as well, which managed to do some pretty impressive and Erlangy things with web services. It's also the best usage of Erlang's record syntax I've ever seen. The demo involved pushing the presently active slide to all the people viewing the demo in their web browser, which I've seen done in Ruby before but I suspect the Erlang version is far less hackish :)

Twitter seemed to be haunting me throughout the conference. I hacked together a script which posted Twitter updates with the #erlangfactory tag to the #erlangfactory channel on freenode. Twitter helped me link up with the people who found the lost AC adapter to my MacBook and also helped me figure out who's Apple remote I accidently snagged after my talk. It was really handy.

However, people also talked to me about my blog post regarding Twitter's switch to Scala, and we all talked about how simple the core problem Twitter is trying to solve actually is and how easily it could be (and has been) implemented in Erlang. The Twitter people did their "due diligence" and decided that Scala was best suited to their needs. Whatever guys, I think Erlang could've help you out considerably. I recall a post on a site like HighScalability (although I can't find it) where Twitter said they evaluated Erlang circa 2007 and the lone developer pitching it couldn't get his prototype to work, and so I guess Erlang isn't applicable there, or something? When the Scala prototype works I guess that's what they use...

The general concensus discussing the matter with various Erlangy people was that Twitter was little more than a massive pub/sub message queue which delivers messages to a backing database (from whence their RoR webapp serves their site), which isn't really that hard of a problem and the kind of problem which is almost ideally suited to Erlang. Yet the Twitter people eschewed Erlang for Scala and on a totally unrelated matter run a perpetually unstable albeit massively hyped service. Way to go, guys.

There's quite a few talks I missed which I want to talk out, particularly a talk given on Haskell. I certainly hope the videos (including mine) get posted soon.

Despite ample conversation about CouchDB there was virtually no discussion of the sort that has Ruby drama queens panties in a bunch, something about a presentation involving scantily clad girls that offended people. So awesome to be in a community that's still small enough to value code over anything else :)

All said it was a great conference and one I look forward to attending (or speaking at again) next year.

Why I don't like Scala

Scala is a hybrid functional/imperative object/actor language for the Java Virtual Machine which has recently gained notoriety by Twitter selecting it as the basis of their future development. My language Reia is also a hybrid functional/imperative object/actor language. You may think given these similarities I would like Scala... but I don't.

I originally tried Scala back in 2007, shortly after I started becoming proficient in Erlang. I was extremely interested in the actor model at the time, and Scala provides an actor model implementation. Aside from Erlang, it was one of the only languages besides Scheme and Io I had discovered which had attempted an actor model implementation, and Scala's actor support seemed heavily influenced by Erlang (even at the syntactic level). I was initially excited about Scala but slowly grew discontented with it.

At first I enjoyed being able to use objects within the sequential portions of my code and actors for the concurrent parts. This is an idea I would carry over into Ruby with Revactor library, an actor model implementation I created for Ruby 1.9. Revactor let you write sequential code using normal Ruby objects, while using actors for concurrency. Like Scala, Revactor was heavily influenced by Erlang, to the point that I had created an API almost virtually identical translation of Erlang's actor API to Ruby as MenTaLguY had in his actor library called Omnibus. MenTaLguY is perhaps the most concurrency-aware Ruby developer I have ever met, so I felt I may be on to something.

However, rather quickly I discovered something about using actors and objects in the same program: there was considerable overlap in what actors and objects do. More and more I found myself trying to reinvent objects with actors. I also began to realize that Scala was running into this problem as well. There were some particularly egregious cases. What follows is the most insidious one.

The WTF Operator

One of the most common patterns in actor-based programs is the Remote Procedure Call or RPC. As the actor protocol is asynchronous, RPCs provide synchronous calls in the form of two asynchronous messages, a request and a response.

RPCs are extremely prevalent in actor-based programming, to the point that Joe Armstrong, creator of Erlang, says:

95% of the time standard synchronous RPCs will work - but not all the
time, that's why it's nice to be able to open up things and muck around at the
message passing level.

Seeing RPCs as exceedingly common, the creators of Scala created an operator for it: "!?"

WTF?! While it's easy to poke fun at an operator that resembles an interrobang, the duplicated semantics of this operator are what I dislike. To illustrate the point, let me show you some Scala code:

response = receiver !? request

and the equivalent code in Reia:

response = receiver.request()

Reia can use the standard method invocation syntax because in Reia, all objects are actors. Scala takes an "everything is an object" approach, with actors being an additional entity which duplicates some, but not all, of the functions of objects. In Scala, actors are objects, whereas in Reia objects are actors.

Scala's object model borrows heavily from Java, which is in turn largely inspired by C++. In this model, objects are effectively just states, and method calls (a.k.a. "sending a message to an object") are little more than function calls which act upon and mutate those states.

Scala also implements the actor model, which is in turn inspired by Smalltalk and its messaging-based approach to object orientation. The result is a language which straddles two worlds: objects acted upon by function calls, and actors which are acted upon by messages.

Furthermore, Scala's actors fall prey Clojure creator Rich Hickey's concerns about actor-based languages:

It reduces your flexibility in modeling - this is a world in which everyone sits in a windowless room and communicates only by mail. Programs are decomposed as piles of blocking switch statements. You can only handle messages you anticipated receiving. Coordinating activities involving multiple actors is very difficult. You can't observe anything without its cooperation/coordination - making ad-hoc reporting or analysis impossible, instead forcing every actor to participate in each protocol.

Reia offers a solution to this problem with its objects-as-actors approach: all actor-objects speak a common protocol, the "Object" protocol, and above that, they speak whatever methods belong to their class. Objects implicitly participate in the same actor protocol, because they all inherit the same behavior from their common ancestor.

Scala's actors... well... if you !? them a message they aren't explicitly hardcoded to understand (and yes nitpickers, common behaviors can be abstracted into functions) they will ?! at your message and ignore it.

Two kinds of actors?

One of the biggest strengths of the Erlang VM is its approach to lightweight concurrency. The Erlang VM was designed from the ground up so you don't have to be afraid of creating a large number of Erlang processes (i.e. actors). Unlike the JVM, the Erlang VM is stackless and therefore much better at lightweight concurrency. Erlang's VM also has advanced mechanisms for load balancing its lightweight processes across CPU cores. The result is a system which lets you create a large number of actors, relying on the Erlang virtual machine to load balance them across all the available CPU cores for you.

Because the JVM isn't stackless and uses native threads as its concurrency mechanism, Scala couldn't implement Erlang-style lightweight concurrency, and instead compromises by implementing two types of actors. One type of actor is based on native threads. However, native threads are substantially heavier than a lightweight Erlang process, limiting the number of thread-based actors you can create in Scala as compared to Erlang. To address this limitation, Scala implements its own form of lightweight concurrency in the form of event-based actors. Event-based actors do not have all the functionality of thread-based actors but do not incur the penalty of needing to be backed by a native thread.

Should you use a thread-based actor or an event-based actor in your Scala program? This is a case of implementation details (namely the JVM's lack of lightweight concurrency) creeping out into the language design. Projects like Kilim are trying to address lightweight concurrency on the JVM, and hopefully Scala will be able to leverage such a project in the future as the basis of its actor model and get rid of the threaded/evented gap, but for now Scala makes you choose.

Scala leaves you with three similar, overlapping constructs to choose from when modeling state, identity, and concurrency in programs: objects, event-based actors, and thread-based actors. Which should you choose?

Reia provides both objects and actors, but actors are there for edge cases and intended to be used almost exclusively by advanced programmers. Reia introduces a number of asynchronous concepts into its object model, and for that reason objects alone should suffice for most programmers, even when writing concurrent programs.

Advantages of Scala's approach

Reia's approach comes with a number of disadvantages, despite the theoretical benefits I've outlined above. For starters, Scala is a language built on the JVM, which is arguably the best language virtual machine available. Scala's sequential performance tops Erlang even if its performance in concurrent benchmarks typically lags behind.

Reia's main disadvantage is that its object model does not work like any other language in existence, unless you consider Erlang's approach an "object model". Objects, being a shared-nothing, individually garbage collected Erlang process, are much heavier (approximately 300 machine words at minimum) than objects in your typical object oriented language (where I hear some runtimes offer zero overhead objects, or something). Your typical "throw objects at the problem" programmer is going to build a system, and the more objects that are involved the more error prone it's going to become. Reia is a language which asks you to sit back for a second and ponder what can be modeled as possibly nested structures of lists, tuples, and maps instead of objects.

Reia does not allow cyclical call graphs, meaning that an object receiving a call cannot call another object earlier in the call graph. Instead, objects deeper in the call graph must interact with any previously called objects asynchronously. If your head is spinning now I don't blame you, and if you do understand what I'm talking about I cannot offer any solutions. Reia's call graphs must be acyclic, and I have no suggestions to potential Reia developers as to how to avoid this problem, besides being aware of the call flow and ensuring that all "back-calls" are done asynchronously. Cyclic call graphs result in a deadlock, one which can presently only be detected through timeouts, and remain a terrible pathological case. I really wish I could offer a better solution and I am eager if anyone can help me find a solution. This is far and away the biggest problem I have ever been faced with in Reia's object model and I am sad to say I do not have a good solution.

All that said, Scala's solution is so beautifully familiar! It works with the traditional OOP semantics of C++ which were carried over into Java, and this is what most OOP programmers are familiar with. I sometimes worry that the approach to OOP I am advocating in Reia will be rejected by developers who are familiar with the C++-inspired model, because Reia's approach is more complex and more confusing.

Furthermore, Scala's object model is not only familiar, it's extremely well-studied and well-optimized. The JVM provides immense capability to inline method calls, which means calls which span multiple objects can be condensed down to a single function call. This is because the Smalltalk-inspired illusion that these objects are receiving and sending messages is completely suspended, and objects are treated in C++-style as mere chunks of state, thus an inlined method call can act on many of them at once as if they were simple chunks of state. In Reia, all objects are concurrent, share no state, and can only communicate with messages. Inlining calls across objects is thoroughly impossible since sending messages in Reia is not some theoretical construct, it's what really happens and cannot simply be abstracted away into a function call which mutates the state of multiple objects. Each object is its own world and synchronizes with the outside by talking by sending other objects messages and waiting for their responses (or perhaps just sending messages to other objects then forgetting about it and moving on).

So why even bother?

Why even bother pursuing Reia's approach then, if it's more complex and slow? I think its theoretical purity offers many advantages. Synchronizing concurrency through the object model itself abstracts away a lot of complexity. Traditional object usage patterns in Reia (aside from cyclical call graphs) have traditional object behavior, but when necessary, objects can easily be made concurrent by using them asynchronously. Because of this, the programmer isn't burdened with deciding what parts of the system need to be sequential and what parts concurrent ahead of time. They don't need to rip out their obj.method() calls and replace them with WTFs!? when they need some part of the system they didn't initially anticipate to be concurrent. Programmers shouldn't even need to use actors directly unless they're implementing certain actor-specific behaviors, like FSMs (the 5% case Joe Armstrong was talking about).

Why build objects on actors?

Objects aren't something I really have a concrete, logical defense of, as opposed to a functional approach. To each their own is all I can say. Object oriented programming is something of a cargo cult... its enthusiasts give defenses which often apply equally to functional programming, especially functional programming with the actor model as in Erlang.

My belief is that if concurrent programming can be mostly abstracted to the level of objects themselves, a lot more people will be able to understand it. People understand objects and the separation of concerns which is supposedly implicit in their identities, but present thread-based approaches to concurrency just toss that out the window. The same problem is present in Scala: layering the actor model on top of an object system means you end up with a confusing upper layer which works kind of like objects, but not quite, and runs concurrently, while objects don't. When you want to change part of the system from an object into an actor, you have to rewrite it into actor syntax and change all your lovely dots into !?s?!

Building the object system on top of the actor model itself means objects become the concurrency primitive. There is no divorce between objects and actors. There is no divorce between objects, thread-based actors, and event-based actors as in Scala. In Reia objects are actors, speak a common protocol, and can handle the most common use cases.

When objects are actors, I think everything becomes a lot simpler.

Twitter: a followup

My last post about Twitter unsurprisingly drew the responses of the super duper microblogging interconnected developers at Twitter.

I'm still confused as to why Starling ever came about in the first place, but their reasoning for the move to Scala is a lot clearer now. They want a statically typed language to manage a larger codebase, and a faster language so they can grow to handle more load. It would seem their entire system is both stateful and high throughput, for which they need a high performance disk logged message queue.

However, for the one stateless part of their system, the webapp, I guess they're going to stick with Ruby, and the good old Matz Ruby Interpreter. JRuby won't run their app for some reason.

That's at least what I digested from the Twitter employees. Thanks for your replies.