Reia: now with "Magic Rebinding"

In Ruby, thanks to its first class syntax for "hashes" and mutable state, it's quite easy to do:

h = {}
h[key] = value

The equivalent code in Erlang is noisier, thanks to immutable state and single assignment:

Dict1 = dict:new(),
Dict2 = dict:store(Key, Value, Dict1).

Since Reia lacks mutable state, it never before had syntax as simple as Ruby's... but now it does!

I have been trying to hold off on adding syntactic sugar like this to my new "minimalist" branch of Reia. However, this is a feature I meant to add to the old implementation, and tried to retrofit it in long after the implementation had grown quite complex, never managing to succeed. I decided to tackle it now, and I'm happy to announce that it works! Furthermore, it can be used in complex pattern matching expressions:

>> m = {}
=> {}
>> (m[:foo], m[:bar], m[:baz]) = (1,2,3)
=> (1,2,3)
>> m
=> {:bar=>2,:baz=>3,:foo=>1}

So what is going on here exactly? Reia is an immutable state language, so surely I'm not mutating the value that "m" references.

In these cases, Reia is "altering" the local variable binding. Each time you change a member of a map ("hash" for you Ruby folks, "dict" for you Erlang folks), a new version of that map is calculated, then bound to "m". Behind the scenes, the Reia compiler is translating these calls into destructive assignments.

Maps, Tuples, and even Lists now support assignments in this way (although Lists only for consistency's sake... I hate to think of people actually setting values in lists by index). Tuples and Lists even support Ruby-style negative indexing:

>> t = (1,2,3)
=> (1,2,3)
>> t[-1] = 42
=> 42
>> t
=> (1,2,42)

I plan on eventually exposing this functionality to user-defined types as well, in the form of "bang methods" on immutable objects. Users of Ruby are likely familiar with them:

>> arr = [1,2,3]
=> [1,2,3]
>> arr.reverse; arr
=> [1,2,3]
>> arr.reverse!; arr
=> [3,2,1]

Here you can see that calling the "reverse" method on an array (without the !) does not modify the original array in-place. Instead, it returns a new array in reverse order. The complimentary "reverse!" method performs an in-place modification of the array.

The "method!" idiom in Ruby is generally used to indicate methods that modify their receivers as opposed to versions which do not. However this is not a solid requirement, and "!" is often added to any methods considered "dangerous". There's no specific meaning to putting "!" on the end of a method and certainly nothing Ruby does differently at the language level.

In Reia, "bang methods" will be a first class language construct, and will always rebind the receiver with the return value of the method. This will provide a simple way to allow "in place" modifications of immutable objects, by having "bang methods" create and return a new immutable object.

It's the best of both worlds: the ease of use that comes from mutable state, with the concurrency benefits of being completely immutable.

Reia: now fully compiled (and sometimes JITed)

One of the most frequent questions I get about Reia is its execution model. Is it a simple parse tree walker? Is it interpreted? Is it compiled?

The old branch of Reia made extensive use of the Erlang metacircular interpreter, which is a parse tree walker. Any code within models or classes, however, was compiled to Erlang bytecode. Reia did autodetect HiPE (the Erlang native code compiler/JIT) when available, and would use it when compiling modules/classes.

The new branch of Reia does not make use of the Erlang metacircular interpreter at all. Instead all code, including any code which is eval'd, is translated into Erlang then compiled by the Erlang compiler. This means Reia is 100% fully compiled, and will compile to native code when your Erlang interpreter supports it.

"AutoHiPE" is off by default for now, if only because HiPE has a slightly greater startup time than the normal BEAM interpreter.

HiPE has some additional problems as well. It has limited platform support. x86-64 is not one of the supported platforms. Given that BEAM is fundamentally a register machine you think it'd be ripe for compilation to native code via something like LLVM.

But for now, enjoy native code compilation on the platforms that support it by passing {autohipe, true} as a compiler option.

The new Reia: now without rainbow-farting Domo-kuns

Well, what can be said: the buzz about Reia has died down, and several people are confused about the state of the project. Is it going forward? Is it being actively developed? What's this Peridot thing?

I recently attended RubyConf and was asked by many people about the state of Reia. My answer was always "wow, I should really blog about that" to avoid repeating the same things to people over and over (after all, the Ruby community emphasizes DRY, right?). Well, here is that blog.

The State of Reia: Forget Peridot

To put it bluntly, Reia is undergoing a ground-up rewrite. This does not mean I am abandoning the previous codebase. Far from it. Instead I am rewriting the parts of it which are problematic, and pulling in code from the previous implementation where it makes sense.

When I talk to various people who are implementing languages for fun, it seems most people are interested in producing one-off experiments with little attention to developing them into "real" languages someday that might actually be used in production by a lot of people. This certainly makes sense, and that's how I started with Reia. However, buzz grew, and so did my investment in the project. Every indicator I've been given has shown me that Reia is something a lot of people are potentially interested in, so half-assing it isn't going to cut it.

The new Reia roadmap calls for reaching complete feature parity with Erlang with as minimal an implementation as possible, then making it rock solid. At this point, while Reia will lack many of the "dream features" of the previous implementation, it will be generally usable as an alternative to Erlang. Once new language features become available existing programs can be modified to make use of them. After all this is done syntactic sugar can be added, and finally, the concurrent object model.

Initially I had thought of splitting off these additional features into a separate language, which I called "Peridot", but after long and careful consideration, this doesn't make sense. The new Reia will start as an alternative syntax for Erlang (with destructive assignment) but will grow to include all of the features I had originally envisioned.

What Was Wrong with the Previous Implementation?

Why rebuild Reia from the ground up? Can't the existing implementation be salvaged and molded into something usable?

There are a number of problems with the existing implementation. Some stem from my lack of knowledge of Erlang itself when I started. Some of them stem from my lack of knowledge of best practices when writing a language compiler in Erlang. Others stem from the organic way the language evolved. But above everything else, the problems stem from one feature I deemed absolutely necessary: eval.

I like eval lots! If nothing else, it exists for one reason: so Reia can have an interactive shell (i.e. a read-eval-print loop, a.k.a. REPL). I spend lots of my time hacking on Ruby code by interacting with it from irb, the interactive Ruby interpreter. I have a very hard time working with languages that do not provide some form of interactive interpreter nowadays.

The biggest problem with implementing eval is that you have to write your own implementation for your language. In the previous version of Reia I tried to sidestep that by using erl_eval, the Erlang metacircular interpreter, as my eval implementation. Unfortunately, to facilitate this, I ended up implementing the entire code loading process in a way which shoved everything to erl_eval. The result ended up looking something like this:



When code entered the system, it was first parsed and converted to Erlang by reia_compiler (the Domo-kuns). For module and class declarations, the code was compiled down to Erlang bytecode (the rainbow farts) which were in turn injected into the toplevel Erlang AST. In other words, the toplevel scope of any Reia file was never compiled, but simply stored as expressions, and where module/class declarations existed, instructions to load the compiled module (which itself was injected directly into the AST) were issued. This provides somewhat Ruby-like semantics for module/class declarations: they're "loaded" at the time they're declared.

The resulting Erlang AST, complete with compiled class/module fragments, was then shoved through the Erlang metacircular interpreter, erl_eval (seen in the diagram as the tornado). As you might guess, compared to compiled Erlang the metacircular interpreter is reaaaaaaaaally slow.

Once everything was said and done, the resulting class/modules were loaded into the Erlang code server, pictured here as a hungry Joe Armstrong going *nom* *nom* *nom*.

Making Reia Faster

As you may guess, an as-yet-unstated goal of this rewrite is to improve the speed of code-loading. Previously, Reia could not have a large standard library, because it took so long to load code. Furthermore, implementing a mechanism for caching compiled bytecode was impossible due to the API structure.

The new code-loading API directly compiles everything, including any code which is eval'd. This not only makes everything significantly faster but also facilitates the possibility of caching and also various bugs surrounding the eval implementation. From what I've gathered elsewhere, most compiled languages generally ditch any form of metacircular interpreter and implement eval by compiling temporary modules.

Doing this in Erlang is hard, because certain expressions in Erlang create things which exist beyond when code is being evaluated, namely processes and lambdas (a.k.a. funs). This was a vexing problem to me for quite some time, but after talking with Robert "Hello Robert" Virding, one of Erlang's creators, I believe I've come upon a workable solution, even if it's a bit of a hack.

Reia will implement its own "garbage collector" process for eval'd code, which periodically checks if all the lambdas/processes created by a particular eval call are no longer in use. If so, it will remove the temporary module. If not, then it will continue to wait. It is not the greatest solution in the world, but it will get the job done.

This means no Reia code will ever go through erl_eval. Everything is compiled. This will make code loading of all sorts, and eval, much faster. There are no longer any rainbow farting Domo-kuns.

What About Neotoma?

When I originally began my rewrite of Reia, I was attempting to redo the parser using Neotoma, a Parsing Expression Grammar (PEG) tool for Erlang, similar to Ruby's Treetop.

I eventually shied away. This had little to do with Neotoma itself, but my own inability to understand PEGs, and the fact that my own inability to understand them was a roadblock in continued development. Because of this, I switched back to more familiar tools: leex and yecc, the Erlang equivalents of lex and yacc.

Neotoma has come a long way and become better than ever. I am still considering using it. I think it would be a great tool for solving a lot of problems that aren't presently solved, like handling Reia expressions within interpolated strings. This is something I might pursue when I am confident that development otherwise is coming along at a steady pace, but at this point, switching to Neotoma is a lower priority for me than developing a rock-solid core language.

Where's the Code?

If you're interested in checking out the latest Reia codebase, it's available on this branch on Github:

http://github.com/tarcieri/reia/tree/minimalist

If you're looking at master, and wondering why it hasn't been touched in months, it's because I'm hacking on the new branch, not the previous implementation.

The new implementation is not generally usable. I am still working out the nasty details of implementing a compiled eval, as well as implementing cleaner compiler internals.

But hey, if you're interested in Reia, check it out and let me know what you think.

RIP "FatELF"

I remember installing Solaris onto a 64-bit UltraSPARC many years ago. When I did it, lo and behold, 32-bit and 64-bit versions of all libraries were installed side-by-side. I could still run the many proprietary 32-bit Solaris apps needed by my coworkers, but we could compile memory-intensive scientific models as 64-bit no problem.

Flash forward to today, and Windows and OS X have both figured this out. Windows 7 uses a similar solution to Solaris, installing both 32-bit and 64-bit versions of all libraries, and having a separate "x86" folder for all programs. OS X uses "universal binaries," which allows builds for multiple architectures to be packaged into the same binary. In either case, everything Just Works and it's great!

Off in Linux land, it's a distribution-by-distribution attempt at solutions for this problem. Some distributions have it figured out, others don't. The solutions chosen by various distributions aren't the same. On some of the more popular distributions there is no Just Works solution to running legacy 32-bit programs on a 64-bit install. Even if you are able to install a base set of 32-bit libraries, installing the many other dependencies of a 32-bit program on a 64-bit system can often be a rather challenging task.

So it was rather disappointing to read that an attempt to add OS X-like universal binary support to Linux, the so-called FatELF executable format, was discontinued today. FatELF offers something Linux desperately needs: a kernel-level solution to the 32/64-bit binary problem, the kind every distribution could automatically fall in line with. The infamous Ulrich Drepper's response to the suggestion of fat binaries for Linux was expectedly blunt:

Yes. It is a "solution" which adds costs in many, many places for a problem that doesn't exist. I don't see why people even spend a second thinking about this.

Yes, Ulrich Drepper, the 32/64-bit binary support problem on Linux is totally and completely solved. It should be no problem to install a 32-bit version of any application on any Linux system today, right? Even if it's running a 64-bit distribution? Yes, that problem does not exist.

Maybe if we all pretend the problem doesn't exist it will go away.

A new direction for Reia: Peridot?

I've always viewed Reia as what I hoped to become a spiritual successor for Ruby in the same way that Ruby was the spiritual successor of Perl. Talking with one of my roommates he pointed out that pearls and rubies are the birthstones of June and July respectively, so an interesting name for the spiritual successor of Ruby would be Peridot. If I had the chance to do it all over again, I'd probably would've named Reia as Peridot instead, but as it stands I've already built up a decent degree of mindshare around "Reia" so renaming the language probably isn't practical. I'll come back to Peridot in a bit.

When I first started using Erlang one of the first things I wanted to do was give it a Ruby-like syntax. It seems like two of the biggest reasons people starting out in Erlang reject it is because of the ugly syntax and single assignment. Lately I've been wondering if there would be value in a language which is semantically identical to Erlang except with destructive assignment and a Ruby-like syntax. Lisp Flavored Erlang has seen a lot of interest and is a much simpler undertaking than Reia because it merely provides an alternative syntax and doesn't try to add new and complex semantics to the language. Perhaps there's a niche for a "Ruby-flavored Erlang" which provides a Ruby-like syntax, destructive assignment, and possibly a bit of syntactic sugar while preserving the underlying semantics of Erlang and not trying to add anything new.

With Reia in its current form I feel like I've bit off a bit more than I can chew. Worse, for the past few months I've been stuck on a particularly difficult problem and also very busy. I feel like perhaps I've bit off a bit more than I can chew implementing Reia, and some bad decisions in the initial compiler design plus my frustration with Erlang syntax have left me wanting to rewrite the compiler as a self-hosted implementation. But I don't think Reia as a language is ready for that yet.

Another thing that has popped onto the scene is neotoma, a Parsing Expression Grammar-based parser generator for Erlang. Ever since I began implementing interopolated strings in Reia I have longed for something like this. I have hacked and kludged my way along implementing interpolating with leex and yecc, but a PEG would solve the issue completely and allow for nested interpolated strings of the sort Ruby supports. This has left me wanting to rewrite the scanner/parser for Reia using neotoma instead.

So what to do? How should I proceed? The idea of a simpler Reia has certainly been bouncing around in my head for awhile. I am seriously thinking of reinventing Reia as something more like Erlang, then continuing on to add things like an object system in a new language: Peridot. There are a number of interesting things this would allow. First, Reia would effectively provide a subset of what's in Peridot, and most Reia programs would be valid Peridot programs. In that regard, Reia would work something like RPython and would make a great bootstrap language for implementing Peridot. A reduced Reia would be much easier to get to production quality than one which incorporates all of the elaborate features I've currently tried to implement. And it would once and for all put to rest the complaints about Erlang syntax and single assignment.

I'm interested to hear what people think about this proposal.

Dear Viacom: You're Doing It Wrong

I've been very excited about the upcoming release of The Beatles: Rock Band after hearing about it earlier this week. It's the first PS3 game sold on disc I'm going to snap up since, well, Rock Band 2. Beyond the simple fact that it's Rock Band loaded with Beatles music (and I love the Beatles), I've been rather impressed by the visual style, particularly this intro video directed by Pete Candeland of Gorillaz music video fame:


Hopefully this video hasn't been flagged for a copyright violation by the time you read this post. It rules

However, I was rather surprised to see that a particular copy of the intro video was flagged for removal from YouTube due to a copyright violation. Go ahead, hit play, I dare ya:



Yes, Viacom has decided that they want to forego free advertising for their upcoming video game in order to defend their copyright. WTF? Somebody doesn't get it. Viacom, this isn't someone infringing your copyright. This is someone providing you with a viral marketing campaign for free. You are effectively telling them: "no, don't advertise our product for free. We don't want that"

I forsee a long uphill battle until old media companies finally realize that viral video distribution is actually a good thing. Eventually they'll be drug kicking and screaming to the realization that piracy is good.

Ubuntu's Jackalope not so Jaunty

I'm not one to write reviews of things like desktop Linux systems typically. In fact, any of you who read my blog for Reia should probably just stop now. But I just tried desktop Linux for the first time in two years, and my experience was anything but pleasurable.

My Background (a.k.a. chance to be a blowhard)

For the past several years OS X has been my desktop of choice. I get a beautiful, slickly animated GUI interface, seamless 3D compositing of all UI elements, nifty commercial software, and Unix underpinnings. Sure, it's proprietary, but I don't give a crap.

That said, I am no stranger to desktop Linux. My first desktop Linux experience was using FVWM on a Slackware 2.3 system back in 1995. So yes: I'm one of those Linux users that survived the transition from a.out to ELF and from libc5 to glibc. I'd try a few different distributions, next RedHat and finally Debian before becoming a Debian person. I tried RedHat 5.0, when they made the switch to glibc, and it was such an unmitigated disaster I destroyed the install CD (purchased from a store) out of rage. That is the lowest low I think I've ever seen Linux reach.

I remember trying out an early Enlightenment, which leaked memory so quickly it completely consumed the 16MB of RAM I had installed at the time. Eventually I would discover WindowMaker, which would be my standby window manager for years to come. I flitted about with OS choices after that, running FreeBSD as my primary desktop for quite some time.

Around 2001 I discovered the Synergy software which lets you seamlessly share a keyboard and mouse across two computers. From then on I loved running two computers, typically one with Windows and one with my *IX du jour. This has remained my standard configuration for quite some time.

Around 2006 I was given a new monitor for work, with a strange 1680x1050 resolution. I was running Debian at the time, ripping my hair out hand editing my X config trying to get it to work properly. I could not for the life of me figure out what was wrong, and this was after spending 5 years as a Linux sysadmin. I decided to give Ubuntu a go. I stuck in the install CD, and BEHOLD it booted straight into X and my monitor was automagically configured to the right resolution! I was awestruck.

I'd been against Gnome for years, but by now it seemed almost downright usable. I actually liked having things like desktop icons! It was pretty nifty.

However, shortly thereafter I would buy a MacBook and ditch desktop Linux entirely. I've been running an OS X/Windows Synergy setup ever since (although now I use OS X exclusively at home)

Fast Forward to Today

Amidst many of my coworkers setting up their computers to dual boot Windows and Linux, I figured I'd do the same. I thought it'd be pretty nifty to have OS X one one computer (which would remain my primary development computer) and Linux on the other.

First I installed Windows, which wasn't without its hiccups but when I was done I was left with a 30GB Windows partition and 220GB free for Linux.

I threw in the Jaunty Jackalope CD one of my coworkers had and started up the graphical installer. I missed the good old text-based Debian installer I had used for over a decade, but hey, it's the 21st century, nothing wrong with graphics, right?

I got to the partitioning step. Now, I've dealt with some pretty bad graphical partition managers in the past. Solaris's was particularly atrocious. At first glance Ubuntu's seemed fine... it recognized I had an NTFS volume and offered me the option to "Install Windows and Linux side by side". I figured this was such a common use case it would just naturally know the right thing to do.

So, I click OK and it pops up a modal dialog asking me if I want to resize my NTFS partition. WTF? Resize my NTFS partition? NO! You have 220GB of free space to work with there, why would you resize my NTFS partition? It offered three buttons I could click: one that said "Go Back" which was highlighted (and I guess the one I was supposed to choose), one that said continue/proceed or something, and the traditional "X" in the corner of the modal dialog window to close it. I clicked the latter, which was the wrong decision.

It then popped up another modal dialog window, informing me it was resizing my NTFS partition to fill the entire disk.



Seriously, are you kidding me? Closing a modal dialog window with the "X" button does NOT MEAN I WANT YOU TO PERFORM A DESTRUCTIVE ACTION. And furthermore, who installs Linux and wants it to resize their Windows partition to eat up the entire disk? Frustrated, I opened up a terminal, started gparted, shrank my Windows partition back down to 30GB, and rebooted to try again fresh.

This time I chose to manually partition my disk (although I still longed for cfdisk) and things seemed to go a little more smoothly for awhile.

After I booted into X for the first time I was prompted to install updates. I hit the "Check" button which prompted for a password. It downloaded a list of updates. I hit "Install Updates". Nothing happened. I clicked it again, some 30 times. Nothing. The button depressed, and that was it. I called a Linux-loving coworker over, he looked at it and shrugged. "I don't use the graphical updater". Yes, popping open a terminal and typing apt-get upgrade was seeming like the way to go here. I clicked "Check" again then "Install Updates". Magically it worked this time.

After all was said and done, my display was not at the right resolution. It popped up a little notification prompting me to install restricted drivers. I installed the nVidia drivers, which completed successfully, then tried to open my display preferences.

An error dialog popped up, saying that display preferences couldn't be launched, and I need to run my vendor tool instead. It said I could hit OK to do so, but when I did, it said there was an error launching the vendor tool, and I needed to run a particular command from the command line.

Are you kidding me? At this point I'm seriously confused: who is Ubuntu targeting? When was the last time Windows or OS X asked me from a modal dialog to pop open a terminal and type some shit on the command line? I tried running the command and got yet another error. Frustrated I rebooted.

Now when I try to go to the display preferences, at first I get an error saying I need to run the vendor tool, and then it launches the nVidia preferences. Only... the native resolution of my monitor is not listed. All of them are below the native resolution of my monitor.

I thought: oh well, I'll just edit my xorg.conf by hand. So I did. And I rebooted. I was still at the same resolution, and the changes I made to my xorg.conf were clobbered by something. I don't know. I reopened the file and they were completely gone. What process overwrote it? I don't have a freaking clue. I remember when Linux gave you a sense of control over what you're doing, but now I feel powerless.

Now, insult to injury: this is the exact same monitor which in 2006 I stuck an Ubuntu install CD into my computer and it launched straight into X at the native resolution. I didn't have to install any restricted drivers. I stuck in the CD and it just worked.

3 years ago Ubuntu had me excited that maybe, finally, desktop Linux was reaching a level of general usability. Now here I am, a power user, and I've run into seemingly intractable problems I can't solve.

Pre-Emptive Anti-Zealot Repellant

Did I go onto forums and ask about my problems? Did I post bugs on Ubuntu's trackers? No. Know what I did? I rebooted into Windows. And here I think I will stay. I freshly installed Windows the same day and had it up and running to my satisfaction in a few hours. Windows is working and I am happy, therefore I don't feel the need to try to help Ubuntu sort out their mess.

All I can say is, without a doubt, Ubuntu 9.04 represents a rather severe regression from my previous experience with using Ubuntu on a desktop. We still continue to run it on our servers at my place of employment and there I have few complaints. However, at this point I cannot see myself running it on a desktop, and worse I've lost my sense that desktop Linux is actually getting better over time.