Wednesday, November 25, 2009

The new Reia: now without rainbow-farting Domo-kuns

Well, what can be said: the buzz about Reia has died down, and several people are confused about the state of the project. Is it going forward? Is it being actively developed? What's this Peridot thing?

I recently attended RubyConf and was asked by many people about the state of Reia. My answer was always "wow, I should really blog about that" to avoid repeating the same things to people over and over (after all, the Ruby community emphasizes DRY, right?). Well, here is that blog.

The State of Reia: Forget Peridot

To put it bluntly, Reia is undergoing a ground-up rewrite. This does not mean I am abandoning the previous codebase. Far from it. Instead I am rewriting the parts of it which are problematic, and pulling in code from the previous implementation where it makes sense.

When I talk to various people who are implementing languages for fun, it seems most people are interested in producing one-off experiments with little attention to developing them into "real" languages someday that might actually be used in production by a lot of people. This certainly makes sense, and that's how I started with Reia. However, buzz grew, and so did my investment in the project. Every indicator I've been given has shown me that Reia is something a lot of people are potentially interested in, so half-assing it isn't going to cut it.

The new Reia roadmap calls for reaching complete feature parity with Erlang with as minimal an implementation as possible, then making it rock solid. At this point, while Reia will lack many of the "dream features" of the previous implementation, it will be generally usable as an alternative to Erlang. Once new language features become available existing programs can be modified to make use of them. After all this is done syntactic sugar can be added, and finally, the concurrent object model.

Initially I had thought of splitting off these additional features into a separate language, which I called "Peridot", but after long and careful consideration, this doesn't make sense. The new Reia will start as an alternative syntax for Erlang (with destructive assignment) but will grow to include all of the features I had originally envisioned.

What Was Wrong with the Previous Implementation?

Why rebuild Reia from the ground up? Can't the existing implementation be salvaged and molded into something usable?

There are a number of problems with the existing implementation. Some stem from my lack of knowledge of Erlang itself when I started. Some of them stem from my lack of knowledge of best practices when writing a language compiler in Erlang. Others stem from the organic way the language evolved. But above everything else, the problems stem from one feature I deemed absolutely necessary: eval.

I like eval lots! If nothing else, it exists for one reason: so Reia can have an interactive shell (i.e. a read-eval-print loop, a.k.a. REPL). I spend lots of my time hacking on Ruby code by interacting with it from irb, the interactive Ruby interpreter. I have a very hard time working with languages that do not provide some form of interactive interpreter nowadays.

The biggest problem with implementing eval is that you have to write your own implementation for your language. In the previous version of Reia I tried to sidestep that by using erl_eval, the Erlang metacircular interpreter, as my eval implementation. Unfortunately, to facilitate this, I ended up implementing the entire code loading process in a way which shoved everything to erl_eval. The result ended up looking something like this:

the previous wonky ass Reia compiler

When code entered the system, it was first parsed and converted to Erlang by reia_compiler (the Domo-kuns). For module and class declarations, the code was compiled down to Erlang bytecode (the rainbow farts) which were in turn injected into the toplevel Erlang AST. In other words, the toplevel scope of any Reia file was never compiled, but simply stored as expressions, and where module/class declarations existed, instructions to load the compiled module (which itself was injected directly into the AST) were issued. This provides somewhat Ruby-like semantics for module/class declarations: they're "loaded" at the time they're declared.

The resulting Erlang AST, complete with compiled class/module fragments, was then shoved through the Erlang metacircular interpreter, erl_eval (seen in the diagram as the tornado). As you might guess, compared to compiled Erlang the metacircular interpreter is reaaaaaaaaally slow.

Once everything was said and done, the resulting class/modules were loaded into the Erlang code server, pictured here as a hungry Joe Armstrong going *nom* *nom* *nom*.

Making Reia Faster


As you may guess, an as-yet-unstated goal of this rewrite is to improve the speed of code-loading. Previously, Reia could not have a large standard library, because it took so long to load code. Furthermore, implementing a mechanism for caching compiled bytecode was impossible due to the API structure.

The new code-loading API directly compiles everything, including any code which is eval'd. This not only makes everything significantly faster but also facilitates the possibility of caching and also various bugs surrounding the eval implementation. From what I've gathered elsewhere, most compiled languages generally ditch any form of metacircular interpreter and implement eval by compiling temporary modules.

Doing this in Erlang is hard, because certain expressions in Erlang create things which exist beyond when code is being evaluated, namely processes and lambdas (a.k.a. funs). This was a vexing problem to me for quite some time, but after talking with Robert "Hello Robert" Virding, one of Erlang's creators, I believe I've come upon a workable solution, even if it's a bit of a hack.

Reia will implement its own "garbage collector" process for eval'd code, which periodically checks if all the lambdas/processes created by a particular eval call are no longer in use. If so, it will remove the temporary module. If not, then it will continue to wait. It is not the greatest solution in the world, but it will get the job done.

This means no Reia code will ever go through erl_eval. Everything is compiled. This will make code loading of all sorts, and eval, much faster. There are no longer any rainbow farting Domo-kuns.

What About Neotoma?

When I originally began my rewrite of Reia, I was attempting to redo the parser using Neotoma, a Parsing Expression Grammar (PEG) tool for Erlang, similar to Ruby's Treetop.

I eventually shied away. This had little to do with Neotoma itself, but my own inability to understand PEGs, and the fact that my own inability to understand them was a roadblock in continued development. Because of this, I switched back to more familiar tools: leex and yecc, the Erlang equivalents of lex and yacc.

Neotoma has come a long way and become better than ever. I am still considering using it. I think it would be a great tool for solving a lot of problems that aren't presently solved, like handling Reia expressions within interpolated strings. This is something I might pursue when I am confident that development otherwise is coming along at a steady pace, but at this point, switching to Neotoma is a lower priority for me than developing a rock-solid core language.

Where's the Code?

If you're interested in checking out the latest Reia codebase, it's available on this branch on Github:

http://github.com/tarcieri/reia/tree/minimalist

If you're looking at master, and wondering why it hasn't been touched in months, it's because I'm hacking on the new branch, not the previous implementation.

The new implementation is not generally usable. I am still working out the nasty details of implementing a compiled eval, as well as implementing cleaner compiler internals.

But hey, if you're interested in Reia, check it out and let me know what you think.

30 comments:

Roger Pack said...

glad to hear about reia. I know the scala guys had some difficulty getting eval to work in a compiled language, perhaps some lessons could be learned there?
Thanks again.
-r

Anonymous said...
This comment has been removed by a blog administrator.
Unknown said...

cctv camera for home
Pestveda | Pest Control
Termite control
Surveillancekart | CCTV System
cctv installation services
best home security camera system

Unknown said...

cctv camera dealers in delhi
cp plus cctv camera online
hikvision camera online
cctv camera installation services in delhi
cctv camera installation services in gurugram
cctv camera installation services in gurgaon
cctv camera installation services in noida
cctv camera security systems
cctv installation services
security systems for home

Prateek Barn said...

I can only express a word of thanks! Nothing else. Because with the content on this blog you can add knowledge. Thank you very much for sharing this information. AvriqAvriq India

mariablack said...

Pc Optimization
Windows Installation
Data Recovery

shabnam praveen said...

Call girls in Kolkata
Call girls in Chandigarh
Call girls in Chandigarh
Call girls in Gurgaon
Call girls in Chandigarh
Call girls in Chandigarh

shabnam praveen said...

Call girls in Lucknow
Call girls in Guwahati
Call girls in Mumbai
Call girls in Jaipur
Call girls in Jaipur
Call girls in Jaipur
Call girls in Bangalore

Sania Kumari said...


Call Girls in Mumbai
Call Girls in Puri
Call girls in Cuttack

Call girls in Bhubaneswar
Call girls in Gurgaon
Call girls in Gurgaon

Sania Kumari said...

Call girls in Jaipur
Call girls in Jaipur
Call girls in Jaipur

Call girls in Jaipur
Call Girls in Mumbai
Call Girls in Bangalore

Archana The best Call Girl said...

********************************************************************************************************
Hi friends ! I'm Archana Kumari. I like to have bed relationship with different persons rather than with a single man for the whole of life. Not only is this my notion , but also is this fact that the secret organs are made for enjoyment. Please follow me on the links given bellow.

|| House wife escorts in Kolkata||Call girls in Kolkata||
||Air hostess Escorts in Kolkata||
||Mumbai Call girl|| Kolkata Call girls||
||VIP Call girls in Kolkata||Escorts in Kolkata||
||Escort services in Kolkata||Escorts in Kolkata

If you are interested in me and my ideology ,then you are invited to my bed. Welcome!
*****************************************************************************************************

Archana The best Call Girl said...

Call Girls in Mumbai
Call Girls in Puri
Call girls in Cuttack

Call girls in Bhubaneswar
Call girls in Gurgaon
Call girls in Gurgaon

Call girls in Jaipur
Call girls in Jaipur
Call girls in Jaipur

Archana The best Call Girl said...

Call girls in Jaipur
Call Girls in Mumbai
Call Girls in Bangalore

Archana The best Call Girl said...


Call Girl in Mumbai
Call Girl in Puri
Call girl in Cuttack

Call girl in Bhubaneswar
Call girl in Gurgaon
Call girl in Gurgaon

Call girls in Jaipur
Call girl in Jaipur
Call girl in Jaipur

Archana The best Call Girl said...

Escorts in Mumbai
Escorts in Puri
Escorts in Cuttack

Escorts in Bhubaneswar
Escorts in Gurgaon
Escorts in Gurgaon

Escorts in Jaipur
Escorts in Jaipur
Escorts in Jaipur

Archana The best Call Girl said...

Escort services in Mumbai
Escort services in Puri
Escort services in Cuttack

Escort services in Bhubaneswar
Escort services in Gurgaon
Escort services in Gurgaon

Escort services in Jaipur
Escort services in Jaipur
Escort services in Jaipur

Archana The best Call Girl said...

********************************************************************************************************
Hi friends ! I'm Archana Kumari. I like to have bed relationship with different persons rather than with a single man for the whole of life. Not only is this my notion , but also is this fact that the secret organs are made for enjoyment. Please follow me on the links given bellow.

Call girls in Jaipur
Call Girls in Mumbai
Call Girls in Bangalore


Call Girl in Mumbai
Call Girl in Puri
Call girl in Cuttack

Call girl in Bhubaneswar
Call girl in Gurgaon
Call girl in Gurgaon

If you are interested in me and my ideology ,then you are invited to my bed. Just go ahead !
***********************************************************************************************************

Archana The best Call Girl said...

********************************************************************************************************
Hi friends ! I'm Archana Kumari. I like to have bed relationship with different persons rather than with a single man for the whole of life. Not only is this my notion , but also is this fact that the secret organs are made for enjoyment. Please follow me on the links given bellow.

Call girls in Jaipur
Call girl in Jaipur
Call girl in Jaipur

Call girl in Jaipur
Call Girl in Mumbai
Call Girl in Bangalore

Escorts in Mumbai
Escorts in Puri
Escorts in Cuttack

If you are interested in me and my ideology ,then you are invited to my bed. Just go ahead !
***********************************************************************************************************

Nannie Co Pam said...

IEEE Project Domain management in software engineering is distinct from traditional project deveopment in that software projects have a unique lifecycle process that requires multiple rounds of testing, updating, and faculty feedback. A IEEE Domain project Final Year Projects for CSE system development life cycle is essentially a phased project model that defines the organizational constraints of a large-scale systems project. The methods used in a IEEE DOmain Project systems development life cycle strategy Project Centers in Chennai For CSE provide clearly defined phases of work to plan, design, test, deploy, and maintain information systems.


This is enough for me. I want to write software that anyone can use, and virtually everyone who has an internet connected device with a screen can use apps written in JavaScript. JavaScript Training in Chennai JavaScript was used for little more than mouse hover animations and little calculations to make static websites feel more interactive. Let’s assume 90% of all websites using JavaScript use it in a trivial way. That still leaves 150 million substantial JavaScript Training in Chennai JavaScript applications.

Anjali Arya said...



Most Indian Men needs a young lady who is hot and in addition clever at the same time. Be that as it may, shockingly most elite girls are exhausting. Their service satisfaction level is so basic, yet can’t strike a discussion making their arrangement lacking something. As of now out Massage Parlor in Delhi They are a great listener, can chat on you on any point and give you such a delicate treatment. Check our other services...
Escorts in Lucknow
Call Girls in Lucknow
Escorts Service in Delhi
Call Girls in Aerocity




robbyjaksan said...

We are a free specialist co-op for all the equipment and programming related issues.We are an outsider organization giving specialized help in programming, PCs, PCs, printers and so forth.
||HP Printer Support Toll Free Phone Number
||hp printer tech support phone number
||hp printer customer service number
||Samsung Printer Support Toll Free Phone Number
||samsung printer tech support phone number
||samsung printer customer service number
||Brother Printer Support Toll Free Phone Number
||brother printer tech support phone number
||brother printer customer service number
||Lexmark Printer Support Toll Free Phone Number
||lexmark printer tech support phone number
||lexmark printer customer service number
||Dell Printer Support Toll Free Phone Number
||dell printer tech support phone number
||dell printer customer service number
||Epson Printer Support Toll Free Phone Number
||epson printer tech support phone number
||epson printer customer service number
||canon Printer Support Toll Free Phone Number
||canon printer tech support phone number
||Canon printer customer service number

salma said...



شركة تنظيف خزانات بالمدينه المنوره

customized gifts for him said...

best baby gifts 2020
gifts for her valentines
valentine gifts for him romantic
best christmas gifts for mom
best gifts for dad 2020
best baby gifts 2020
rustic pillows for bed
decorative pillows for sofa
geometric pillow target
linen cushion covers uk

rima said...

siliguri escorts
siliguri escort
siliguri female escorts
siliguri escort service



siliguri escorts
siliguri escort
siliguri female escorts
siliguri escort service



siliguri escorts
siliguri escort
siliguri female escorts
siliguri escort service



siliguri escorts
siliguri escort
siliguri female escorts
siliguri escort service



siliguri escorts
siliguri escort
siliguri female escorts
siliguri escort service

No #1 Gurgaon Escorts Service Agency-Gurugram said...




We are here to serve you hot Call Girls in Gurgaon and Escorts in GurgaonOur Girls are fresh, friebdly, educated and interested in pleasure with our clients. Check our other Services...
Escorts in Gurgaon
Escorts in Gurgaon
Escorts in Gurgaon
Escorts in Gurgaon
Escorts Service in Gurgaon

Kelly Cain said...

Interpages

Guest Blogger

Guest Blogging Site

Guest Blogging Website

Guest Posting Site

Best Boarding School in Dehradun said...

Nice Blog. Thanks for sharing.

Top Residential School in Dehradun

How Can Working Parents Help Their Children’s Education In School ?

LL said...

http://www.unlimitednovelty.com/2009/04/why-i-dont-like-scala.html

LL said...


There is a new mod that allows multiple new functionalities that were not available originally on the default version.

https://flosshype.com/fmwhatsapp/

Himah Hamih said...

Appreciating the persistence you put into your blog and detailed information you provided.
Thanks for sharing! Glad to read your posts. Thumbs up👍!!

online internship
online internships
watch internship online
online internship for students
the internship online
online internship with certificate
online internship certificate
python online internship
data science online internship