Tuesday, April 5, 2011

Reia: Now with a PEG parser

I haven't blogged about Reia in so long. Mea culpa! It's about this time that people start to think "I haven't heard about Reia in awhile. It must be dead." No, Reia is very much alive my friends, and making some pretty interesting progress... I hope to be able to announce some important new features around Reia's 3rd birthday. Stay tuned for that, but first a preview of things to come...


An experimental branch of Reia is now available which uses Neotoma to generate its parser thanks to some pretty impressive contributions by Graeme Defty. Neotoma is a Parsing Expression Grammar (PEG) based parser generator for Erlang, inspired by Packrat. Neotoma's author Sean Cribbs has also released Neotoma 1.5 with contributions by Graeme Defty which uses Erlang binaries internally and is substantially faster than the previous version.

Why does a PEG matter? PEGs make short work of problems that are extremely complex to handle when you attempt to subdivide the problem of parsing into separate scanning and parsing steps, such as how a traditional lex/yacc (or in Reia's case, leex/yecc) parser would operate. If you've ever attempted to look at the source code of the Ruby parser, you'll find some strange and confusing feedback between the lexer and the parser.

What's this for? Well the foremost reason is: interpolated strings. Ruby allows you to embed arbitrary Ruby code inside of any string using the #{...} syntax. This is a really great feature and one I managed to half-assed implement in Reia because I strongly believe in its awesomeness. However, Reia's implementation is a bit brittle and has a lot of implementations. Just recently I switched to the latest greatest version of leex which ships with the Erlang runtime and had to make some rather arcane changes to some code that uses leex internals, just to continue to support a partially functioning string interpolation mechanism.

Now, toss in some more fun complexities: the awesome /.../ regex literal syntax. We all love it, but why doesn't every language have it? What you may not realize is that this syntax is ambiguous in certain cases with the / and /= operators like it is in other languages like JavaScript. Now, for added fun, toss in the fact that regular expressions can interpolate just like strings, and you're beginning to see the makings of a gramatical nightmare.

All of these things are easy with a PEG. PEGs blend tokenization with parsing and allow comprehension of a much wider range of languages than is possible without pulling your hair out using lex/yacc-like tools, and much more than that, it's a very natural process with PEGs. PEGs are also by nature right-recursive, something that works quite well in immutable state languages like Erlang that have to build their syntax trees from right-to-left due to the nature of immutable singly linked lists. These lists only let you append elements on the left.

This sounds well and good, but unfortunately I have some bad news. Even after a few rounds of performance tuning, using an experimental branch of Neotoma that uses binaries internally instead of lists, the PEG parser is still about an order of magnitude slower than the leex/yecc version. Talking to Neotoma's author Sean Cribbs, it sounds like Neotoma might be doing an excessive amount of unnecessary copying internally. If PEGs pique your interest and you know a bit of Erlang you might take a peek at Neotoma and see if you can find some potential performance optimizations.

I would still like to merge the peg branch of Reia, and the PEG grammar fixes a lot of quirks in the present yecc grammar, but I'm still holding out until the performance is improved to closer to the leex/yecc speeds.

9 comments:

phil pirj said...

Congradulations on that. Huge respect to Graeme.
Reia is still my point of interest, and I have some experience with Neotoma as well. Hope to have some time to play with it this spring.

Aliya said...

Surveillancekart | CCTV System
Pestveda | Pest Control
Termite control
cctv installation services
best home security camera system

emma said...

cctv camera dealers in delhi
cp plus cctv camera online
hikvision camera online
cctv camera installation services in delhi
cctv camera installation services in gurugram
cctv camera installation services in gurgaon

shabnam praveen said...

Call girls in Kolkata
Call girls in Chandigarh
Call girls in Chandigarh
Call girls in Gurgaon
Call girls in Chandigarh
Call girls in Chandigarh

shabnam praveen said...

Call girls in Lucknow
Call girls in Guwahati
Call girls in Mumbai
Call girls in Jaipur
Call girls in Jaipur
Call girls in Jaipur
Call girls in Bangalore

call Girl said...

Call girls in Gurgaon
Call girls in Lucknow
Call girls in Guwahati

Call girl in Gurgaon
Call girl in Lucknow
Call girl in Guwahati
Bangalore call girls


Gurgaon call girls
Lucknow call girls
Guwahati call girls
Bangalore call girls


Gurgaon call girl
Lucknow call girl
Guwahati call girl
Bangalore call girl

Escorts in Gurgaon
Escorts in Lucknow
Escorts in Guwahati
Escorts in Bangalore


Escort services in Gurgaon
Escort services in Lucknow
Escort services in Guwahati
Escort services in Bangalore


Escort service in Gurgaon
Escort service in Lucknow
Escort service in Guwahati
Escort service in Bangalore


Gurgaon escorts
Lucknow escorts
Guwahati escorts
Bangalore escorts

Naincy Kumari said...

This is so much amazing .so,you can also visite my links.
call girls in kolkata
russian escorts in kolkata
kolkata call girls
call girls kolkata
escorts in kolkata
kolkata escort
housewife escorts in kolkata
college call girls in kolkata
call girls in park street
escorts in park street

call girls in gurgaon
call girl in gurgaon
gurgaon call girls
escorts in gurgaon
gurgaon escorts
gurgaon escort
Escort services in Gurgaon
Escort service in Gurgaon
russian escorts in gurgaon
housewife escorts in gurgaon
college call girls in gurgaon

Call girls in Noida
Call girl in Noida
Escorts in Noida
Escort services in Noida
Escort service in Noida
Noida Escorts
Noida Escort
Noida Call girls
Noida Call girl

call Girl said...

Gurgaon escorts
Lucknow escorts
Guwahati escorts
Bangalore escorts
call girls in bangalore

Gurgaon escort
Lucknow escort
Guwahati escort
Bangalore escort

Faridabad call girls
Faridabad call girl
Faridabad escorts
Faridabad escort
Escort services in Faridabad

Escort services in Faridabad
Call girls in Faridabad
Call girl in Faridabad
Faridabad escort services
Faridabad escort service

Escorts in Faridabad
Escorts in Faridabad
Escorts in Faridabad
Escorts in Faridabad

Escort in Faridabad
Escort in Faridabad
Escort in Faridabad
Escort in Faridabad

Escort services in Faridabad
Escort services in Faridabad
Escort services in Faridabad
Escort services in Faridabad

Escort service in Faridabad
Escort service in Faridabad
Escort service in Faridabad
Escort service in Faridabad

siska said...

Togel Sydney
Togel Singapore
Togel Hongkong
Prediksi Togel
Buku Mimpi
Angka Jitu Togel