Tuesday, July 26, 2011

The Trouble with Erlang (or Erlang is a ghetto)


This is a blog post I have been meaning to write for quite some time. I lament doing so because I've made a considerable time investment into the Erlang infrastructure and really love some of its ideas. Erlang has done a great and still unique job of synthesizing a number of concepts in a very interesting way. But after using the platform in various capacities for some 4 years now, there are some glaring issues I think need to be called out.

Records suck and there's no struct/map data structure

Erlang has a feature called "records" which uses the preprocessor to give you something akin to a struct or map, i.e. a way to access named fields of a particular object/term within the system. As far as I can tell, there's pretty much universal agreement within the community that this is a huge limitation, and several proposals have been made to remedy the problem. The requested feature has typically been referred to as a "frame", and several proposals for implementing frames have been floating around for several years. Yet no action has been taken on the problem.

So why doesn't Erlang have frames? While Erlang is an open source project, its implementation and release cycle are managed by Ericsson, the company that created it, and Ericsson just doesn't seem to care. I'm not sure what Ericsson's priorities are when it comes to adding features to Erlang, but in my opinion they're doing a worse job of engaging the community than Oracle has been doing with Java. I hate Oracle as a company, but so far it feels like they've actually done a fairly good job managing Java development and moving Java forward. I can't say that at all with Ericsson, and frames are the quintessential example of this.

Erlang sucks at managing memory

Once upon a time I looked upon BEAM's design as the future pattern all virtual machines would follow. I strongly encourage you to read that post before taking issue with anything I have to say in regard to this matter. I have completely reversed my opinion since the time I write that post.

The other night I tweeted "If you're looking for a language that gets multicore concurrency right, look at how Azul implemented Java on their Vega architecture" and I definitely stand by that. Azul is a company that got a lot of smart hardware and software people together and had them work on designing a custom system which would scale to hundreds of CPU cores (up to 768 of them), heaps that topped 500 GB (up to 768GB), and had the GC pause only 10-20ms at a time. The realtime performance characteristics Azul managed to eek out of their system lead them to often describe their GC as "pauseless".

Where Azul was scaling up to 768 CPUs in 2007, Erlang was crapping out around 15 CPUs in 2009. For everything Erlang had to say about the importance of immutability and messaging in concurrent systems, and despite Joe Armstrong's promise that "your Erlang program should just run N times faster on an N core processor," it turns out that on the Erlang VM the N core processor promise had an upper bound of around 15.

Why is this? Erlang implements its own memory allocator and can't take advantage of libraries like tcmalloc to provide better multithreaded heap management. I can't fault a language VM like BEAM for doing this save for the fact that what Erlang provides is relatively crappy.

Erlang has done a fairly decent job given the constraints it was working within. Erlang wanted to provide a soft realtime system, and managed to create one that works on commodity architectures, unlike the Azul Vega appliances which require custom hardware. However, Azul has managed to port their version of the JVM to x86 hardware with their Zing Architecture, which wraps the JVM in a separate runtime container which uses software transactional memory to replace the hardware transactional memory found on the Vega appliances. It's higher overhead but provides similar guarantees. Java also provides the RTSJ specification for building realtime systems in Java.

Both Zing and RTSJ demonstrate that Erlang's approach to building a realtime garbage collected system, using separate heaps per process, isn't necessary to still provide realtime characteristics. Erlang's approach of using separate heaps is nonstandard and comparatively hard to optimize because most other systems are using a shared heap model. Azul's Vega architecture shows that shared heaps can scale up to hundreds of CPU cores and hundreds of gigabytes of heap while still providing realtime characteristics. Even more exciting is that AMD's Fusion architecture, which they're implementing in conjunction with ARM, provides read and write barriers at the hardware level necessary to provide a system like Azul using commodity hardware.

However, I think everything I just said is moot for the majority of applications. People building messaging systems want the best performance possible but don't typically have software realtime constraints. The Erlang VM's approach to soft realtime made a design decision which hampers its messaging speed, namely the use of separate heaps, which requires messages be copied from one heap to another. This means the Erlang VM does not provide zero-copy messaging. Every time you send a message from one Erlang process to another, some amount of data must be copied.

Erlang has partly mitigated this problem by providing a separate shared heap for binaries, which are the Erlang type for arbitrary blobs of binary data. This means if you ensure the majority of data you move around doesn't contain anything of significant size except binaries, perhaps this won't be a problem. However, if you're moving large collections of numbers around (Erlang's strings-as-lists-of-integers come to mind), messaging will be comparatively slow compared to a zero copy system.

Various solutions to this have been proposed for BEAM, such as switching from a shared-nothing heap to a shared heap or a hybrid heap (where message-passed objects are copied once), however the Erlang garbage collector is not suitable for managing shared/hybrid heaps and would need to be rewritten for the task, and nobody has managed to get the shared/hybrid heaps working with Erlang's SMP scheduler, or rewritten the garbage collector to be more suitable to the task of managing a shared/hybrid heap.

A potential solution to this? Erjang, an implementation of Erlang on the JVM, provides zero copy messaging using the Kilim library for lightweight threads.

JIT? What JIT?

Erlang has a "JIT" compiler called HiPE, which is mostly hype. I put JIT in quotes because HiPE is mostly an Erlang-to-native-code compiler with a limited set of backends which does a pretty bad job of optimizing and can't use runtime profiling information to improve the quality of the native code it generates in the way JIT compilers like HotSpot are able to. Calling HiPE a just-in-time compiler is a stretch as it is for most part an ahead-of-time native compiler for Erlang. The quality of native code produced by HiPE can be so poor that it's often outperformed by the userland bytecode interpreter implemented in BEAM.

HiPE can perform a very limited set of optimizations. In particular, Erlang code is factored into modules, and HiPE's inliner is unable to inline natie code across modules. This is due to HiPE's lack of a deoptimizer (a.k.a. deopt), or a way to translate JITed code back into bytecode, which is necessary in general but particularly necessary in Erlang for cases like hot code swapping. Deopt support is a feature of many JIT compilers in languages more popular than Erlang, most notably the HotSpot compiler on the JVM. Google's V8 virtual machine for JavaScript added deoptimization support as part of their "Crankshaft" compilation infrastructure.

Erlang isn't general purpose

Erlang hates state. It especially hates shared state. The only facility provided by the language for dealing with shared state in Erlang is called "Erlang Term Storage" and provides a Judy array that several Erlang processes can talk to. The semantics of ETS are fairly awkward and using it directly is difficult. Erlang has a baked-in database called Mnesia which is built on ETS. Mnesia's performance characteristics aren't great but it provides a friendlier face for ETS. These are the only solutions to shared state baked into the language.

What should you do if you want to deal with a shared-state concurrency program in Erlang? The general advice is: don't. Erlang isn't designed for solving shared-state concurrency problems. If you encounter a shared state concurrency problem while developing your Erlang program, sorry, you picked the wrong language. Perhaps you should move along... and Clojure offers you some great ways to tackle shared state concurrency problems.

The syntax is atrocious

I think this one goes without saying. That said...

Let me come at this from a different angle than you're probably expecting: I've recently started working with Clojure, and I have to say, I really think Erlang would've been a lot better off with a Lisp-like syntax than a Prolog-inspired syntax. To-date Erlang is the only popular language with a Prolog inspired syntax and all of the awkward tokens and gramatical constructions make me wish it just had a simple Lispy syntax. This has been implemented in Robert Virding's Lisp Flavoured Erlang, which is very cool and worth checking out.

That opinion might come as a surprise, because the main project I was developing in Erlang was Reia, a Ruby-like syntax and runtime for Erlang. I've discontinued this project, for many reasons, one of which is because it's been surpassed in features and documentation by a similar project, José Valim's Elixir. After years of working on Reia, I've really grown to believe I'd rather spend my time working on a language which incorporates Erlang's ideas, but on the JVM with mutable state.

The Erlang cargo cult would love to hang me out to dry for even saying that... so let me address it right now.

Immutable state sucks and isn't necessary for Erlang-Style Concurrency

Immutable state languages force object creation whenever anything changes. This can be partially mitigated by persistent data structures, which are able to share bits and pieces of each other because they're immutable. This works, for example, when attempting to create a sublist that consists of the last N elements of a list. But what if you want the first N elements? You have to make a new list. What if you want elements M..N? You have to make a new list.

In mutable state languages, performance problems can often be mitigated by mutating local (i.e. non-shared) state instead of creating new objects. To give an example from the Ruby language, combining two strings with the + operator, which creates a new string from two old ones, is significantly slower than combining two strings with the concatenating >> operator, which modifies the original string. Mutating state rather than creating new objects means there's fewer objects for the garbage collector to clean up and helps keep your program in-cache on inner loops. If you've seen Cliff Click's crash course on modern hardware, you're probably familiar with the idea that latency from cache misses is quickly becoming the dominating factor in today's software performance. Too much object creation blows the cache.

Cliff Click also covered Actors, the underpinning of Erlang's concurrency model, in his Concurrency Revolution from a Hardware Perspective talk at JavaOne. One takeaway from this is that actors should provide a safe system for mutable state, because all mutable state is confined to actors which only communicate using messages. Actors should facilitate a shared-nothing system where concurrent state mutations are impossible because no two actors share state and rely on messages for all synchronization and state exchange.

The Kilim library for Java provides a fast zero-copy messaging system for Java which still enables mutable state. In Kilim, when one actor sends a message, it loses visibility of the object it sends, and it becomes the responsibility of the recipient. If both actors need a copy of the message, the sender can make a copy of an object before it's sent to the recipient. Again, Erlang doesn't provide zero-copy (except for binaries) so Kilim's worst case is actually Erlang's best case.

The limitations of concurrent objects in Reia were solved using mutable state in my Celluloid concurrent object library for Ruby, but that deserves a blog post in and of itself.

Single assignment is just as problematic as destructive assignment

Erlang doesn't allow destructive assignments of variables, instead variables can only be assigned once. Single assignment is often trotted out as a panacea for the woes of mistakenly rebinding a variable then using it later expecting you had the original value. However, let me show you a real-world case that has happened to me on several occasions which wouldn't be an error in a language with destructive assignment and pattern matching (e.g. Reia).

There exists a complimentary case of mistaken variable usage to the afforementioned problem with destructive assignment. In single-assignment programs, it involves mistakenly using the same variable name twice excepting the variable to be unbound the second time:

The first pattern matching expression binds the Foo variable to something. In the second case, we've mistakenly forgot Foo was already bound. What's the result?

exception error: no match of right hand side... 

We get no compiler warning in this case. This is the type of error you only encounter at runtime. It can lay undetected in your codebase, unless you're writing tests. Know what other problem writing tests solves? Mistaken destructive assignments.

Single assignment is often trotted out by the Erlang cargo cult as having something to do with Erlang's concurrency model. This couldn't be more mistaken. Reia compiled destructive assignments into Static Single Assignment (SSA) form. This form provides versioned variables in the same manner as most Erlang programmers end up doing manually. Furthermore, SSA is functional programming. While it may not jive with the general idealism of functional programming, the two forms (SSA and continuation passing style) have been formally proven identical.

The standard library is inconsistent, ugly, and riddled with legacy

Should module names in the standard library be plural, like "lists"? Or should they be singular, like "string"? Should we count from 1, as in most of the functions found in things like the lists module, or should we count from 0 like the functions found in the array module? How do I get the length of a list? Is it lists:length/1? No, it's erlang:length/1. How do I get the Nth element of the tuple? Should I look in the tuple module? Wait, there is no tuple module! Instead it's erlang:element/2. How about the length of a tuple? It's erlang:tuple_size/1. Why is the length of a list just "length" whereas the length of a tuple is "tuple_size"? Wouldn't "list_length" be more consistent, as it calls out it works on lists?

When we call erlang:now() to get the current time, it returns {1311,657039,366306}.  What the hell does that mean? It's a tuple with three elements. How could time possible need three elements? A quick look at the documentation reveals that this tuple takes the form {Megaseconds, Seconds, Microseconds}. Separating out Microseconds makes sense... Erlang has no native decimal type so using a float would lose precision. But why split apart Megaseconds and Seconds?

Once upon a time Erlang didn't support integers large enough to store the combination of Megaseconds and Seconds, so they were split apart. The result is a meaningless jumble of three numbers, which you have to run through the confusingly named calendar:now_to_local_time/1 function to get a human meaningful result, which doesn't tell you what time it is now, but instead takes the tuple that erlang:now/0 returns as an argument and will spit back meaningful {Year, Month, Day} and {Hour, Minute, Second} tuples.

Legacy in the grammar

Try to use "query" as an atom in Erlang, e.g. {query, "SELECT * FROM foobar"}. What happens?

syntax error before: ','

This is because 'query' is a reserved word which was reserved for Mnemosyne queries. Never heard of Mnemosyne? That's because it's an archaic way of querying Erlang's built-in database, Mnesia, and has been replaced with Query List Comprehensions (QLC). However, it remains around for backwards compatibility.

You can't use "query" as a function name. You can't tag a tuple with "query". You can't do anything with "query" except invoke a deprecated legacy API which no one uses anymore.

Strings-as-lists suck

Erlang provides two ways of representing strings. One is as lists of integers, which is the traditional way that most of the library functions support. Another is binaries. Erlang has no way of differentiating lists of integers that represent strings from lists of integers that are actually lists of integers. If you send a list of integers in a message to another process, the entire list of integers is copied every time. On 64-bit platforms, every integer takes up 64-bits.

The obvious solution here is to use binaries instead of lists of integers. Binaries are more compact and exist in a separate heap so they aren't copied each time they're sent in a message. The Erlang ecosystem seems to be gradually transitioning towards using binaries rather than strings. However, much of the tooling and string functions are designed to work with list-based strings. To leverage these functions, you have to convert a binary to a list before working with it. This just feels like unnecessary pain.

The abstract concept of lists as strings isn't inherently flawed. In many ways it does make sense to think of strings as lists of characters. Lists as strings would probably make a lot more sense if Erlang had a native character type distinct from integers which was more compact and could avoid being copied each time a string is sent in a message like a binary. Perhaps in such a system it'd be possible to avoid transcoding strings read off the wire or completely transforming them to a different representation, which is costly, inefficient, and often times unnecessary (yes, this is a problem with Java too).

There's no "let"

Want a local binding in Erlang? Perhaps you've used let for this in a Lisp. What happens when you try to do this in Erlang? Even attempting to use "let" in Erlang just yields: syntax error before: 'let'

Once upon a time Erlang was supposed to get let bindings, and the "let" keyword was set aside for this purpose. But much like frames, it never happened. Instead, let is now an unimplemented reserved word which just breaks your programs.

There's no "nil"

In Clojure, I can write the following: (if false :youll-never-know).  This implicitly returns "nil" because the condition was false. What's the equivalent Erlang?

Erlang forces you to specify a clause that always matches regardless of whether you care about the result or not. If no clause matches, you get the amazingly fun "badmatch" exception. In cases where you don't care about the result, you're still forced to add a nonsense clause which returns a void value just to prevent the runtime from raising an exception.

Where do I go from here?

Again, I want to emphasize that I have a great deal of respect for Erlang conceptually. But at this point I'd like to take what I've learned and go elsewhere with it. One direction I've gone is the Celluloid concurrent object library for Ruby. You can read more about it in the original blog post I wrote about Celluloid, which is a bit out-of-date at this point. I have a forthcoming blog post which should dive a bit deeper into Celluloid's guts and how it can do things which aren't possible in Erlang.

As you've probably guess from the references sprinkled throughout this post, I'm learning Clojure. I'm a fan of the JVM and Clojure provides a great functional language for leveraging the JVM's features. I think the sort of things that I'd be writing in Erlang I'll try writing in Clojure instead. Clojure has elegant Lisp syntax. Clojure has maps. Clojure has powerful facilities for dealing with concurrent shared state problems. Clojure has great semantics for safely managing mutable state in a concurrent environment. Clojure has real strings. Clojure has let. Clojure has nil. Clojure runs on the JVM and can leverage the considerable facilities of the HotSpot JIT and JVM garbage collectors.

I'd also like to try my hand at creating a JVM language, especially with the impeding release of Java 7 this Thursday. Java 7 brings with it InvokeDynamic, a fast way to dispatch methods in dynamic languages, and considerably eases the difficulty of implementing dynamic languages on the JVM. Stay tuned for more details on this.

482 comments:

«Oldest   ‹Older   401 – 482 of 482
jerry123 said...

great blog
inspirational quotes

Raju said...

This is one of the best blog I have read. Thanks for making my day :) And yes it’s a request from one of your reader i.e me, keep posting such great and informative blogs. I would love to refer my others friends also to visit your blog and share it on social media channels. Thanks one again Sweetie!!!
hawaii smoke shops
smok g priv accessories

krishna singh said...

I am new here. I like your post very much. It is very usefull post for me.Are you looking for cell phone repair services. We fix iPhones, iPads, Samsung Galaxy Note. Broken screen repair or Cracked LCD Glass? Home, Speaker or Volume button not working.
phone fix near me
iphone repair store

Unknown said...

Thanks for your post.
Datameris is the
best digital marketing agency
 & best seo agency in india

BEST WEB DEVELOPMENT COMPANY

chopal said...

Hello! This is my first visit to your blog! This is my first comment here, so I just wanted to give a quick shout out and say I genuinely enjoy reading your articles
chopal

Mahiescorts said...

Hi firends My Name Is Mahi verma And I Am 24yrs Old Escort Girl Living In manali And Here Is An Independent Escorts Girls Service In manali Escorts So u Can Call Us

And Book An Call girls Get your Doorstep.
manali Escorts
manali Call Girls
Escorts In manali
Call Girls In manali
https://mahiescorts.com/

Money said...

Hello! This is my first visit to your blog! This is my first comment here, so I just wanted to give a quick shout out and say I genuinely enjoy reading your articles. Your blog provided us useful information. You have done an outstanding job
thigh flask holder

كيمو نور00201201726286 said...

شركة مكافحة حشرات بجازان
شركة مكافحة حشرات بجازان
شركة نقل عفش بجازان
شركة عزل اسطح بجازان

كيمو نور00201201726286 said...



شركة عزل أسطح بنجران
شركة تنظيف خزانات بنجران
شركة تنظيف منازل بنجران
شركة تنظيف بنجران
شركة رش مبيدات بنجران
شركة نقل أثاث بنجران
شركة مكافحة حشرات بنجران
شركة كشف تسريات المياه بنجران



Rhianne Jhane said...

I see something really special in this site.

www.freesitemaker.net/how-to-make-a-website/ecommerce-website-structure

tcavapor said...

Nice post ! grateful for your blog post. You will find a lot of approaches after visiting your post
smoke shops honolulu
aspire breeze accessories
atomizers for mods

siddhi manali said...

hi firends My Name Is sidhhi sharma And I Am 24yrs Old Escort Girl Living In Manali And Here Is An Independent Escorts Girls Service In Kullu Manali Escorts So u Can

Call Us And Book An Call girls Get your Doorstep.
Manali Escorts
Manali Call Girls
Escorts In Manali
Call Girls In Manali
http://kullumanaliescort.in

kajalmodels said...

Dehradun Escorts Might Give The Required Content To You. These Lovely Beauties Are Professionally Trained To Keep You Happy In Various Styles Of Manners. Your blog is so glorious, Can visit our site
Dehradun escort
Mussoorie escort
haridwar escort
rishikesh escort
gomti nagar escort
Bhopal escort

jain123 said...

It’s really a great and helpful piece of information. I’m satisfied that you just shared this helpful information with us.
Please stay us up to date like this. Thank you for sharing
Click here: First Ad Position Google Search

Assignment help Australia said...

Wondrous article thanks to providing us, we appreciate you, Get the best assignment help in whole over the world by Ph.D. experts at affordable prices. Our assignment writing services are 100% risk-free and are assured by 100%. assignment help

suhani arora said...

You Can Appoint haridwar escort Girls For Getting Pleasure. Our Collection Manages To Keep Three Most Demanded And Fascinating Classed Of Escort Service In haridwar. The Love To You Directly In The Form Of Escort Service In haridwar http://www.suhaniarora.in

suhani arora said...

You Can Appoint haridwar escort Girls For Getting Pleasure. Our Collection Manages To Keep Three Most Demanded And Fascinating Classed Of Escort Service In haridwar. The Love To You Directly In The Form Of Escort Service In haridwar http://www.suhaniarora.in

Sejwal Technologies said...

This article is very nice. Thanks for post.
web designing agency in dwarka
web designing agency in delhi
Web designing company in delhi
Web designing company in dwarka
best web designing agency in dwarka
best web designing agency in delhi
best Web designing company in delhi
Web designing company in dwarka

This article is very nice. Thanks for post.
website designing company in delhi
best website designing company in delhi
website designing company in dwarka
best website designing company in dwarka
website designing and development company in delhi
best website designing and development company in delhi
website designing and development company in dwarka

This article is very nice. Thanks for post.
best website designing and development company in dwarka
Best Portal Designing in delhi
Best Portal Designing in dwarka
Best Portal Designing and development in delhi
Best Portal Designing and development in dwarka
Portal Designing and Development in Delhi
Portal Designing and Development in Dwarka
Best seo in delhi

Sejwal Technologies said...

This article is very nice. Thanks for post.
best seo in dwarka
best seo in bharthal
best seo service in delhi
best seo service in dwarka
seo service in delhi
seo service in dwarka
seo service in bharthal

This article is very nice. Thanks for post.
Best SMO in delhi
best SMO in dwarka
best SMO in bharthal
best SMO service in delhi
best SMO service in dwarka
SMO service in delhi
SMO service in dwarka
SMO service in bharthal

This article is very nice. Thanks for post.
Best Facebook campaign in delhi
best Facebook campaign in dwarka
best Facebook campaign in bharthal
best Facebook campaign service in delhi
best Facebook campaign service in dwarka
Facebook campaign service in delhi
Facebook campaign service in dwarka
Facebook campaign service in bharthal

Sejwal Technologies said...

This article is very nice. Thanks for post.
Best Digital marketing agency in Dwarka
Best Digital marketing agency in Delhi
Best Digital marketing Company in Delhi
Best Digital marketing Company in Dwarka
Digital Marketing agency in delhi
Digital Marketing agency in dwarka
Digital Marketing agency in Bharthal

Digital Marketing agency in Gurgaon
Best Internet Service provider in dwarka
Best Internet Service provider in bharthal
Best Internet Service provider in Plam Vihar
Best Internet Service provider in Bijwasan
Best CCTV Services in dwarka
Best CCTV Service provider in bharthal
Best CCTV Service provider in Plam Vihar
Best CCTV Service provider in Bijwasan

angelisaka97 said...

Permainan poker pastinya banyak di kalangan remaja hingga dewasa yang sangat menggemari permainan poker, apalagi dalam 1 id game ada banyak permainan kartunya silahkan kunjungi situs kami untuk merasakan kenyamanan dalam bermain.

daftar idn poker 88

daftar idn poker88


Daftar idnplay


idn poker apk


poker idn deposit pulsa


DAFTAR POKER99

daftar akun idn poker99

buat akun poker99

daftar akun idn poker 99


LOGIN POKER99

cara login poker99

login poker99 via website

login poker99 via mobile

akun poker99


APLIKASI IDN POKER 99

aplikasi poker99

aplikasi idn poker99 terbaru

siddhi manali said...

hi firends My Name Is sidhhi sharma And I Am 24yrs Old Escort Girl Living In Manali And Here Is An Independent Escorts Girls Service In Kullu Manali Escorts So u Can

Call Us And Book An Call girls Get your Doorstep.
Manali Escorts
Manali Call Girls
Escorts In Manali
Call Girls In Manali
http://kullumanaliescort.in

siddhi manali said...


hi firends My Name Is sidhhi sharma And I Am 24yrs Old Escort Girl Living In Manali And Here Is An Independent Escorts Girls Service In Kullu Manali Escorts So u Can

Call Us And Book An Call girls Get your Doorstep.
Manali Escorts
Manali Call Girls
Escorts In Manali
Call Girls In Manali
http://kullumanaliescort.in

siddhi manali said...

hi firends My Name Is sidhhi sharma And I Am 24yrs Old Escort Girl Living In Manali And Here Is An Independent Escorts Girls Service In Kullu Manali Escorts So u Can

Call Us And Book An Call girls Get your Doorstep.
Manali Escorts
Manali Call Girls
Escorts In Manali
Call Girls In Manali
http://kullumanaliescort.in

درب ضد سرقت سنا درب said...

مدل درب ضد سرقت به دلیل شکل ظاهری و ساختاری که دارا می باشد بسیار مقاوم تر و امنیت بیشتری را نسبت به درب های معمولی دارا است . این درب های ضد سرقت از نظر ساختار کلی دارای ورق های فولادی در خود می باشد ، قفل ها و لولا ها دارای جنس درجه یک نسبت به مدل دربهای معمولی می باشد .
درب ضد سرقت

The Gabay Group said...

כתיבה מעולה, אהבתי. אשתף עם העוקבים שלי.
קבוצת גבאי אלי גבאי

Granite Store said...

Thanks For Sharing this Information.
Best Regards
Granite Store

Bio - Granite Store is a Calgary based stone company dealing in Granite Countertops Calgary,Marble Countertops Calgary, Stone Countertops Calgary & Quartz Countertops Calgary.

Granite Store said...

Thanks For Sharing this Information.
Best Regards
Granite Store

Bio - Granite Store is a Calgary based stone company dealing in Granite Countertops Calgary,Marble Countertops Calgary, Stone Countertops Calgary & Quartz Countertops Calgary.

Unknown said...

Thanks

סילברגייט

Unknown said...

شركة تسويق الكتروني
صيانة كريازي
صيانة جليم جاز
صيانة فريجيدير

شركة MK
صيانة الاسكا
صيانة سيمنس
صيانة فريش
صيانة توشيبا
للتسويق الالكتروني
شركة تسويق الكتروني
01015375663 واتس اب

Jatin Sharma said...

Study in Canada
Best Canada Education Consultants
Best Canada Immigration Services
Overseas Education Consultant
Study in Canada Consultant
AAR Overseas
AAR Overseas
AAR Overseas
Canada Education Consultant
Study in Canada
Study in Canada Consultant Near Me
Study in Canada
Study in Canada Consultant

preethi minion said...

nice..
afghanistan hosting
angola hosting
afghanistan web hosting
bahrain web hosting
belize web hosting
india shared web hosting

preethi minion said...

italy web hosting
suden web hosting
tunisia hosting
uruguay web hosting
inplant training in chennai

Unknown said...


Hello Friends My Name is Mahi Verma And I Am An Independent call girl living in manali and i am doing Part Time Job In Mahi Escorts Agency Here Is Lot Of Call Girls Avialable In this agency So U can Call Us And Book An Escorts. Manali Escorts
Manali Call Girls
Escorts In Manali
Call Girls In Manali

Cinema furniture said...
This comment has been removed by the author.
Unknown said...

Heart to Heart Introductions is a local matchmaking service. At Heart to Heart you can meet professional singles and learn dating tips with a local matchmaker.
Heart to Heart Introductions
Heart to Heart Introductions
Heart to Heart Introductions
Heart to Heart Introductions

Home-service said...

رقم صيانة كريازي 

صيانة ثلاجات كريازي

صيانة غسالات كريازي

صيانة ديب فريزر كريازي 

صيانة توشيبا العربي

رقم صيانة ثلاجات توشيبا 

صيانة غسالات توشيبا

صيانة شاشات توشيبا 

رقم صيانة ال جي 

صيانة غسالات ال جي 

صيانة ثلاجات ال جي 

صيانة شاشات ال جي 

مركز صيانة الاسكا 

صيانة غسالات الاسكا 

صيانة ثلاجات الاسكا 

صيانة ديب فريزر الاسكا

Sweet Page Hosting Trays said...

הדעות שלי קצת חלוקות בעניין הזה אבל ללא ספק כתבת מעניין מאוד
מגש אירוח מתוקים

Aleena said...

מזל שנתקלתי בכתבה הזאת. בדיוק בזמן
טבעות אירוסין מיוחדות

Mahzar said...

BMIT – BSc Medical Imaging Technology – Here is the details about Best BSc Medical Imaging Technology Colleges In Bangalore. If you are looking to study in Bangalore, the below link will redirect to a page that will show the best BMIT colleges in Bangalore
BSc Medical Imaging Technology Colleges In Bangalore

sheweta sinha said...

This is actually a very informative article – not like most of what I see online. Thanks for the free share and looking forward to reading your updates! simply wow
Karol Bagh Escorts
Paharganj Escorts
Malviya Nagar Escorts
Friends Colony Escorts
Punjabi Bagh Escorts
South Delhi Escorts

Unknown said...

We Provide Excellent Opportunities For Your Children To Connect To Nature, The Very Best Of Early Years Foundation Stage And Forest School Teachings.
Mighty Oaks Day Nursery and Preschool
Mighty Oaks Day Nursery and Preschool

Cinema furniture said...

לגמרי פוסט שדורש שיתוף תודה.
שולחנות אוכל

Comfy furniture said...

מזל שנתקלתי בכתבה הזאת. בדיוק בזמן
פינות אוכל מעוצבות

Unknown said...


Hello Friends My Name is Mahi Verma And I Am An Independent call girl living in manali and i am doing Part Time Job In Mahi Escorts Agency Here Is Lot Of Call Girls Avialable In this agency So U can Call Us And Book An Escorts.
Manali Escorts
Manali Call Girls
Escorts In Manali
Call Girls In Manali

Amediciercosmetic said...

אין ספק שזה אחד הנושאים המעניינים. תודה על השיתוף.
מזותרפיה לשיער

Deep Bhardwaj said...
This comment has been removed by the author.
Deep Bhardwaj said...

Contact Deep Bhardwaj, leading SEO expert in India. We have 6+ years of experience in providing top SEO services in Delhi to our clients across Delhi and India. Visit our website to request a free quote today.

NEHASHARMA said...

I like your incredible post. I wanted to embrace this kind of article. likewise, without a doubt the first occasion when I visit our site. likewise, I fulfilled to be here, an obligation of appreciation is all together for sharing this astonishing post.
callgirlsnumbers
southdelhicallgirls
sexescorts
callgirlsdelhincr
delhiescort-girl
kimdelhi

Planner Productions said...

תודה על השיתוף. מחכה לכתבות חדשות.
הפקת אירועים

Babysitting said...

אין ספק שזה אחד הנושאים המעניינים. תודה על השיתוף.
שטיח לחדר ילדים

brokertome said...

כל מילה. תודה על השיתוף, מחכה לעוד פוסטים בנושא.
השקעה בנדלן בארץ

manali escorts by mahi gill said...

Hi firends My Name Is mahi gill And I Am 24yrs Old Escort Girl Living In manali And Here Is An Independent Escorts Girls Service In manali So u Can Call Us And Book

An Call girls Get your Doorstep.
manali Call Girls
Escorts In manali
Call Girls In manali
http://www.mahiescortservices.com/

raj1234 said...

Hello! This is my first visit to your blog! This is my first comment here, so I just wanted to give a quick shout out and say I genuinely enjoy reading your articles

antique jewellery

manali escorts by mahi gill said...

Hi firends My Name Is mahi gill And I Am 24yrs Old Escort Girl Living In manali And Here Is An Independent Escorts Girls Service In manali So u Can Call Us And Book

An Call girls Get your Doorstep.
manali Call Girls
Escorts In manali
Call Girls In manali
http://www.mahiescortservices.com/

gemcreature said...

nice blog
dolphin ankle bracelet

chopa;l said...

Hello! This is my first visit to your blog! This is my first comment here, so I just wanted to give a quick shout out and say I genuinely enjoy reading your articles
WhatsApp Stickers

Unknown said...

Thanks for sharing.

מומחה קידום אתרים

raju35 said...

This is my first comment here, so I just wanted to give a quick shout out and say I genuinely enjoy reading your articles. Your blog provided us useful information. You have done an outstanding job
thigh flask holder

1 DAY PAINTERS said...

Hello Everyone,

I like this article, It is very useful to me. I’m working with 1daypainters.pro. 1daypainters.pro is the best Calgary Painting Company providing Stucco Painting Calgary, Exterior House Painting Calgary & Knock Down Ceiling Calgary. Hire us if you are looking for Stucco Painters, House Painters & Calgary Painters.
https://1daypainters.pro/
Calgary Exterior Painting companies
Knock Down Ceiling Calgary

DFS Services said...

Hi Everybody
I am new here this community. I like your article. I am working for DFS Services. we are best service provide.
thank you
Stucco Painters Calgary

Granite Store said...

Hello Everyone,
I like your aritcal. We are beat service provided Granite Store. I am new here this community.
thank you

marble countertops calgary

Safety Shoes said...


Hi Everyone,
I am new here this community. I am working prime work boots. it's very useful boots. I like your article.we are best service provided.
Thank you

gumboots

Inoventic Creative Agency said...

Hello Admin!

Thanks for the post. It was very interesting and meaningful. I really appreciate it! Keep updating stuffs like this. If you are looking for the Advertising Agency in Chennai | Printing in Chennai , Visit Inoventic Creative Agency Today..

Bloxi said...

אין ספק שהפוסט הזה דורש שיתוף. תודה.
תמונה מודפסת על עץ

Inoventic Creative Agency said...

Hello Admin!

Thanks for the post. It was very interesting and meaningful. I really appreciate it! Keep updating stuffs like this. If you are looking for the Advertising Agency in Chennai | Printing in Chennai , Visit Inoventic Creative Agency Today..

Unknown said...

Great Post very Nice.

123movies

Unknown said...

i love your post.

123 movies

nada ahmed said...

صيانة اريستون
صيانة وايت بوينت
ا صيانة ال جي
صيانة وايت ويل
صيانة زانوسي
صيانة سامسونج
صيانة يونيفرسال
صيانة كريازي

nada ahmed said...

صيانة الاسكا
صيانة سيمنس
صيانة فريش
صيانة توشيبا
للتسويق الالكتروني
شركة تسويق الكتروني
شركة تسويق الكتروني
صيانة كريازي
صيانة جليم جاز
صيانة فريجيدير

شركة MK
01015375663 واتس اب

Deep Bhardwaj said...

Great Post. Please keep sharing with us.


seo consultant services

Prempal Singh said...

Nice post, Thanks for sharing with us.

Regards

Prempal Singh

Best SEO Expert in India | SEO Expert India | SEO Freelancer India.

alasksecurity said...


תודה על השיתוף. מחכה לכתבות חדשות.
מערכת אזעקה לבית

Unknown said...

We have worked with many businesses in New Zealand and abroad and we have found that although there has been massive growth in technology, most small to medium sized business owners have been left behind.

Website Development Company in New Zealand
Software Development Company in New Zealand
E-Commerce Development Company in New Zealand
Content Writing Company in New Zealand
Digital Marketing Company in New Zealand
Pay Per Click Company in New Zealand

oxigen said...

Oxigen BC Private Limited Company is India's Largest CSP Provider, which works in all the states of India to open customer service point of all banks. Such as - sbi, boi, bob, pnb etc.

CSP Apply
CSP Online Application
Online CSP Apply
CSP Registration
CSP Online Application
CSP Provider

Unknown said...

Ombre Powder Glow brows is a technique performed with the assistance of a Master Liner Machine and provides a soft appearance that is both natural looking and long lasting. This brow type typically lasts 1-2 years.

Microblading
Permanent Makeup
Microblading Los Angeles
Eyebrows
Eyebrow Tattoo
Cosmetic Tattoo
Ombre Eyebrows
Powder Eyebrows

Reputation management said...

מעולה. תודה על הכתיבה היצירתית.
ניהול מוניטין באינטרנט

Dellinspiron52 said...

This post is very nice thanks for sharing its very good post
Website: Best Smartphone

SEO Karachi Pakistan said...

We are the best home services company in the Kingdom We provide cleaning services of the best equipment and manpower and transport services of the finest packaging and storage and 
شركة نقل عفش بجدة
شركة تنظيف بالبخار بجدة
شركة تنظيف خزانات بجدة
شركة تنظيف بجدة
شركة مكافحة حشرات بجدة
شركة نقل عفش بجدة

sattamatka52 said...

Thanks for sharing this blog here. It seems really very informative. It’s really a great and helpful piece of information. I’m satisfied that you just shared this helpful information with us.
website: satta king

التسويق الالكتوني said...

شركة تنظيف بالقطيف

شركة تنظيف شقق بالقطيف
شركة تسليك مجاري بالقطيف
شركة تنظيف فلل بالقطيف
شركة مكافحة حشرات بالقطيف
شركة كشف تسربات المياه بالقطيف
شركة مكافحة النمل الابيض بالقطيف

manali escorts by mahi gill said...

Hi firends My Name Is mahi gill And I Am 24yrs Old Escort Girl Living In manali And Here Is An Independent Escorts Girls Service In manali So u Can Call Us And Book

An Call girls Get your Doorstep.
manali Call Girls
Escorts In manali
Call Girls In manali
http://www.mahiescortservices.com/

«Oldest ‹Older   401 – 482 of 482   Newer› Newest»