Tuesday, July 26, 2011

The Trouble with Erlang (or Erlang is a ghetto)


This is a blog post I have been meaning to write for quite some time. I lament doing so because I've made a considerable time investment into the Erlang infrastructure and really love some of its ideas. Erlang has done a great and still unique job of synthesizing a number of concepts in a very interesting way. But after using the platform in various capacities for some 4 years now, there are some glaring issues I think need to be called out.

Records suck and there's no struct/map data structure

Erlang has a feature called "records" which uses the preprocessor to give you something akin to a struct or map, i.e. a way to access named fields of a particular object/term within the system. As far as I can tell, there's pretty much universal agreement within the community that this is a huge limitation, and several proposals have been made to remedy the problem. The requested feature has typically been referred to as a "frame", and several proposals for implementing frames have been floating around for several years. Yet no action has been taken on the problem.

So why doesn't Erlang have frames? While Erlang is an open source project, its implementation and release cycle are managed by Ericsson, the company that created it, and Ericsson just doesn't seem to care. I'm not sure what Ericsson's priorities are when it comes to adding features to Erlang, but in my opinion they're doing a worse job of engaging the community than Oracle has been doing with Java. I hate Oracle as a company, but so far it feels like they've actually done a fairly good job managing Java development and moving Java forward. I can't say that at all with Ericsson, and frames are the quintessential example of this.

Erlang sucks at managing memory

Once upon a time I looked upon BEAM's design as the future pattern all virtual machines would follow. I strongly encourage you to read that post before taking issue with anything I have to say in regard to this matter. I have completely reversed my opinion since the time I write that post.

The other night I tweeted "If you're looking for a language that gets multicore concurrency right, look at how Azul implemented Java on their Vega architecture" and I definitely stand by that. Azul is a company that got a lot of smart hardware and software people together and had them work on designing a custom system which would scale to hundreds of CPU cores (up to 768 of them), heaps that topped 500 GB (up to 768GB), and had the GC pause only 10-20ms at a time. The realtime performance characteristics Azul managed to eek out of their system lead them to often describe their GC as "pauseless".

Where Azul was scaling up to 768 CPUs in 2007, Erlang was crapping out around 15 CPUs in 2009. For everything Erlang had to say about the importance of immutability and messaging in concurrent systems, and despite Joe Armstrong's promise that "your Erlang program should just run N times faster on an N core processor," it turns out that on the Erlang VM the N core processor promise had an upper bound of around 15.

Why is this? Erlang implements its own memory allocator and can't take advantage of libraries like tcmalloc to provide better multithreaded heap management. I can't fault a language VM like BEAM for doing this save for the fact that what Erlang provides is relatively crappy.

Erlang has done a fairly decent job given the constraints it was working within. Erlang wanted to provide a soft realtime system, and managed to create one that works on commodity architectures, unlike the Azul Vega appliances which require custom hardware. However, Azul has managed to port their version of the JVM to x86 hardware with their Zing Architecture, which wraps the JVM in a separate runtime container which uses software transactional memory to replace the hardware transactional memory found on the Vega appliances. It's higher overhead but provides similar guarantees. Java also provides the RTSJ specification for building realtime systems in Java.

Both Zing and RTSJ demonstrate that Erlang's approach to building a realtime garbage collected system, using separate heaps per process, isn't necessary to still provide realtime characteristics. Erlang's approach of using separate heaps is nonstandard and comparatively hard to optimize because most other systems are using a shared heap model. Azul's Vega architecture shows that shared heaps can scale up to hundreds of CPU cores and hundreds of gigabytes of heap while still providing realtime characteristics. Even more exciting is that AMD's Fusion architecture, which they're implementing in conjunction with ARM, provides read and write barriers at the hardware level necessary to provide a system like Azul using commodity hardware.

However, I think everything I just said is moot for the majority of applications. People building messaging systems want the best performance possible but don't typically have software realtime constraints. The Erlang VM's approach to soft realtime made a design decision which hampers its messaging speed, namely the use of separate heaps, which requires messages be copied from one heap to another. This means the Erlang VM does not provide zero-copy messaging. Every time you send a message from one Erlang process to another, some amount of data must be copied.

Erlang has partly mitigated this problem by providing a separate shared heap for binaries, which are the Erlang type for arbitrary blobs of binary data. This means if you ensure the majority of data you move around doesn't contain anything of significant size except binaries, perhaps this won't be a problem. However, if you're moving large collections of numbers around (Erlang's strings-as-lists-of-integers come to mind), messaging will be comparatively slow compared to a zero copy system.

Various solutions to this have been proposed for BEAM, such as switching from a shared-nothing heap to a shared heap or a hybrid heap (where message-passed objects are copied once), however the Erlang garbage collector is not suitable for managing shared/hybrid heaps and would need to be rewritten for the task, and nobody has managed to get the shared/hybrid heaps working with Erlang's SMP scheduler, or rewritten the garbage collector to be more suitable to the task of managing a shared/hybrid heap.

A potential solution to this? Erjang, an implementation of Erlang on the JVM, provides zero copy messaging using the Kilim library for lightweight threads.

JIT? What JIT?

Erlang has a "JIT" compiler called HiPE, which is mostly hype. I put JIT in quotes because HiPE is mostly an Erlang-to-native-code compiler with a limited set of backends which does a pretty bad job of optimizing and can't use runtime profiling information to improve the quality of the native code it generates in the way JIT compilers like HotSpot are able to. Calling HiPE a just-in-time compiler is a stretch as it is for most part an ahead-of-time native compiler for Erlang. The quality of native code produced by HiPE can be so poor that it's often outperformed by the userland bytecode interpreter implemented in BEAM.

HiPE can perform a very limited set of optimizations. In particular, Erlang code is factored into modules, and HiPE's inliner is unable to inline natie code across modules. This is due to HiPE's lack of a deoptimizer (a.k.a. deopt), or a way to translate JITed code back into bytecode, which is necessary in general but particularly necessary in Erlang for cases like hot code swapping. Deopt support is a feature of many JIT compilers in languages more popular than Erlang, most notably the HotSpot compiler on the JVM. Google's V8 virtual machine for JavaScript added deoptimization support as part of their "Crankshaft" compilation infrastructure.

Erlang isn't general purpose

Erlang hates state. It especially hates shared state. The only facility provided by the language for dealing with shared state in Erlang is called "Erlang Term Storage" and provides a Judy array that several Erlang processes can talk to. The semantics of ETS are fairly awkward and using it directly is difficult. Erlang has a baked-in database called Mnesia which is built on ETS. Mnesia's performance characteristics aren't great but it provides a friendlier face for ETS. These are the only solutions to shared state baked into the language.

What should you do if you want to deal with a shared-state concurrency program in Erlang? The general advice is: don't. Erlang isn't designed for solving shared-state concurrency problems. If you encounter a shared state concurrency problem while developing your Erlang program, sorry, you picked the wrong language. Perhaps you should move along... and Clojure offers you some great ways to tackle shared state concurrency problems.

The syntax is atrocious

I think this one goes without saying. That said...

Let me come at this from a different angle than you're probably expecting: I've recently started working with Clojure, and I have to say, I really think Erlang would've been a lot better off with a Lisp-like syntax than a Prolog-inspired syntax. To-date Erlang is the only popular language with a Prolog inspired syntax and all of the awkward tokens and gramatical constructions make me wish it just had a simple Lispy syntax. This has been implemented in Robert Virding's Lisp Flavoured Erlang, which is very cool and worth checking out.

That opinion might come as a surprise, because the main project I was developing in Erlang was Reia, a Ruby-like syntax and runtime for Erlang. I've discontinued this project, for many reasons, one of which is because it's been surpassed in features and documentation by a similar project, José Valim's Elixir. After years of working on Reia, I've really grown to believe I'd rather spend my time working on a language which incorporates Erlang's ideas, but on the JVM with mutable state.

The Erlang cargo cult would love to hang me out to dry for even saying that... so let me address it right now.

Immutable state sucks and isn't necessary for Erlang-Style Concurrency

Immutable state languages force object creation whenever anything changes. This can be partially mitigated by persistent data structures, which are able to share bits and pieces of each other because they're immutable. This works, for example, when attempting to create a sublist that consists of the last N elements of a list. But what if you want the first N elements? You have to make a new list. What if you want elements M..N? You have to make a new list.

In mutable state languages, performance problems can often be mitigated by mutating local (i.e. non-shared) state instead of creating new objects. To give an example from the Ruby language, combining two strings with the + operator, which creates a new string from two old ones, is significantly slower than combining two strings with the concatenating >> operator, which modifies the original string. Mutating state rather than creating new objects means there's fewer objects for the garbage collector to clean up and helps keep your program in-cache on inner loops. If you've seen Cliff Click's crash course on modern hardware, you're probably familiar with the idea that latency from cache misses is quickly becoming the dominating factor in today's software performance. Too much object creation blows the cache.

Cliff Click also covered Actors, the underpinning of Erlang's concurrency model, in his Concurrency Revolution from a Hardware Perspective talk at JavaOne. One takeaway from this is that actors should provide a safe system for mutable state, because all mutable state is confined to actors which only communicate using messages. Actors should facilitate a shared-nothing system where concurrent state mutations are impossible because no two actors share state and rely on messages for all synchronization and state exchange.

The Kilim library for Java provides a fast zero-copy messaging system for Java which still enables mutable state. In Kilim, when one actor sends a message, it loses visibility of the object it sends, and it becomes the responsibility of the recipient. If both actors need a copy of the message, the sender can make a copy of an object before it's sent to the recipient. Again, Erlang doesn't provide zero-copy (except for binaries) so Kilim's worst case is actually Erlang's best case.

The limitations of concurrent objects in Reia were solved using mutable state in my Celluloid concurrent object library for Ruby, but that deserves a blog post in and of itself.

Single assignment is just as problematic as destructive assignment

Erlang doesn't allow destructive assignments of variables, instead variables can only be assigned once. Single assignment is often trotted out as a panacea for the woes of mistakenly rebinding a variable then using it later expecting you had the original value. However, let me show you a real-world case that has happened to me on several occasions which wouldn't be an error in a language with destructive assignment and pattern matching (e.g. Reia).

There exists a complimentary case of mistaken variable usage to the afforementioned problem with destructive assignment. In single-assignment programs, it involves mistakenly using the same variable name twice excepting the variable to be unbound the second time:

The first pattern matching expression binds the Foo variable to something. In the second case, we've mistakenly forgot Foo was already bound. What's the result?

exception error: no match of right hand side... 

We get no compiler warning in this case. This is the type of error you only encounter at runtime. It can lay undetected in your codebase, unless you're writing tests. Know what other problem writing tests solves? Mistaken destructive assignments.

Single assignment is often trotted out by the Erlang cargo cult as having something to do with Erlang's concurrency model. This couldn't be more mistaken. Reia compiled destructive assignments into Static Single Assignment (SSA) form. This form provides versioned variables in the same manner as most Erlang programmers end up doing manually. Furthermore, SSA is functional programming. While it may not jive with the general idealism of functional programming, the two forms (SSA and continuation passing style) have been formally proven identical.

The standard library is inconsistent, ugly, and riddled with legacy

Should module names in the standard library be plural, like "lists"? Or should they be singular, like "string"? Should we count from 1, as in most of the functions found in things like the lists module, or should we count from 0 like the functions found in the array module? How do I get the length of a list? Is it lists:length/1? No, it's erlang:length/1. How do I get the Nth element of the tuple? Should I look in the tuple module? Wait, there is no tuple module! Instead it's erlang:element/2. How about the length of a tuple? It's erlang:tuple_size/1. Why is the length of a list just "length" whereas the length of a tuple is "tuple_size"? Wouldn't "list_length" be more consistent, as it calls out it works on lists?

When we call erlang:now() to get the current time, it returns {1311,657039,366306}.  What the hell does that mean? It's a tuple with three elements. How could time possible need three elements? A quick look at the documentation reveals that this tuple takes the form {Megaseconds, Seconds, Microseconds}. Separating out Microseconds makes sense... Erlang has no native decimal type so using a float would lose precision. But why split apart Megaseconds and Seconds?

Once upon a time Erlang didn't support integers large enough to store the combination of Megaseconds and Seconds, so they were split apart. The result is a meaningless jumble of three numbers, which you have to run through the confusingly named calendar:now_to_local_time/1 function to get a human meaningful result, which doesn't tell you what time it is now, but instead takes the tuple that erlang:now/0 returns as an argument and will spit back meaningful {Year, Month, Day} and {Hour, Minute, Second} tuples.

Legacy in the grammar

Try to use "query" as an atom in Erlang, e.g. {query, "SELECT * FROM foobar"}. What happens?

syntax error before: ','

This is because 'query' is a reserved word which was reserved for Mnemosyne queries. Never heard of Mnemosyne? That's because it's an archaic way of querying Erlang's built-in database, Mnesia, and has been replaced with Query List Comprehensions (QLC). However, it remains around for backwards compatibility.

You can't use "query" as a function name. You can't tag a tuple with "query". You can't do anything with "query" except invoke a deprecated legacy API which no one uses anymore.

Strings-as-lists suck

Erlang provides two ways of representing strings. One is as lists of integers, which is the traditional way that most of the library functions support. Another is binaries. Erlang has no way of differentiating lists of integers that represent strings from lists of integers that are actually lists of integers. If you send a list of integers in a message to another process, the entire list of integers is copied every time. On 64-bit platforms, every integer takes up 64-bits.

The obvious solution here is to use binaries instead of lists of integers. Binaries are more compact and exist in a separate heap so they aren't copied each time they're sent in a message. The Erlang ecosystem seems to be gradually transitioning towards using binaries rather than strings. However, much of the tooling and string functions are designed to work with list-based strings. To leverage these functions, you have to convert a binary to a list before working with it. This just feels like unnecessary pain.

The abstract concept of lists as strings isn't inherently flawed. In many ways it does make sense to think of strings as lists of characters. Lists as strings would probably make a lot more sense if Erlang had a native character type distinct from integers which was more compact and could avoid being copied each time a string is sent in a message like a binary. Perhaps in such a system it'd be possible to avoid transcoding strings read off the wire or completely transforming them to a different representation, which is costly, inefficient, and often times unnecessary (yes, this is a problem with Java too).

There's no "let"

Want a local binding in Erlang? Perhaps you've used let for this in a Lisp. What happens when you try to do this in Erlang? Even attempting to use "let" in Erlang just yields: syntax error before: 'let'

Once upon a time Erlang was supposed to get let bindings, and the "let" keyword was set aside for this purpose. But much like frames, it never happened. Instead, let is now an unimplemented reserved word which just breaks your programs.

There's no "nil"

In Clojure, I can write the following: (if false :youll-never-know).  This implicitly returns "nil" because the condition was false. What's the equivalent Erlang?

Erlang forces you to specify a clause that always matches regardless of whether you care about the result or not. If no clause matches, you get the amazingly fun "badmatch" exception. In cases where you don't care about the result, you're still forced to add a nonsense clause which returns a void value just to prevent the runtime from raising an exception.

Where do I go from here?

Again, I want to emphasize that I have a great deal of respect for Erlang conceptually. But at this point I'd like to take what I've learned and go elsewhere with it. One direction I've gone is the Celluloid concurrent object library for Ruby. You can read more about it in the original blog post I wrote about Celluloid, which is a bit out-of-date at this point. I have a forthcoming blog post which should dive a bit deeper into Celluloid's guts and how it can do things which aren't possible in Erlang.

As you've probably guess from the references sprinkled throughout this post, I'm learning Clojure. I'm a fan of the JVM and Clojure provides a great functional language for leveraging the JVM's features. I think the sort of things that I'd be writing in Erlang I'll try writing in Clojure instead. Clojure has elegant Lisp syntax. Clojure has maps. Clojure has powerful facilities for dealing with concurrent shared state problems. Clojure has great semantics for safely managing mutable state in a concurrent environment. Clojure has real strings. Clojure has let. Clojure has nil. Clojure runs on the JVM and can leverage the considerable facilities of the HotSpot JIT and JVM garbage collectors.

I'd also like to try my hand at creating a JVM language, especially with the impeding release of Java 7 this Thursday. Java 7 brings with it InvokeDynamic, a fast way to dispatch methods in dynamic languages, and considerably eases the difficulty of implementing dynamic languages on the JVM. Stay tuned for more details on this.

854 comments:

«Oldest   ‹Older   801 – 854 of 854
Damini Sharma said...

She is a most beautiful young Aerocity Escorts for incall and outcall Service. consequently here you are watching her figure and you can envision by seeing her figure how she appeared to be super hot, whoever needs to have an ideal model Aerocity Escorts than in this issue he attempts to locate her information. you will be greatly amped up for seeing her when you locate her before you. Independent Aerocity Escorts are demonstrating some representation of pictures and these pictures are taken and edited yet you can see total pictures if you do get in touch with us.

Petersons said...

There are certainly a lot of details like that to take into consideration.


garage door repair blog

شركة الراية للخدمات المنزلية said...


https://rokn-alabrar.com/%d8%b4%d8%b1%d9%83%d8%a9-%d8%aa%d9%86%d8%b8%d9%8a%d9%81-%d9%85%d9%86%d8%a7%d8%b2%d9%84-%d8%a8%d8%a7%d9%84%d8%b1%d9%8a%d8%a7%d8%b6-0559481301-%d9%88%d9%85%d8%ac%d8%a7%d9%84%d8%b3/

A house cleaning company in Riyadh: All your requests you want will be found in a cleaning company in Riyadh, as it is the best company that has the best detergents, and equipment capable of cleaning the house, with the highest degree of purity and beauty.
شركة تنظيف منازل بالرياض
شركة تنظيف منازل بالرياض:جميع طلباتك التي تريدها سوف تجدها في شركة تنظيف بالرياض ، حيث انها أفضل شركة تمتلك أفضل المنظفات، والمعدات القادرة على تنظيف المنزل ،على اعلى درجة من النقاء والجمال .

شركة الراية للخدمات المنزلية said...

https://www.elmagdclean.com/
شركة تنظيف منازل بالرياض
شركة المجد كلين الرياض من افضل و ارخص شركات تنظيف المنازل بالرياض التي تتميز بالمهارة و الخبرة فى مجال تنظيف المنازل فنحن فى هذا المجال منذ 10 سنوات مما جعلنا نكتسب خبرات كبيرة جدا فى تنظيف المنازل و أيضاً عملنا الدائم فى تنظيف الفلل و القصور الكبيرة جعل مهمة تنظيف الشقق أمراً سهلا جداً و ليس هذا فقط بل لأن شركة المجد كلين الرياض تمتلك من العمالة المنزلية من لهم خبرة كبيرة فى مجال تنظيف المنازل و مع ذلك و مع مرور الوقت فنحن نحرص على ان يكتسب هؤلاء العمال خبرة أكبر من السابق عن طريق المزاولة المستمرة من خلال تنظيف عدد كبير من المنازل و الشقق و البيوت و القصور داخل مدينة الرياض و هذا ما تتميز به شركتنا المجد كلين الرياض افضل شركة كلين الرياض على مر تلك السنوات و هو الفارق بيننا و بين شركات تنظيف المنازل فى الرياض

Petersons said...

Your experience is great and learning for other peoples.


garage door repair Waterloo

Nio Stars Technologies LLP said...

Thanks for sharing...Keep it up..
Nio Stars Technologies LLP is the best
Website Development Company in Pune having a team of a very creative designers and developers. Hire us for creative website design, web application development & digital marketing services too.

Damini Sharma said...

There are different sorts of Aerocity Escorts. On the off chance that you are looking through various types of classification, at a similar spot then it is the perfect spot for you since we have a group of various Aerocity Escorts are dispersed among the classifications and you may discover without meandering on this web see the accompanying sorts of escorts Services are related with Aerocity Escorts Agency, for example, given underneath.

Unknown said...

Great blog, keeping me from working. All the Best


Top Garage Doors 

Shaimaa Elsadek said...

شركة مكافحة بق الفراش بجازان
شركة ترميم اسفلت بجازان
شركة ترميم أسقف بجازان
شركة مكافحة صراصير بجازان
شركة تنظيف منازل بجازان
شركة اعمال لياسة بجازان
شركة اعمال سباكة بجازان
شركة جلي بلاط بجازان
شركة تنظيف فلل بجازان

Petersons said...

מאמר מצוין נהניתי מכל רגע של קריאה

חברת ניקיון

Micky john said...

thanks for giving that type of information.
Finance Magazine

David Smith said...

There are certainly a lot of details like that to take into consideration.

אטרקציות לאירועים

Jack Harry said...

thanks for giving that type of information.
Finance Reports Magazine

Jack Harry said...

Thanks for giving such a wonderful informative information.
dryer vent cleaning pittsburgh

Jack Harry said...

Thanks for giving such a wonderful informative information.
dryer vent cleaning pittsburgh

Unknown said...

thanks… I’ve been bookmarking them for a while now and just decided to create a post to provide them to others…


air duct cleaning pittsburgh

David Smith said...

Post is very informative,It helped me with great information so I really believe you will do much better in the future.

Garage Door Doctor

Jack Harry said...

Thanks for giving such a wonderful informative information.
pittsburgh garage door repair

mahmoud said...


مؤسسة مفتاح الدقة للخدمات المنزلية

شركة عزل اسطح بالرياض

شركة تنظيف كنب بالرياض

شركة تمديد غاز مركزي بالرياض

mahmoud said...

شركة زهرة المدائن للخدمات

شركات التنظيف بالرياض

معلم دهان ابواب حديد بالرياض

معلم دهان ابواب خشب بالرياض

Micky john said...

Nice service there are just few who are providing this service great job.
garage door installation brantford

Micky john said...

Nice content
plumber in tremont

Micky john said...

valuable Content.
reddit essay writing service

Joseph Thomas said...

bowling crew hack You made good thing and share this with others. Im gonna beat you all! Thanks for resources

Tech said...

I really enjoy every part and have bookmarked you to see the new things you post. Well done for this excellent article. Please keep this work of the same quality.
Artificial Intelligence course in Chennai

Micky john said...

I like this post, And I figure that they have a great time to peruse this post, they might take a decent site to make an information, thanks for sharing it with me.
garage door pros

Huongkv said...

Đặt vé máy bay tại Aivivu, tham khảo

vé máy bay đi Mỹ giá rẻ

đặt vé máy bay giá rẻ từ mỹ về việt nam

giá vé đi nha trang máy bay

vé may bay phú quốc

ve may bay di Hue re nhat

Micky john said...

This is my first-time visit to your blog and I am very interested in the articles that you serve. Provide enough knowledge for me.
dryer vent cleaning pittsburgh

Micky john said...

מאמר מצוין נהניתי מכל רגע של קריאה
Gapps ישראל

Sumer said...

Nice Blog !!.....
Shot blasting machine manufacturer's in India. shot blasting machine is the leading supplier, and exporter of shot blasting machine, shot peening, shot blasting room, shot blasting hopper, shot blasting cabinet for sale, portable shot blasting machine price in India. A Shot blasting machine is enclosed equipment designed for abrasive blasting for cleaning and preparing rough surfaces.
See more:
Shot Blasting machine price in India
Shot Blasting Machine Manufacturer's

harpreetmahajan said...


Chennai Escort Services

Call Girls in chennai

Chennai Call Girls

Escort service in chennai

Call Girls in Baroda +91 9163765477 said...

ashram road escorts service

escorts service paldi

call girl service naroda

escorts service bodakdev

call girl service vastral

escorts service odhav ring road

vastrapur escorts service

call girl services south bhopl

prahlad nagar escorts services


Call Girls in Baroda +91 9163765477 said...

Escort Services in Ahmdabad

Baroda Escort Services

escort service in vadodara

escort service in alkapuri

prahlad nagar escorts services

sg highway escorts services

escorts services cg road

call girl service maninagar

Call Girls in Baroda +91 9163765477 said...

ashram road escorts service

escorts service paldi

call girl service naroda

escorts service bodakdev

call girl service vastral

escorts service odhav ring road

vastrapur escorts service

call girl services south bhopl

prahlad nagar escorts services


Shaimaa Elsadek said...



شركة غسيل مكيفات بجازان
شركة رش دفان بجازان
شركة تعقيم بجازان
مؤسسة قص وتخريم الخرسانة بجازان
شركة رش مبيدات بجازان
شركة مقاولات بجازان
شركة كشف تسربات المياة بجازان
شركة تنظيف مجالس بجازان
شركة عزل اسطح بجازان
شركة عزل خزانات بجازان

Unknown said...

thamks for sharingoverseas education consultants in delhi

admin said...

check my accounts: pinterest - gumroad - vingle - instapaper
tashilaty

Unknown said...

قفل استخری
قفل درب حیاط
چشمی دیجیتال

Damini Sharma said...

Several players deal in supplying the most efficient Aerocity Escorts and then have a great time. On several occasions, the interested client's interaction takes place with the second-rate form of adult love provider. This, in the long run, does not turn out to be a good solution. Females that are offered are not interested in serving the client properly. A lot of argument takes place between the client and the erotic Delhi Escorts provider. To counter it, the correct source is required to interact with it. Now, the hot babes will be approached easily.

سونجول said...

شركة نقل اثاث بالرياض 0500091013 ارخص شركة نقل عفش – إدارة سعودية
تقوم شركة نقل عفش بالرياض بتغليف العفش باعلى واجواد انواع الخامات المستخدمه كالاسترتش نقوم باستخدمه فى
حالات تخزين العفش حتى لا يودئ الى تلف الاثاث فى حالا ت التخزين لفترات طويله كما نستخدم الكرتون
المضلع والفوام والبابليز وسلتيب ثميك للف الاثاث وحمايته من التجريح والخدش كما
تستخدم شركة شركة نقل عفش بالرياض عربيات مجهزه لنقل الاثاث وهناك مقاسات مختلفه من تلك السياره
تقوم شركة نقل عفش بالرياض بتغليف العفش
فك وتركيب اثاث المنزل
تغليف الأثاث
خدمات التنظيف
افضل شركة نقل عفش بالرياض
نقل الأثاث الي أبعد المحافظات و الأماكن
أفضل و أرخص شركة نقل أثاث بالرياض
هدف شركة القمة هو راحة العميل من خلال تقديم خدمة شاملة و مميزة بأرقي أسلوب و دون أن يتحمل العميل أي من متاعب النقل فهي تمتلك .
سيارات مجهزة علي اعلي مستوي المفتوح منها و المغلق .
عمالة مدربة مكونة من نجاريين و فنيين متميزيين .
الأوناش الهيدرولكية التي تصل الي الأدوار الشاهقة .
مجموعة من أكفء قائدي السيارات الحاصلين علي رخص قيادة يستطيعون تحمل القيادة في كافة الطرق و السفر الي أبعد المحافظات .
لديها ايضا جميع خدمات الفك و التركيب و التغليف و النقل و الرفع والتنزيل ( يدوي _ أوناش هيدروليكية ) ا بواسطة أكبر المتخصصيين و تحت اشراف مجموعة من أكبر المشرفيين .
افضل شركة تخزين عفش بالرياض
وكما توفر شركة نقل عفش بالرياض شخص لتنظيم العفش داخل السياره عن طريق وضع الزجاج فى اماكنه
والرخام فى المكان المخصص لها معنى تلك الكلام ان شركة مفكو ايجيبت تضمن لك توصيل الاثاث
بدون خدش واحد اوكسر كما اننا لانحصل على اى اموال الا بعد اتمام العملية تماما وبعد ارضاء العميل هدفنا هو
وصول اثاث الشقق والفيلال والقصور والشركات والفنادق والمحلات الكبراء فى جميع المحافظات بدون خدش واحد فقط.

كما انه منوف لدى شركتنا خدمة تخزين الاثاث فلدينا الاماكن الجهزه لاستيعاب تخزين الاثاث لاي مده ولاطول فترة يريدها العميل فلدينا الامان المجهز للحفاظ على الاثاث من الرطوبه وتغيير الجو والحشرات لدينا امهر العماله الثي تقوم بخزين الاثاث باحدث الطرق والاساليب

نقل اثاث بالرياض
[URL=https://elawaeil.com/%D8%B4%D8%B1%D9%83%D8%A9-%D9%86%D9%82%D9%84-%D8%A3%D8%AB%D8%A7%D8%AB-%D8%A8%D8%A7%D9%84%D8%B1%D9%8A%D8%A7%D8%B6-%D8%B9%D9%85%D8%A7%D9%84%D8%A9-%D9%81%D9%84%D8%A8%D9%8A%D9%86%D9%8A%D8%A9/]نقل اثاث بالرياض[/URL]

Damini Sharma said...

If you go online and explore the internet, so many Delhi call girls would offer their services with your desires. But here, it is necessary to make sure that you have got a girl of your choice. Don't avoid any crucial points when you start looking for a sexy Aerocity Escorts you should spend a little of your time on the internet and choose the one after comparing their services and other related things. If you are in Delhi and want to hire high-class and sexy Delhi Escorts, you should stop for a while and check whether you are hiring the right one. Many professionals are providing their services to satisfy their male clients' sexual desires.

Micky john said...

A useful post shared.
garage door installation brantford

moumita said...

Excellent and nice post. It will beneficial for everyone. Thanks for sharing such a wonderful post. Buy Instagram Followers India

Gaurish said...

Great information, i was searching of this kind of information, thank you very much for sharing with us.

Craigslist Posting Service for Car Dealers |

Gaurish said...

This is really amazing website that I have been found on google regarding website Blog Commenting sites. and I would like to thank admin who also given us to post the link on his side.
Lubbock moving company |

Gaurish said...

This was something I was looking for, really helpful, and great work is done. Thank you so much for sharing such valuable information.
Car Auction Software |

Nessco India said...
This comment has been removed by the author.
Nessco India said...

Nesscoindia A Trusted and Audited Exporter that manufactures and supplies a wide range of Paper Cup Making Machines pairing with the latest technology. We provide a High-Speed Paper Cup Machine Series, Paper Plate Making Machine, Face Mask Making Machine, Flexo Printing Machine, Roll Die Cutting Machine, PE Coating Machine, Paper Bag Machine, Paper Straw Machine, and other machines in the disposable paper range. We are the best paper cup machine manufacturers in India.


High Speed Paper Cup Making Machine
"Fully Automatic" Paper Cup Making Machine
Paper Plate Making Machine
Roll Die Cutting Machine
Flexo Printing Machine
Disposable Paper Cup Making Machine


Nessco India said...

Nessco India is an audited and trusted manufacturer and exporter of a high range of disposable Cup Making Machines. With over 40 years of experience in the production of disposable paper cups, NESSCO is not a relatively new name but an old master in the manufacturing industry. Nessco disposable products making machines are embedded with the latest technology. Our high-speed disposable product-making machines listed below.


Disposable Paper Cup Making Machine
Paper Glass Making Machine
Beer Cup Making Machine
Tea Coffee Cup Making Machine
High Speed Paper Cup Making Machine
"Fully Automatic" Paper Cup Making Machine
Paper Container Bowl Making Machine
Paper Plate Making Machine
Flexo Printing Machine
Roll Die Cutting Machine

Unknown said...

Hi! I really like your content Your post is really informative. I have learned a lot from your article and I’m looking forward to apply it with my children and in my class too.
main mumbai result
matka tips

Unknown said...

This was a fantastic blog. A lot of very good information given,. I have saved this link and will return in a couple of months, when I need to build my first blog. Thank you for the information.
balaji day result
matka charts

Unknown said...

RC car tips
Official Result BD

online reputation said...

It's always exciting to read articles from other writers and practice something from their websites.
garage door springs north dallas

online reputation said...

It's always exciting to read articles from other writers and practice something from their websites.
garage door springs north dallas

«Oldest ‹Older   801 – 854 of 854   Newer› Newest»