Saturday, November 01, 2008

Erlang/Scala... gateway to State Machine Oriented Programming



For the past couple of weeks, my team has been working on a version two of one of our apps.

The first version was basically a Python application, that called out to a Java REST webservice. This webservice talked to asterisk via AMI.

There was also some python AGI scripts also for doing automated IVR sessions.

It worked, but there where a lot of ideas after the first version was done. So I did what any sanity loving team lead would do.... branched out git repo and started work on version 2. Now, our version 1.0stable is available for clients, but we're shooting to quickly start recommending version 2.0 and already, just after 3 weeks of very hard work, we're smiling and approaching the end.

New Beginnings Architecture
The structure of the new application is this:

The Java AMI REST webservice, has been replaced by an Erlang/OTP/Mnesia/Mochiweb webservice. If you have any idea what those terms mean, you can already see that I've gained a lot of features.

My Prioritized FIFOs are now distributed across multiple nodes... at almost zero coding cost (I didn't find a comfortable way of doing this in Java). (We'll be releasing the Erlang AMI library next Month as open source software).

The python AGI scripts have been replaced with Scala AGI application modules, seated in a novel FastAGI server built also in scala. (This server will also be released as open source).

The rest of the application is a Comet/Oribited/JQuery/Pylons over mod_wsgi mix.

Why Oh Why?!!!
Now, this is the fun part... why?

For the last 1.5+ years I've been working exclusively in Telecoms integration middleware and SDP space, building and deploying services for very agrressive customers, one of them now the fastest growing Telecoms company in Nigeria (if not Africa). And that experience left me with some major impressions:

  • My approach was still error prone.
  • The problems are simple but exacting
  • There is NO ROOM for NON-EXACTNESS... ZERO!!!
I learnt this lessons, rather shockingly, since I consider myself a carefull artisan, always striving to improve my art, and having done so more than a rather no-too-small percentage of others :)

Enter Application Patterns
The first thing I noticed was that all the applications follow a particular pattern, and I rightly set about building a framework to easily enable writing of these applications... I'm pretty proud of that framework and its humming silently in more than 3 different sized telecoms networks.

The framework has adapters to connect to various network elements, USSD Gateways, SMS Gateways, MSCs, Mediation Servers, etc and convert all incoming messages into a common format (based on HTTP URLS). These requests are routed on the framework router to the Service endpoints... the Handlers.

The handlers are the core of the framwork and do all the SCP, AAA, etc, connections and business logic.

After writing a couple of the handlers, more similarities stood out, so I kept fine-tuning that part of the framework untill very little of it was left to be customized for just the business logic. The plan is to eventually develop a DSL that can be used to write that part and deploy in realtime.

Where Did I Go Wrong
Along the way, I noticed that some of the applications would still evade my exception handling. Also, once in a while, when traffic would spike, I would get some errors which I *shouldn't* be getting. I would
rise to the occasion and fix them, but after one pretty hairy issue with some lost transactions, I really began to question my entire foundations.

There seemed to be something amiss, I couldn't seem to translate the very simple logic flow into similar Java code, without some corner and edge suprises (and I unittest A LOT!). Something was definitely missing.

I would solve the problem, but decided that I needed to change something very fundamental.

Does This Tunnel Have An End?
That was when I started my gazillionth trek up the Functional programming hill - out of desperation. I had dabbled in Haskell 2 years ago and left off, because there where no practical libraries. I'd had a brief fling and excitement with Ocaml around the time I dropped Haskell, but dropped it too, though I really liked Ocaml and was more productive, I dropped it because it didn't have good libraries (these situations are changing as I write... just last month or so, both Ocaml and Haskell have annouced Battries Included projects... hooray).

I had also touched Erlang but hurriedly dropped it... I mean... ewwwwwww.... was that even readable syntax? I had played with various dialects of lisp, (common lisp, newlisp, a bit of scheme), but somehow, my inner programmer didn't gel with any of these, at least not yet. Maybe it was just the timing, but I like to think that I just wasn't ready, I hadn't faced my own waterloo class of problems yet.

The Matrix Has You... errr... Scala
Anyway, this time, as I started exploring other options again, something was different. I was looking out for a language that:
  • Would allow me to write code without carrying tooo much state information about, so I didnt' get shot in the foot when I least expected
  • Had very broad library coverage
  • If possible allow me build on what I'd already done.
  • Could allow me build my dream distributed persistence framework (in my class of problems, Map/Reduce hence Hadoop/HBase/CTable is not the solution... I was looking for a mature DHT or something better with object serialization support (Scalaris,Mnesia!).
  • Easily allow me scale out without worrying about concurrency issues, locks, synchronication, etc.
Anyways, that was when I first came across Scala the language. (

Scala combines very strict typing, functional and object oriented paradigms with very sexy type inference into one badass language that runs on the JVM, (and I hear .NET CLR too).

I couldn't believe it, so I took my time dipping my feet in the pool and sampling.

Learning Strategy... the Stalker/Prey Approach
There is a way that I like to learn when I'm really really serious. I search to web for all sort of references and tutorials, blogs, articles and what nots about my prey (well... technology doesn't sound as cool :) ), and then take my time, going thru all the trivial examples, code snippets, hanging out on IRC, etc... just to immerse myself in the feel on the technology. When I have a reasonable feel or the technolofy and community, I now dive in head first via the "recommended text or tutorial".

Long story short, I began to stalk Scala. Funnily, in so doing I became the prey! I know... its weird, but eventually, Scala and I just "clicked". I don't know what did it, if it was the special sweet spot on the JVM, or the combined OOP and Functional paradigms, or the fact that I could import and just use my Vast Array of company code that had been writen already in Java. I don't know but we just clicked. And I knew I had to see this one through.

My first plan was to investigate how to write a servlet in Scala and see if I could eventually swap out my Java handlers with Scala handlers. I started investigating that productively, but kept happening along another big mine that blew up in my mind...

Erlang: A Second Coming
As I read and played with Scala, the Actors library implementation just kept coming up and how it was a clone of something that was native in Erlang. Well, I decided that if Scala stole it from Erlang, it had to be worth finding out the main thing itself. I made up my mind, and I picked up a second victim.

Still syntax-wary from my last attempt at Erlang, I went for the stalking now with questions on my mind.

Then a funny thing happened. This time, I waltzed over the Erlang syntax... and I still don't know why, but that syntax just seemed different, instead of weird. After I kept at it for a couple of days, the veil simply lifted.

So What's The Bling?
Erlang has so much goodies its hard to imag ine that its open source. Its usually features that rich, that make companies like Microsoft, keep close tabs on their languages.

Anyway, my easy way of explaining Erlang is this: Erlang feels like a domain specific language for highly available and highly scalable applications with very low error rates and high integration to other environments.

Take some time and read that again. Out of the box, erlang allows you to very cheaply create concurrent applications that can be distributed on mulitple network nodes. Its a very basic concept, not an advanced language feature. (spawn, !)

These processes are encouraged to be built for failure, so you don't try to prevent your processes or nodes from dieing (A concept I call Fighting Your Exceptions). Instead you encourage them to die, quickly! Erlang provides process parenting/monitoring as another very basic concept, to allow you restart dead processes. (spawn_link, link, monitor, supervisor)

Finally, what I'd always loved in C and detested in Java (low-level bit manipulations) are present in a very convenient data type: Binaries

This means a lot to me. I used to fall back to C/D to write low-level binary protocol clients, and use the library to build a proxy-daemon which spoke plain-text, and then connect to that from Java, Python, etc... now I just do the entire back-ends in Erlang!

Programming As State Machine Construction
Finally, I come to the main point.

Has any of these helped me solve my problems with my previous approach? Hell Yeah!!!

  • Immutability in Scala/Erlang helps me avoid a very nasty class of concurrency problems... race conditions in multithreaded Java code.
  • Immutability also encourages localized variable definitions, which encourages what I call "terminal functions". These are functions whose scopes terminate in themselves. The have all they need inside them, without looking for a global or class wide state. Everything is either passed in or initialized there. And debugging them is WAAAAAAYYYYY simpler :)
  • Higher Order Functions, Anonymous functions and Currying, provide some of the bassaddest refactoring tools known to man (and woman... if she can program :) ). Its just sooooo cool refactoring code in these languages. Even Python has to take a back seat sometimes, and that is saying a LOT because Python is darned flexible.
  • Tail Recursion (probably my favourite feature): Allows me to write pure State Machines. Some event driven, some acceptor state machines. This has been my biggest boost in the last couple of months. I sit down and come up with 3 types of diagrams:
  1. An application layering diagram, that gives me the various service layers and defines the protocols, contracts and messages between each layer.
  2. A State Transition Diagram or and State Table, which describes completely all the states for each layer and the messages and transitions between them
  3. A flowchat for each state in each state State Machine in each application layer. I now sit down and virtually translate the diagram into code that works without suprises!! I never felt this was possible in such a short time.

I use the Erlang/OTP framework, especialy gen_server, gen_fsm and gen_event, a lot to simplify my life and use Scala Actors intensively. I plan to try out the Scala/OTP library that is being built here:

I'm in the business of software, and one of our company motos and policies is to solve hard problems that plague our clients. We continually come up with innovative solutions and deliver them, and most of them end up being used by lots of concurrent users.

Scala and Erlang have shown in a very short time, that the future will be pretty different from what we currently know today. I think the era of the Swarm is upon us, where most of our applications are going to be used by a myriad of users. The ability to be able to write correct code that is easily debuggable and run in a scalable environment is going to be more and more of a differentiating factor, and hopefully, that can translate into pre-forty year old retirement ;)

In all seriousness, if you've not looked at a Functional programming language for your choice platform, you do your self a serious disservice, which can only be fixed by looking into one, SERIOUSLY as soon as possible.

Ace... out.

Labels: , , , , ,


At Saturday, November 01, 2008 1:35:00 PM, Anonymous Anonymous said...


I've actually been working on AMI-Erlang-ReST type of a project. I have found it to be a joy (except with the GetConfig and UpdateConfig parsing).

My old version used Ruby and even though the ReST part was easyer it just can't beat the robustness of Erlang when it comes to socket communication.

At Saturday, November 01, 2008 8:56:00 PM, Blogger Ulf Wiger said...

Very nice article!

At Monday, November 03, 2008 12:05:00 PM, Blogger Essien Ita Essien said...


The AMI interface is kinda klunky, but I guess there are reasons for it to be the way it is... I keep toying with creating a protocol aware proxy that speaks JSON and is event aware, unlike AJAM.

Once I wrap up this project, I'll publish my AMI library, the more the merrier ei? :)


Thnx. That means a lot... especially coming from you :)

At Wednesday, November 12, 2008 12:13:00 AM, Anonymous Anonymous said...

wow .. very useful information.
Thanks for sharing.


Post a Comment

<< Home