Thursday, December 04, 2008

New Project - URLCron

Following up my quiet annoucements yesterday on Erlami, Erlcfg and Fastiga, I've just pushed another project out to github... UrlCron.

The way I architechture out platforms, closely mimicks the way UNIX programs are advisedly written... i.e. Small Programs Which Do One Thing, Do Them Well, And Can Read From Standard In and Write To Standard Out In A Simple Format.

In our case, most of our applications usually are made of up many RESTfull webservices, all talking to each other via HTTP Calls and JSON responses. The UIs are all Heavily ClientSide javascripts, that also talk to the various webservices. Normally, each application will have one authoritative application server, which is also a RESTfull webservice, that is directly responsible for knowing *everything* about this particular application.

For instance, one of our media campaign engines has an appserver which deals with authentication, authorization, media setup, etc for various front ends (HTTP clients, SMS clients, IVR clients, etc).

Following this very very loose coupling and disconnected architechture, we tend to evolve lots of small stand alone systems. And a lot of times, we need to set timeouts, callbacks, schedules a lot of activities that should be deferred etc. This can all quickly degenerate into complexity and incompatibility and lots of duplicate code, and can become very language dependent. Also, the option of using the venerable cron daemon to call urls from shell scripts is not very nice, and is too much of a hack.

Enter URLCron. UrlCron is designed to fit into our disconnected mesh of services, to allow us to schedule a call from any service to call any other service and then store the results. The scheduling service can come back at a later datetime to check the status.

In its current implementation on github, its just a few days old and is not yet mature, but the entire concept works end to end.

The readme on github contains a lot of ideas of where I would like to logically take this to, and some notes on architechture and design.

Someone else wants to have fun poking around at this? Go crazyyyy!!!!

Ace out!

Labels: , , , , , ,

Wednesday, December 03, 2008

Sweet November

November was a good month. Yeah really was... and yeah i know... the world is in general economic recession, but hey... Obama won!!! :)

Anyways... last month, saw three of our inhouse projects quietly released on github. My github page is at:

http://github.com/essiene

I'll just use this post to talk briefly on the three projects.

ErlAMI

ErlAMI (pronounced Erl AM I) was my first serious Erlang project that was not hot off the tutorial presses :)

I cut my proverbial Erlang teeth on this project. It is an Asterisk AMI protocol library done in Erlang/OTP and trust me, I've learned a huge load just doing this project.

I needed to build an automated dialler based on Asterisk as part of our inhouse product stack, and defintely needed to talk to Asterisk. This was a project that I had started in Java a couple of months back, but after taking the red pill, I decided that the Matrix had me, and I just had to do this in Erlang.

At the time I started it, I was still reading Joe Armstrong's "Programming Erlang", so walking through the git
history will feel like a trip through Joe's book.

I started by rolling my own structures and processes everywhere, learning new features of the language and using them, learning eunit and other things and rolling them in.

Eventually, I had built my own poor man's OTP. I had my primitive supervision trees, FSMs, servers, even a poor straw man's event manager and handler framework.

Impressive, but as I finally completed the OTP parts of the book, I realized I had to port to OTP because of the immense gains, and just the raw amount of iron clad tested code I would be basing on. I did just that, and the result is the current Erlami.

This one will take a while to mature, but i'm continually finetunning my ideas and feeding them back here, and there are some funny bugs and corner/edge case issues here and there, which will get cleaned up with more use.

Erlcfg

This is the new kid on the block, and an idea I just had to execute... plus, since last year, I've made it a job of mine to learn compiler tools on every language that I know and use. In this case, I just had to get my hands on leex and yecc.

The idea is simply my own extrapolation of the Java properties file format.

Normally a properties file looks like:

some.key = value
some.other.key = other value
some.other.other_key = other other value

I like this because it is naturally namespaced, is eye friendlier than XML and my java SDP platform uses properties files strictly for configuration. The only problem is verbosity, which is caused by non-nesting, i.e. The format does not allow itself to be refactorable. For instance, examining:

some.key = value
some.other.key = other value
some.other.other_key = other other value
some.other.other.key = other value

It would be more maintainable to rewrite this as:

some = {
key = "value";

other = {
key = "other value";
other_key = "other other value";

other = {
key = "other value";
};
};
};

Also, this bears some resemblance to JSON and in some way YAML. Well... erlcfg aims to do just this, and add something else... VARIABLES! Checkout the README for a good complex config file example.

Currently, the syntax is very strict (notice the extra assignment operators and semicolons... annoying!) to help make parsing easier. But since I have achieved my initial goals, I'll go back and make it easier to write.

Also, I have some interesting plans, like adding XML DTD type support, or simple type annotations to help in verification of config file syntax, easily, this would allow you to define:

server {
port = integer;
listen = list(string);

log {
data = string;
level = atom;
}
}

And the use this to verify

server = {
port = 111;
listen = ("192.167.10.1", "192.168.10.1");

log = {
data = "/var/log/applog";
level= debug;
};
};

I would then love to be able to verify the file by doing:

$ erlcfg --verify /var/lib/configfile.dtd /etc/config.conf

This is still a nice dream right now, but trust me to keep hacking at this till it is done. Did I also mention that it supports all erlang terms except the tuple?


Fastiga

This one is a scala project. It is an implementation of an Asterisk FastAGI container in scala.

This really came about from me having a very specific way I wanted to build my AGI applications, i.e. like the Simple State Machines that they are. Basically, almost like you would build a parser from a BNF specification, I wanted to build my AGI applications to match my state diagrams.

So I set out to eliminate IFs with MATCHes, build case classes that would allow those matches and employ scalar Actors and Tail Recursion to make it all work nicely. Also, using reflection, we can host multiple AGI applications inside just one container, which currently is deployed inside of another servlet container, hence allowing more packagable code and promoting better code reuse.

You work on your AGI logic, let us handle the running and hosting framework.

Its still a work in progress, but the main ideas are realized and it is being used heavily in-house.

We actually have a *thin* framework around it, (much like the OTP is a framework around Erlang) which
I'll tidy up and make a part of the project.


Looking Forward

Honestly, more things are coming down the pipe... lots of ideas and tools that we're using inhouse, which I believe will be valuable to some other team of harrassed developers like us. :)

So this is hoping for a more productive December and onwards.

Ace... out!