Friday, October 17, 2008

Client/Server design and implementation: Discovering the symmetry in implementation

I experienced a paradigm shift in my design approach recently, which I'd like to put into writing, so I can formulate the rough thoughts in my head better.

I've been working on an Erlang AMI library implementation for a couple of weeks now.

I set out initially to write a simple AMI Client library that I could use to implement my application. On the way, I had to learn eunit so I could thoroughly test as I built, not to mention, quickly build test the small blocks before integrating them.

After I built a working prototype (with plenty of bugs which I was well aware of), I knew I had to unittest this properly, if I wanted to have confidence in the code not to mention bragging rights for well tested code.

First a problem
The problem with testing external connections like this is usually bootstrapping. If I have asterisk installed on my dev machine, writing the tests is not a problem. But if J. Random Hacker grabs my source, and
out of habit types:

make test

They're only going to get pass marks only if they have asterisk configured, and by some telepathic ethos among hackers, also set up his asterisk instance to have the same username and password as my own, among other things (channels, etc). Unfortunately, this doesn't happen as much as we'd love it to :).

There is a way around this using Mock frameworks (which I've never used by the way :D). Anyways, I did what any hacker worth his mettle would do, I sucked it up, stashed the client code in its buggy state, and went ahead to implement a simulator from the same code base.

You what?!!!

Yup... I'm building AmiSym, which is an Asterisk AMI Simulator, and it shipswith the ami library (so you get a client and server for the price of just the client). Now onto the cool parts of doing this.

Advantage Number 1 - More experience
You can never over emphasise the value of increased experience when solving a domain specific problem. In my case, I'm basically solving the AMI problem a second time, even though it is from another perspective
(more on that below). This give me more exposure to the problem, than I would have had if I'd done a one-time-coding-pass and written the client library.

Infact, I end up tackling the AMI library problem 3 times. Once when I implemented the prototype. Once when I implemented the simulator. And once when I reimplement the client with the experience I've gained from implementing the simulator.

Advantage Number 2 - The other side
If I had stuck to writing just the client, I would probably come up with a well done client (I mean, I love to consider myself a not-so-shabby programmer), but now that I am also implementing the server side, I get
to see the problem from both sides of the coin. And that has impacted the way I understand the communication and the mechanisms that the client has to implement because I'm designing both the producer and consumer.

For instance, I created an internal data structure to store the Key: Value pairs that are AMI responses, with a simple hack to deal with things like action: command results, which contain extra data that
don't come in Key: Value pairs.

This worked satisfactorily for as far as I was just a client. But once I started implementing the simulator, I decided I had to use the same data structure on both sides of the connection, immedietly, I found out my
Data structure wouldn't be symmetric. That Is, I couldn't do:


and then


This was a problem, since it would mean, either two code bases, or just mean that I should redesign my INTERNAL REPRESENTATION, which is what I did.

So I now have a more robust Data Structure, because I took a trip to the other side.

Advantage Number 3 - Borrowing From the other side
If these were the only advantages, the code would already be better off, but there is more. My design of the simulator State Machine and internal processes, end up being very sensible. I attribute this to the fact that my first attempt at writing the simulator is in actualty, my second attempt at tackling an AMI library core.

Now, going back to the client, and I'm borrowing a MIRROR image of the simulator process in the client. Simply put, this makes a *lot* of sense. It makes sense for a Client and Server of the same protocol, to be mirror images of each other, such that when plugged together, and a message in inserted into any them, it will loop through the virtual ring that is formed and "theoritically" come back in the same representation.

This is all kind of abstract, but just think of two halves of a hoop that fit perfectly together... or YinYang... each groove in one is complemented by an extrusion on the other, so they're really just each other, only turned inside out.

I think that approaching any problem space from this perpective produces a much more robust and complete design and coded implementation. Or to state it in cooler terms... realizing the Zen of YinYang in
Client/Server systems yeilds better code and a better design.

Advantage Number 4 - Modularity
The final advantage I want to bring up is extreme modularity. Usually, one can relate to modularity in its more common form. Take for instance a problem where we write a generic TCP framework that allows one to inherit and extend to either an HTTP or SMTP module.

This I would like to call forward-divergent modularity, which refers to a single code base being made modular to allow the core functionality to be easily diverged down an irreversible path.

This is not type of modularity I gained. The type of modularity I'm referring to, which I'd like to call cyclic-divergent modularity would best be described with the help of some funky diagrams. Enjoy :P

The figure above shows what I have termed Forward-Divergent Modularity. Each of UNIQUE CODE A and B have a common core or set of common core modules, which do similar stuff (like open a socket, setup a generic session, store client_id/ip mappings in a HashMap, etc), but then each of them have their unique portions (SMTP protocol versus HTTP protocol).

I call it forward-divergent, because a virtual message travelling through this system, would either start from CORE and move towards A, or move towards B. A system that is structured this way is an Either/Or system. Nothing would make a single message cross camps. You would not be able to "virtually" LOOP the above system at the UNIQUE CODE POINTS without introducing a protocol converter (mostly impractical) which
confirms that even though they have the same base, they're different logical systems.

Now onto cycle-divergence.

The figure above shows what I term cyclic-divergent. As you can see, this entire system turns out to be a Single virtual system, even though there is some uniqueness in A and B.

In this case, a "virtual" system message can be inserted at any point, and it should theoritically, go through various transforms, but by the time it returned to its originating point, it should be back to the same format it originated in.

In Conclusion - Dude Stop Being Abstract
Ok, I'll stop. :)

I don't know about you, but I'm always thinking about code I write, and realisations like these, make me a better systems designer and implementor. For instance, now that I know that I'm actually building a single virtual unit, I have been able to identify that common core in my code base now.

My work from here on out is to widen the common core as much as possible, so that the unique parts are very small. This will result in a more robust client library and server library, that both depend on a well tested and shaken core. This is most decidedly better than implementing one in absense of the other would have resulted in.

As an aside, I'm not so sure I would have seen this using a mock framework (I've never used one, so I don't know). This is not an anti-mock framework rant, just an observation anyways. My personal conclusion then is that whenever one is implementing a protocol-ish library, there could be a lot to gain from also building the Other Side of the protocol in the same code base. The important point of note is IN THE SAME CODE BASE.

So if I were to write an HTTP server, I would also write a client in the same code base, and use that in my testing. Too much work? Yeah, maybe... but its fun... and its got its perks :)

Peace out.

Labels: , , , , , ,


At Saturday, October 18, 2008 5:24:00 PM, Blogger Anupam Kapoor said...

oh yeah. i have been doing this (not in erlang btw) for quite sometime now. i was under the impression that this is the "standard" way to go about doing things anyways.

one more cool thing here is that, if you have the core parts as a library, then you can have the client act as it's own server !

At Tuesday, October 21, 2008 11:22:00 AM, Blogger Essien Ita Essien said...

Didn't know it was kinda a "standard" way of going about things. Damn... so I'm late to the party :)

But like you say... yup! Wayy cool.

I guess it just tickles the symmetric artist in me :)


Post a Comment

<< Home