Sunday, April 17, 2005

Software design by Evolution - What Charles Darwin Had to say on programming.

Software design by Evolution - What Charles Darwin Had to say on programming.

There is the age old advice of sitting down to figure out every single thing about a system before attempting to build it. This works in lots of well structures that don't have changing requirements, and are usually cast in stone once the design is
approved. The problem is that in software systems, requirements keep changing, while delivery dates keep sliping.

It seems then that this method that works so well in a field like say, Civil Engineering and Construction is not doing so well in Software Development. A lot of other very intelligent people have talked a lot on the pros and cons of Detailed Initial Design, so I won't give it
much attention here. Instead i will talk about a method i've been using while coding, and has allowed me to challenge my clients with statements like, "This project can pretty much go any direction you want it to go.", "We can change anything, any how you want it". And just incase, you have not yet figured it out yet, i actually have been encouraging scope creep from my clients :). Should I add at this point that once faced with that possibility, they actually didn't take up the offer. (Uhh... don't try this with your clients... you may b biting off much more than you can chew ;) )

Anyway, the point is that it is possible to code flexibly, in such a way that features are not frowned upon, and the resulting code is still very well built and maintainable. I call this method 'Design By Evolution'. Actually, this is just a fancy name for Bottoms-Up programming, or what i usually love calling, the "UNIX" way, just for the simple reason that I first saw evidence of its existence in the way UNIX programs are written.

So what is Design by Evolution?
In a nutshell, it is a method of code design, where existing code naturally determines the next evolutionary step (which happens to be new features, etc). Also its most important feature or concept is that you NEVER design what you're not building immmidietly. In Agile development practices, i think this is called deferment or something like that.

Now, let me take it slightly deeper. Traditionally, if we want to design a program that accepts and parses various structure, read from a database, and the results stored into a file. We might want to start by building a super class that all the data
structures will inherit from, that will have methods for reading from the database, writing to file, etc. This is traditionally good OOP design. In 'Design By Evolution', I wont commit to the super class untill the need comes up. For instance, if in my first release, i will only be dealing with one type of data structure, Traditional design will be best served if i already have the super class built, at this first release, but design by evolution begs to leave this decision until it is unavoidable. So i would release my first version with just support for the single data type.

In a second version, if its justifiable to build a super class then i do, but if its not, i still hold off untill it becomes unavoidable. This may seem like concious bad design, but that's because I've already mentioned the constrasting traditional approach. In practice what this means, is to consider ONLY the problem that is before you, with NO THOUGHT of implementation detail on problems that have not yet arisen. My good friend Stan always says, "We'll cross that mountain when we get there".

What this does is many fold. First of all, it removes the overhead that comes with trying to figure out a solution so that you don't get stuck latter on. I say that this worrying is MOST of the time, premature. If you keep to good coding practices of modularity and small contained non-overlapping units, you'll be able to tear down and rebuild sections of code that need
to be torn down and rebuilt, without breaking other parts of the code. If indeed it ends up being a lot of work to do, i say that its more advantageous to do this work latter when it is needed, than earlier when its not needed.

The other aspect of Design by Evolution is the fact that its easier to build an EarthQuake resistant Airport (like Kansai Airport :D ), if you've actually been exposed to Earth Quakes, than if you learnt about Earth Quakes from School. This is because real life is soooo different from imagination. Relating this to programming and design, its usually easier to have a
code base without a feature, and then work the feature into the code base, within the design limitations afforded by the existing code base, than it is to design a feature complete code base and begin to build it.

This is because no amount of design can envisage every possible situation (except in the most trivial of cases), so there is almost always going to be a need to redesign and change initial design doing actual coding. Because of this reality, it is usually better to only design
and build what needs to be built at the time, and leave the rest for latter.

At this stage, It best i make this point. Design by Evolution is a very aggressive coding methodology, and is very non-forgiving of bad programming practices. If you make basic coding mistakes like tight-coupling of functionality, Design By Evolution will burn you. If you code without a rigorous testing methodology like Test Driven Development, Design by Evolution will burn you. If you leave unrefactored sections of code as you add to your existing code base, Design by Evolution will burn you. In fact, i could actually say that the foundations of Design by Evolution are 'loose-coupling', 'Refactoring', 'Testing', 'Testing' and 'Testing'.
If you mix 'loose-coupling', 'Refatoring' and aggressive testing, what you come up with are extremely stable reusable components. These can be libraries or smaller programs that can be glued together (which is the preferred 'UNIX' way), or both. Software that is built like this is very stable and flexible.

There is something that must be said about how the code evolves. When you defer design decision until the need arises, the existing code base imposes restrictions on what you can/should do or avoid. This is actually better since the code can be looked at to be choosing its own evolutionary path. If we had to make this decision before the code is built, we're bound to be a bit off the mark, but leaving this decision to be virtually made by the code itself, brings out the best combination of stable code and stable extension than would be possible otherwise.

Probably, the greatest advantage of this approach is the fact that there is no virtual closure on the code imposed by prior design. What do i mean? Well, when you first start by designing everything, you make some irreversible decisions/trade-offs, thus killing off some evolution paths that _may_ be desirable latter. On the other hand, leaving the code to evolve stage by
stage, while abiding by loose coupling, only kills of evolutionary paths for the individual parts, which could be easily reopened by reimplementing those parts. This way, we don't close code ourselves into a corner, and even if we do, we can just break down that corner without breaking the rest of our code.

[ADDENDUM: I added this following part, after suddenly realizing that Design By Evolution is not a word i came up with, but its been in existence in the Agile Programming Camp. I decided to take a closer look at all that Martin Fowler, and others have to say about it. Also in this time, I have started a 'Go Agile' initiative with my team of coders, and we're
aggressively studying the existing body of knowledge from the Agile Camp, with a view to integrate into our not at all agile methods. I'm actually hoping that one of them will read this material :)]

The other thing about Design By Evolution is that it can greatly aid Feature Driven Development. By letting your evolutionary blocks be feature driven, your client can see immidiet results, and your Time TO Market can be greatly reduced. In another
article I'll talk about Feature Driven Development.

As a conclusion, I'll just say this, "Agile Development is the future of an aggressively fast-paced market, where requirements can change in a whim, and by murphy's law, will change when you're least prepared to meet the change." To embrace Agile development might be the only way that software developers can hope to meet this reality head-on, but embracing
this methodologies is not for the lazy or the light hearted. You can ruin your business totally, if you don't take Agile development on its proper foundations of 'Loose Coupling', 'Refactoring', and 'Testing'.

[ Uhh... Mfon, i'm still owing you a Refactoring Tutorial :-w, and Farhan, I owe you some screenshots... dang!!! Ok... they're coming in the next week. This link is better now :) ]


At Monday, April 18, 2005 1:02:00 AM, Blogger Mfoniso said...

OK. That was funny! The whole time I read the article I was busy thinking I have to remind you about the article on refactoring. So tell me, where did you buy your copy of "teach yourself future-mind-reading in 21 days?". I'd like a copy.

*nix net prog on da wei* ;)

At Monday, April 18, 2005 1:02:00 AM, Blogger Mfoniso said...

This comment has been removed by a blog administrator.

At Wednesday, April 20, 2005 6:50:00 AM, Blogger Bunmi said...

Great article Essien. I think you bring up many points that are worth discussing amongst other developers. In fact our dev meeting today had some interesting talks on the topic.

I don't think you highlighted the need for Continuous integration in your article. That is also a key process, especially when you're not working solo on a project. For successful Agile implementation, continuous integration is key, coupled with testing and refactoring.

You brought up a point like "...This is actually better since the code can be looked at to be choosing its own evolutionary path...", Isn't that a little overemphasized? you may scare away some traditionalist I think...I'm thinking what you mean here is that, since with Agile methodologies such as XP, your latest codebase is your best and most up-to-date documentation ( and since it's simple, refactored and very readable), then if you had to extend a feature, your current design, as it exists in current code will determine the direction of the new change... I just point this out because I feel that this approach, without extremely good refactoring techniques is quite scary - you could be building a Monster otherwise LOL.
Good job!

At Wednesday, April 20, 2005 9:52:00 AM, Blogger Essien Ita Essien said...

Hmm... i see how traditionalists can be scared away :) good point. I think your re-wording of what i was trying to convey is exactly correct. Thnx.

I had choosen to word it this way tho, because of the weird way my brain works ;), and when developing BerryTime especially and 'scode'/'libscode' which i basically 'peered' with Stan on ('scode' is a small utility in C for easing environment based buffer-overflow attacks on vulnerable programs on Linux. libscode grew during development of scode). When working on these, there were times when i wanted to add a new feature, and i would be wondering how to do it, when suddenly, while looking thru the existing code, there would seem to be an obvious way that keeps to the rest of the way the code was built. Its a nice feeling i tell you :) Even nicer, you just refactor based on rule of thumbs, and when adding the next feature, the last function you refactored out, becomes the core you need to add the new feature. This you would never see if you didn't refactor.

I think the point you make is valid.

On the importance of refactoring, it can not be overemphasised in this matter. Refactoring is the core of good programming, and of-course without testing, you can't refactor with your mind at rest. I'm currently preparing an article on Refactoring.

Continous Integration is i think what this whole article is actualy about, even tho like you said, i didn't explicitly Highlight it. (unless i dont get what you mean by continous integration, maybe you could throw more light here). I think i should hold back my comments till i'm sure i understand what you exactly refer to as Continous Integration.


Post a Comment

<< Home