Essien Ita Essien's Blog

New Project - URLCron

2008-12-04T19:54:00.000+01:00

Following up my quiet annoucements yesterday on Erlami, Erlcfg and Fastiga, I've just pushed another project out to github... UrlCron.

The way I architechture out platforms, closely mimicks the way UNIX programs are advisedly written... i.e. Small Programs Which Do One Thing, Do Them Well, And Can Read From Standard In and Write To Standard Out In A Simple Format.

In our case, most of our applications usually are made of up many RESTfull webservices, all talking to each other via HTTP Calls and JSON responses. The UIs are all Heavily ClientSide javascripts, that also talk to the various webservices. Normally, each application will have one authoritative application server, which is also a RESTfull webservice, that is directly responsible for knowing *everything* about this particular application.

For instance, one of our media campaign engines has an appserver which deals with authentication, authorization, media setup, etc for various front ends (HTTP clients, SMS clients, IVR clients, etc).

Following this very very loose coupling and disconnected architechture, we tend to evolve lots of small stand alone systems. And a lot of times, we need to set timeouts, callbacks, schedules a lot of activities that should be deferred etc. This can all quickly degenerate into complexity and incompatibility and lots of duplicate code, and can become very language dependent. Also, the option of using the venerable cron daemon to call urls from shell scripts is not very nice, and is too much of a hack.

Enter URLCron. UrlCron is designed to fit into our disconnected mesh of services, to allow us to schedule a call from any service to call any other service and then store the results. The scheduling service can come back at a later datetime to check the status.

In its current implementation on github, its just a few days old and is not yet mature, but the entire concept works end to end.

The readme on github contains a lot of ideas of where I would like to logically take this to, and some notes on architechture and design.

Someone else wants to have fun poking around at this? Go crazyyyy!!!!

Ace out!

Sweet November

2008-12-03T01:10:00.000+01:00

November was a good month. Yeah really was... and yeah i know... the world is in general economic recession, but hey... Obama won!!! :)

Anyways... last month, saw three of our inhouse projects quietly released on github. My github page is at:

http://github.com/essiene

I'll just use this post to talk briefly on the three projects.

ErlAMI

ErlAMI (pronounced Erl AM I) was my first serious Erlang project that was not hot off the tutorial presses :)

I cut my proverbial Erlang teeth on this project. It is an Asterisk AMI protocol library done in Erlang/OTP and trust me, I've learned a huge load just doing this project.

I needed to build an automated dialler based on Asterisk as part of our inhouse product stack, and defintely needed to talk to Asterisk. This was a project that I had started in Java a couple of months back, but after taking the red pill, I decided that the Matrix had me, and I just had to do this in Erlang.

At the time I started it, I was still reading Joe Armstrong's "Programming Erlang", so walking through the git
history will feel like a trip through Joe's book.

I started by rolling my own structures and processes everywhere, learning new features of the language and using them, learning eunit and other things and rolling them in.

Eventually, I had built my own poor man's OTP. I had my primitive supervision trees, FSMs, servers, even a poor straw man's event manager and handler framework.

Impressive, but as I finally completed the OTP parts of the book, I realized I had to port to OTP because of the immense gains, and just the raw amount of iron clad tested code I would be basing on. I did just that, and the result is the current Erlami.

This one will take a while to mature, but i'm continually finetunning my ideas and feeding them back here, and there are some funny bugs and corner/edge case issues here and there, which will get cleaned up with more use.

Erlcfg

This is the new kid on the block, and an idea I just had to execute... plus, since last year, I've made it a job of mine to learn compiler tools on every language that I know and use. In this case, I just had to get my hands on leex and yecc.

The idea is simply my own extrapolation of the Java properties file format.

Normally a properties file looks like:

some.key = value
some.other.key = other value
some.other.other_key = other other value

I like this because it is naturally namespaced, is eye friendlier than XML and my java SDP platform uses properties files strictly for configuration. The only problem is verbosity, which is caused by non-nesting, i.e. The format does not allow itself to be refactorable. For instance, examining:

some.key = value
some.other.key = other value
some.other.other_key = other other value
some.other.other.key = other value

It would be more maintainable to rewrite this as:

some = {
key = "value";

other = {
key = "other value";
other_key = "other other value";

other = {
key = "other value";
};
};
};

Also, this bears some resemblance to JSON and in some way YAML. Well... erlcfg aims to do just this, and add something else... VARIABLES! Checkout the README for a good complex config file example.

Currently, the syntax is very strict (notice the extra assignment operators and semicolons... annoying!) to help make parsing easier. But since I have achieved my initial goals, I'll go back and make it easier to write.

Also, I have some interesting plans, like adding XML DTD type support, or simple type annotations to help in verification of config file syntax, easily, this would allow you to define:

server {
port = integer;
listen = list(string);

log {
data = string;
level = atom;
}
}

And the use this to verify

server = {
port = 111;
listen = ("192.167.10.1", "192.168.10.1");

log = {
data = "/var/log/applog";
level= debug;
};
};

I would then love to be able to verify the file by doing:

$ erlcfg --verify /var/lib/configfile.dtd /etc/config.conf

This is still a nice dream right now, but trust me to keep hacking at this till it is done. Did I also mention that it supports all erlang terms except the tuple?

Fastiga

This one is a scala project. It is an implementation of an Asterisk FastAGI container in scala.

This really came about from me having a very specific way I wanted to build my AGI applications, i.e. like the Simple State Machines that they are. Basically, almost like you would build a parser from a BNF specification, I wanted to build my AGI applications to match my state diagrams.

So I set out to eliminate IFs with MATCHes, build case classes that would allow those matches and employ scalar Actors and Tail Recursion to make it all work nicely. Also, using reflection, we can host multiple AGI applications inside just one container, which currently is deployed inside of another servlet container, hence allowing more packagable code and promoting better code reuse.

You work on your AGI logic, let us handle the running and hosting framework.

Its still a work in progress, but the main ideas are realized and it is being used heavily in-house.

We actually have a *thin* framework around it, (much like the OTP is a framework around Erlang) which
I'll tidy up and make a part of the project.

Looking Forward

Honestly, more things are coming down the pipe... lots of ideas and tools that we're using inhouse, which I believe will be valuable to some other team of harrassed developers like us. :)

So this is hoping for a more productive December and onwards.

Ace... out!

Erlang/Scala... gateway to State Machine Oriented Programming

2008-11-01T06:19:00.000+01:00

WARNING: LONG ARTICLE AHEAD, GRAB YOUR FAVOURITE BEVERAGE.

Background
For the past couple of weeks, my team has been working on a version two of one of our apps.

The first version was basically a Python application, that called out to a Java REST webservice. This webservice talked to asterisk via AMI.

There was also some python AGI scripts also for doing automated IVR sessions.

It worked, but there where a lot of ideas after the first version was done. So I did what any sanity loving team lead would do.... branched out git repo and started work on version 2. Now, our version 1.0stable is available for clients, but we're shooting to quickly start recommending version 2.0 and already, just after 3 weeks of very hard work, we're smiling and approaching the end.

New Beginnings Architecture
The structure of the new application is this:

The Java AMI REST webservice, has been replaced by an Erlang/OTP/Mnesia/Mochiweb webservice. If you have any idea what those terms mean, you can already see that I've gained a lot of features.

My Prioritized FIFOs are now distributed across multiple nodes... at almost zero coding cost (I didn't find a comfortable way of doing this in Java). (We'll be releasing the Erlang AMI library next Month as open source software).

The python AGI scripts have been replaced with Scala AGI application modules, seated in a novel FastAGI server built also in scala. (This server will also be released as open source).

The rest of the application is a Comet/Oribited/JQuery/Pylons over mod_wsgi mix.

Why Oh Why?!!!
Now, this is the fun part... why?

For the last 1.5+ years I've been working exclusively in Telecoms integration middleware and SDP space, building and deploying services for very agrressive customers, one of them now the fastest growing Telecoms company in Nigeria (if not Africa). And that experience left me with some major impressions:

My approach was still error prone.
The problems are simple but exacting
There is NO ROOM for NON-EXACTNESS... ZERO!!!

I learnt this lessons, rather shockingly, since I consider myself a carefull artisan, always striving to improve my art, and having done so more than a rather no-too-small percentage of others :)

Enter Application Patterns
The first thing I noticed was that all the applications follow a particular pattern, and I rightly set about building a framework to easily enable writing of these applications... I'm pretty proud of that framework and its humming silently in more than 3 different sized telecoms networks.

The framework has adapters to connect to various network elements, USSD Gateways, SMS Gateways, MSCs, Mediation Servers, etc and convert all incoming messages into a common format (based on HTTP URLS). These requests are routed on the framework router to the Service endpoints... the Handlers.

The handlers are the core of the framwork and do all the SCP, AAA, etc, connections and business logic.

After writing a couple of the handlers, more similarities stood out, so I kept fine-tuning that part of the framework untill very little of it was left to be customized for just the business logic. The plan is to eventually develop a DSL that can be used to write that part and deploy in realtime.

Where Did I Go Wrong
Along the way, I noticed that some of the applications would still evade my exception handling. Also, once in a while, when traffic would spike, I would get some errors which I *shouldn't* be getting. I would
rise to the occasion and fix them, but after one pretty hairy issue with some lost transactions, I really began to question my entire foundations.

There seemed to be something amiss, I couldn't seem to translate the very simple logic flow into similar Java code, without some corner and edge suprises (and I unittest A LOT!). Something was definitely missing.

I would solve the problem, but decided that I needed to change something very fundamental.

Does This Tunnel Have An End?
That was when I started my gazillionth trek up the Functional programming hill - out of desperation. I had dabbled in Haskell 2 years ago and left off, because there where no practical libraries. I'd had a brief fling and excitement with Ocaml around the time I dropped Haskell, but dropped it too, though I really liked Ocaml and was more productive, I dropped it because it didn't have good libraries (these situations are changing as I write... just last month or so, both Ocaml and Haskell have annouced Battries Included projects... hooray).

I had also touched Erlang but hurriedly dropped it... I mean... ewwwwwww.... was that even readable syntax? I had played with various dialects of lisp, (common lisp, newlisp, a bit of scheme), but somehow, my inner programmer didn't gel with any of these, at least not yet. Maybe it was just the timing, but I like to think that I just wasn't ready, I hadn't faced my own waterloo class of problems yet.

The Matrix Has You... errr... Scala
Anyway, this time, as I started exploring other options again, something was different. I was looking out for a language that:

Would allow me to write code without carrying tooo much state information about, so I didnt' get shot in the foot when I least expected
Had very broad library coverage
If possible allow me build on what I'd already done.
Could allow me build my dream distributed persistence framework (in my class of problems, Map/Reduce hence Hadoop/HBase/CTable is not the solution... I was looking for a mature DHT or something better with object serialization support (Scalaris,Mnesia!).
Easily allow me scale out without worrying about concurrency issues, locks, synchronication, etc.

Anyways, that was when I first came across Scala the language. (www.scala-lang.org)

Scala combines very strict typing, functional and object oriented paradigms with very sexy type inference into one badass language that runs on the JVM, (and I hear .NET CLR too).

I couldn't believe it, so I took my time dipping my feet in the pool and sampling.

Learning Strategy... the Stalker/Prey Approach
There is a way that I like to learn when I'm really really serious. I search to web for all sort of references and tutorials, blogs, articles and what nots about my prey (well... technology doesn't sound as cool :) ), and then take my time, going thru all the trivial examples, code snippets, hanging out on IRC, etc... just to immerse myself in the feel on the technology. When I have a reasonable feel or the technolofy and community, I now dive in head first via the "recommended text or tutorial".

Long story short, I began to stalk Scala. Funnily, in so doing I became the prey! I know... its weird, but eventually, Scala and I just "clicked". I don't know what did it, if it was the special sweet spot on the JVM, or the combined OOP and Functional paradigms, or the fact that I could import and just use my Vast Array of company code that had been writen already in Java. I don't know but we just clicked. And I knew I had to see this one through.

My first plan was to investigate how to write a servlet in Scala and see if I could eventually swap out my Java handlers with Scala handlers. I started investigating that productively, but kept happening along another big mine that blew up in my mind...

Erlang: A Second Coming
As I read and played with Scala, the Actors library implementation just kept coming up and how it was a clone of something that was native in Erlang. Well, I decided that if Scala stole it from Erlang, it had to be worth finding out the main thing itself. I made up my mind, and I picked up a second victim.

Still syntax-wary from my last attempt at Erlang, I went for the stalking now with questions on my mind.

Then a funny thing happened. This time, I waltzed over the Erlang syntax... and I still don't know why, but that syntax just seemed different, instead of weird. After I kept at it for a couple of days, the veil simply lifted.

So What's The Bling?
Erlang has so much goodies its hard to imag ine that its open source. Its usually features that rich, that make companies like Microsoft, keep close tabs on their languages.

Anyway, my easy way of explaining Erlang is this: Erlang feels like a domain specific language for highly available and highly scalable applications with very low error rates and high integration to other environments.

Take some time and read that again. Out of the box, erlang allows you to very cheaply create concurrent applications that can be distributed on mulitple network nodes. Its a very basic concept, not an advanced language feature. (spawn, !)

These processes are encouraged to be built for failure, so you don't try to prevent your processes or nodes from dieing (A concept I call Fighting Your Exceptions). Instead you encourage them to die, quickly! Erlang provides process parenting/monitoring as another very basic concept, to allow you restart dead processes. (spawn_link, link, monitor, supervisor)

Finally, what I'd always loved in C and detested in Java (low-level bit manipulations) are present in a very convenient data type: Binaries

This means a lot to me. I used to fall back to C/D to write low-level binary protocol clients, and use the library to build a proxy-daemon which spoke plain-text, and then connect to that from Java, Python, etc... now I just do the entire back-ends in Erlang!

Programming As State Machine Construction
Finally, I come to the main point.

Has any of these helped me solve my problems with my previous approach? Hell Yeah!!!

Immutability in Scala/Erlang helps me avoid a very nasty class of concurrency problems... race conditions in multithreaded Java code.

Immutability also encourages localized variable definitions, which encourages what I call "terminal functions". These are functions whose scopes terminate in themselves. The have all they need inside them, without looking for a global or class wide state. Everything is either passed in or initialized there. And debugging them is WAAAAAAYYYYY simpler :)

Higher Order Functions, Anonymous functions and Currying, provide some of the bassaddest refactoring tools known to man (and woman... if she can program :) ). Its just sooooo cool refactoring code in these languages. Even Python has to take a back seat sometimes, and that is saying a LOT because Python is darned flexible.

Tail Recursion (probably my favourite feature): Allows me to write pure State Machines. Some event driven, some acceptor state machines. This has been my biggest boost in the last couple of months. I sit down and come up with 3 types of diagrams:

An application layering diagram, that gives me the various service layers and defines the protocols, contracts and messages between each layer.
A State Transition Diagram or and State Table, which describes completely all the states for each layer and the messages and transitions between them
A flowchat for each state in each state State Machine in each application layer. I now sit down and virtually translate the diagram into code that works without suprises!! I never felt this was possible in such a short time.

Conclusion
I use the Erlang/OTP framework, especialy gen_server, gen_fsm and gen_event, a lot to simplify my life and use Scala Actors intensively. I plan to try out the Scala/OTP library that is being built here: http://github.com/jboner/scala-otp/tree/master

I'm in the business of software, and one of our company motos and policies is to solve hard problems that plague our clients. We continually come up with innovative solutions and deliver them, and most of them end up being used by lots of concurrent users.

Scala and Erlang have shown in a very short time, that the future will be pretty different from what we currently know today. I think the era of the Swarm is upon us, where most of our applications are going to be used by a myriad of users. The ability to be able to write correct code that is easily debuggable and run in a scalable environment is going to be more and more of a differentiating factor, and hopefully, that can translate into pre-forty year old retirement ;)

In all seriousness, if you've not looked at a Functional programming language for your choice platform, you do your self a serious disservice, which can only be fixed by looking into one, SERIOUSLY as soon as possible.

Ace... out.

Client/Server design and implementation: Discovering the symmetry in implementation

2008-10-17T11:59:00.003+01:00

I experienced a paradigm shift in my design approach recently, which I'd like to put into writing, so I can formulate the rough thoughts in my head better.

Background
I've been working on an Erlang AMI library implementation for a couple of weeks now.

I set out initially to write a simple AMI Client library that I could use to implement my application. On the way, I had to learn eunit so I could thoroughly test as I built, not to mention, quickly build test the small blocks before integrating them.

After I built a working prototype (with plenty of bugs which I was well aware of), I knew I had to unittest this properly, if I wanted to have confidence in the code not to mention bragging rights for well tested code.

First a problem
The problem with testing external connections like this is usually bootstrapping. If I have asterisk installed on my dev machine, writing the tests is not a problem. But if J. Random Hacker grabs my source, and
out of habit types:

make test

They're only going to get pass marks only if they have asterisk configured, and by some telepathic ethos among hackers, also set up his asterisk instance to have the same username and password as my own, among other things (channels, etc). Unfortunately, this doesn't happen as much as we'd love it to :).

There is a way around this using Mock frameworks (which I've never used by the way :D). Anyways, I did what any hacker worth his mettle would do, I sucked it up, stashed the client code in its buggy state, and went ahead to implement a simulator from the same code base.

You what?!!!

Yup... I'm building AmiSym, which is an Asterisk AMI Simulator, and it shipswith the ami library (so you get a client and server for the price of just the client). Now onto the cool parts of doing this.

Advantage Number 1 - More experience
You can never over emphasise the value of increased experience when solving a domain specific problem. In my case, I'm basically solving the AMI problem a second time, even though it is from another perspective
(more on that below). This give me more exposure to the problem, than I would have had if I'd done a one-time-coding-pass and written the client library.

Infact, I end up tackling the AMI library problem 3 times. Once when I implemented the prototype. Once when I implemented the simulator. And once when I reimplement the client with the experience I've gained from implementing the simulator.

Advantage Number 2 - The other side
If I had stuck to writing just the client, I would probably come up with a well done client (I mean, I love to consider myself a not-so-shabby programmer), but now that I am also implementing the server side, I get
to see the problem from both sides of the coin. And that has impacted the way I understand the communication and the mechanisms that the client has to implement because I'm designing both the producer and consumer.

For instance, I created an internal data structure to store the Key: Value pairs that are AMI responses, with a simple hack to deal with things like action: command results, which contain extra data that
don't come in Key: Value pairs.

This worked satisfactorily for as far as I was just a client. But once I started implementing the simulator, I decided I had to use the same data structure on both sides of the connection, immedietly, I found out my
Data structure wouldn't be symmetric. That Is, I couldn't do:

AMI RAW DATA => RESPONSE PARSER => INTERNAL REPRESENTATION

and then

INTERNAL REPRESENTATION => DESERIALIZER => AMI RAW DATA

This was a problem, since it would mean, either two code bases, or just mean that I should redesign my INTERNAL REPRESENTATION, which is what I did.

So I now have a more robust Data Structure, because I took a trip to the other side.

Advantage Number 3 - Borrowing From the other side
If these were the only advantages, the code would already be better off, but there is more. My design of the simulator State Machine and internal processes, end up being very sensible. I attribute this to the fact that my first attempt at writing the simulator is in actualty, my second attempt at tackling an AMI library core.

Now, going back to the client, and I'm borrowing a MIRROR image of the simulator process in the client. Simply put, this makes a *lot* of sense. It makes sense for a Client and Server of the same protocol, to be mirror images of each other, such that when plugged together, and a message in inserted into any them, it will loop through the virtual ring that is formed and "theoritically" come back in the same representation.

This is all kind of abstract, but just think of two halves of a hoop that fit perfectly together... or YinYang... each groove in one is complemented by an extrusion on the other, so they're really just each other, only turned inside out.

I think that approaching any problem space from this perpective produces a much more robust and complete design and coded implementation. Or to state it in cooler terms... realizing the Zen of YinYang in
Client/Server systems yeilds better code and a better design.

Advantage Number 4 - Modularity
The final advantage I want to bring up is extreme modularity. Usually, one can relate to modularity in its more common form. Take for instance a problem where we write a generic TCP framework that allows one to inherit and extend to either an HTTP or SMTP module.

This I would like to call forward-divergent modularity, which refers to a single code base being made modular to allow the core functionality to be easily diverged down an irreversible path.

This is not type of modularity I gained. The type of modularity I'm referring to, which I'd like to call cyclic-divergent modularity would best be described with the help of some funky diagrams. Enjoy :P

The figure above shows what I have termed Forward-Divergent Modularity. Each of UNIQUE CODE A and B have a common core or set of common core modules, which do similar stuff (like open a socket, setup a generic session, store client_id/ip mappings in a HashMap, etc), but then each of them have their unique portions (SMTP protocol versus HTTP protocol).

I call it forward-divergent, because a virtual message travelling through this system, would either start from CORE and move towards A, or move towards B. A system that is structured this way is an Either/Or system. Nothing would make a single message cross camps. You would not be able to "virtually" LOOP the above system at the UNIQUE CODE POINTS without introducing a protocol converter (mostly impractical) which
confirms that even though they have the same base, they're different logical systems.

Now onto cycle-divergence.

The figure above shows what I term cyclic-divergent. As you can see, this entire system turns out to be a Single virtual system, even though there is some uniqueness in A and B.

In this case, a "virtual" system message can be inserted at any point, and it should theoritically, go through various transforms, but by the time it returned to its originating point, it should be back to the same format it originated in.

In Conclusion - Dude Stop Being Abstract
Ok, I'll stop. :)

I don't know about you, but I'm always thinking about code I write, and realisations like these, make me a better systems designer and implementor. For instance, now that I know that I'm actually building a single virtual unit, I have been able to identify that common core in my code base now.

My work from here on out is to widen the common core as much as possible, so that the unique parts are very small. This will result in a more robust client library and server library, that both depend on a well tested and shaken core. This is most decidedly better than implementing one in absense of the other would have resulted in.

As an aside, I'm not so sure I would have seen this using a mock framework (I've never used one, so I don't know). This is not an anti-mock framework rant, just an observation anyways. My personal conclusion then is that whenever one is implementing a protocol-ish library, there could be a lot to gain from also building the Other Side of the protocol in the same code base. The important point of note is IN THE SAME CODE BASE.

So if I were to write an HTTP server, I would also write a client in the same code base, and use that in my testing. Too much work? Yeah, maybe... but its fun... and its got its perks :)

Peace out.

Quick link to creating custom Erlang behaviours

2008-10-12T02:56:00.001+01:00

One of the things I liked about Python when I learnt it back then, was
how it exposed what it was doing under the hood in a simple way.

For instance, passing self as the first argument of class methods made
lots of sense coming from an Abstract Data Type use case with C
Structs,
I could immedietly see the relationship.

Anyway, some parts of Erlang are like that too... and trust me... its
a really really small and concise language (too concise? :)),
anyways... if you've
ever tried to do any sort of Frameworky thingy in an OOP language,
you've had to use Interfaces or something similar (hopefully you've
done this right? and not left
your users to figure out at [compile | run] time why things are
borking sooooo badly).

Anyways... since Erlang has no classes and there are nothing like
Interfaces, they implement a very nice feature called Behaviours which
do some thing similar to Interfaces. At least, tthat's the
simplest way to think about Erlang Behaviours.

Here's a quick link showing how to create yours... short and straight
to the point: http://www.trapexit.org/Defining_Your_Own_Behaviour

Asterisk AMI Library in the works

2008-10-12T02:05:00.001+01:00

We're building an interesting application in the office, and Asterisk is
a big part of that application.

The component I'm working on communicates with Asterisk via AMI mostly
for originating calls, and I have a version of it in Java working with
some small bugs here and there.

I decided some weeks back to rebuild that component in Erlang, so I'm
currently working on an Asterisk AMI library in Erlang. This is my first
"real" app in erlang (The ring benchmark exercise in Programming Erlang
doesn't count :) ).

For me, when working on a project, I *really* love to get unit tests up
and running. They are a great way to "break the ice" and really really
really help keep code maintainable and keep regressions in check as
codebase grows. So last week, I finally downloaded and went thru the
docs for Eunit, and started using it, first for the Ring benchmark (I
should post that code somewhere) exercise and now for the AMI library.
Its pretty neat and simple.

Since I'm still new to Erlang, I'm learning and applying so this
library. The library will eventually be open sourced and then I'll be
found out for what I really am... a cheap fraud who can't code to save
his left pinkie :)

Anyways... just finished some unit testing and all works, so I'm going
to bed in high spirits :)

--Zzzzzzz

Back Online

2008-10-11T20:25:00.001+01:00

Well, I'm back to blogging and the whole online presence thing.

But there was a reason I deliberately stopped blogging. I realized that
the internet is a very interesting medium of communication. People that
don't even know you will get to read the things that you write.

This is all cool, but the problem is that no matter how shallow or
uncooked your theories (or reasonings) are, some people who are just too
lazy to fathom things out by themselves, will swallow what you have to
say, hook, line, sinker and dare I say boat with the captain and crew!!!
I decided then to stop blogging until I was sure that I had something
fairly well cooked to contribute to the community at large.

So, why am I back? Am I saying that now I'm a rock star programmer or
something akin to that? Well... one small part of me wishes that were
the bare truth, but I really can't judge my "rock-starNESS" (or
awesomeness to boot!), but I can honestly write about a wider breath of
interesting topics that I have had a lot more close encounters with,
than before...

Just to give a brief preview, I have spent the past 1year+, working on
an SDP (Service Deployment Platform) that is now in production
deployment in a number of tel cos here. That has been probably the most
satisfying experience I've gained, and has in turn led me onto my
current path which I toe as a programmer. On this path, I've picked up
and continue to pick up things like Parsers (JavaCC, Apaged, Lex/Yacc),
Distributed, Redundant and Reliable Systems and their construction
(DHTs, Heartbeat, Mnesia, MySql replication), Functional Programming
Languages (Ocaml, Scala, Erlang), OTP (Did i mention Erlang already?!).

I have learnt sooooo much technology in a short while and waaaaaay much
more about crafting reliable systems and managing code complexity to
produce easily evolving code bases that you don't have to pray to God
each time you do an upgrade.

From here on out, I'll be blogging a bit more often, not sprouty and
all authoritative, but just staying on the stuff that I've learnt and
now know. Hopefully, these will make for some fun reading and mayhap...
helpfull.

RSS Support for Neengine

2006-06-27T23:10:00.000+01:00

Yay!!! I just completed validating the RSS generated by Neengine,
therefore Neengine now had RSS Feed support. You'll notice the RSS feed
icon now if you use a real browser (*hint* Firefox).

Also, Thelda now properly mirrors to a specified blogger.com url. I'm
thinking of writing an e17 module that will call thelda and periodically
post blogs from a folder to Neengine... hmmmm... I'll see about that
(seems essienitaessien.com is becoming a technology franchise :)

Anyways, I don't think I should be repeating any RSS Howto here, so I'll
just point to some of the many fine articles available if you want to
know how to do this too:

<ul>
<li><a
href="http://www.peterfreitag.com/item/465.cfm">http://www.peterfreitag.com</a></li>
<li><a
href="http://www.kbcafe.com/rss/rssfeedstate.html">http://www.kbcafe.com/rss/rssfeedstate.html</a></li>
</ul>

The first of the links is a quick primer and you can get up and running
in no time. The second one contains some more information that is
interesting to know.

Oh, and I quickly learned that you REALLY WANT TO VALIDATE your feed.
You can do this over at <a href="http://feedvalidator.org">Feed
Validator</a>

Ok... with that out of the way, I think I to really work on the comment
spam validation next. This means, I have to implement or borrow a <a
href="http://en.wikipedia.org/wiki/Captcha">CAPTCHA</a> implementation.
That's all that's really holding Neengine back from being ready for a
v1.0 I think. Anyways, I'll have to work on that this week.

In other news, I'm beginning work on a Forums module. I will probably be
testing it here on essienitaessien.com, but its to be used in client web
communities.

Time to go clean-up the soul... aka... reading C code and possibly
refactoring.

return;

My kingdom for an SMS Port

2006-06-23T00:33:00.001+01:00

I'm currently awake, putting the finishing touches to the MIDP
application I've been working on since last week. I posted last week
about the rapid speed with which I quickly whipped up a 90% working
demo. At that point, what was left was finally sending collected
information via SMS to a remote application server listening via SMS.

To do this, you'd use the J2ME Wireless Messaging API (WMA). The api
seems rather simple and I looked at the code in the WMA Demo example
that comes with the j2me Wireless Tool Kit. I adapted the main sending
functionality there into a function of mine to send text... and that's
where my problems began (I know that in retrospect).

Apparently, according to the SMS specification, more than one SMS
application can recieve at the same phone number simoultaneously, just
like TCP/IP sockets. To differentiate the different applications, the
sms url addressing scheme looks just like the usual http scheme we're
all used to, complete with the port numbers, so I can send an SMS to a
phone number at sms://+123456:10 . When the SMS gets to the phone, an
application listening at sms port 10 will recieve this. All well and
good it seems, but not so for me... when I suddenly discovered that SMS
allows for port numbers, I started wondering what would happen if I
don't specify the port number? Or which port the default sms recipient
application listens at? Basically, b/cos I knew that in HTTP, not
specifying a port number means you're specifying port 80, I got worried
and tried to find out. I didn't find this out directly, instead what I
found out was that if your app is LISTENING, and it couldn't careless
about the port the message comes in on, it should be set to listed at
port 0.

Armed with this half baked information, I went on to assume that sending
to port 0 then should be okay, if you're not interested in sending to
any particular port! Well, needless to say, my first version didn't
work. I didn't get an exception or anything, the application just
behaved as if it were sending, and quit rapidly, but nothing happened.
On the Sony-Ericsson phone I was initially testing on, the phone kept
asking me if I wanted the application to RECIEVE messages! This was odd,
since i was trying to SEND, anyways, I just thought the phone was being
stupid and that all was ok, and I went on... it didn't work.

I decided to try on a Nokia 6600, but when I tried to move the jar over
via bluetooth, it didn't even install. In my "infinite" wisdom ;), I
believed we were having problems with code signing (hilarious huh?).
Anyways, I convinced my boss to start the process of obtaining a
verisign certificate last week, and I went on to relax and wait.

By Monday this week, I realized the certificate would probably not be
getting to us b4 Wednesday, when I was due for a second client demo. On
tuesday then, I decided to go stay over with some of my friends (Bugzy,
Nnamo, ROM and co :P) who also write j2me apps on and off. That nite,
Bugzy and Myself first tried to deploy his j2me apps on another 6600 and
they deployed... that was when I realized the Nokia issue was due to the
MIDP version. I had been building for MIDP2.0 all the while, so I
changed my setting to build for MIDP 1.0, CLDC 1.0 with WMA 1.0, and
rebuilt the app, fixing what needed fixing, then I deployed to the Nokia
properly... and boom!!! It worked!!! The sms left the Nokia phone... we
were ecstatic.

We then deployed to a sony-ericsson and met disappointment... it didn't
send. After some tinkering, we changed the port number from 0 to 1, just
for the sake of trying something different... and w00t!!! It worked!!!
PHEW! I tidied up the code for the demo and went for the demo yesterday.
It went well BUT there was a problem, the SMS message arrived at the
application with some 8 extra bytes infront of the commands we were
sending! Incidentally, sending the message to another phone, there were
no extraneous bytes... so what could be wrong :(.

I was just frustrated and annoyed, but hey... I had to figure it out. I
started scouring materials, and probably read over 10 WMA tutorials
today, that all said the same thing, and who's example code looked
exactly like my current code, though they were all using port 5000. I
decided to use port 5000, but nada... no show. Very depressing. Finally,
my collegue at the office, Lanre suggested we try to setup NowSMS on his
box, since he has a trial version and see if that could help. Grudgingly
I agreed.

10 minutes latter, we have NowSMS listening for messages, and I send a
command from the application, and check the incoming messages to NowSMS
and what do I see? THE SMS COMES IN AS A BINARY MESSAGE AND NOT AS A
TEXT MESSAGE!!!

That is sooooo suprisingly amazing... but finally I at least know why
the bytes. I try to read thru the specs to see where I may have made a
mistake, and finally, I come to a conclusion that J2ME WMA
implementation has a bug, since I had specified
MessageConnection.TEXT_MESSAGE!!! Finally, I do what I do sometimes when
all logical explanations have flown out the window... I deleted the sms
sending function I had, and proceeded to write another one. I just
cleared the page, and without thinking of my former implementation I
wrote a smaller one, and just tested again... and BOOM!!! IT WORKS!!! I
see the messages into NowSMS come in as TEXT and not as BINARY!!!
Amazing... I look thru my former code and the new one... and the
difference? Nothing... hey.. then why is this worki... uhh... wait... in
my new code, I FORGOT to add port numbers!!!!! You mean...? Yup. The
problem was the port numbers I was adding. I SHOULD NOT have added them
AT ALL. Infact, if I hadn't added them at all, the very first time I
tested my app on the sonyericsson, it would have worked!!!!!

Well, needless to say, this was an anti-climax. I was just too relieved
and rushed to do the demo, and everything works well finally. I'm
currently awake putting finishing touches to the look, input
verification, etc.

The moral lesson of this long twisted story? Uhhh... I'm not too sure
what that should be, but I guess "when all fails, do something
illogical." is not a bad moral lesson :)

Seriously though, I still don't know why I hadn't tried without port
numbers loooooooooooooong ago. Now at least, I know better, and I'm
putting this here incase someone else has the same problem.

Anyways... njoi

OLPC board in da haus!

2006-06-23T00:33:00.000+01:00

Two days ago Khaled of the OLPC project was at our office. They're in
Nigeria trying to wrap up the arrangement with the Nigerian government.
He came with some friends from Alteq (Alteq is an IT company in Nigeria
operating out of Abuja, who also use Open Source software end-to-end to
provide their solutions - Really cool guys).

Anyways, Khaled came to drop one of the prototype boards with me, so I
can play around with it. I immidietly transfered one of the older OLPC
version of fedora I had lying about onto a USB drive and tried to boot
off of the board, but I got a GRUB error... meh!!! I'be been busy with
my MIDP app, so no time to tackle it squarely yet. I should be done with
the darn MIDP application tomorrow anyways, so I should be a bit free to
tackle the board.

Its very small and cute. I should post images here... as soon as I work
in image or the gallery section of essienitaessien.com.

Ahhh well... we had some very interesting conversations and some really
cool ideas flowing back and forth... its always that way when open
source folks meet to talk. You'd think that coming from different parts
of the world, we'd have little to crack wit about... but you'd be sooooo
wrong. Its amazing, but at least for IT/geek types, who spend time on
newsgroups/irc/forums, there has emerged a kind of sub-culture which is
immedietly evident once they meet and start conversing. We really had
fun... its a pity they had to head back to Abuja that day. Khaled should
be swinging by Lagos tommorrow or Saturday probably, so we should be
able to catch some drinks this evening without the pressure of work
deadlines :)

I'll err.. try to *hic* keep everybody *hic* posteddddd *hic*

Neengine Update - Fun With Cheetah Templates

2006-06-12T10:47:00.000+01:00

Finally, Neengine is looking cleaner and more visually palatable. Thanks
to a few hours on Googling on HTML formatting with Cheetah Templates.
I've been mulling moving over to templetor which seems will come by
default in webpy 2.0, but the '4 space' indentation thing actually irks
me. I can understand it in Python as a language, but as a hard
requirement in a templating system, its a bit irritating. The real
problem though is that right now, the errors caused by getting
indentation wrong are not accurately reported, and will probably lead
you to frustratingly look at other things. Since I'm not motivated to
hack that area, I'll just wait for Aaron to fix it, before I evaluate it
again :D

Anyways, I got to find out that Cheetah can control how special
characters in its place holders are treated, using something called
Output Filtering. With output filtering, you can either specify an
output filter as an argument to the rendering function, or you can do it
in the Template itself, using the #filter directive to cheetah.

By default, web.py treats HTML tags specially, hence rendering them
literally, instead of treating them as HTML. But using the two
directives: #filter ReplaceNone, #filter WebSafe, you can turn this
behaviour on and off. With the 'ReplaceNone' filter, you switch to
Cheetah's default behaviour, which is to only treat Python's 'None' as a
special character, replacing it with an empty space, every other thing
is rendered as is, so HTML tags will be rendered as HTML tags. With the
'WebSafe' filter, all characters that that can have special meanings on
the Web are escaped so as to be rendered visually literal.

I currently switch between these two when rendering Neengine's HTML
contents. Ofcourse, this is after turning linebreaks into and HTML break
tag. Currently, I still have to add transparent support for embedding
HTML into the posts, so it is escaped intelligently. I'm not even going
to think of how I'll do that now, till probably later in the week (if
i'm in the mood).

With that, Neengine is a lot more mature now. The only critical things
left now before I can declare a version 1.0 Neengine is Anti-Spam
validation for comments, so that I can enable back the comments section,
and RSS feed support. Other features like embedding images in posts will
probably come in a major version bump, since this will likely break the
database schema.

For now though, I'm off to hack thin client images :)

*enjoi*

Unix Network Filesystems Suprise

2006-06-09T21:25:00.000+01:00

Ever since I started work on this Thin Client distro, I've been kinda
skeptical about the kind of application performance I would get.

I decided to ignore it and work on and cross over the speed barrier when
and if it hits me in the face, and I was really expecting a huge punch.
Coupled with the fact that I was testing the distro out with a USB
bootable version of it, everything was just slow and ungainly.

The first time I tried to mount the network filesystem via samba on the
USB based distro, and run OpenOffice off of it, I almost passed out just
from the time I was waiting for openoffice to show me the damn splash
screen!!! Well... that really discouraged me, but hey... I went on to
put stuff together.

As at today, I just had to test that baby on an IDE device, so I
hijacked an old harddrive, dd'd my image over to it and proceeded to
boot off of it. The first thing I decided to try was Opera (which our
client really wants), and WOW HOT DAMN!!!! I was shocked. The thing was
blazing fast!!! Just to see how Opera running off of a networked
filesystem performed against a networked Firefox as well, I went ahead
to run Firefox too... and tho it was slower to start up than Opera, I
was also impressed.

I went on to run some other applications like QT Designer which I was
expecting to take forever and a half to start... and my o my... It
displayed the splash screen almost immedietly!!!

Finally, I went to try OpenOffice and it took like 25 seconds to show me
the splash screen (which is basically about the same time that MS Office
apps take to show me their splash screen on a Windows Box!!!) Well, at
the end of today... I'm really tripped about Network Filesystem
performances. I actually wrote an initscript helper app 'wltcfsd'
(Wazobia Linux Thin Client FileSystem Daemon) to make configuring and
persisting my network filesystem settings on the thin client easier.

I can say... I'm really loving thin clients. Infact, i'm loving it so
much I'm thinking of setting up all the translators to run off thin
clients at the office :P

Also, I've been thinking of clustering too... check the scenario... Thin
clients, connecting to a huge cluster over a 1000baseT network or
something... WOW!!!! That there is the power of linux.

Ahhh... now I have to go become a couch potatoe... I don't think i'm
partying this weekend. I have me 24 on DVD, I have to finally add
intelligent formatting to Neengine, and I have some refatoring patch I
was thinking about for e17 modules gadcon clients.

Seems I'll just be coding thruout this weekend.

t'sall good

Thin Client Success

2006-06-09T15:14:00.000+01:00

Hurray for persistence!!! I've finally gotten the thin-client distro to
a point we can actually deploy.

My main worry (apart from the hardware fiasco i mentioned in a previous
post), was getting the network filesystem right. I was worried about
performance and all those other small things, but eventually, it seems
to a large degree SMBFS mounts solves the immediate need of our first
prospective client.

I'm impressed at the speed of stuff like Opera, Skype, etc, running over
the SMBFS mounted partition. Veeeeerrry cool stuff. What I did was
basically to modify kadischi (the Fedora LiveCD generator), to stop
immediately after building the filesystem and after running
post_install_scripts.

At this point, I run my own customization scripts, and then make a
squashfs image. I also build my initrd system from an updated ALE
(http://datavibe.net/~essiene/ale) initrd system.

I'll have to do a full overhaul on ALE, as its serving me in ways I
didn't even think of previously... another hurray for lazyness by prior
doing!!! :P My initrd image, creates a unionfs of the readonly squashfs
filesystem and some other read-write images which I use to persist
config data, and once I boot, my custom rc.d script runs and mounts my
server drives over SMBFS and I have my full system.

As at today, the config file for the mounting is just that... a simple
txt file. I'm sure the clients will want a GUI config program accessible
from the menu... if i'm not passed out over the weekend, I'll do that up
for Monday :)

For now, I'm wrapping up everything nicely, and I'll probably drive down
to do a techie demo for the techs over at our first prospective
thin-client client (huh? :P)

return;

Wazobia Linux Thin Client Edition - Problems

2006-06-06T13:55:00.000+01:00

For the past couple of weeks, I've been deriving an embedded distro
based on Wazobia Linux which ofcourse is based off of Fedora Core. I've
tested this out on an HP hardware we layed our hands on for testing
purposes and everything has been working fine.

Recently though, we had to order a prototype of the hardware we intended
to use and I proceeded to boot off of it and WHAM!! BHAM!!! Problems.
I'd gotten so comfortable on my test platform that I'd forgotten that
the test platform was i586 but the target platform is i386. First, I
rebuilt the kernel... and I get it to boot. Next was rebuilding the
initrd image. That one was a huge battle, since I had some other
problems with tty(s) that I didn't show themselves till I tried booting
off of the target platform.

Right now, I realise we have two options. Either we rebuild all the
binaries for the target platform (a lot of drudgery involved :( ) or we
select our test platform (or similar) as the new target platform.
Arggghhhh!!! And i'm supposed to be having a demo later today :(. I
guess I'll just go ahead and demo with the test platform, and worry
about the rest later.

Its really annoying though... ahh well... at least I now have a way
forward. *yawn* return;

Thelda/Neengine

2006-05-31T22:32:00.000+01:00

Two nites ago, I stayed up and worked out the last few show stoppers in my

News/Blogging engine (Neengine). Today, Its up for testing on
http://www.essienitaessien.com.

I've disabled comments for now, till I work out Comment Spam Avoidance
(CSpA(tm)) :P.

Since Neengine is a RESTfull webapp, posting an article is a simple HTTP
post. To facilitate this, I have a python library and client - Thelda.
Currently, there is no authentication in place, and I'm working on
that... but I'll probably put up my todo engine first, dunno... just
depends on what brand of hashish I'm hashing out gaswise when I sit down
to code at nite :P

For a bit more on the name Thelda/Neengine... see
http://www.essienitaessien.com/news/53

return;

Holiday Kingdom

2006-05-29T20:40:00.000+01:00

I think Nigeria is a Holiday Kingdom!!! I can't even count how many
Public holidays and non-work days we've had this year alone. Its simply
amazing that we are still earning salaries :)

Anyways, I had to be a good citizen and join in the celebration of
Democracy Day today, which meant no work (in the 9 - 5 sense of the
word, but hey... we know better right? ;)) I actually came in late from
Abeokuta today, I'd lost my phone some 3 fridays ago, so I had to go
pick up my SIM pack, so I can recover the number from MTN services center.

In other news. I've been fine tuning the blog/news engine and I should
have it online sometime this week on http://www.essienitaessien.com.
Once that is done, I'll be posting directly there and probably have the
posts mirrored here for archiving purposes, hey... Google has enough
space... why reinvent the flat tire?

In yet other news. I was having some issues last week with a IDE Disk On
Module (DOM) chip, while trying to put my thin client distro onto it.
Well, the nice manufacturers have sent me the instructions I needed and
I'll soon have it splinkin and splankin... :D

In yet other news... uhhh... what other news? sheesh!!!

Ace... out!

New Job, New Begginings

2006-05-26T02:04:00.000+01:00

I've always loved Linux since my last year in college when I discovered
it, and though I used it extensively in my personal endeavours, and
aggressively introduced it anywhere I worked (routers, etc), I'd always
dreamed of working in a Linux and Open Source based company where I can
use Open Source technologies that I've come to know and trust.

Well, up to late last year, I worked at a .NET shop somewhere in
Abeokuta, Ogun State in Nigeria. While I must say, I did some stuff that
others may call cool, I never really enjoyed myself fully (apart from
those few times when I conived with Dan to introduce some Python into
the works ;) or that other time when I worked with Kevin on a BlackBerry
application talking to a PHP webservice... now those were cool). I still
went along with my personal mantra - Real Hackers Hack Back Home At Nite
At What They Love!!!

This year however, all that has changed. January this year, I had an
exciting job offer that I snapped up, not neccessarily b/cos of the pay
(which was indeed higher than my then pay, but not by that much of a
margin), but b/cos of what I'd be doing. To cut the long story short...
I resigned from where I was, and I now work at Leapsoft Limited and I'm
the Lead developer(/engineer?) on Wazobia Linux
(http://www.wazobialinux.com). YAY!!!

For me, this has been a PHEW!!! experience. A lot of what I'd done while
playing around on my lappie at nite, I now HAVE to do at work. For
instance, I'd studied Pacman (the package manager that comes with
ArchLinux), and understood how to build a fresh system from that (I even
created a project based on that, targetting small footprint systems (ALE
: http://datavibe.net/~essiene/ale). Currently at Wazobia Linux, I've
HAD to study the rpm and Anaconda system and I've built something akin
to what ALE did, that we now use to automate our distro building
process. Cutting Long to short - its damn exciting!!!

The other thing that keeps triping me, is that finally, i'm having to
build solutions on Open Source technologies for a Living (and I'm
getting paid to do it). All the *l337* stuff I used to play around with,
I now have to do them on a deadline, and I've finding out ways to master
my tools better, and work even faster.

This brings me to my main gripe about Windows... I don't hate
Windows/M$ technologies... NO!!! What I hate is that they're not
mallaeble (well, they're mallaeble to an extent, but this don't come
cheap!). Alot of the information and resources you need to turn your m$
tool into what you want, and not neccessarily what the creators foresaw,
is almost too steep to go after seriously. Infact, its soo steep that a
lot of very inquisitive ppl just dumb themselves down and say "Is it
worth the struggle?" after trying for a while.

I've had friends who back in second year at college, we used to all play
around with computers, thinking how cool it would be to do this, and
that and the other, if we only found out how, and a lot of them stopped
having fun with their systems a long time ago. I'm not refering to
gaming fun, or watching movies :) that still happens, I'm talking of the
raw fun you have when you've just pulled off one *crazy* hack on your
system. On Linux and other opensource related technologies/platforms,
inquisitive guys like me basically have information overload. With or
without money to buy good books (like we used to be back in school), its
possible to scrounge round, raise a buck, go to a cyber cafe and
download from tldp.org, print and go back to your system. That was the
joy of discovering Linux for me. I COULD RESUME PLAY. Today, I'm lucky
to be working on this kind of project day in day out.

Ofcourse, the software industry is in a serious state of flux now.
Microsoft announcing 3 betas, Google growing bigger, Linux systems no
more just toys, the web undergoing the rennaissance of immense
proportions, its going to be interesting to see how all these play out
in a couple of years, but i personally believe OpenSource will keep
getting stronger overtime. I don't think its 100m sprint, I think its
more like a Marathon or a Cross Country race.

return;

Happy Birthday Me (What? A website?)

2006-05-25T16:15:00.000+01:00

Yay!!! I'm one year older today. And I have just the perfect birthday
present for me - A new website at: http://www.essienitaessien.com (Stan
says the url is pretty user unfriendly... I say meh!!! :P)

njoi.

One more year approaching

2006-05-24T03:58:00.000+01:00

Thursday 25th Of May is my birthday!

Hmmm... just wait a minute. I'm adding yet another year to the chronos
matrix of my life... ah well... I think this one year has been quite good.
In retrospect, I've gone way deeper into programming than I expected I
would, yet in some ways, I didn't go as deep as I really wanted to.

Anyways, I'll just think more about this and deliver this with the
birthday present I've _built_ for myself. I'll unveil this on Thursday
morning, with
all the untold gist and hush-hush crap i've been with-holding all these
months.

Catchya on thursday

(hmm... does this feel like an Apple or Google Pre-press release? :P)

Ace out.

Breaking Radio Silence

2006-04-07T15:39:00.000+01:00

I have been on self imposed radio silence since the beginning of this year.

I have been becoming ONE with the NULL VOID!

Right now I have gained oneship with the null as you will see in
subsequent posts, but perfect oneness with the null void has eluded
me... and i think its because I was striving to cast myself into a ready
made void pointer. Instead, I shall now cease to strive...

I feel the voidness at the periphery... it is but like the fluttery
kisses of a See Nymph ever so near, yet never there.

I cease to strive and instead I choose to break radio silence.

may you become one with the null void.

return;

Fun with SQLite - Embedded never so easy

2005-12-13T13:32:00.000+01:00

Over the weekend, I had to help a friend out with a little assignment (again :-)), to write an address book in C. Anyways, I opted to use SQLite, my reasons being (in no particular order):

I’d heard how simple it was
I’d never tried it and was itching
I’d written a production app with mysql embedded C library about 2 yrs ago, and was curious to see how SQLite felt like
I just needed an excuse to try SQLite out, and the C API at that

My prognosis in a nutshell: AWESOME!!!

I basically was writing C SQLite code 5 minutes after reading their “Getting started in 5 minutes guide”, and though I was on the irc channel (#sqlite on freenode), I didn’t ask any questions… it was all clear and straight forward – except that part when my AUTOINCREMENT didn’t work on a field declared as *int*, till I realized it was supposed to be *integer* - meehhhh!!!

Anyways, the tipping point for me was how it deals with rows returned in a query.

Typedef for a function pointer sqlite3_callback, that looks like:

int func(void * data, int numcols, char** colvals, char** colnames);

Basically, if you pass this function pointer to sqlite3_exec when executing a query that returns rows, this call back will be called for each row returned! This is very cool really. It’s a massive relief from the pointer gymnastics I had to do with the mysql4 C api, to get at my returned rows back then.

Anyways, I don’t know if this exists yet, but with all the data structure work I’ve been doing recently, I have been thinking of a patch to SQLite that would instead take a call back that looks like this:

int func(void * data, int numrows, sqlite_list);

the sqlite_list, would actually be a linked list of hash tables. Each node in the list will be a returned row, and each node, would actually be a hashtable of the columns. This would be even EASIER to program against.

Currently, I’m not even going to deceive myself and say I’m going to write that patch… I’m not even thinking about it in any of my main processing threads… instead its been swapped out into some fscked up area of my mind… where it will stay until such a time as I have a *voila* experience with the SQLite source code (which I intend to read anyways… but that intention is stored in that same fscked up area of my mind… so you get the real picture of things)

Anyways… I did finish the address book rather nicely, and it was fun to call over one of my C hating colleague to see how it worked. Ope just laffed me to smitherins :-)

The fun part of the project was modeling the entities in the application. We have this Object Persistence Framework we use in our C# apps in the office, and I basically built something close to that in C, to do the project… and it again reaffirmed my belief that learning various programming paradigms leaves you a MUCH better programmer. Taking a look at the Address Book source code, looks SOOOOO much like OOP code written in any modern language… the only difference will be that no member access operator is being used. Infact, it looked like doing Python without the namespaces :)

Conclusion… SQLite is cool, so am I :-)

Essien out.

yirae: lists complete

2005-12-13T13:18:00.000+01:00

Yesternite, I was able to wrap up the episode on Singly and Doubly linked lists for libyirae. I’ll upload the code later in the evening.

Some cool things happened as usual during the coding process, that allowed me to refactor out more Node manipulations, like Make_Next_Node, Make_First_Node (this for doubly linked lists, as singly linked lists don’t have any differentiating features for the *first* node), etc.

Most important of all, was that implementing the Doubly Linked List class ydList.c allowed me to see my algorithms more clearly, and had me going back to ysList (the singly linked class), to clean stuff up.

Right now, I have a number of todo’s but my next target Data Structure is a Hashtable.

Just before I get there though, I have to refactor the core operations of the linked lists class, I have a lot of operations, push, pop, insert, insertsorted, etc, and I think its not necessary. I want to settle on some core ops that can be used to perform the rest. For instance. Insert(), should be able to be used to implement both Append() and Pop(), even though that’s not what I’m doing currently. But before I move on the larger issue of Hashtables I have to clean up that (and probably, since it will be childs play, implement a Stack and Queue class).

After cleaning up the operations, I want to make the lists, generic. Right now, the DATA member of the lists are just ints. I want to make these into void * so I can use the lists as a generic carrier for _any_ type of data class.

Also, related to the immediate above point, I’m not sure how to structure the Hashtable. I have two options…

struct yHashTable {

void *data;

char *key;

struct yHashTable *next;

}

struct yhtData {

void *data;

char *key;

}

Struct yHashTable {

struct yhtData *content;

struct yHashTable *next;

}

There seem to be advantages for each implementation. In 1 above, I have simplicity of representation and initial implementation, but I will have to rewrite most of my linked list code to support stuff like sorting by key, etc. in 2 above, I can pass in a different function pointer each time I need to sort, one for sorting by data, one for sorting by keys, and I won’t need to modify my linked list code much when inheriting to create the HashTable. I tend to favour method 2 above though, but I’ve not made up my mind yet which way to go. I’ll finish that operations clean up first, and then when I get to that bridge, I’ll just have to cross it.

It appears I’m still on target for my end of yr goal for Data Structures… the one I’m apprehensive about is Binary Trees. I don’t know *anything* about them apart from the fact that they’re pretty tall, and come in two flavours (b i n a r y t r e e s…. get? :;) ) I guess that pretty much sums up my understanding of those :D

Anyways… back to other things now… I’ll post when yirae-0.3 is uploaded.

Essien… out.

libyirae: API updated to clearly differentiate a list from a node

2005-12-07T10:58:00.000+01:00

I put my words to action yesternite, and updated all the functions to properly differentiate b/w a Node and the List itself. The NODE objects are used internally in the List implementation, and keep things clean, for the user of the library, all you’ll ever need to do, is to use the LIST and its functions.

It was actually fun to do. The fun part was, change all the declarations, recompile, and run the test, see the SegFaults, and start making them all go away one at a time :-), it doesn’t get better than that… ohhh… it just did… VALGRIND!!! Yup, I also had to plug mem leaks and once again we’re VALGRIND certified to boot!!! :P. I’ve really learnt a tremendous lot doing all this.

There is a decision I’m dilly-dallying on… Currently, I have the NODE functions and LIST functions in one file still…. I have to do basic File Content refactoring obviously, the question is now… how closely do I couple the two? I’m trying to think ahead to when I start inheriting (tonite to be precise), close coupling will entail doing something like:

Pull the list functions out of ysnode.c into yslist.c
Define the move the struct _ysNode definition into ysnode.h
beable to to call struct->next, etc from yslist.c

loose coupling will entail:

Pull the list functions out of ysnode.c into yslist.c
leave the struct _ysNode definition in ysnode.c (hence hiding it from yslist.c)
defining accessor functions ysnGet_Next(), ysnSet_Next(), ysnGet_Data(), ysnSet_Data() for the NODE objects
using these accessors from yslist.c

Why am I dilly-dallying? Theoritically, loose coupling is the way to go, but it just seems like an unnecessary hoop deep in my gut… hmmm… especially since I’m not even sure how the inheritance is going to work yet till I start coding. Anyway, I think I just made up my mind. I’ll stick with loose coupling, and be ready to change my mind if things start getting unnecessarily complex. Yup… that’s what I’ll do. That means, by tonite, I should start on the doubly linked lists implementation, then will release 0.3

I’m looking forward to close of work already :D (don’t tell Bunmi I said that ;) )

Essien out.

Directory Services for the rest of us.

2005-12-07T10:46:00.000+01:00

If you've every worked with M$ Active Directory, you'll appreciate the need for directory services to be easy to deploy.

I've worked extensively with Active Directory, and maybe I was just being lazy, but I could never actually get myself gingered up to configure OpenLDAP.

I also kept hearing good things about Novell eDirectory, but heck, I can't lay my hands on it to try it (but I hear its pretty tite).

Last year, RedHat bought AOL's Directory Server, and planned to OpenSource it... I heard that news, and just put it at the back of my mind as probably one of the most important Linux announcements (for the back office), for last year. We didn't hear from them again till earlier this year, when they made a first release.

That release was the basic LDAP engine without the admin console and a few other extras. The reason they gave was there are still parts of the code that are not able to be opensourced at that time, and they were still working to make it good to go.

Anyway, this morning, I saw this via an article on lxer.com: http://directory.fedora.redhat.com/wiki/FDS10Announcement

I'm yet to play with it, but this is a version 1.0 release, with everything feature complete for a first milestone. Its a great acheivement for RedHat, and the OpenSource community in general. I want to thank the guys at RedHat for being a great OpenSource company (even though i still think Fedora is bloated ;) - I mean, everyone is entitled to _MY_ own opinion right? :P).

Anyway, getting serious, I'm going to definitely try this baby out. Then I'll be able to know if the wait was worth it (in my guts, i feel it was).

Once more... thank you RedHat.

Essien out.

Listening To Code: The importance of proper nomenclature

2005-12-06T14:19:00.000+01:00

Is that even the spelling of Nomenclature up there? ;;)

[hmm… it apparently is correct]

Yesterday nite, as I implemented Pop() for yirae singly linked lists, I realized that my return value couldn’t be a ysNode as all my other functions till then had been doing.

Basically, I’d been doing stuff like:

ysNode n = NULL;

n = ysNode(n, 5);

As you can see, I pass in the node (which is actually the head node of the linked list), and I return the new head of the list. This works well, until I get to POP(), which needs to pull off the first item on the list, and traditionally return it, but unlink it from the list. I couldn’t do:

n = ysPop(n);

Doing that will loose the data I’m popping. Eventually, I had to do this instead:

int data = ysPop(&n);

And my declaration for ysPop is:

int ysPop(ysNode*);

I notice that this is very neat and efficient, and I don’t like my functions having conflicting calling semantics, so I start to implement all my other functions to match this semantic, since I can’t make Pop() match their semantics, so Push() becomes:

ysNode n = NULL;

ysPush(&n, 5);

succinct and powerfull!!!

There is just one problem… now, I have to remember to append the ampersand infront of all my nodes that I’m passing to the various functions… like so: ‘&n’ instead of just ‘n’, since I’m now passing a pointer to a ysNode. Ofcourse, I’m apt to forget that, and infact, many of my ‘make’ invocations remind me of that with an error/warning. Thinking of how to solve this, I begin to think of creating a typedef for ysNode*.

Before I continue, a bit of background on ysNode.

The definitions if you look in ysnode.c:

struct _ysNode {

int data;

struct _ysNode* next;

}

typedef _ysNode* ysNode;

Now, you can see what my basic Node element struct looks like. Ofcourse, I could have been using struct _ysNode throughout my functions, but my code will quickly become ungainly, so according to SAGE advice, I typedef struct _ysNode* as ysNode, and I’m using it throughout. Now, I realize I also have to have a ysNode*, which is essentially a struct ysNode**, I sat back and started thinking… hmm…. Have I made a mistake somewhere? Why do I find myself having to define ysNode*?

BAAMM!!! It struck me.

A linked list is a collection of items, each item having a reference (or knowing how) to find the next item. Each of these items is a NODE, but the NODES are a distinct ENTITY from the LIST itself. From the way my code is starting to look, there seems to be a LinkedList type struggling to emerge, infact, its shadow is already present in &n (which is a ysNode*), but I hadn’t recognized it yet. Immediately I realized this, I also realized that my public functions were doing the wrong things, namely: exposing nodes to the user of the library.

What do I mean. From what I just said previously, it would seem obvious that a NODE is contained in a LIST, but that could as well as just be an IMPLEMENTATION detail… I could by stroke of Genius (AKA Insanity ;) ), also come up with an implementation of Lists that doesn’t use NODES :), I just wonder what would be used, but anyway, the point is that the design is porous as it is exposing the NODES to all eyes. My *public* functions then, SHOULD operate on Lists, and not Nodes. The question is then, what is a list, and what is a node?

At the Domain level, a LIST is a collection of Nodes, but at the programmatic level, a LIST is just a pointer to the HEAD of a node… does this sound familiar? Try this, an Array is C, is just a Pointer to the first element of an Array!!! Yup, that a nice zinger… once I realized this, a lot of things just fell into place.

The title of this article, is taken from the fact that I WOULD have missed this, if I wasn’t paying attention to what the code was trying to say to me. And the code was able to talk to me clearer because I gave it a good voice to start with – THE NAMES OF THE VARIABLES N STRUCTURES. Its very important to properly name your objects to as accurately as possible model the problem domain you’re trying to solve. If you do this, the code gains a voice, so to speak, and you can read it back and learn from it. Just imagine if I’d stuck to my earlier plan of calling ysNode, yLinkedList. I would NEVER have had the paradigm shift I had in the way I did. Infact, immediately after naming it yLinkedList, I just thought… no… this doesn’t feel right, all I’m passing around are nodes, so I renamed them, and its come back to pay me good for that decision.

Its like this, I start out implementing functionality, with a particular rule of thumbs, along the way, I happen to implement some stuff that I don’t exactly plan for from the scratch (b/cos I’m paying attention to refactorings and good coding techniques), then a pattern emerges which I didn’t set out to create. I recognize the pattern and want to optimize it or at least refactor it properly, and in so doing I set out to understand the pattern. The understanding of this pattern, then teaches me something much more than what I knew at the onset of the project. This is my code beginning to talk to me.

This may all sound poetic and stuff, but in practice, these things happen all the time… your domain problem becomes clearer as you set out to define the problem in terms of a programmable solution. This is where bottoms-up coding is strongest… as you build small units, you can recognize the patterns that are forming and seek to understand them. If you follow Tops-Down coding, chances are that most of what you’re doing, you’ve already thought of it outside the advantage of being embedded in the problem’s solution, therefore, you’ll miss the tiny details, which are likely to make the solution *elegant*. Ofcourse you may argue that elegance in software is an unreal panacea, or a Holy Grail of sorts, and can’t be achieved in production scenarios, but only in small one on one projects… my answer to that is that, in a proper dev house, the *production software* is just a combination of many *one on one* modules – any project manager will agree with that description. The question then is how to make the coders benefit from the fact that they still end up working on their individual one-on-one projects.

Well… I’m still learning, and still growing, and by Jove… Data Structures seems to be a wonderfull way to end 2005 :-)

Essien

yirae: uploaded

2005-12-06T13:47:00.000+01:00

The code to yirae is uploaded:

http://datavibe.net/~essiene/downloads/yirae-0.2.tar.gz

libyirae: simple sorting of singly linked lists

2005-12-06T12:19:00.000+01:00

And so the story goes…

I have finally done it yippee!!! :-) Source will be available at http://datavibe.net/~essiene/downloads/yirae-0.2-tar.gz before the end of the day.

(needless to say, i built this on a GNU system, so gcc in any of its incarnations will work [cygwin, ming, etc]

After my last ungainly experience with those-pesky-lists™, I had to go follow most of the links I had bookmarked to see how guys have been tackling sorting of singly linked lists. Everyone seemed to agree that they were harder than sorting doubly linked lists (at least I have to believe that, so I don’t feel dumber than I’m feeling presently :)). Anyways, there are a number of approaches, but I’ll try to explain the one I like best.

First imagine you have a list like:

1->3->2

When traversing this list, and you come across 2, you just have to swap it with 3 and you’re done. Imagine instead that you have:

1->3->2->5->1->7

A single pass would go something like this:

You don’t touch 1, then you swap 3 and 2, you then swap 5 and 1 and you don’t touch 7… the result looks like:

1->2->3->1->5->7

Tada… its still not sorted, you’ll have to pass thru this list at least two more times, before you get the desired:

1->1->2->3->5->7

This raises a number of issues… 1st you have to repeat this process basically till you can verify that all your elements are in order… for a long list, this is just not acceptable. Anyways… this was the most obvious approach, and its ridden with a number of issues.

The other not so obvious approach is this. First, imagine that you have an InsertSorted() function that will insert an element such as to keep its place. It doesn’t sort an existing list, but it will not add to the confusion if its not sorted. So if I have:

1->5->7->3

And I call InsertSorted(2) on this list, it will result in:

1->2->5->7->3

Once I have a method like that, I can proceed to one of two implementations for the sorting:

Method 1 – the least brain tickling path
=============================
1. iterate thru the list
2. POP each node you meet, OFF the list (remember POP removes a node)
3. InsertSorted each node you POP into a new list
4. return the new list :)

Nice and easy, this is the current implementation I have. Though, I’ve been wondering about the cost of the new list in the whole process. Also, my current implementation has another Node memory overhead (I’ve explained that one in the code). There is a second method, that at first glance, seems more efficient, though after thinking thru it, the efficiency can just to tied down to the process of creating a new linked list that the first method incurs, if that cost can be shown to be zero, then there is basically no loss or gain as far as I can think, anyway, the second method goes thus:

Method 2 – using only one list
============================
1. iterate thru the list
2. if you find any node out of sorted order, Unlink the Node (unlink doesn’t destroy the Node, just removes it from the list and returns the Node to you)
3. InsertSorted the node back into the list.
4. return the list

This method seems to be more efficient than the method 1, but I’ve not done any testing/profiling so I don’t know for sure. After thinking for a while, there is really no memory usage difference, as in the first case, when the new list grows as the old list decreases. Also, the creation of a new list isn’t a costly operation at all by my implementation.

If you look thru the code, you’ll see that my public functions deal with data not nodes, so I POP data, and I Insert data hence I always have to create an underlying Node object to help me out… I’m thinking of having utility versions of all these operations that pop Nodes and Insert Nodes, so I can reuse the memory without free’ing one only to allocate it later…

Anyways, the code is free to look at. I hope it helps some other person, especially if (s)he’s doing an assignment ;)

Oh finally, the thing that makes me happy is that ynode_test.c is totally valgrind approved :D, I hear LinkedLists are always a point where guys leak memory, but I guessed I closed mine up well… maybe not NewsWorthy™, but definitely something for me to be proud about.

Essien out!

Singly Linked List Class - libyirae genesis

2005-12-05T15:25:00.000+01:00

/*-------------------ABOUT--------------------*/
A collection of data structures, implemented for the sake of learning. Yirae, is a contamination of the Efik work Y're, which means to link or join. I chose the name b/cos the first structures implemented for yirae was a singly linked list. The library is called libyirae :)

*---------------------------------------------------*/

/*-----------------Makefile--------------------*/
all: libyirae.a test
@echo "All Done"

test: ysnode_test
@echo "All Test Done"

ysnode_test: ysnode_test.c libyirae.a
gcc -o ysnode_test ysnode_test.c -I. -L. -lyirae

libyirae.a: ysnode.c ysnode.h
gcc -c ysnode.c
ar -rcs libyirae.a ysnode.o

clean:
rm *.o
rm *.a

/*---------------------------------------------------*/

/*-------------------ynode.h-------------------*/
#ifndef YNODE_H
#define YNODE_H

typedef struct _ysNode *ysNode;

ysNode ysCreate(void);
void ysFree(ysNode);

ysNode ysPush(ysNode, int);
ysNode ysRemove(ysNode, int);

void ysSwapNodes(ysNode*, ysNode*);
ysNode ysSwap12(ysNode);

int ysLength(ysNode);
void ysPrint(const ysNode);

#endif
/*---------------------------------------------------*/

/*----------------ysnode.c---------------------*/
#include

struct _ysNode {
int data;
struct _ysNode *next;
};

typedef struct _ysNode *ysNode;

ysNode ysCreate()
{
ysNode head = malloc(sizeof(struct _ysNode));

if(head) {
head->data = 0;
head->next = NULL;
}

return head;
}

void ysFree(ysNode head)
{
ysNode current = head;
ysNode next;

while(current) {
next = current->next;
free(current);
current = next;
}
}

ysNode ysFind_Last(ysNode head)
{
ysNode current = head;

while(current) {
if(current->next == NULL) {
return current;
}

current = current->next;
}
}

ysNode ysPush(ysNode head, int data)
{
ysNode n = ysCreate();

if(n) {
n->data = data;
n->next = head;
head = n;
}

return head;
}

ysNode ysRemove(ysNode head, int data)
{
ysNode current = head;
ysNode prev = NULL;

while(current) {
if(current->data == data) {
if(prev) {
prev->next = current->next;
free(current);
break;
} else {
head = current->next;
free(current);
break;
}
}

prev = current;
current = current->next;
}

return head;
}

ysNode ysCopy(ysNode node)
{
ysNode n = ysCreate();
if(n) {
n->data = node->data;
n->next = node->next;
}
return n;
}

int ysLength(ysNode head) {
int count = 0;
ysNode current = head;

while(current) {
count++;
current = current->next;
}

return count;
}

void ysPrint(ysNode head)
{
if(!head) {
printf("(null)\n");
} else {
ysNode current = head;
while(current) {
printf("%d->", current->data);
current = current->next;
}
printf(".\n");
}
}

/*-------------------------------------------------------*/

/*-------------------ynode_test.c----------------*/

#include
#include

int main()
{
ysNode l = NULL;

/* using ysPush() eventually, faster than Append, that has to Find_Last() first, before appending*/
l = ysPush(l, 5);
l = ysPush(l, 10);
l = ysPush(l, 7);
l = ysPush(l, 23);

printf("The list is now %d nodes long as shown below: \n", ysLength(l));
ysPrint(l);

printf("Starting removals\n");

l = ysRemove(l, 7);
ysPrint(l);

l = ysRemove(l, 23);
ysPrint(l);

l = ysRemove(l, 5);
ysPrint(l);

l = ysRemove(l, 10);
ysPrint(l);

printf("Removals finished\n");

ysFree(l);
return 0;
}

/*------------------------------------------------*/

On implementing singly linked lists

2005-12-05T15:16:00.000+01:00

This weekend, on account of a friend’s assignment, I had to delve into implementing linked lists. (Yeah, I admit, I’ve never taken time to study the implementation of these babies, all I knew was the concept). Jumping in with my usual optimistic, how hard can it be… I mean, I’ve written kernel modules to boot ;)

Anyways, to be honest… the singly linked list was not hard to implement at all. I got all the basic operations, Create(), Free() (oh… I love to valgrind my stuff, it gives me a good feeling ;) ), Add() and Remove(), these were relatively painless (for the Add(), I had to implement a Find_Last(), so I could always append at the end of the list, being that I wasn’t holding a tail pointer anywheres).

Anyway, the other requirement was for the list to be always sorted in ascending order. And away I went, I mean, how hard can sorting and swapping of pointers and s*** be? I’ll just add this teeny… weeny… BAAAM!!! I hit a road block!!! Suddenly, my understanding of pointers is being stretched to the limit, and much as I would hate to admit on OpenInterWeb™, I DON’T KNOW POINTERS LIKE I THOUGHT I DID!!!

It was a sobering moment, as I kept deceiving myself, fighting with a very annoying segment of code, trying to get the sorting to work and I finally admitted to myself… I’ve been butt kickd!!! DAMN!!! You can imagine how intelligent I was feeling then.

Anyways, I’ve managed to bookmark a lot of articles on advanced pointers and linked lists, and data structures in general (I mean, when I kick this guy in the patella, I’m off to the Races with Hash Tables, and binary trees!!!).

One of the good things I found out tho, was that traversing a list with a for is not quite so neat as with a while. I’ve been so used to all the foreach, for foo in bar ;), constructs, my initial reaction was to traverse with a for loop. When doing things like Remove(), that need a reference to a previous pointer, its somehow *cleaner* to use a while loop, at least, it made my code cleaner when I changed the fors to whiles.

Anyways, I think every hacker worth his salt, should try to implement a couple of things just for the fun of understanding them. And if possible, eat your own dog **** on a couple of projects, so I’m making up a list of stuff I NEED to do before this year runz out finally J

Complete the Darned Sorted Singly Linked lists,
Inherit and implement Doubly Linked Lists
Implement Basic Hashtables
implement a Basic Binary tree

Wow!!! Now that’s a tall order!!! That means, I’m parking/suspending all other hacking endeavours for the next two weeks :(

The other thing I started thinking of, and I’m not so fixated on it now, but I wouldn’t mind attempting an implementation, is a Garbage Collector. This one is really of interest to me, as I’m very MemoryLeak conscious these days.

Oh well… lets see what happens in the next two weeks. I must win I tell you.

Sourceforge New Look

2005-12-02T10:43:00.000+01:00

I've been lazying my behind and didn't even know sourceforge has a brand
new look!!! WHEEEEEEEEEEEEEe!!!!!

It really looks cool. I just got the monthly mail sent to project admins
today, and looked at the site, and its really cool. I hope other aspects
of usability in the backend have been worked on as well... for instance,
I just couldn't get my sourceforge webspace to work :(

Anyways, I'll be checking that out later.

Oh... Pysystray looks even cooler now :)

https://sourceforge.net/projects/pysystray/

Essien out.

INIT Issues : More reasons why I love Linux

2005-12-01T21:46:00.000+01:00

A couple of days ago, I was playing with the LIVE CD image i've created. It boots properly off the HDD when I vector to it using a basic initrd image. The problem is that once It gets written to a CD, funky stuff begin to happen, all b/cos a CD is a read-only medium.

When I started playing with the LIVE CD creation, I hadn't really taken that into consideration. So I had to scourge the web looking for ideas on how ppl handled that. Why I was scouring, I was trying to take the lazy way out - I went into my /etc/rc.sysinit, and started commenting out lines that would cause a disk write (I just wanted to see a darned login screen... i mean... what's wrong with that? :) )

Anyway, after one of my many chroots into the image, i tried to boot into the image, and realised i'd borked something badly. I then tried to boot back into my system and TADA!!!!! i got:

INIT: "cannot execute /etc/rc.sysinit"

Anyway, to cut out all the dramatics, I had a very unusable system.

Now, lemme qualify, when Linux doesn't boot normally, there are a MYRIAD of ways to boot and access your files, especially when you use GRUB as your bootloader. Anyway, I tried doing a chroot from the original install cd which can serve as a minimal rescue cd, and all the permissions where in place... nothing looked out of the ordinary, still i got the silly message!!!

I did what i normally do with Linux problems, I just let it alone, and for a the last few days, i didn't get to boot linux when i got back home from work (I work on .NET on M$ by day anyways ;) ). Finally I decided to give it one last shot. I sent a message to the archlinux mailing list, asking for an idea on what could be wrong. While waiting for an answer, i hit google, and i came across a nearly similar thread in a RedHat ml and on the Arch forums. Both just confirmed my suspicions, something WAS amiss, in the case of the RedHat guy, the file that INIT was looking for, wasn't physically there... in the case of the ArchLinux guy, the file was there, but permissions were denied, and none of these was my scenario. To make matters worse, I'd been running /etc/rc.sysinit on that partition when accessed from the install cd without issues.

Finally, i decided to take it home. I just used grub to set my init=/bin/bash, once i got the bash prompt, i attempted to run /etc/rc.sysinit, and BOOP!!!, it borked nicely. It couldn't find /bin/sh!!! This was the problem... the intepreter at the shebang was /bin/sh!!! I looked, and it wasn't there... i prolly symlinked it out without knowing when i was doing multiple chroots from the root image :((

Anyway, I had to reboot the kernel in rw mode (other than the default ro mode), and use pacman to reinstall bash, and get back that symlink, and voila... its all good my man!!! All GOOD!!!

The moral lesson? Use a flexible OS, that allows you to know all about it without buying expensive books etc... this was what saved me, and this is Why i LOVE Linux.

with BlinDvws, i would prolly have had to reinstall (at least a repair install of XP :( )

Anyway... Linux R0ckz and so do I :)

Essien out

Pacman kicks serious arse

2005-11-24T13:35:00.000+01:00

On of the more deceptive qualities of Open Source software is the
userbase of a particular product. People tend to swarm to what others
are already using, and accept all that is there without much of a
thought about alternatives. The bad part of it is when the authors of
the software don't look out at alternatives to see if they could improve
theirs by learning a thing or two from others.

Case in Point... Linux Package Management Systems.

My distro hopping ended for me when I discovered ArchLinux at least 3yrs
ago or there about. Before then, I had breifly used Gentoo for about 6
months, and I loved the auto-dependency resolution it did. I also felt
cool about the compiling backthen ;), untill I found ArchLinux and found
I didn't HAVE to compile UNLESS I realy WANTED to. For me, Pacman was
Portage without the compiling (and ohh... ArchLinux PKGBUILD files are
so what I expect, as opossed to Gentoos Ebuilds which I kept telling
myself I was going to get into, but that's not the point). The thing is
after using Portage for a while and loving it, and then using Pacman and
adoring it, I'd not really given thought to other distros and their
related package managers... untill recently.

I've had a brush with apt and yum. And frankly, Pacman kicks all their
collective arses in the areas where their functionality overlap. My main
gripe with apt, is the various interfaces (which after using apt for
almost more than a month, I'm still confused when I need to search for
instance, (apt-cache? apt-get? apt-foobar?). I'm all for small programs
that do one thing and do it well, but I'm also all about interfaces for
productivity, and everyone in a while the apt-bleh family of programs
trip me on guessing which should be doing what. Ok... that seems to be
my only gripe with apt actually :) (Its not as intuitive to me as Pacman
is, prolly b/cos i've wrked with Pacman for the past 3yrs? Though I can
vaguely recall grabbing the whole of pacmans major operations after
about 2 or 3 *schemes* thru the man page. I'm still procrastinating when
I'll sit down and STUDY the apt-bleh family of man pages). Its ok, but
well... i prefer pacman in that regard.

For yum, I think it has a more succinct interface than apt, and its
easier to get a grasp of its options, almost and compact as pacman too.
My overwhelming gripe with yum is all the *i think* uneccessary things
it does each time. Maybe there's and option to turn the attitude off,
but i dunno right now. Each time i hit yum install foobar, or yum search
bleh, or yum erase st0pd3l00pz. It refreshes its archives (? or what is
that thing it does each time, fetching base, updates, etc, EVERY
FRIGGING TIME!!!). Its just annoying especially over an SSH connection
that is breaking like burnt, crusty pancakes. I know yum is trying to
build dependencies, etc, but does it build it dynamicaly? I dunno. For
pacman, pacman packages installs these info in a DB of sorts, so it can
easily get back this info, in a very very minute fraction of the time it
takes yum to grab its own dependency information. So far, that's the
only gripe I have with yum.

In summary, I know I'm seriously biased toward ArchLinux, but my head is
not so far gone up my behind that I can't stick it out and do a balanced
assessment (and its not like i set out to do an assessment anyway... its
just my first reactions from my guts). Lets hope in the coming days as I
continue to use apt and yum, I can find admirable features which can be
patched into Pacman, so it kicks even more serious arse...

In parting...

yum: Hey pac... what are you doing? Those boots look huge
pacman: uhhh... could you please just stand over here next to apt
apt: okay d00d, we're here... what do we do next.
pacman: just turn around, with your backs to me, and bend over just a
bit... uhh.. too much... aha... perfect.

*B00T*

------------------------------
LMFAO

Coding Guidelines... MUST READ!!!

2005-11-21T21:11:00.000+01:00

http://thc.org/root/phun/unmaintain.html

sfdisk hacking

2005-11-21T15:00:00.001+01:00

Still in a bid to create a self installing router, I’ve had to play around with sfdisk for autopartitioning and stuff. After reading thru the man pages, getting an idea, I grabbed the source of the ArchLinux installer and looked at how Judd did the auto partitioning parts… it looks cool.

My main problem is that I want to be able to detect the size of the disk, then create various partitions sized in percentages of the disk total size. The ArchLinux installer routine doesn’t do this, instead of uses some fixed sizes for /boot, swap and the whole rest of the disk for /.

I’m still looking for the best way to get the disk size (maybe I’ll even go with parted eventually). Anyway, I’ll try stuff out maybe tmrw nite or something… if that works… then the automated router installation CD will be finished sooner than I was expecting.

Good Bye DevFS (sob... sob.. sob)

2005-11-21T15:00:00.000+01:00

A lot of folks complained about DevFS and all its issues in the kernel tree... but I really wasn't one of those guys. For me DevFS solved a real problem: When I built initrd images, I didn't have to worry about populating the /dev tree before the kernel was booted up.

I don't really know how UDev based initrds handle this, I've been a bit lazy about finding out, untill last weekend, when I had to build an initrd image for an automated router installer. I tried to build with the lattest ArchLinux 2.6.13 kernel (DevFS was finally removed from this kernel build). I hit a snag, and I hatted the pre-DevFS days of copying some /dev files over manually when building the initrd, and I have this very neat scripts that do everything for me (part of ALE, which i've been revamping seriously as I get to do more work these days). Anyway, I finally chicken'd out just to get things working, and fell back to using 2.6.12 kernel.

I'm going to take out some time to find out more about UDev (FINALLY!!!), since its finally shoved itself in my face... blegh!!!

ok... back to code .NET Web forms DataLists now. :)

mod_rewrite voodoo :)

2005-11-16T14:19:00.000+01:00

`` Despite the tons of examples and docs, mod_rewrite is voodoo. Damned cool voodoo, but still voodoo. ''

-- Brian Moore
bem@news.cmc.net

The above is one of my best quotes so far, after encountering mod_rewrite. Anyways, onto the meat.

From an anonymous comment on my post yesterday, someone wanted to do something like this using mod_rewrite:

http://www.abc.com/sub ========> http://sub.abc.com

My former solution had to do with some general redirect/MIME forcing and then some PHP trickery, which actually works. But there's even a more mod_rewrite (ty) based solution... the secret is in the use of the RewriteCond expression which I didn't know of when I replied that post initially.

The problem initially was that with RewriteRule, you can only access the part of the url that comes after the host name. For instance, in http://foo.com/bar, you only have access to /bar in RewriteRule. To do this, we actually need access to the foo.com part as well. This is where RewriteCond comes in handdy.

With RewriteCond, you can basically execute a RewriteRule as long as the condition is satisfied. There are special condition variables that you have access to like HTTP_HOST (which we'll be using here), and a bunch of other apache server/environment variables.

Ok... the rule look like this:

RewriteEngine on #this turns on the RewriteEngine... duh!!! :)
RewriteLog /path/to/logfile # i like to turn this on, to see what actually happens
RewriteLogLevel 1 #when testing out a new rule, i set this to 5 so i can see what's happening step by step

RewriteCond %{HTTP_HOST} ^www\.([a-z0-9]+)\.com$ #this will match, www.bleh.com, www.anything.com, www.wassup.com, well...you get the idea. Note that I applied the () grouping so i can access the matched variable later as %1

RewriteRule ^/([a-z0-9]+)$ http://$1.%1.com/ [R,L] #now i pick up the /sub part of the url, and use that to generate the redirect url. The i apply the two flags [R, L]. R means force a redirect. L just makes this the final rule that this match will traverse.

Lets lay it out again without all the nasty/helpfull comments.

RewriteEngine on
RewriteLog /path/to/logfile
RewriteLogLevel 1

RewriteCond %{HTTP_HOST} ^www\.([a-z0-9]+)\.com$
RewriteRule ^/([a-z0-9]+)$ http://$1.%1.com/ [R,L]

Yup... try this out, and see the deep vodoo. On my test apache host, if i do:

http://www.mrouter.com/sub

I end up getting http://sub.mrouter.com

kewl o yeah... voodoo? DEFINITELY!!!

Essien out.

guess who's back

2005-11-15T22:50:00.000+01:00

so you thought you had me huh? And i was gone for good? well... i'm back.

Seriously though, a lot of things has been going thru my mind this last
weeks, and I thought it was better i sort them out before i went on with
bloggin as usual.

Anyways... now, i'm back bloggin. And for a first taste, i've been
playing with mod_rewrite for apache.. and DANG!!! Its sooo cool.

Pysystray progress - Lessons learned so far

2005-10-05T15:15:00.000+01:00

pysystray-0.6.0 and pysystray-0.6.1 (a bugfix release) are out in the wild now :)

(http://datavibe.net/~essiene/pysystray)

The major work in the 0.6.X branch is to make the menu's modifiable on the fly.

Grab n Play at: https://sourceforge.net/project/showfiles.php?group_id=145849

Now, this is what brings me to the crux of this post. On my own, there are some plans I had for pysystray, these plans are no where near what it is now. My point is that my own scope when compared to what pysystray can currently do in its relatively short lifetime, was terribly myopic. Its amazing what real life situations can do to a software project, even a very small one like pysystray.

This brings me to another thought. Imagine that I was a company, producing pysystray. Ofcourse, one can say that my users will also eventually create the ecosystem akin to the one i've described. I agree. But the question is, what chances would a single guy like me, have of convincing the next Joe Schmoe to actually Buy n License pysystray? Well, I really don't know the true answer to that, but i have a feeling that it has been easier to build an ecosystem b/cos of the BSD license.

That said, I think the most important difference is this. Under a Buy n License arrangement, (aka. Get em, lock em in ;) ), it would seem that once I have my customers, locked in and dependent on me, I am likely to drive the code in any direction that I see very fit, regardless of wether this is neccessary or not. I could decide to begin to add features, and even begin to dream up integration into other products of mine that i'm dreaming up. I can then use that product as a leverage to move other products of mine into the unsuspecting consumers hands. This seem like a wise Marketing move to me. Eventually what could happen, is that small simple pysystray, could be Pysystray Professional 2005 :-), and i'll have standard versions, enterprise editions that can connect to Oracle and even SAP!!! :-) (i can't even say that with a straight face!!! LMAO!!! )

Anyway, you get the picture. Let me paint the second picture. In the second case, the code is opensource, so my selfish interests can be uneccessarily served, because once i begin to move in a direction that is undesirable, someone else can very rightly grab the source and fork the project, and i'll be left with a product with no users :). Instead of being selfish, I have to learn to listen to the community that is growing around the project. Some of the users are using it for some very serious projects, and will have really demanding requests to make. Its in the interest of the project, to try to satisfy these real needs as much as possible. That way, the small project gets better and more usefull, and everyone goes home happy :)

Let me say here, that the first scenario doesn't neccessarily always happen, but it can happen (and has happened too many times than is good). I belive software should be as small and simple as possible (and no smaller/simpler), serve one purpose (and better serve it well), and be reusable in ways not originally imagined by the creator. If software does this, it will proove over time to have survival insticts of a virus, as it will keep finding itself used by larger projects.

Software written in this way, is easier managed too, and is simpler to finish too. I'm beginning to find that some of my biggest beef with some software, is my total inability to bend them to my vastly superior will (yeah right!!). But seriously, I'm not dumb, and I want to be able to do things the arthur didn't even think possible. This is my idea of Genetic Mutation / Natural Selection as applied to software. If your software can't easily mutate to serve new needs (b/cos there will always be new needs), it is doomed before it is even released.

In conclusion, I've been learning that there is truly "Nothing New Under The Sun", as most ideas we think are new, are slight variations of already known ideas. If you combine this with "Scratch Your Itch", then you can really have a successfull project. If you build something just for the sake of building it, its almost likely to fail, or become stuck in a rut later in life. If you build something you actually use yourself, chances are you'll lovingly take care of it, and strange as it may sound. Successful opensource software have a loving arthur, who painstakinly guards against dirty code, wrong ideas, etc, but most importantly, keeps the code lean and mean and relevant.

Phew!!! That was a mouthfull...

Essien... out.

XMLHttpRequest - Good or Evil

2005-10-05T11:49:00.000+01:00

Bunmi (http://www.inthrill.com/weblogs/bunmi) sent this article around: http://www.devx.com/webdev/Article/28861

They've been having fun with studying AJAX at Artemis (http://www.artemis-solutions.com) recently (i'm jealous :) ). Anyway, that article arthur ticked me off the wrong way, especially with his conclusions... I thought he was going to be objective and balanced, but he wasn't, so here goes my two kobo reply.

I think its unfair to paint Ajax as unsafe in the face of similar data that CAN and using the authors phrase, HAS been ganered from users using Desktop apps too. Truth is, once youre on the internet, Security and Privacy gain a whole new meaning. I mean, the term "spyware" came up way before AJAX. Assuming that the Web was safe before AJAX (or safer), is a false premise, check out what cross-site-scripting can achieve. Agreed with AJAX you can do these things easier than before, that doesnt mean that without AJAX these things totally cant be done.

One may argue that with spyware, you have to install them... well... have you ever decompiled/disassembled that program you use most on your system, to make sure its not spying on you? Oh... i c... you took the Software Companies word for it... And now you're not going to take their word for it if they tell you their website is not spying on you? Whatever... Me, I don't believe any of them anyway... I just limit how much of my info can be compromised so at anytime the amount of info I loose is really minimal. How do I do this? I TRY to allow only OpenSource software have access to my most personal data, and where I can't, I don't bitch if I can't fix it. I'm I still very safe? Ofcourse NOT!!! What kind of world would this be if we were all totally safe? Unrealistic... that's the truth of Life... deal with it.

It is also not ground enough for the author to say: *Are you not yet fearful and angry? Not ready to rise up and strip the XMLHttpRequest code from your browser?*

As a matter of fact, I think thats just a dumb conclusion (IMNSHO). Every Network Admin worth his salt, infact, any Engineer or human that deals in innovations, knows that when adding features to a product/design, most features can be used for Good as well as for Bad. If you asked me, I would say that in Nature everything comes in ionized (for want of a better expression) pairs Positive Side n Negative Side.

Take the most basic of human abilities Choice. This is the single factor that allows us to have great men like G. Washington, A. Lincoln, Essien Ita Essien (no thats not a typo :P), its also the same factor that allows us to have some of the worlds worst Tyrants like A. Hitler, J. Stalin, etc. Is choice a Bad Thing? Ill say a resounding NO!!! But Im guessing that the Arthur of that article is likely to recommend that the UN recommend all Nations of the world to pull out Choice from all New Human Beings. Soundz hilarious right? Exactly, thats what reading that pessimism laden paper feels like to me.

I would appreciate an article that highlights the potential security holes, and a way for users to be wary of them better than an out-on-out attack on AJAX (just b/cos its a new technology, and wasnt pushed to the fore by some *traditional* big name, b/cos i'm almost certain that where this angst is coming from), and a conclusion that we should Strip Out XMLHttpRequest from our browsers!!! That just silly.

Anyways, any browser that likes, can go ahead and pull out XMLHttpRequest from its implementation and then sit back and watch satisfactorily as its Market Share torpedos downward (Oh, and if you've somehow missed the news, Opera the browser is free now too!!!). Yeah, and they can quote me as having said so (who am i? baaah. Go away thats not important YET :-) ). XMLHttpRequest has come to stay. AJAX is a technology that grew up in the wild of real software solutions (as opposed to a carefully designed paradigm), as such, it has _Darwinically_ earned its right to survive and my o my survice it will, or mutate into something even more crazy it will.

My advice for all Nay Sayers? Wake up shesssh get real this is NOT _Duh Maytreex_, this is actually _Duh Reelwhurled_ :-) and as such, AJAX is here to stay.

Oh and lest I forget, better get more educated on safe browsing habits, and dont worry, the next generation of Data Protection is just around the corner trust me, I use Firefox!!!

Essien... out.

Web 2.0 - What do you think it is?

2005-10-04T11:17:00.000+01:00

Bunmi (http://www.inthrill.com/weblogs/bunmi) recently sent a mail around the office that related to Web 2.0, I replied and in so doing, crystalized my own thoughts for Web 2.0. Here is my unaltered reply below.

Web 2.0 is the emerging webscape that is basically driven by small, decentralized, hackable and open platforms. It incorporates a lot of things, but the thing that seems to have really got it kicking off is AJAX. At the very core of web 2.0 is dynamism (and dynamic frameworks with its starchild RoR) , transparent data (RSS, Atom), simple frameworks (a brave new RESTafarian web as opposed to SOAPsuds :-)), and oh! Did I mention a connection back to Google? :-)

I like to think of it as the desktop, slowly distilling its most important parts, onto the browser. What I mean is that Web 2.0 aims at pulling the basic underlying things we do on our desktop, and making them doable from the browser. This doesnt mean well soon have M$ Office running in Firefox / Opera / IE, but it means that a lot of what we do today on the desktop, will be increasingly done in a browser, hence making the most important platform of the next couple of decades, NOT the Operating System, but the BROWSER.

This also means relevance in this emerging techscape is going to be judged a lot on hackability / trustability (I define hackability as the property of a platform that allows it to be easily extended/modified/maintained, without relying on lots of theory and/or _advanced_ tools. I consider trustability of a platform, as the property that allows it to be easily extended/modified/maintained, without loosing peace-of-mind. Think continous_integration, test_driven_development, worse_is_better, release_frequently_release_early, feature_driven_development.

This is ofcourse my own understanding of the current webscape and events I see unfolding in the industry. Whatever eventually happens, Im sure glad to be alive and coding in these times == >

@(^_-)@

Burning My Way To A Distro

2005-09-23T13:42:00.000+01:00

15 CDRs and counting!!!

This was just at last nite... i burnt about 10 of them last nite.

What am I talking about? Oh yeah, I've been building a customized linux distro based on ArchLinux for a targetted application for a while now.

I already have the ALE (see my Links sidebar) framework, so this hasn't been a total from the scratch affair. Thanks to ArchLinux's overwhelming KISS principle, anyone who's interested can do stuff like that. Trus me... its not rocket science :-). Anyways, the plan is to embrace and extend ALE. So once this is done, i'll supplement ALE such that, I can redo this by just changing a config file, and boom!!! (Ok... so exploding stuff isn't quite so good, but you get the picture.

Last nite, I ensured the partioning works properly (i've disabled FTP mode from the base arch installer for now, just to keep things simple), and I'm making the actions follow themselves. Currently, on the Arch Installer, each time you finish an action, it brings you back to the main menu, which is ok for me really, but for my target audience, It will be less ambigous, to actually go thru the steps one after the other.

Everything works okay now, so I'll need to update the base packages to pull in the lattest packages available now, and then i've been thinking of the Config Files tweaking. Dunno how i'm going to solve that yet. I've been thinking of just dropping some sane default values there, and having a seperate system program to help tune the configs. Dunno, not decided yet, so i'll leave it as it is.

By tonite, I should finally install successfully on the first victim box :), then I'll spend the weekend tidying up the dialogs, and perhaps finetuning the configuration mechanism.

Essien out!!!

pysystray on Daily Python

2005-09-21T16:37:00.000+01:00

This was an interesting suprise on Daily Python (http://www.pythonware.com/daily/), The list for today 2005-09-21, lists pySystray-win32's lattest sourceforge release announcement there. Yipee!!!

Its actually a humbling experience for me... *sniff*

I'll just go cry now in private... this is a special moment :-)

pysystray-0.5.2 released

2005-09-21T11:27:00.000+01:00

Just updated sourceforge, with the lattest version of pysystray.

This release is basically concept complete with what I initially set out to acheive. Its amazing that most of the time, I didn't really plan what would be in the next version, but I just went with my gut.

I now have balloon tooltips, on_load event callbacks, etc, so I can fully write small service like apps that totally sit on the Windows Notification Area, all in python.

My first thought is an IRC impersonation of me :), this guy will just notify me if i'm doing something important and someone in a channel he's watching says something that might interest me :), ofcourse, there's the ever-vapourware 'phiba' (tm), that may eventually get written now.

bahhh... i'm just a lazy humbug :)

Essien... out!!!

Customising ArchLinux

2005-09-12T20:29:00.000+01:00

I've recently had to build a custom distro aimed at a specific purpose. To do this, I quicly visited my sources for ALE (http://datavibe.net/~essiene/ale), as a starting point to build the LIVE root image. This was actualy a no brainer, as all it involved was building the pacages I wanted, and changing the ALE config files.

The intersting part has so far been building the CD installer.

In according to the Zen Of Laziness (TM) of which i'm an avid practitioner, I am basing the CD installer off the basic official ArchLinux installer ISO. Hacking it has been fun so far. What i'm basicaly doing, is to tweak the setup script, so some options that I don't need are hidden, and some sane defaults are used in configuring most of the config files. Since this is a targetted distro, its wiser to use them Sane defaults.

Once I finish this, i'm going to tie it into ALE somehow. The goal is to be able to build a CD installer for a custom distro in all of 5 minutes, levering off ALE's config files, etc. :)

I currently have to vanilla box to test the installer on :-( so I'm having to just print messages where work would be done... (not cool!!! :( ), Ii'll still make sure everything works well, then maybe I'll give wendy (my old laptop), a run for here money :), poor ole gal, she really served me faithfully (and still has some important data, but bolivia (my new laptop), totally kicks wendies behinds ;)

If i keep working at nights, I should be able to get this completed this weekend, with minimum fuss. What's actually slowing me down, is the testing... I wonder if there's any other way I could do this.... hmmm....

pysystray new releases

2005-09-12T20:20:00.000+01:00

pysystray is undergoing some maturing out in the wild, that it prolly would not have achieved indoors :)

http://datavibe.net/~essiene/pysystray

https://sourceforge.net/projects/pysystray/

Originally I had intended pysystray as an alternative to writing real services. So a pysystray application would just sit in the system tray and do what a service would have normally done (yeah... i know... its not the best place to do that... but heck it rocks).

Anyways, releases upto 0.4.2 focused on ironing that out smoothly.

Last week, there was a need on the mailing list, to use pysystray in a more traditional sense, as part of a larger GUI project. In this case, enough wasn't exposed via the API for controlling the underlying _App() object. Solving that was an intersting excercise though, and on Friday, i released 0.5.0, which introduced a new Object systray.Control() (for want of a better name :-) ). This allows more traditional things like Control.show(), Control.hide(), Control.enable(), Control.disable(), etc.

Today, i've just released 0.5.1 (worked on this over the weekend), to iron out some kinks in 0.5.0.

Currently, while waiting for feedback, i'm thinking of cleaning up the Control api in the next 0.5.x release.

Ofcourse, all these, have not broken API yet for the already existing systray.App().

Also, in the process of doing this, i've been learning more about how win32api actually works. Its not facinating (and i think its too verbose), but at least, i finally am understanding the shindig! :)

Silent Nigh, Hackers Nigh :)

2005-09-12T20:11:00.000+01:00

Yup, i've been quiet for a while. Been quite busy on a number of fronts, and interesting things are happening too. Be giving a flurry of updates in the next few days.

E-Modules tutorial on edevelop.org

2005-08-29T14:49:00.000+01:00

The E-Modules exploration I wrote here: http://essiene.blogspot.com/2005/08/exploring-e17-modules-part-i.html, has now moved to http://www,edevelop.org

I've cleaned it up a bit more, removed the funky disclaimer (which i actually miss O.o :) ). Generally, I think that version is cleaner, but i've maintained a link to the original.

In future, I will also keep posting the raw versions here, while I will clean-up and post another version to edevelop.org

Very Cool Thoughts On Coding

2005-08-26T18:10:00.000+01:00

I just stumbled across these links, and I must say, I love Kevin Barnes' way of putting things.

http://www.journalhome.com/codecraft/8106/#c2752

http://www.journalhome.com/codecraft/7775

Exploring E17 Modules - Part I

2005-08-26T15:23:00.000+01:00

DISCLAIMER: This is not your traditional tutorial where someone that knows somthing tells you about it. No. In this case, this is someone learning about something and writing down thoughts. That it happens to feel somewhat like a tutorial is a side effect. That said. Njoi :-)

After much procrastination, yesternite, i was plain restless, I looked at the numerous projects I have at different stages of completion on my box, and decided to instead look at something different: E17 Modules.

After my fair success with pysystray(http://datavibe.net/~essiene/pysystray), I've been thinking of doing something similar in E17. The only problem is having the time to sit down to understand what is really going on. So yesterday, I took the first plunge.

Its amazing what barriers knowledge can pose, and consequently, what barriers you can surmount, when you have the knowledge. My main problem earlier had been setting up a build environment for a module. I hadn't really understood the working of Autotools (from the angle of the developer), so a few weeks back, I did a Google for AutoTools tutorials and came up with a free book :). After reading upto chapter 12, I really was *enlightened* :-), and finally, looking around the e_modules source tree, I suddenly understood how to insert my own module into the whole e_modules build process. (I'll expantiate on this latter, just incase you are like me, and didn't quite grab it at first).

Anyways, my approach is a very bare bones approach. I took the in-tree weather module, stripped it to its barest components. And created a module i called 'hello' :). And from there, I'll start building up as I keep striving to understand the whole architecture.

My first findings (which are not really news anyways):

1. There are some functions that MUST be supplied by the module (this is akin to implementing an Interface if you're thinking OOP). e_modapi_init(), e_modapi_shutdown, e_modapi_save(), e_modapi_info(), e_modapi_about().

2. Each of these functions takes a single argument. A pointer to an E_Module.

3. The E_Module structure, contains all the information you need to know/tweak for your module.

4. The order goes like this:
a. When you load your module, nothing in your code is called (same when you unload your module). The work starts when you Enable your module or when you Start/Restart E while your module is marked as Enabled
b. When your module gets started some funky stuff happens to load the module into memory, and the first function from your module to be called in e_modapi_info(). In e_modapi_info(), you're supposed to set distinctive characteristics of your module. Like the icon file, the edje file, and some other things like that. In my bare module. I didn't do anything there... just to get stuff running.

c. Next, E's config system queues your e_modapi_save() to be called at a latter time when the system is idle. (Thnx to raster for this explanation, I had it wrong before). e_modapi_save() is actually supposed to be used to persist data used by your module. (more on this latter). In this bare module. I left this blank as well.

d. Finally, you e_modapi_init() is called. Remember this function takes an E_Module *, but returns a void*. Some important things need to happen here. Firstly, You _SHOULD_ return a valid pointer to a data structure that will be used by your module. If you return a NULL the init will fail. You can be sure that if you return anything that points to memory you can't account for you'll have funky results ;) (Be my guest... try this, and lemme know how long it took for you CPU to melt :-) ).
The strategy here, is to create a simple struct that will contain all the info that you need to do you stuff, and return a pointer to that struct. What E does, is take your return value and store it in the E_Module->data attribute, where it will always be available from that point onwards.
Imagine you have this.

typedef struct _hello {
char* message;
} hello;

typedef hello* Hello;

Let that struct be your module Hello struct. for e_modapi_init(), we'll have this:

void*
e_modapi_init(E_Module *m)
{
Hello h = NULL;

/*normally you check for api version here and bail out if version is wrong. lets just ignore that for now*/

h = (Hello) malloc(sizeof(hello)); /*you should usually ecapsulate this in a function like hello_new() that does this and some other stuff for you.*/

if(!h) {
e_error_dialog_show("Hello Module", "Initialize Failed!"); /* I'm adding this so we can see what's going on.*/
return NULL;
}

e_error_dialog_show("Hello Module", "Initialized Successfully!");
e_error_dialog_show("Hello Module", "Hello World!"); /* Finally our own e_module hello world... now the world is at peace ;) */
return h;
}

That's a very basic e_modapi_init() to get you going. When you being thinking of saving and restoring data, a bit more will need to be done here, and when you throw in menus, GUI, config stuff... that baby can begin to do realy much more... for now tho.. this is all you need.

e. After init returns. You module should be fully running. There are two more important hooks that you need to take care of. e_modapi_save(), and e_modapi_shutdown(). Like I mentioned earlier, e_modapi_save() is used to persist data. And we don't really want to do any persisting now, so we ignore it. On the other hand e_modapi_shutdown(), will be called when we Disable our Module, or when E is going down (for good, or for a restart). What we need to do in shutdown() is to release all memory we claimed in init(). But how do we get that data? The E_Module struct has a number of attributes we should get very friendly with, but right now, what we're iterested in, is the data struct we initialized in init(), That data struct is stored in the 'data' attribute of our E_Module struct. In our case this should suffice.

int
e_modapi_shutdown(E_Module *m)
{
Hello h = NULL;

h = m->data; /* this is how we get back our Hello struct from the module subsystem. There are other attributes that hold other things, face, menu, etc. ignore for now */

free(h);

e_error_dialog_show("Hello Module", "Shutdown Completed");

return 1;
}

In our case, that simple function will do our shutdown for us.

f. Lets quickly run over the info, save, and about. Remember, e_modapi_info() is the first function that is called in your module. Its supposed to help setup information on your module, like label, icon_file (which shows up in the menu), etc. about is called when the about menu item is clicked. Typically should just give info on your module and exit, but hey.... you can write a webserver inside there if you're so inclined :), just return 1 when you're done. For save/info, we'll do nothing. Here we go:

int
e_modapi_info(E_Module *m)
{
return 1;
}

int
e_modapi_save(E_Module *m)
{
return 1;
}

int
e_modapi_about(E_Module *m)
{
e_error_dialog_show("Hello Module",
"A hello wold module for E17. To appease the gods of tutorials.\n"
"If we don't have this, we may be banished to debugging the great "
"Aiii Eee for the rest of our days! \n\n"
" Visit: http://essiene.blogspot.com/2005/07/travellers-log-1.html "
"for more information :)"
);

return 1;
}

5. Finally comes the part that kept me from doing this for a very long time. I didn't understand the compilation process. Well, now I understand it a bit more, thanks to reading up autotools (btw, FREE BOOKS ROXS :-) )

The _easiest_ way *THAT I* found to compile my module was to add it to the e_module tree from CVS. If you already have that proceed. Else, check http://www.get-e.org for instructions on how to grab E from CVS. You only need the e_module tree really. So this could be a quick start here. The password for anonymous is blank. Remember that.

$ cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/enlightenment login
$ cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/enlightenment co e17/apps/e_modules

These two commands will check out e_modules for you:

The e_modules directory will contain this directory src/modules, where all the other modules exist.
a. Create a new directory called hello in the src/modules directory.
b. Type your source files in here. Traditionally, the files are named e_mod_main.c and e_mod_main.h (for any custom headers you want to define and function prototypes). Remember to include the correct header files, e.h for all the e stuff, and any other functions you use. (stdlib.h for malloc and free)
c. You'll need a file Makefile.am, that is used by autotools to generate Makefile.in which is used by configure to generate Makefile. (pretty twisted huh? Yup... if you don't understand it that is :) )
d. For that, you can copy the Makefile.am from the flame module directory, into your directory, and rename all instances of 'flame' to 'hello'
e. Some files are mentioned in that Makefile.am that you don't have currently, so i suggest that you copy module_icon.png from the flame module over as well. Don't wory, it won't break anything.
f. Now move into the parent directory (modules), and edit the Makefile.am there. add 'hello' to the end of the line that says SUBDIRS = . (this enables your module folder to be processed)
g. Step upto the e_modules root folder and edit configure.in. At the bottom of the file, there will be a set of lines that look like: src/modules/flame/Makefile. Add your line after the rest, but before the square brackets. Something like this should suffice: src/modules/hello/Makefile
h. Save and exit configure.in
i. If you fetched from CVS, this is the time to type ./autogen.sh in the e_modules root (if you downloaded a tarball, then just go ahead and type ./configure, specifying the neccessary options to configure/autogen.sh)
j. go ahead and make with make install
k. load the module like you would load other modules
l. enable it. You should see your message boxen now :)

Well, that's how far I got last night. Before I started delving into the E sources trying to understand more stuff, also asking questions on IRC. In the next log, i'll attempt to tackle persisting information, and maybe adding menus and or faces. I hope this helps someone out there :)

Its Official GTalk Rox

2005-08-24T16:19:00.000+01:00

Threaded handlers in Pysystray

2005-08-24T12:42:00.000+01:00

One of the important TODO's I have for pysystray is making the menu item callbacks/handlers threaded. To acheive this, I thought of the simlplest most unobstrusive way to add it, and I decided to go with Decorators.

Uptill yesternite, I didn't really get Decorators, but back at home, Stan (my flat mate) and I took a look at the PEP, and suddently It was all clear.

I then wrote this simple decorator:

>>> import thread

>>>

>>> def run_threaded(func):

>>> def wrapper(*args):

>>> thread.start_new_thread(func, tuple(args))

>>> return wrapper

To run any function in a seperate thread of execution you'd do this:

>>> import time #just for a sleeping example :)

>>> @run_threaded

>>> def looper(x):

>>> for i in xrange(0, x): #gotta love generators :)

>>> print i

>>> time.sleep(2)

now running:

>>> looper(10)

will run it in a seperate thread of execution.

With me finally understanding Decorators, I now have just metaclasses left to understand before I can say I'm trully upto step on python 2.4's features. But man... you just gotta love this language... Its sooooo malleable.

I've added the run_threaded decorator to pysystray, but i've not released this version yet... I'm going to refactor the files a bit more, and try to attack one or two more todo's before i release 0.5.0.

Btw, If you're wondering why this is an important addition... Currently, if a MenuItem callback/handler takes too long to complete, it hangs the UI, and the pop-up menu no more shows up. This will also be helpfull for the on_load event handler so it also doesn't lock up the UI.

Oh.. btw, I'm thinking of using the Google API message notifier for pysystray's notifier... If its possible. I'll start groking the SDK next :)

Google Rulez!!!!

2005-08-24T12:20:00.000+01:00

OMG!!!!

Yesterday, I installed Google Desktop 2, allowed it to index Bolivia (my laptop), and was just tripping all over it, and was just about to post praises, but guess what happened?

Google Talk was released this morning, and DAMN!!! I've fallen all over myself on this dual move.

Everyone at my office already crying 'Death to Yahoo Chat' just by using Google Talk Beta!!! Its amazing, and the Voice Talk feature just plain blew us all away. Yahoo Chat is toooooo darn heavy... sheesh!!!

The things that Google does are now bodering on Magic ;), I mean, with Google Desktop, who needs that Start Menu anymore? Just type the beginnings of your program name, and its finds it, you press enter and 'launcho!!!'

Google Desktop also has notifiers for my Outlook mails, Gmail, everything, and even an Outlook search plugin... which totally rox!!! Ofcourse, i've disabled my Outlook notifier and i'm using Google's Notifier now.

My homepage is now set to Google Desktop's Homepage, which is totally what I love.

What's next? Oh well.... there's the SDK with Python bindings :D, so I guess you can already percieve what's next... oh yeah... baby... we're gonna hack the hell outta that SDK :D

Oh.. and Google Talk is keeping with Open standards, supporting XMPP/Jabber... cool... Man, these guys are seriously throwing open the field... now a major IM provider is behind Jabber, I wonder how Yahoo and MSN respond to that... not that I care anyways :D

Yahoo made a good move with konfabulator, as i'm still going to keep my Konfabulator weather widget on my desktop, but hey... that's about all i'm going to be using from them :D

The world is getting more intersting... i'm just wondering what next... Googlix? A Google Linux Distro? O.o... they have the world by the balls already... and the world doesn't seem to mind ;)

Oh... boy.. exciting times :)

pysystray progress

2005-08-23T12:01:00.000+01:00

Yesterday, on request by some users, I changed the licensing to BSD style. There is now a 0.4.2 release, which just contains that change.

This seems to be the least problematic license out there, and I really don't have to wrangle my head around the specifics. I just want to write code that others can use, period.

There is now a mailing list: http://lists.sourceforge.net/lists/listinfo/pysystray-users

There is now a couple of things to do, cheif among them is to make the menu items actions threaded, so that it doesn't lock up the UI while long actions are being performed. That and Adding an on_load, event that can be set to start a process once the App starts. Those are my highest priority features before i start considering a version 1.0.

pysystray on sourceforge

2005-08-22T12:19:00.000+01:00

I've been quiet for a while. Deliberately too.

In that time, quite a lot of things have happened, all not too good, but some very exciting.

I'm currently hosting pysystray on sourceforge. The homepage is at http://datavibe.net/~essiene/pysystray

I tried everything in my power to get the sourceforge supplied webspace to host the homepage for the project, but nada.... it didn't work. So i gave up and went along with my webspace.

So far, i've had some good feedback, and a major project seems to want to use it, I just got mail from someone on their dev team this morning, just clarifying the licensing :)

Cool huh? Yeah... makes me feel good that pysystray is usefull for someone afterall.

I'll update with what happens to that :)

more EET progress, update on D Xlib

2005-08-05T14:56:00.000+01:00

Got some more lines of code in yesternite. Moving a bit slowly... i have this project at work that is tasking me out.

Anyways, i've added support for image data to the EET File. In the course of this, i had to create two struct ImageHeader, and Image, to make it all cleaner to use. Image may become a class with its own methods in a future incarnation/refactoring cycle. I can smell it already, but i'll delay the decision untill its obvious. Next come the DataDescriptor stuff i've been avoiding :D. Once i'm able to do those, i'll give myself an ECCTD certification. What's ECCTD? Duhhh!!! Essien Certified C To D Porter... DOH!!! :)

On the D Xlib front. Dejan (http://dejan.lekic.org), has finally gotten around to trying out what i've done so far, and declared we're good to go for the next step. Incidentally, we've decided to put a chill pill on Directly making Xlib D, OOP. What i think is that it will happen when we start the next phase... uhh.. yeah... This was just a precursor to another project. Porting DFL to Linux (http://www.dprogramming.com/dfl.php).

Just incase you haven't yet figured it out... that's how the XLib to D port came up :). Vathix, who is the author of DFL would want to have DFL on Linux, so Dejan and I are helping out, and in the process, hopefully filling a great need.

Anyways, we're ready to start moving on DFL, and we're going to be modelling it after dotGNU's Xsharp (A from scratch implementation of the .NET System.Windows.Forms functionality for the dotGNU project).

Dejan should setup subversion for Xlib D, and we'll be on our way... lets see how much work we can punch in this weekend.

exml : first feelings...

2005-08-05T14:09:00.000+01:00

First of all Libxml2 is a great library. I truly respect it.

Ok… that said, the C API … it has too many gotcha’s… sheesh!!!

Lets say I have a file ‘root.xml’ that looks like this

<root>
<node1>
<node2>
foo
</node2>

<node2>
bar
</node2>
</node1>
</root>

Using libxml2 directly, I can get a structure representing the whole document with

xmlDocPtr doc = xmlParseFile(“root.xml”);

To get the root node:

xmlNodePtr node = xmlDocGetRootElement(doc);

currently, node will contain information about the <root> level.

If I try to printf() node->name, it correctly reports ‘root’. Nil Problemo.

My problem arises when I start trying to walk the tree. xmlNodePtr has a member called children,
And its supposed to return the children node. Anyways… if I do:

node = node->children;

I expect it to now be pointing at <node1> right? Well, its not… instead when I try to printf() node->name at this point,
It gives me ‘text’.

Ok… before some libxml2 guru crucifies me, I’m sure there’s something I’ve not read the documentation for, but hey… that’s the behaviour I expect… or am I missing something?
Don’t answer… but I guess I am. I kept trying to play with this for a couple of hrs, trying to recursively walk thru any document… and this behaviour kept tripping me.

Anyways, I just thought.. hmm… what is in exml? Afterall what do I have to loose? I have the entire enlightenment cvs tree on my system anyways…. So away I went.

Using exml, I was pleasantly surprised at how it matched up with my thinking and my expectations (I belive my expectations of API’s are usually very high).

Anyways…. To get the entire document with exml I do:

EXML* doc = exml_new();
exml_file_read(doc, “root.xml”);

Almost same as libxml2, and a bit more verbose I admit. I would have preferred it being structured after a constructor paradigm.

Now, I can query the current point that the document is at. Via functions… so if I do:

printf(“tag = %s\nvalue = %s\n”, exml_tag_get(doc), exml_value_get(doc));

nice and easy… ofcourse… you can still an object that refers to the current node, by:

EXML_Node* node = exml_get(doc);

And the previous can be done with:

printf(“tag = %s\nvalue = %s\n”, node->tag, node->value);

So far, nothing really different.

The tripping part is if I want to get the <node1> part of the document. EXML has very handy functions for clearly moving around the document unambiguously. (I don’t know if libxml2 has these as well, but I didn’t wait to find out once I discovered exml’s ease of use)

exml_down(), exml_up(), exml_next(), exml_next_nomove(), exml_goto(), exml_goto_top();

each of these move you in a specific direction without suprising results, and return the value of the next node tag as a char*, or NULL if there is no next node.

Still trying to drill down to ‘foo’ and ‘bar’ from our xml document, I would do this:

//I’m not checking for errors or freeing memory as I’m going, in real code, I’d do these.

EXML* doc = exml_new();
exml_file_read(doc, “root.xml”); //right now we have the whole document, and we’re at <root>
exml_down(doc); //we’re now at <node1>
exml_down(doc); //we’re now at <node2> so we can read and get ‘foo’

printf(“Expecting ‘foo’, got : ‘%s’\n”, exml_value_get(doc));

exml_next(doc); //we’re now at the second <node2> so we can read and get ‘bar’

printf(“Expecting ‘bar’, got : ‘%s’\n”, exml_value_get(doc));

well…that’s it basically… that’s just it.

All in all EXML.h is just 81 lines of defs, white space included. Also, I didn’t need any primer to figure out how to use this library. I just looked at the header file and it was clear (Rule Of Least Surprise).

To my lazy mind… this is currently THE xml C library for me now. The only thing for some other folks may be that it depends on Ecore, another part of the EFL, which actually depends on Evas and EET as well, for me tho… I already depend on the EFL, so I don’t mind one bit.

Did I remember to mention that its built ontop of libxml? Yup… it’s a very nice piece of work… building on the solid layer that is libxml2, but not having all the gotcha’s etc.

Ofcourse, there’s no inherent support (YET) for XPATH, XSLT, etc, and all the goodies that libxml2 already has, still, it works so far, and in less than 15 minutes after looking at the header file for the first time, I already had the weather.com parsing library upto the point where I can inteprete simple search queries…

Nice? You bet!!!

Essien, out.

Ajax Paper

2005-08-05T13:57:00.000+01:00

http://www.adaptivepath.com/publications/essays/archives/000385.php

This seems to be one of the earliest and authoritative on the AJAX phenomenon. Good read.

C Xml Processing

2005-08-04T13:30:00.000+01:00

Started trying to grok XML Processing in C, and so far, I’m learning libxml2. Seems interesting, though really painfull initially ( I mean, coming from ElementTree and the links of Amara that are available in Python, libxml2 C Api can be very primitive). But now, its not looking that bad at all, infact, I can say I’m enjoying myself… currently I’m turning my attention to using it to grab information from the weather.com webservice (I expect to have a libweathercom from that experiment). Then ofcourse, there is the conversion to D (if it hasn’t already been done)… who says life is boring? :)

Essien.. out!!!

EET progress

2005-08-04T13:27:00.000+01:00

I’ve moved over the hoop that was causing me grief in my EET File class. Its done now, and unittested. I’ve also discovered that apparently a file can’t be opened for Appending… that’s somewhat odd, there are some File Mode flags for READ, WRITE, READ_WRITE.I’ve not been able to use READ_WRITE to successfully open a file. Tired asking around #edevelop. No one answered authoritatively, but ‘Codewarrior’ suggested I may be right.

Anyways, that won’t stop me from moving ahead.

There are some tricky contstructs still infront, so, my path for the rest of the week is cut out for me I guess. Interesting? YOU BET!!! :)

Xlib Classes for D

2005-08-03T11:37:00.000+01:00

Yesternite, I got around to start encapsulating Xlib C functionality into D Classes.

Still working from the simple line drawing example, I was able to pick out the necessary layers for a basic Xlib application, and have translated these to classes. So far, I had to create there to work:

Display

This seems to be the basis of what you end up seeing. I’m still reading up on what it really is, but I think it’s the basic abstraction of what the XServer actually controls.

Screen

For my program to run, I needed to get the current BlackPixel, WhitePixel values from the DefaultScreen of the Display. So my display class allows me access to the Screens (encapsulated as the Screen class)

Window

Now, those little boxen you see on you monitor. The ones with the close buttons, etc, are actually the windows. Actually I got to find out that even buttons, scrollbars, etc, are all windows under Xlib.
To create me a window I need a parent, and the display has a DefaultRootWindow, which I use as the parent for the top level window of my little line drawing app. Display class has a property, DefaultRoowWindow (encapsulated as a Window class)
Also windows are not visible or usable until after successfully Mapped by X.

This one confused me initially, as I first thought it was another surface encapsulation. Reading the Xlib programming manual in the wee hours of this morning, cleared this up for me.
A GC (Graphic Context), is an encapsulated resource that holds information about drawing primitives, for example, line width, line style, foreground color, etc.
To draw a line, I create a simple GC from the Window, and call its DrawLine method.

Yesternite, I was a bit lazy about encapsulating the Events immidietly, actually I felt that my initial understanding of the Events wasn’t very good, and I may not do a good job encapsulating, so I left them raw as they are. I’ll read more on them tonite, and do some more work. Once I have the Events also encapsulated, I’ll proceed to write the various unittests for the small functionality present in these classes, and I think I’ll do a File refactoring, it seems those classes are gonna get really big, I don’t want them all in the same single file.

Once I’m done with the UnitTests tho, I’ll move on to the second example program, and add features to the classes to get it to run.

Btw, I didn’t get to touch the EFL to D port last nite… I have some design decisions to make on EET, and I’m thinking I’ll take the very raw and straightforward way forward for now… when I’m more used to the class, I’ll come back and refactor.

Essien, out!

Yahoo! Widgets

2005-08-01T12:21:00.000+01:00

You know... ever since Google took the world by storm (uuhh... make that by search... or is it by searching for a storm? :) ), they've left A LOT of competitors behind... and not just in the dust... but in the Sand Dunes.

But i think i can say that Yahoo has been a worthy competitor... they keep responding well, (after all Yahoo's and Google's founders are all Stanford products right?).

Anyways, that aside, Yahoo has done something very cool... they acquired Konfabulator and set it free. I used to use Konfabulator pre 2.0 on my old laptop (wendy), and used to utter some PG Expletives anytime the expiration notice showed up on my box. When i got my new laptop (bolivia), i swore they wouldn't see my boxen anymore... no matter how cool they were (and they're ooooooo sooooo cool). Well a couple of weeks ago, i found out that Yahoo has acquired them, and set it free. So i can USE IT WITHOUT paying (call me cheap-arse, cheap-stake... whateva... that's your problem)

What is Konfabulator? Uhh... check out: http://www.konfabulator.com

what can you do with it?

That screenshot is worth a thousand how-to's (but you don't really need a thousand how-tos on generating a screenshot) [that statement doesn't mean anything, so please don't try to make any meaning out of it:) ]

Weekend Hacks

2005-08-01T11:12:00.000+01:00

I had an interesting and fun filled weekend... really.

First, I completed the first stage of the Xlib D port. And i have the first sample D Xlib program that draws a line to the screen, waits 10 secs and exits.

This is just the first stage.

The next stage involves making the library *FEEL* OOP and feel like D instead of C. This will involve a continous process of introducing classes, etc, to encapsulate most of the pointers flying around.

The hard part is done anyways, this part is a matter of continous integration, and good judgement. I'm starting that this week, and I'll have to be studying core Xlib programming to understand the application layering, so i can do at least a half decent job. No rushes here.

I'll make these available on my download area within the week.

On the D EFL port, i've been creating the D layer for EET, and ran into some pointer related issues, as i try to wrap up the function calls into D Classes, but along the way, i learnt more about D's invariants, pre and post conditionals (Contract Programming), this stuff is VERY VERY COOL. When used properly, it really reduces bugs, and helps find bugs faster, combine this with D's inbuilt UnitTesting, and i really don't know why i should use any other language for Midlevel Application Development (Low level is owned by C, and High Level is owned by Dynamic Languages like Python). If you really feel the need for speed... don't take a cup of Java, or try to Sharpen your C, no no no no no, just take the next alphabet... D. :)

Needless to say, i felt pretty accomplished this weekend, and rewarded my self with a nice move.... SIN CITY... the movie is sorta weird... but its really cool, and my, was Mickey Rourke, Marv? That guy was my main guy in the movie.

Ok.... off to work now :)

The Birth Of Google

2005-07-28T12:09:00.000+01:00

http://www.wired.com/wired/archive/13.08/battelle.html

Very interesting read.

More D port progress

2005-07-27T12:36:00.000+01:00

efl.edb is now complete for a first version. All natice Edb.h funcionality has been wrapped up with D functions/methods to give them a D feel.

I've started on the Eet.h. This also shouldn't take time. I'm going to avoid Evas for now, and instead probably focus on all the smaller libs, Epeg, Epsilon, Imlib (not soo small), Embryo. If i survive thru all these, i think i'll be in the right frame of mind to tackle the big boys, Evas, Ecore, Edje, Ewl.

Something is happening though as i'm doing all this. EFL is getting very clear. There is this thought of a Python port in my mind too. If I last thru all these a Python port will be very feasible and clearer too.

On the Xlib front. I'm continuing with the #define constants. I should be thru tonite or tmrw, then enter a thourough testing phase to see if i can transliterate C programs to D using Xlib.d; Once that is successfull, then i'll start to wrap the functions into D classes, etc.

Traveller's Log 1

2005-07-27T12:14:00.000+01:00

The sacred book of hacking says: "Every coder worth his modules, should have time that is *his* own to spend mashing at the stuff that interests him. That is the only path to true greatness"

There are they who wander in this path, but get sidetracked by the Great Coperate (tm). It is rumoured that some of them have been seen in the dungeons of the Great W$ debugging the Legendary Beast... Ai EE, that accepts malformed HTML, and spits your CSS back in your face. Needless to say, great is their misery.

Consider instead, J. R. Hacker of the Noob, of the order of the Freshman, of the order of the Unemployed. He it is said of, to have followed the path of X. It is said he was visited in a dream by the GNOME, the great protector of Gtk. Weather it be true, we shall never know. But we know it was he who initiated the GNOME Project. Today, he has been granted the venerable post of VP at the Wovell.

Thus spake the great master Google and I listened to his result pages. From them I have found the path to the SourceForge. I have learned to drink from its great repository with the cup of CVS and SVN. Master Google has thought me the rudimentary yet arkane art of building using the secret autotools and raw Makefiles.

I have learned to sit at the feet of other students of Great Master Google who meet at the IRC. Great scholars of the order of the #c, #python, #bash, #linux, #edevelop, #archlinux, and a great many other sects that are more numerous than any one human can possibly know. But the Great Master Google knows them all.

In the hallways of the #c, i have learnt the secret arts of glibc, and devoted myself the the art of libdl. I have learned to grok the PEPs of the Python dot Org to understand what the Masters say in #python. Now i understand the way of the List-Comprehension, and I can swim through any Generator Expression. There is still the secrets of the Meta-Classes and Decorators to be mastered though. But i have found that sitting under the ElementTree, one can feel the Zen of XML.

Finally I also have started my journey thru the desert of OSS. That great Oximorous Multiverse. My sword in hand, is built from Raw Linux found in the KISS valleys of http://www.archlinux.org. I have extended the blade and stuck a custom -mm kernel in the handle. I carry about a small Vim anvil in case i need to craft any C, D, Python, Bash tools. My backpack contains additional PHP and Perl hammers among a lot of other contraptions that master Google has given to me over the years.

My quest I know not where it will lead me, but I am intent on finding that path to true greatness.

Edb.h to Edb.d

2005-07-26T12:21:00.000+01:00

Anyway, just to Test my understanding of the .h to .d conversion process thoroughly, yesternite, i basically started and finished porting Edb (from the enlightenment EFL) to D. And you know what? I'm extremely IMPRESSED with those libraries. I was expecting to have lots of #defines to copy across, but NO NO NO NO NO!!!

Raster and guys are using really good C programming Semantics, with little used but beneficial features of C. Instead of using #define foo baa, like everyone else does, they use C's enums... And looking thru the header files, they read like a cool users manual. WOW!!! I'll tell you something... that library is WELL designed.

also, i've wrapped up the Edb into an Edb class, to make it feel more like a D programmer would expect. and the Edb class is ready as a version 0.0.0.0.1 :)

What is left are three functions:

char **e_db_dump_multi_field( char *file, char *file2, int *num_ret);
char **e_db_dump_key_list( char *file, int *num_ret);
char **e_db_match_keys(E_DB_File *db, char *pattern, int *num_ret);

I have to make these module level functions, and give them D calling semantics, and encapsulate away the raw pointers.

And oh.. here's a sample program that worked for me yesternite:

/*-------------------------------------------------------------------------*/

import std.stdio;
import efl.edb;

int main(char[][] args)
{

Edb db = new Edb("afile.edb");

db.set("Magnus_i", 55);
db.set("Magnus_f", 55.59);
db.set("Magnus_s", "fifty-five");

db.close();

Edb db2 = new Edb("afile.edb");

int ival = db2.get_int("Magnus_i");
float fval = db2.get_float("Magnus_f");
char[] sval = db2.get_string("Magnus_s");

db2.close();

writefln("Magnus = %d\t%f\t%s", ival, fval, sval);

return 0;
}

/*-------------------------------------------------------------------*/

Oh... and i forgot to mention... since D has inherent support for unittests, the Edb.d module contains its own unittests.

Why am i doing this? I'm learning... Duh!!!

This means to me that i totally understand the C.h to D.d conversion process, and hey... its fun.

Once i totally wrap up the C parts of Xlib.h, then I can start giving them D objects to make D Xlib programming more fun.

Essien... out.

Xlib, C to D port: Third Iteration

2005-07-26T11:58:00.000+01:00

O blimey!

When i said i would probably start again one more time on the Xlib C to D port... i sort of felt it. I'm not so sure i meant it, but i had to do exactly that this weekend.

When the Mythical Man Month says, plan to start from scratch at least once... its very true. Things get clearer if you not scared of starting over at least once. Infact... just plan for it.

I completed the second iteration of Xlib and had Xmd.d, X.d and Xlib.d as at Saturday, in the wee hours of morning. Then i tried to port a simple Xlib C program that draws a line to its D version. It was then my suspicions of gcc -E were confirmed. Most of the convenience macros and #define constructs were missing. The convenience macros are not really a problem, since D semantics will mean having to rely less on such trickery, but the #define constants are really important, as they are mostly values that are unique to a function etc.

Thus i decided to look at h2d at http://www.dsource.org/projects/h2d/

I tried it out, and my... it made my work even easier, and converted all the structs, typedefs, types for me. phew!!! time shaver, but still, the script it uses 'deeify' is still subject to the gcc -E problem, since it still employs it.

My third iteration, involved:

1. Using h2d's 'deeify' to run convert the .h file to a .d file.
2. Visually inspecting the .h file and copying over all the #define constants that have been left out by 'deeify'
3. convert the copied over #defines to either const declarations or enums (this part is still hard boring work)

Using this, i quickly went thru Xmd.h and X.d again, and they are ready in their first version. The last is Xlib.d, which has quite a number of boring constants to work on. I'm going to probably whip up a script to do those. But now that everything is *really* clear, i'm making progress.

Lazzyness is not a bad habit! Slothfullness is :)

2005-07-22T11:59:00.000+01:00

So what’s the difference?

Hehe… in my world, I classify Laziness as the unwillingness to do more work than you feel you need to do. Slothfulness, in this same world of mine, is the unwillingness to do work that must be done.

You still don’t see the difference? Lemme explain. I’m slothfull about my Laundry (trust me… you don’t want to stick your nose near my laundry bad :) ), but I’m totally Lazy about Programming.

Porting XLib to D, is a case in point. From my limited understanding of the conversion process, I jumped in, took one look at Xlib.h, saw that it included X.h, and Xmd.h, and I decided to start by converting X.h to X.d

All well, and good, till I started doing the actual conversion. I started seriously replacing complex #defines, #ifndefs, etc, with version() statements, etc, and got stomped on a number of them, put TODO: markers on them for a next iteration, and tried to move as fast as I could. In the first two days, I completely had X.h converted to X.d (a whooping 3000+ line file!!!). That proves I’m not slothfull by my definition.

Well, next was Xmd.h to Xmd.d, and I started… full of energy and enthusiasm… and then I realized… heck… at this rate… I’ll NEVER finish this stuff!!! How did I come to that realization? Well, all this files are VERY LARGE. Then my lazy side kicked it… my hands just REFUSED to move on the keyboard again, and I started thinking… there MUST be something I’m not doing correctly. I considered writing a bash or python script to take the drudgery away, but then there were still so many special cases, and my RegEx is not FBI class RegEx (whatever FBI class RegEx is :) ).

It was then I started looking closely at the docs and found that there was a recommended step I was indeed missing out.

There is a recommendation to run the Preprocessor before conversion actually starts.. hmm… how did I ever miss that? Will it simplify my work? I went to find out.

I ran gcc –E on X.h and OMG!!! The result was less than 20 lines of code… I immediately began to suspect the process.
I repeated this on Xmd.h and DAMN! The same thing… all the code was disappearing, and some files like Xosdefs.h turned up totally empty. I popped into irc, asked around a bit, and someone said I may have empty files if I’m not referencing anything from those files.

Keeping that in mind, I reapeted that on Xlib.h, which pulled in code from X.h and Xmd.h, and I compared the sections it pulled in, and they were the same as the ones I got when I ran gcc –E on the independent files. Also, Xlib.h wasn’t empty like the other files, but it was just left with struct defs and typedefs…

Wow!!! That is a whole lot less work than my previous approach. So I’ve already clobbered the old sources, and started again. I’m almost certain I may start afresh at least once more as things get clearer, but it’s a whole lot more clearer. What I want to do is to download digital mars C compiler and use that to run the Preproccessor stage and compare the two outputs.

Currently my work is cut out for me, I’ve written a quick and dirty bash script, that does most of the typedef replacements that are obvious, and the type mappings (unsigned long long -> ulong for instance).

I’m sure I’ll be able to test my example code this weekend, so I’m still shooting for that.

Back to laziness and sloth. I was lazy about continuing, but not slothfull about doing the necessary research. Also, my continuing along with my HARD WORK approach, will seem to prove that I’m not lazy and not slothfull. The truth is, once I refuse to AT LEAST CONFIRM a suspicion, then I’m slothfull. This is probably one of the key factors that separate smart hackers from the regular drone next desk :).

Maybe I’m not your uberKewl smart hacker, but hey…I’m MY UberKewl smart hacker :)

Essien… Out.

XLib D port, Jython BlackBerry apps

2005-07-21T09:43:00.000+01:00

Progress on porting XLib to D, took a minor step forward yesternite.

I've completed X.h -> X.d conversion, and dmd reports no errors. That's a little victory. Now, X.h actually includes Xmd.h, which has some *wicked* preprocessor defs, like:

#if defined (_LP64) || defined(__alpha) || defined(__alpha__) || defined(__ia64__) || defined(ia64) || defined(__sparc64__) || defined(__s390x__) || (defined(__hppa__) && defined(__LP64__)) || defined(__amd64__) || defined(amd64) || defined(__powerpc64__) || (defined(sgi) && (_MIPS_SZLONG == 64))
#define LONG64 /* 32/64-bit architecture */
#endif

My translation to D, looks so far like this:

version( _LP64 || __alpha || __alpha__ || __ia64__ || ia64 __sparc64__ || __s390x__ || __hppa__ || __LP64__ __amd64__ || amd64 || __powerpc64__ ) {

version = LONG64;
}

Ofcourse, you'll note that i don't have any representation for the compound && that is or'd into that expression..:

|| (defined(sgi) && (_MIPS_SZLONG == 64)

I'm not sure how to add that to the version() construct. What is really stomping me, is how to cater for the _MIPS_SZLONG == 64, since i can't specify expressions in a version() conditional. Hmmm....

Anyone with any ideas? I'm still thinking about this. I'll continue tonite, and i'll move on to other parts... everything doesn't have to be picture perfect for a first release, which will happen when i'm able to run the tutorial at:

http://tronche.com/gui/x/xlib-tutorial

I won't need those defs to get it work on my box, which is a humble AMD Anthlon64 :), but we'll need them eventually if we're really serious about portability.

Btw, yesterday, i was also trying to get a Jython BlackBerry sample to build, and suprisingly, i got quite far, only to be stomped by a preverification error. java.lang.NoClassDefFoundError. I used jythonc to build a .java file out of my .py program, and proceeded to use rapc.exe (RIM's custom tool for creating .cod files, needed by the BlackBerry).

rapx.exe actually calls another program preverify.exe, and then calls a java compiler, then does some funky stuff. My problem, happened at the preverification stage. it was preverifying the jython.jar archive and borked. The good news is, i think its just a path thing i have to sort out, as when i also tried to preverify net_rim_api.jar, without using the IDE, it also borked on the same error.

I'll give this another shot sometime tonite. If this woiks, this WILL be SOOOOOOO cool... my next step will be generic MIDP and CLDC apps in jython! WHOOOOO!!!

Ok... Essien... out.

Revisiting D

2005-07-20T12:42:00.000+01:00

After not doing active D coding in a while, I’ve been revisiting it gently in the past weeks. What slowed me of initially was there was nothing I was doing with it productively.

For my scripting and generic one-off projects that I see no reason why I need to compile, I use the venerable Python, and at work, I work with C#. And back at the shell, I still use serious Bash scripting. So apart from the proof of concept D sprint that I set for myself, I really didn’t have any project to use it for.

Well, it seems that is going to change. Someone on (Dejan@irc.freenode.net#d), suggested the creator of DFL (Vathix@irc.freenode.net#d), do a Linux/Unix port of DFL, utilizing Xlib. So, I’ve jumped the opportunity to put my D to the test.

What we’ll be doing is basically porting XLib to D so D GUI toolkits like DFL, can be built easily on Linux.

The truth is, what we’re doing (thanks to D being a practical and realistic language), is not as hard as its sounding… infact, all we have to do, first convert all the necessary XLib header files to d modules, then we can work on making the D modules more ‘D-Like’, adding object oriented paradigms etc, but the first part, is actually boring translation if you may J

I started somestuff last nite… and my first target is to be able to import X11.Xlib (analogous to #include <X11/Xlib.h>), for this, I have a whole lot of modules to convert. I’ve moved part of X.h to X.d, I’ll continue tonite.. and hopefully in two nites time, I should be able to write a basic XLib app in D.

In other news, as I started working on this yesternite, it became clearer how I would approach porting the EFL to D. Infact, I don’t even need the whole EFL right now, I just want to make the test module that comes with e17, be writable in D, then I’ll go from there. I’m not going to code that yet, I’ll wait for things to become clearer from the XLib effort, then I’ll let my inner-coding-monkey™ get to work on that J

Essien out.

Kernel Issues, e17 autobuild script

2005-07-19T11:45:00.000+01:00

I just solved a very frustrating issue that showed up with my Kernel.

First of all, my laptop (a Compaq R4000), has issues with the default installed ArchLinux Kernel 2.6 (I’ve not tried 2.4).

I had to build a kernel during the install to get my Keyboard to work – yes, keyboard… I have no idea what happened. I’ve even tried doing a visual examination of the diff b/w the two kernel .config’s but I came up with no clue… anyway… at least it worked, with one funny glitch: My time was really accelerating, so when I ran that kernel for a while, and rebooted into windows, I’d have kerberous issues b/cos of time sync.

Last week, I decided to solve that problem once for all, by repeating a compile from my FailSafe kernel config. In doing that… I felt I should do some more optimizations and playing around, and I disabled PNP_BIOS support, and left only PNP_ACPI. Well… this cause me untold headaches, as my Network card stopped working… the driver would load properly, but I’d just get an SIOCADDRT: Network Unreachable.

Very frustrating… I searched high and low, and had no clue what was causing it, until I ran across some Redhat user threads, and someone mentioned IRQ issues that had given him similar errors, then I closely looked at my dmesg, and noticed a complain about IRQ handling with ACPI and some drivers that may still be using the old method. That was when I recalled my visual diffs of the Arch kernel and my own config, that PNP_BIOS was OFF… well… long story short… I enabled it back and its working again… and I’ve learnt something new!

On other fronts, I’d been using Rensel’s Repository to feed by e17 needs on ArchLinux for a while now, but after noticing that he only seems to build against Rasterman’s releases on freedesktop.org, I decided to roll mine (I’d tried this before and had serious compile issues).

Anyway, this time, I was prepared, and after getting from CVS, I built the eet, edb, evas and ecore, they all worked, tho I had an eye opening problem with ecore finding evas, and I finally mastered the use of the ‘—with-bleh-config=/path/to/bleh-config’ argument to ./configure.

Now I have a tried and tested autobuild script for e17, that builds using ‘pacman’, and installs to my system. I did most of the work over the weekend, but still had some issues with some packages finding Ewl headers, and my result from ‘/usr/bin/ewl-config’ was really messed up. Yesternite, I stayed up till 4.00am, finetuning that script, and currently… its in its simplest form, with a library, and two actually builder scripts. And to test stuff, I uninstalled my last successful build, setup the build process and went to sleep. When I woke up around 7.30, I just logged in and typed ‘startx’… and yup! It works beautifully.

I also managed to write my first e17 module today (actually I just modified the snow module, using the test module as a guide :) ), at least it works.

So now, e17 is back up to current spec, and Dapo wants it tonite too :).

Ahh… I’m thinking of modifying this script to work with Autopackage (http://www.autopackage.org)

Anyways, I guess I have my work cut out for me.

Essien, out.

We've caught a virus

2005-07-15T13:12:00.000+01:00

At my office.. we’ve caught a virus!

I swear I had nothing to do with it…

But currently… everyone is talking Python, Linux, Open Source (when I mentioned this to Stan, we agreed to keep our big mouths shut, and see where all this was leading to)

I mean, core Windows guys who would not even look at Open Source stuff, suddenly going out of their way to learn Linux… and not just any distro, ArchLinux (http://www.archlinux.org).

You probably wont guess what their default window Manager is: e17 (infact, Dapo has asked me to build Arch Packages of the lattest e17 from cvs… wow! These guys are really demanding ;) )

Uhh… and yup… they’re already trying Mono, MonoDevelop and are seriously excited… infact… Dapo said, “The world is getting interesting”, this was when I told him about WINE (http://www.winehq.org), well… I just told him, “Its always been interesting… you just didn’t know about it then”.

Any way, did I say I had nothing to do with it? I guess I lied… I’ve been doing passive evangelism…. The sort that Microsoft will use when they release Longhorn – They’ll force you to use it… force you nicely J

Well, I released PMUtils today to the rest of the guys in the office, made it look VERY COOL J, and it uses the lattest libpm (python version), I’m too lazy to update the C# version (uhhh… this means… I just don’t want to do it :D), this means that to have publishing features for myPM, you HAVE TO install python, elementtree, and my libraries, libpm and systray. To make things cleaner, I’ve used inno setup, to pack these nicely.

So now, everyone at Splashers Technologies (http://www.splasherstech.com) uses a Python program everyday ;)

That is VIRAL Marketting. And we’ve caught the Linux/Python/OpenSource virus :)

I learn from the best.

Eclipse Help Plugin for D

2005-07-11T19:28:00.000+01:00

I went playing with Eclipse this weekend (again! :) ).

Anyways, I don't see the reason of using Eclipse and being a developer if one doesn't know how to create a plugin. But then again, one can't just go about creating plugins just for the sake of plugins.

Any way, i've been having his 'itch' recently. My most used eclipse plugin PyDev, doesn't have all documentation that comes for instance with PythonWin or the Standard IDLE. Also, the eclipse plugins for the 'D' programming language doesn't have any documentation/reference.

I went to work, to see how easy it would be to extend the Eclipse help system, and the screeny above is the result of that effort, for the D programming Language.

To be honest, i really can't say i totally grok the stuff, but at least i have the idea. I've just downloaded the HTML docs for Python, and will over the couple of days, try to work them in.

I'm also going to clean up the D docs and contribute them to the eclipseD guys.

Am i having fun? You bet...

Essien out.

Java on LInux for Client, Inno Setup

2005-07-07T19:54:00.000+01:00

Finally Dapo has figured out all he needs to run a Java TTS (Text To Speech) App on Linux, and he's set for his Demo in Lagos this weekend.

I'm finishing up his Linux setup, and his environment is:

Archlinux/e17 (yuh baby.!)/Eclipse/Tomcat/MySQL/Java

(I'm wishing we could have added Jython to that too :) )

Kewl huh? you bet.

Also, I decided to master Inno Setup once for all, and I finally got it working, and have used it to package PMUtils 1.0. Its rather flexible and i prefer the setup package it produces to the default msi's we build with .NET... uugghh!!!

Anyways, i'm using the proper setup on my box now, am noticing some usability issues... i guess i'll hit guys with it tmrw.

Nokia Development with Eclipse

2005-07-07T12:44:00.000+01:00

I've been working on this gently ever since i finally got the hang of Eclipse. Last nite, i stayed up, read some pdfs on Programming style and stuff like that, and continued on trying to get this to work.

Basically, i learnt that the Nokia Development Environment could plug into Eclipse as well as other popular Java IDEs. I've downloaded and setup the Nokia Development Environment, and in the process, also Downloaded and setup the EclipseME plugin.

Now, i can create a Nokia MIDP Project (but i don't know where to go from there :) ) Also, I have options of creating a MIDlet suite, but when i try to do that, eclipse complains that there is no platform to build the MIDlet for. I'm still looking into this.

Anyway... It looks like i'm actually set for developing Nokia apps, and i have the 7210 SDK (which i can test with my phone :) ). What i need now, is a sort of wholesome primer... From looking at the docs that come with the SDK, its a lot like writing a BlackBerry App, i just need to know how to create the entry point.

I'll scrounge around some more this weekend, download lots of articles from Forum Nokia, and spend next week reading thru them... am sure i'll get it all working by next week... Ahh... i feel a whole new frontier coming for me.

By the way, i'm itching to get the new Nokia Internet Tablet, once its officially released. Its a pure Linux Based Beast... And from what i've seen via Google and various blogs... its going to be one fun hackable platform. Maybe i'll just get this instead of the Zaurus I was thinking off... hmm....

Python Speech API for Windows

2005-07-06T12:22:00.000+01:00

Dapo has been working on this Java app that should read text out of a webpage, and my! Has he had issues with Java!

He’s gone from issue to issue, and he wishes he could have done it all in C# / .NET (He’s really our local .NET guru J ).

Anyway, I started feeling slightly jealous, and set out to see how hard it would be to achieve in Python. I hit my repository – AKA – Google J, and typed in Python Speech API.

There were myriads of results. I picked the first one that happened to be a recipe from the Python Cookbook on ActiveState. : http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/93025

Well, the second link I checked out was a module called pyTTS, which is a thin layer atop Microsofts SAPI framework. : http://www.cs.unc.edu/~parente/tech/tr02.shtml

To say that I was shocked will be an understatement… this is what the code to read a line looks like with pyTTS:

#-----------start of code -------------#

import pyTTS

tts = pyTTS.Create()

tts.Speak(“Unbelievable right? But I’m speaking!”)

#------------uhh… end of code :D -------#

That’s it!!!

I called Dapo over, to look at the sample Python Code, and we all started laughing… I mean it was so simple it was absurd.

The point here, is not *JUST* that it is only three lines of code. The point is that I most python libraries feel the same way. They do most of the underground work for your Default Scenario (which you end up using 90% of the time), and if you still want to get raw access… be my guest… its always there. The CookBook recipe still shows that without pyTTS, I could still build something simple and very much Pythonic from scratch, in VERY LITTLE lines of code. Its definitely amazing to have this much power at your finger tips.

If there is one thing I’ve learnt from Python, it is the power in succinctness. IMOHO, any developer that is not harnessing this succinctness yet from any source, Perl, Python, Ruby, Groovy, etc, is really wasting valuable time on things that he/she doesn’t have to spend time on.

Now, this is not exactly running out of a web page (So its not exactly Dapo’s *Current* problem… but a Python Webapp, is same as a Python Desktop app, and I don’t have to battle Java Applets like he’s doing now… poor guy J )

Now, what’s the next cool thing I’ll do with this? Hehehe… I think ‘misctray’ is going to start reading my tutorials and reading materials for me J

More systray utils

2005-07-06T12:21:00.000+01:00

Yesterday, I decided I needed to clean up some of my remote pop3 mailboxes I’ve not checked in a while.

Not a problem, but I have a very slow link, and downloading 13.8Mb of mail is not something I’d like to wait after work to do. So well, I got home last nite, and welded together another systray based utility I named ‘misctray’. Currently, It has just 2 uses… check the number of mails on a particular account, and allow me to delete the mails without downloading them first (basically a looped ‘DELE’ J ).

Tested it out this morning, and boom! It works. ‘misctray’ has me thinking tho… I think I’d like a framework, where I can write arbitrary python scripts, drop them in a ‘mistray plugins’ folder, and misctray would pick them up automagically, and create a menu/menu-item out of them. That way, I can have one consistent launcher for my python scripts, right on my Windows System Tray.

Hmm…nice idea… I like it, I’ll do it.

PMTray 1.0 with some screenshots

2005-07-04T18:13:00.000+01:00

Well, well, well, sometimes we make plans and totally don't end up following them, because a better plan came up all of a sudden.

That's basically happened to me over the weekend.

Remember i had planned to start working on the now very much hypothetical and soon to be not so vapourware 'phiba' :). Well, i found another immediate need, PMTray. PMTray is my lattest addition to the PMUtils package i currently use to manage our time and interact with our web based time management tool.

I quickly put this together with the systray module (that dude has realy gotten usefull).

Using systray to solve a real problem also helped me to refactor it for better usability, and what i really need to add now before i can satisfactorily declare systray-1.0.0 is 1) A status message akin to Yahoo Messenger Notifications (when folks log on and off, but as beautifull as Outlooks Mail notifier messages.
2) Support of an OnApplicationLoad event, so i can run a full service off the systray.
3) Support for Notification Balloons (This is not really too important if i can achieve the first one. Infact, any of 1 or 3 will be cool for a version 1.0.0, and both WILL ROCK!)

The screenies sort of speak for them selves. (pmtray is the cool - yup, cool, green icon nearer the system clock)

1. pmtray is idle
2. pmtray - showing menu options
3. pmtray - synchronizing with WebPM

ok... gtg, i really have this pain in my neck... i think its a cold :(

systray sample app screenshots

2005-06-30T18:37:00.000+01:00

I decided to post some screenies of the sample app included in the systray package source.

1. The first shot identifies the sample app (its the 'S' icon).

2. The second shot showcases MenuOptions.

3. The third is me in the midst of selecting an option that will change the app icon.

4. The App icon has been changed (And the Hover Text as well). The current icon has a white 'S' the other one had a black 'S' with an outer glow.

Anyways... just a sample app. staty tuned... 'phiba' coming your way soon.

systray 0.2.1

2005-06-30T15:28:00.000+01:00

The topic says it all. And as usual... i had to stay up... this time i think i just stayed till 1.00am... i'm getting lazy i guess :)

I have built up systray-0.2.1, and its looking good.

Here's the changelog so far.

0.2.1
=============
1. Make Menu() and MenuItem() have default values for 'title' in constructor, so they can be blank.

0.2.0
=============
1. Add MenuItem.add_submenu
2. implement the _App class inheriting from SysTrayIcon.SysTrayIcon
3. Added text and icon property to _App class which call refresh_icon, automatically
4. Modify App.start() to use _App() instead of SysTrayIcon.SysTrayIcon
5. Change App.start() to correctly set window_class_name of _App class
6. Refactor out MenuBase.
7. Reimplement MenuItem as a child of MenuBase, removing irrelevant attributes
8. Implement Menu as a chile of MenuBase, seperate from MenuItem.
9. Extend App to support both App.add_menu() and App.add_menuitem(), for unambigous usability
10. Modify App to have a default on_quit handler that does nothing, so it now always exits properly.
11. Rename default_onclick(s) to default_handler(s).
12. Create a module object DEFAULT_HANDLER to be the default_handler accessor object
13. Make App and MenuBase use DEFAULT_HANDLER as their default handlers for on_quit() and onclick() respecively
14. Added doc strings for _App and App, MenuBase, Menu, MenuItem.
15. Extend Menu to support SubMenus and MenuItems explicitly
16. Incoporate distutils framework for installation.

0.1.0
=============
1. Initial Release
2. Create systray module
3. Build App class
4. Build MenuItem class

The source comes with a sample application, here is the code:

#--------- systray sample ---------------
from systray import *
import time

mitem_txtchg = systray.MenuItem('Change Text')

def txtchg_onclick(s):
s.text = 'systray sample - ' + str(time.clock())

mitem_txtchg.onclick = txtchg_onclick

menuitem_published = systray.MenuItem()
menuitem_published.icon = 'published.ico'

def menuitem_published_onclick(s):
s.icon = 'published.ico'

menuitem_published.onclick = menuitem_published_onclick

menuitem_default = systray.MenuItem()
menuitem_default.icon = 'default.ico'

def menuitem_default_onclick(s):
s.icon = 'default.ico'

menuitem_default.onclick = menuitem_default_onclick

menu_icon = systray.Menu('Icon')
menu_icon.add_menuitem(menuitem_published)
menu_icon.add_menuitem(menuitem_default)

st_sample = systray.App('systray sample', 'default.ico')

st_sample.add_menuitem(mitem_txtchg)
st_sample.add_menu(menu_icon)

st_sample.start()

#--------------------------------------------------------#

at this stage, i think i've side tracked enough from 'phiba'. What is phiba? phiba was why i got looking for the SysTrayIcon.py module in the first place... its supposed to be a home brewed blogger butt... ooopss.. blogger bot :)

I'll start work on this sometime tonite i guess... i have the blogger API docs, and now i have 'systray'. I think that was the last missing piece. i think i should have something exciting soon :)

Eclipse, Python at Work and uhhh.. EMR

2005-06-29T21:04:00.000+01:00

Today, I 'officially' had reason to work with Eclipse, PyDev and Python. :)

We were modifying the Hospice Backend, so i had to redo the import scripts which were crafted in Python. I had done this previously using PythonWin IDE, but since i've fallen inlove with Eclipse (yeah, now i admit it is cool, it has a rather high coolness factor too ;) ), I Decided to import the project into Eclipse and continue from there.

Yup... It turned out very painless and infact, very beneficial. I think that's it, Eclipse/PyDev has become my Python development Environment of choice. Now all that needs to be added is a way to access PyDoc easily, probably integration of all the docs that come with PythonWin into the Eclipse Help system, and I guess I won't open any other python Python IDE/IDLE/SHELL again ;)

woooo... exciting times.

Btw, today, I officially joined our EMR team. Phew! Everyone dreads that project, b/cos its so big, somewhat badly designed... and fixes keep coming in everyday.

Well today, I worked with Seun and did some fixes. One of the Refactorings I added today, was for implementing Warnings before closing forms with Changed data. I took one look at the stuff and told Seun... man... we can't be copying and pasting... Immedietly I went to work on an interface ISaveClose, and we integrated it first into the New Allergies form and the New Medications form. It works very nicely. Tmrw, we'll make all the other forms that need to do that, use it.

I'll actually like to go one step further, build a class that implements that inteface already, and all the what nots, so by making each of the forms just inherit from that class, we will be good to go.

Why this was never done before? hehe... that's the million dollar question :)

Boy... my back aches... my tummy rumbles... i really gtg.

Python and the Windows System Tray

2005-06-29T09:27:00.000+01:00

Continuing with my exploration of small neat and usefull python hacks for windows, I went searching the web for how to use python to write an app that would run on the Windows System Tray.

Shift over... its Googlistic All The Way :)

I came across this from Simon Brunning : http://www.brunningonline.net/simon/blog/archives/SysTrayIcon.py.html

Quickly downloaded it, and yesternite (yup... i'm becoming a Pythonire - that's a poor vampire spin off :D ), I took it for a Test Drive.

After a quick understanding of how it works, I tried to write a small app, and in the process I quickly refactored out a small library, that makes using it a bit more Intuitive to me.

The module which i dubbed 'systray' looks like this:

import SysTrayIcon

def default_onclick(s):
return

class SysTrayApp:
def __init__(self, title, icon=None):
self.icon = icon
self.title = title
self.menu = []

def start(self):
SysTrayIcon.SysTrayIcon(self.icon, self.title, tuple(self.menu) ,default_menu_index=1)
return

def add_menuitem(self, m):
self.menu.append(tuple(m))
#print tuple(self.menu)
return

class MenuItem(list):
def __init__(self, title):
list.__init__(self)
self.append(title)
self.append(None)
self.append(default_onclick)

def __set_icon(self, icon):
self[1] = icon

def __set_onclick(self, handler):
self[2] = handler

icon = property(None, __set_icon, None, "Icon Property")
onclick = property(None, __set_onclick, None, "OnClick Event Handler")

Easily use this module, you can do the following:

#-------------------------------

import systray

mitem_sample = systray.MenuItem('Sample Menu')

st_sample = systray.SysTrayApp('systray sample', 'default.ico')
st_sample.add_menuitem(mitem_sample)
st_sample.start()

#--------------------------------------

To add icons and menu actions to those menu items, modify the code as follows:

import systray

mitem_sample = systray.MenuItem('Sample Menu')
mitem_sample.icon = 'default.ico'

def sample_onclick(systrayicon):
systrayicon.title = "I have been clicked"
return

mitem_sample.onclick = sample_onclick

st_sample = systray.SysTrayApp('systray sample', 'default.ico')
st_sample.add_menuitem(mitem_sample)
st_sample.start()

#---------------------------------------------

Well, that's it.

I want to use this for some nifty utils, like a network monitoring daemon, a monitor for pmserv, etc.

hmm... fun times ahead i say.

GNU GOATS on JAVA

2005-06-28T18:19:00.000+01:00

Like i said earlier, these are exciting times. And if i hear anymore person say that lack of patents stiffles creativity, i'll personally truss em!

After crying about the Java Trap (http://www.gnu.org/philosophy/java-trap.html), the GNU community has been responding on many fronts, as a lot of Free software these days are written in Java. The result is GCJ and libgcj (which comes with GCC 4.0).

If you're a bit used to UNIXy nomenclature, libgcj may strike you as odd, b/cos its normally c/c++ libraries that are named like that, since they are compiled to native code. Well... GCJ can compile Java code to native code... Yes sir, you heard me well. In this manner, Java programs can run REALLY FAST.

libgcj is then the implementation of the Java library, and is currently a result of two GNU projects collaborating well (the original libgcj guys and the GNU Classpath hackers).

What does this mean for me as a coder, well, i now have a standard free platform to write Java code (uhh... i'm not exactly a Java freak tho). But most important to me, is that i can compile OpenOffice and Eclipse and run them as native binaries on Linux (and to some extent on windows using Ming). Now that is what i call innovation. You would wonder why Sun didn't do this? Or why .NET uses the same principle as Java (no native code). Well, they all tell you its compile once run everywhere, but in practice we know its not usually that straight forward, and its more true with Java than with .NET anyway, so why bother?

I personally don't know, and right now, i don't care :) Once I can run my Java programs that i NEED on my preferred platform, at native speeds, the bigwigs can go do what ever they like.

Business as we know it is changing fast, and innovations like this are the reason. Like I said earlier, these are exciting times.

Eclipse Weekend, pmutils - introducing pmserv

2005-06-28T17:58:00.001+01:00

Last weekend, i decided to really take Eclipse for a drive. And i have one word. It is USEFULL. Note i didn't say cool. I'm yet to feel the coolness factor, though one plugin nearly got me there (the vim plugin - yeah... i'm a nerd so what? :) )

What I did was to install eclipse and setup the lattest version of PyDev, and then migrated my development of libpm and pmutils, to Eclipse.

My first really tripping moment was the Refactoring Functionality. It is really cool. What I ended up doing mostly is, all the places i know i want to factorize out a function, I use the Refactoring -> Extract Method. Then examine the resulting function to see if its good enough (sometimes, there are unnesseccarry arguments, etc). But, its really a time-shaver :)

I firstly cleaned up libpm and pmutils somemore, dealing with most of my todos, then started facing pmserv, which was intended as a windows service. i had layed out a basic service framework, but there was still work to be done, methods to extract from the existing pmutils, etc.

I did all of this in Eclipse, and last nite, i stayed up till 2.30am, to finish pmserv, and it works beautifully.

One of the things that really tripped me with eclipse was the External Tools feature. I added the python console (ofcourse), and a windows command prompt, which i used while developing the service (installing, starting, stoping). I really needed to include the windows Service Manager and Event Viewer too, and a PyDoc viewer, but didn't get around to doing those... I guess there's always latter, as i'm planning to write a blogging service too :)

All in all, i'll say, Eclipse is a really serious utility, and this is another boon for opensource development, and the ubiquitious J. Random Hacker. Thanks to tools like eclipse and all the budding tool smiths out there, anyone can start doing very usefull stuff, without paying a dime, and what's more... this things are actually industry standard.

Exciting Times are ahead i tell you, i'm really happy to be alive and coding in this era.

Eclipse Weekend, pmutils - introducing pmserv

2005-06-28T17:58:00.000+01:00

Last weekend, i decided to really take Eclipse for a drive. And i have one word. It is USEFULL. Note i didn't say cool. I'm yet to feel the coolness factor, though one plugin nearly got me there (the vim plugin - yeah... i'm a nerd so what? :) )

What I did was to install eclipse and setup the lattest version of PyDev, and then migrated my development of libpm and pmutils, to Eclipse.

My first really tripping moment was the Refactoring Functionality. It is really cool. What I ended up doing mostly is, all the places i know i want to factorize out a function, I use the Refactoring -> Extract Method. Then examine the resulting function to see if its good enough (sometimes, there are unnesseccarry arguments, etc). But, its really a time-shaver :)

I firstly cleaned up libpm and pmutils somemore, dealing with most of my todos, then started facing pmserv, which was intended as a windows service. i had layed out a basic service framework, but there was still work to be done, methods to extract from the existing pmutils, etc.

I did all of this in Eclipse, and last nite, i stayed up till 2.30am, to finish pmserv, and it works beautifully.

One of the things that really tripped me with eclipse was the External Tools feature. I added the python console (ofcourse), and a windows command prompt, which i used in developing the service. I really needed to include the windows Service Manager and Event Viewer too, but didn't get around to doing those... I guess there's always latter, as i'm planning to write a blogging service too :)

All in all, i'll say, Eclipse is a really serious utility, and this is another boon for opensource development, and the ubiquitious J. Random Hacker. Thanks to tools like eclipse and all the budding tool smiths out there, anyone can start doing very usefull stuff, without paying a dime, and what's more... this things are actually industry standard.

Exciting Times are ahead i tell you, i'm really happy to be alive and coding in this era.

Solution Layering

2005-06-25T13:13:00.000+01:00

How many times have you actually felt that you were doing too much work while programming?

Hmmm... ok... i know we all feel overworked, and most of us derive a kind of sadistic pleasure just from the feeling of overworked pressurized must-meet-dealine kinda crap :) but sometimes, its plain unneccessary.

Actually, as you grow as a coder, you begin to relish 'elegant' solutions, that not just showcase how smart you are, but also reduce the number of keystrokes you hit and stuff like that (at least, i crave for that elegance in my solutions). What keeps hitting me everytime, is the importance of layering as an engineering art, and the importance of recognizing funcionality meant for those layers and keeping them there.

I'll give a couple of examples.

First of all, imagine you have a database table with two fields [id, name]. You also have this program that is putting data into the database. Supposing you're reading the data you have, from a couple of CSV files, collected from different secretaries in the same company. As you're about to start running your import script, you're suddenly alerted that there could be duplicate information in those CSV's and you have to ensure it doesn't mess up the database. What do you do?

I'm not sure what the no brainer answer is for a lot of ppl, but lets look at the alternatives.

1. Modify your script to (a) Hold the names in memory as it is reading, and compare new names with those already in memory to ensure duplicates don't occur (DONT LAUGH :) )

2. Before each insert, try to retrieve the same name from the database, just to see if its already there. If its not there (its UNIQUE), then insert.

3. Just make the stupid field in the database UNIQUE and don't touch you script at all.

Looking back at the three alternatives now, its almost a no-brainer see which is the most trivial and probably better (at least for me the programmer) solution.

Let's look at this solutions again before I finalize my point.

Solution 1.
===========

In solution 1, we were trying to do EVERYTHING in our code. Without basically, if we like it or not, or we realize it or not, we're going to need to replicate some functionality of a database (in memory data store). Once we commit to that solution, we're introducing a LOT of possible bugs, since our once simple import script will become an import script with an Embedded Database (albeit a poor man's database), and of course since its a poor man's database, our program and we the programmer will suffer from that decision. What will probably save us from this condition is if we realize this, and remember that usually, our programs should STRIVE to DO ONE THING, and DO IT WELL. Also, if there is ANOTHER program/library, that DOES WHAT WE WANT DONE, BETTER than we do it, WE OWE OURSELVES to TRY TO USE IT. Enough said. (Did i hear someone say Opensource?)

Solution 2
===========

In solution two, the change to our program is much simpler, because we're exploiting a property of our Database Server (querying for prior existence), before we insert. We can see that once we involve the Database System in anyway, our headaches automagically reduce to a very negligible minimum. This is almost acceptable, and in some cases of software layering is very ok. At least, we've pushed the real data manipulation to the system that was designed for data manipulation... the database itself.

Solution 3
============

The only reason why solution 2 is not the best solution is because Solution 3 is possible. Probably not in all cases, as there maybe some stripped down version of an embedded database to be used in a low resource application that will not have the ability we exploit in Solution 3, but as long as it exists, it is a preferred solution. Here, we totally shift EVERYTHING to the database layer. This is true layering. And our first advantage is ultimate simplicity... we DONT ALTER OUR CODE AT ALL. Now, that is a benefit i'm proud of. On the other hand, we have to be aware that this can be done, else we can't benefit from this. This brings to mind one of the nuggets i came across once about being a good programmer. It said to LEARN ABOUT THE ADVANCED FEATURES OF YOUR LANGUAGE(s). The idea being that most of those advanced features are easier ways of solving very real problems.

Now for a quick summary.

Its always beneficial for us everyone if we can properly design our solution systems to exploit designed-for strengths of various components of our systems. Infact, if we don't, we almost invariably build weaker systems. This is why working harder is not always a good thing, but working smarter usually translates as a problem solver.

Now, at times, it is not obvious that there is a better layer to handle your problem, so what do you do? Well, i usually think about my implementations, and everytime i find my self doing too much work, it *smells* wrong. I may not know the best solution immedietly, but i do leave TODO's and other kinds of hints in my code, so if a wiser me or a wiser someone else comes along, that code can be improved latter. Its usually better to roll-out a quick dirty solution within your time constraint than a better solution out of budget time. Your programming buddies may hail you for that, but your boss will chew you out, and you may have cost your company an actual job.

The key is to COMMIT TO IMPROVEMENT. If you identify a problem that you don't have the mental resources to solve now, not it down, and solve it latter. If on the other hand, you CAN solve it now, please do, or your head WILL Fly :)

Ok... now, repeat after me... "I am done reading this blog entry for today..."

More Python and Linux at work

2005-06-25T12:33:00.000+01:00

Recently, i've been having more and more opportunity to use Python at work... mostly for scripting a lot of stuff for myself, as the needs were arising. Especially data analysis.

Well, i had another more "official" data transformation or more accurately data migration project recently. I had to migrate a very flat file database built in Access, to a more Relational Model solution, still built in Access. And it was to be a quick job. So here I was, itching to use Python (i mean... there was no way i was going to be doing that in C#... just plain old forget it... its too complicated and error prone.)

First thing i did to assuage my concience was to state the problem aloud, and ask if anyone had ANY MEANS that would solve this problem elegantly without my use of Python (Everyone knows Stan and I love Python), so well, we brainstormed for sometime, and honestly, there was no simpler faster solution (I knew that already, but i just wanted to be sure i wasn't being merely self-serving).

Long story short... I did the project in Python. To make things even more interesting, Dan Lash (one of the new consultants at Artemis, and most importantly the consultant on this project), is also a Pythonista, needless to say, we had real fun on this project, and NO BUGS, just suggested implementation changes. Nice wrap up.

On the case for Linux, i'm still keeping this underwraps a bit, but we've been pursuing a lot of OpenSource solutions for some new projects coming down the line, of which i can't really say much of at the moment. But I'm currently configuring E17 (Yup.. The new Enlightenment), for Dapo. The last week, we were setting up Eclipse/Java/Tomcat/MySql, the whole works on Linux and stuff like that...

Pretty interesting and cool. I'm getting to do more of what i really love i guess... could still be much more coming the next few weeks too.

Truce indeed.. who am i decieving?

2005-06-22T16:41:00.000+01:00

ahh... i really don't feel too good.

i'm afraid i'll have to take at least one day off... arrgghh! There goes my stainless record.. anyways... my good friend Ugo put me in perspective, when he told me to stop playing Superman, and just go lie down and get better...

i think he's right...

Btw, just some random updates... i have been doing a full recon of website authoring recently... its been a while since i did that. And i'd been really growing tired of Table Kung-Fu (as Ryan neatly described that art!).

Anyways, DIVS, CSS, Ajax, the whole thing... i'm even thinking of picking up Ruby On Rails... ahh... anyways... this malaria fever thing is slowing me down... this is just why i hate to turn in sick :(

Bahh... i better just sign off now...

Malaria, Truce and libnetprog (What the heck!!!)

2005-06-21T13:22:00.000+01:00

i'm feeling a bit under the weather...

Its like i have this running bet with Malaria, and every year, it tries to get me down at least once... there were times when i used to be gotten down... but these days... we sort of lock into a stale mate.

I keep going to work... being slightly affected with a nagging headache, and probably teary eyes that hurt. On my part.. i increase the intensity of my exercises, take more Vitamin C, drink lots of water... and work harder during the day than before.

This truce of convenience lasts for about 3 or 4 days, and eventually i'm back to normal. This is waaaaay much better than being enBEDDED for about 1wk. #:-s

Anyways... i'm in the middle of one now... started yesterday, and i still worked till 10.30pm yesterday, on one of those small and interesting utility projects.

In finishing that project, i had to dig up a library i had started work on a while back, and clean it up, extend it, and use it. I call it 'libnetprog', and was my way of 'pythonizing' or 'c-izing' .NET network programming. I TOTALLY HATE STREAMS!!! why Java, and .NET decide to NOT encapsulate if i will never know. Anyways..., libnetprog.NSTcpClient() [No Streams TcpClient()] is an encapsulation of System.Net.Sockets.TcpClient() but adds Send() and Recv() that allows more natural communication by abstracting the NetworkStream away from the user. That and the conversion that goes on with System.Text.Encoding.ASCII

libnetprog.SMTPClient() is another cool class. And it has a very usefull static function SendMail(), that is inspired by PHP's mail() function.

Anyways... lots of UnitTests in that library, and its looking good and stable. Its a 'driven-by-need' library, which i'm only extending as the need arises... i guess i should put this up on my website download area.

Ahh.... that reminds me... i've stumbled across a very nice way of layering GUI applications... its just a natural extension of Refactoring and Bottoms Up programming (Aggressive Unix Programming), applied to GUI's, and so far, all the stuff i'm building i'm building like that.

I *think* they're very stable (since i can trivially UnitTest the GUI using this method), and i end up having reusable components too... i'll blog about that latter... let me get the entire picture very clear :)

btw, good to be back.

Alive and Kicking

2005-06-13T21:03:00.000+01:00

just a short note... i'm still alive and kicking... or well.. coding :)

fast blog updates coming soon.

BOLIVIA!!!

2005-05-09T09:35:00.000+01:00

WHOOOOO!!!

Bolivia just came in this morning :)

She's my new Laptop. A sexy Compaq R4000.

Right now, its running Windows Only..... Uggh!! I'll fix that over the weekend. Put Archlinux + Enlightenment D17.

I can hardly wait to see Enlightenment on this baby.... un... one... two... UNO!!

Right now, i'm paying my last rights to Wendy (My Older Laptop a buxom Dell Inspiron 2500), She served me well too, and stuck with me thru my earliest of Open Source Projects (ALE, PTSP and currently imlib2 python bindings and edje python).

She's done good, but i'm eager to put Bolivia to the test... with a sweet 15.4" widescreen, coding has never been this good.

Ahhh...

Ok ok ok... let me get back to work now... this week is going to be full of collaborations btw Wendy and Bolivia as I slowly romance Bolivia :D

ok ok ok....

I'm off now :)

Ohh... and yeah... this post is made from Bolivia :)

XSL Transformations AT WORK - Literally

2005-05-06T12:07:00.000+01:00

During our ‘Geekify’ session on Wednesday, Stan took us on XSL transformations and it was like… WOW!!!!

You know… XML is cool and all, but probably XSL is really what makes XML sooo usefull. Its so interesting.

Also, just to satisfy the Geek Child in me, I designed an XML format and accompanying XSL stylesheet for our daily SCRUM updates. Those where beginning to look messy, and I’d had in mind to write a Python frontend, to generate the report, but once Stan did the XSL tutorials, I was like… HEY!!! That’s what I need. Akin is currently using the XML raw, but he’s interested in Building a GUI frontend, to generate the XML for him. I mean, he takes notes at scrum, so he’s the one to make his life easier.

The other part of this is that, with all this ‘make_life_easier’ projects I’m introducing at the office, its interesting to see how other members of my team are reacting… it seems hacking is back in fashion at the office, and I’m beginning to notice it in how we’re working.

Ahh… the rise of the hackers :)

____________________________________________________________________________________

[x for x in range(0,200) if (lambda y: len([z for z in range(2,y-1) if y%z == 0]) == 0)(x)]

Python In The Office

2005-05-04T13:58:00.000+01:00

I’ve been finding ways to make my life easier, more interesting and generally more efficient.

Ok… that’s just an excuse to introduce my lattest work plaything… Python.

If you know me at all, you’ll know that I would love to work daily with Python, but I work in a C#/.NET outsourcing, so it’s sort of impossible right? I mean since IronPython is still in early alpha, there’s nothing I can actually do about that, right?

Well, I’d not been really using my brain earlier. Now that I am actually turning on my brain full force, by aggressively getting lazy about repetitive tasks in any incarnation, I’ve began to really see uses of Python all around what I do almost on a daily basis.

It started with writing libpm, which is a Python library to interface with a Webservice which we use to do some reporting at work. Well, my next focus was some really dreary analyses I’ve been doing on a database project. I run some huge queries, and produce files and I’m supposed to compare for accuracy and other stuff. Well, when I’m troubleshooting, I get to do a lot of other fancy stuff to those files, and I’d been doing all these visually all this while!!!

Last week, I had to get back to the project, as it has over-dragged behind other projects, and for the analyses, I said NO NO NO, I’m not doing this visually again. I mean, its not even efficient enough, so I dug up Python and started building up small functions and small programs. As a result of that I now have a little module called ‘kamba’ that does some really interesting text processing. The most exiting is a function align, that if given two lists in the form of:

[1,2,3,4,5,6,7,8,9,10,11]

[1,3,5,6,9,10]

Will return a third list which aligns the second one against the first one, like:

[1, ,3, ,5,6, , ,9,10, ]

This single functionality is a realllll time shaver :D, and combined with other stuff has helped me get more usefull results in my analyses.

I think I’ve discovered in doing this, that hackability of a platform depends on a lot of factors, but most importantly are the tools available and the mindset of the said hacker.

I really am not thrilled by Windows generally, but hey, Python on Windows is helping me get some of the fun I normally catch on UNIX.

The second thing is that flexible tools affect mindset. For instance, If I didn’t have Python available to me, there is no way I would have written those tools in a language like C# or similar. Why? Well… there’s a lot of inherent overhead in setting up the project, and just thinking thru annoying issues like type definitions and other things that are not directly related to the problem you’re trying to solve.

This is not exactly a bad thing considered on its own, I mean, there’s the argument that you should use another tool to do that kind of thing. Well… this brings me to my point. My point is that a lot of programmers these days (well, at least more than I would like to think about), don’t have any kind of dynamic programming tool at their disposal, because there’s this prevalent all or nothing mindset.

Those kind of people are more apt to resist programmatic changes. They don’t think in terms of scripts and quick solutions. They’re not used to making small programs that do only one thing and do it well, or really to writing small functions that do one thing and do it well.

Basically, by sticking to and working with non-malleable programming languages their mindset is also non-malleable. These kinds of programmers find it hardest to think Agile or think in the dynamic way needed to really drive software development these days.

Ofcourse you could say this is just my opinion, and I’d agree with you, but I’ll point out that I’ve met plenty of these kinds of programmers.

My recommendation is that every programmer, should learn more than one language. And these languages should be best spread along different programming paradigms. For instance, if someone learnt to program with Java, and has now learnt C#, yeah… that’s ok… but this person has NO IDEA whatsoever of how a program can be written without classes. This programmer also is more likely to have more complex approaches to problems than others that are exposed to say C, Ruby, Python, Perl, and shell scripting.

Anyways…. Work is currently more interesting.

pyimlib2

2005-05-04T13:50:00.000+01:00

Yesterday nite, at the perfect witching hour( 12.00 midnite), myself and stan started work on building a python extension module for imlib2.

Imlib2 is a very robust and interesting image manipulation library is part of the new generation Enligtenment Family – Enlightenment Foundation Library.

It was a very enlightening experience (pun intended – mfon, I just had to say that :P). Our approach which we’re trying to keep as Agile as possible so we see quick results, was to pickup the EFL Cook Book, and go thru the examples on imlib2, then for each example, port enough functionality to a python extension to allow the example to be done in Python totally.

As at yesternite, we have just one function left to complete example one.

So far we have:

imlib2.load()

imlib2.clone()

imlib2.save()

imlib2.get_size()

tonite, we’ll add imlib2.blend().

What we’ve been thinking is that it will be more natural to have an Imlib2Image object(), that will have all these functions as methods. For now, we’ve not totally decided how to implement this, but I don’t think we’ll build the object in C. its more likely that when we have the functions in python completely, then we’ll build a higher level Python Library abstracting that into an Object. That’s the more natural way that I’m thinking of it.

Ahh… this application layering thing is making lots of sense now.

____________________________________________________________________________________

[x for x in range(0,200) if (lambda y: len([z for z in range(2,y-1) if y%z == 0]) == 0)(x)]

The Force Is Strong With This One

2005-05-04T09:55:00.000+01:00

If you've not yet guessed, my favourite programming languages are C,Python and D. I have no real order there, as it depends on what i'm doing, but with those three languages i have a whole spectrum of problem areas covered.

Recently, i've been thinking of adding one more layer to projects i do in C. A python layer.

The thougt process follows from trying to make a system as programmable and explorable as possible, and to reduce the amount of time needed for a hacker to get to actually hacking the system. What trips me silly about python everytime, is the live intepreter you get with it, so one can interogate libraries while learning about them. You don't have to use the usuall - READ_FIRST->Code->Compile->PRAY_MURPHY_IS_BUSY->Execute cycle. You can actualy grab a library, and fireup the intepreter, print the __doc__ string, and start figuring out how to go, if you get stuck, google, then come back and continue.

To a large extent you can do this in other languages, but Python with its live intepreter takes it to a new level... trust me :)

Anyway, this is a model of programming in c i'm trying to formulate:

[Applications (C-Programs,shellscripts, GUIs,etc)]

| | |

| [c-command-line-utility][Python Layer]

| | |

[ c-library-layer ]

|

[ problem scope ]

At the bottom-most of this layer, is the problem we're trying to solve. Say for instance, manipulating Netfilter on LInux in userspace.

The first layer, is made up of a well designed C-Library (in this case, libipq), that properly abstracts every kind of interaction with the linux Netfilter layer. Once the library is in place, theoritically, the problem is solved, but normal users can't use it yet. Applications have to be written.

On unix, there is a common practice, that these libraries normally are accompanied by command-line utilities, that are almost like proof-of-concepts for the libraries. Some of these tools are so advanced that every single thing doable in the library is doable by the commandline utility with just an option switch and the proper argument. (I should state here that sometimes, probably most times i guess, these libraries are never solo projects, but actually arise while on the path to craft the command-line tools)

Traditionally, after the command-line tool layer is available, it qualifies as an Application, well..., as a System software. At this stage, system administrators can use shell scripting, perl or even python to drive the command-line utility in ways the creators never thought was possible. Finally, GUIs can be tacked on as a Front-end to many of these utilities, and we have a complete appliation stack.

The problem is that once in a while, there exists a feature that can not by its very nature be exposed by the command-line-utility. Apart from that, it may not be desirable for any reason to write a much higher level application in C. I personally prefer to leave that layer for languages like Python and D... Actually Python ;)

This is where the other layer, the Python Layer comes in. I placed it at the same layer as the command line utility, because it interacts with the c-library-layer for you, but unlike the command-line-utility, it can have access to every single functionality of the c-library, hence allowing even more to be done at a higher level in the stack, without having to fall back to raw C.

I'm currently literally putting my money where my mouth is, by building a python layer for 'scode' i call, pyscode. Scode is a personal project of mine (hmm... i should put it up on my website next weekend.)

I’ve also concluded a python extention to another library I have (a much smaller one I use for managing packed inet addresses and dotted notation), called ‘addrtool’.

so far, i'm having fun learning how to extend Python with C. It trully a path to increasing the Force withing you :D

Ok... so much for my ramblings.

____________________________________________________________________________________

[x for x in range(0,200) if (lambda y: len([z for z in range(2,y-1) if y%z == 0]) == 0)(x)]

Python Windows Services

2005-04-25T18:54:00.000+01:00

We had project last friday to write a windows service that would poll an Exchange mail box and write to a database. It was executed in C#. Well... i got thinking... hmmm... can't python be used to do this? Because if it can, then that project would not last more than a couple of hours (actualy, the C# project shouldn't also last more than a couple of hours if you know what you'r doing, but hey.. i have to have something to say about why i decided to do it in Python write? i mean.... I LOVE Python may not cut it for some folks :) )

OK... lets get to the meat of it.

I first hit google, looking for links to python and windows services. I then found some mailing list archives and in one of them, there was a reference to a chapter on Windows Services in the book, Programming Python on Win32.

Well... i had access to the book, and went headlong. Long and short of it, 15 mins of perusing latter, I had a windows service running. Lets attempt to get this stuff to work together now.

First there are two very important modules that helps our lives out. win32service and win32serviceutil

We're going to write a service that just writes to the event log every 10 seconds. U'll have to stop this service soon or else, you'll run out of EvenLog space ;)

First some important pre-parations and background on python-windows services. There is a program called pythonservice.exe, that actually handles everything that concerns windows services written in Python. I'm walking a thin rope here with this explanation now, but this is how i understand it, just don't quote me in a conference where there are other Pythonistas and Lords Of the Eggs, I'll deny it vehemently ;)

The way i figure it, pythonservice.exe is to python services what the usual python.exe is for all other normal python scripts. Normally on windows, once .py is associated with python.exe, each time you run a script, the python intepreter, python.exe is called to 'intepret' it. For services, apparently some funky stuff goes on, so instead of calling python.exe as the intepreter, pythonservice.exe is called as the intepreter (well, not exactly intepreter i guess, but it is the process that runs the services.) You can also look at it like this: You say service start, windows identifies it as a python service, and starts pythonservice.exe passing it parameters to find the service itself. Pythonservice.exe locates the service, and starts it. As far as windows is concerned, it is running a process called PythonService.exe (You'll see that without special tricks, when writting to eventlog, PythonService is the one that does the writing)

Now the preceeding means that windows has to know to associate python services with pythonservice.exe . This is essentially called registeration. So pythonservice.exe must be registered with windows to handle python windows services. to do that, locate where the python win32 extensions are on your box. They'll probably be in your site-packages folder. Look for the win32\ subdirectory, and you'll locate pythonservice.exe sitting somewhere there:

Mine is at C:\Python24\Lib\site-packages\win32\pythonservice.exe

you can change to that directory:

C:\Python24\Lib\site-packages\win32

and then do: pythonservice.exe /register

You should see a message about registering the Python Service Manager.

After that, if there are no errors, we're ready to plunge.

First, we need to import the two all important modules.

For those who don't understand Python... the lines beginning with #, are just comments.

import win32service
import win32serviceutil
import time

#at this point. We're ready to go.
#Put simply, a python windows service inherits from win32serviceutils.ServiceFramework
#simply extending that class, sets up all you ever need to do.

class aservice(win32serviceutil.ServiceFramework):
_svc_name_ = "aservice"
_svc_display_name_ = "aservice - It Does nothing"

def __init__(self,args):
win32serviceutil.ServiceFramework.__init__(self,args)
#at this point service is created pefectly.
#you could stop here and jump to setting up the '__main__' section,
#but you wont be able to stop your service, and it won't do anything.
#at the very least, you need to implement SvcDoRun() and better still SvcStop():
#This next attribute is used when its stopping time.
self.isAlive = True

def SvcDoRun(self):
import servicemanager
while self.isAlive:
#remember when i said you needed only two modules?
#well... i think i lied. If you're going to do anything
#usefull, you're going to obviously need more modules.
#This next module servicemanager, has some funny
#properties that makes it only to be visible when
#the service is properly setup. This means it can't be imported
#in normal python programs, and can't even be imported
#in the Global Namespace, but only in local functions that
#will be called after the service is setup. Anyway,
#this module contains some utilities for writing to EventLog.

servicemanager.LogInfoMsg("aservice - is alive and well")
time.sleep(10)
servicemanager.LogInfoMsg("aservice - Stopped")

def SvcStop(self):
#before you stop, you'll want to inform windows that
#you've recieved a stop signal, and you're trying to stop.
#in the windows Service manager, this is what shows the status message as
#'stopping'. This is important, since SvcDoRun() may take sometime before it stops.
import servicemanager
servicemanager.LogInfoMsg("aservice - Recieved stop signal")
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
self.isAlive = False #this will make SvcDoRun() break the while loop at the next iteration.

if __name__ == '__main__':
win32serviceutil.HandleCommandLine(aservice) #this line sets it all up to run properly.

Uhh... just incase you haven't guessed. That is all for the service.
The next part is installing and starting it. You can save this as aservice.py

cd to the directory where it is saved and do:

aservice.py install

Note that 'aservice.py remove' will remove the service

you can start and stop with Windows Service manager or:

aservice.py start

and

aservice.py stop

OK... that's it... play around, flesh out... anything.

You may or may not have figured out that the entire functionality of the serivice gets started from SvcDoRun()

That's all folks... i hope this is usefull :)
Hey... i just had a brainwave. I'll repeat the code here without any comments :)

import win32service
import win32serviceutil
import time

class aservice(win32serviceutil.ServiceFramework):
_svc_name_ = "aservice"
_svc_display_name_ = "aservice - It Does nothing"

def __init__(self,args):
win32serviceutil.ServiceFramework.__init__(self,args)
self.isAlive = True

def SvcDoRun(self):
import servicemanager

while self.isAlive:
servicemanager.LogInfoMsg("aservice - is alive and well")
time.sleep(10)
servicemanager.LogInfoMsg("aservice - Stopped")

def SvcStop(self):
import servicemanager

servicemanager.LogInfoMsg("aservice - Recieved stop signal")
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
self.isAlive = False

if __name__ == '__main__':
win32serviceutil.HandleCommandLine(aservice)

Note that if you're going to copy and paste this stuff, you may have some white space issues, since i'm just typing straight and editing here.

gtg