Monday, April 25, 2005

Python Windows Services

We had project last friday to write a windows service that would poll an Exchange mail box and write to a database. It was executed in C#. Well... i got thinking... hmmm... can't python be used to do this? Because if it can, then that project would not last more than a couple of hours (actualy, the C# project shouldn't also last more than a couple of hours if you know what you'r doing, but hey.. i have to have something to say about why i decided to do it in Python write? i mean.... I LOVE Python may not cut it for some folks :) )

OK... lets get to the meat of it.

I first hit google, looking for links to python and windows services. I then found some mailing list archives and in one of them, there was a reference to a chapter on Windows Services in the book, Programming Python on Win32.

Well... i had access to the book, and went headlong. Long and short of it, 15 mins of perusing latter, I had a windows service running. Lets attempt to get this stuff to work together now.


First there are two very important modules that helps our lives out. win32service and win32serviceutil

We're going to write a service that just writes to the event log every 10 seconds. U'll have to stop this service soon or else, you'll run out of EvenLog space ;)

First some important pre-parations and background on python-windows services. There is a program called pythonservice.exe, that actually handles everything that concerns windows services written in Python. I'm walking a thin rope here with this explanation now, but this is how i understand it, just don't quote me in a conference where there are other Pythonistas and Lords Of the Eggs, I'll deny it vehemently ;)

The way i figure it, pythonservice.exe is to python services what the usual python.exe is for all other normal python scripts. Normally on windows, once .py is associated with python.exe, each time you run a script, the python intepreter, python.exe is called to 'intepret' it. For services, apparently some funky stuff goes on, so instead of calling python.exe as the intepreter, pythonservice.exe is called as the intepreter (well, not exactly intepreter i guess, but it is the process that runs the services.) You can also look at it like this: You say service start, windows identifies it as a python service, and starts pythonservice.exe passing it parameters to find the service itself. Pythonservice.exe locates the service, and starts it. As far as windows is concerned, it is running a process called PythonService.exe (You'll see that without special tricks, when writting to eventlog, PythonService is the one that does the writing)

Now the preceeding means that windows has to know to associate python services with pythonservice.exe . This is essentially called registeration. So pythonservice.exe must be registered with windows to handle python windows services. to do that, locate where the python win32 extensions are on your box. They'll probably be in your site-packages folder. Look for the win32\ subdirectory, and you'll locate pythonservice.exe sitting somewhere there:

Mine is at C:\Python24\Lib\site-packages\win32\pythonservice.exe

you can change to that directory:

C:\Python24\Lib\site-packages\win32

and then do: pythonservice.exe /register

You should see a message about registering the Python Service Manager.

After that, if there are no errors, we're ready to plunge.


First, we need to import the two all important modules.

For those who don't understand Python... the lines beginning with #, are just comments.




import win32service
import win32serviceutil
import time

#at this point. We're ready to go.
#Put simply, a python windows service inherits from win32serviceutils.ServiceFramework
#simply extending that class, sets up all you ever need to do.

class aservice(win32serviceutil.ServiceFramework):
_svc_name_ = "aservice"
_svc_display_name_ = "aservice - It Does nothing"

def __init__(self,args):
win32serviceutil.ServiceFramework.__init__(self,args)
#at this point service is created pefectly.
#you could stop here and jump to setting up the '__main__' section,
#but you wont be able to stop your service, and it won't do anything.
#at the very least, you need to implement SvcDoRun() and better still SvcStop():
#This next attribute is used when its stopping time.
self.isAlive = True

def SvcDoRun(self):
import servicemanager
while self.isAlive:
#remember when i said you needed only two modules?
#well... i think i lied. If you're going to do anything
#usefull, you're going to obviously need more modules.
#This next module servicemanager, has some funny
#properties that makes it only to be visible when
#the service is properly setup. This means it can't be imported
#in normal python programs, and can't even be imported
#in the Global Namespace, but only in local functions that
#will be called after the service is setup. Anyway,
#this module contains some utilities for writing to EventLog.

servicemanager.LogInfoMsg("aservice - is alive and well")
time.sleep(10)
servicemanager.LogInfoMsg("aservice - Stopped")

def SvcStop(self):
#before you stop, you'll want to inform windows that
#you've recieved a stop signal, and you're trying to stop.
#in the windows Service manager, this is what shows the status message as
#'stopping'. This is important, since SvcDoRun() may take sometime before it stops.
import servicemanager
servicemanager.LogInfoMsg("aservice - Recieved stop signal")
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
self.isAlive = False #this will make SvcDoRun() break the while loop at the next iteration.

if __name__ == '__main__':
win32serviceutil.HandleCommandLine(aservice) #this line sets it all up to run properly.



Uhh... just incase you haven't guessed. That is all for the service.
The next part is installing and starting it. You can save this as aservice.py

cd to the directory where it is saved and do:

aservice.py install

Note that 'aservice.py remove' will remove the service

you can start and stop with Windows Service manager or:

aservice.py start

and

aservice.py stop


OK... that's it... play around, flesh out... anything.

You may or may not have figured out that the entire functionality of the serivice gets started from SvcDoRun()


That's all folks... i hope this is usefull :)
Hey... i just had a brainwave. I'll repeat the code here without any comments :)

import win32service
import win32serviceutil
import time

class aservice(win32serviceutil.ServiceFramework):
_svc_name_ = "aservice"
_svc_display_name_ = "aservice - It Does nothing"

def __init__(self,args):
win32serviceutil.ServiceFramework.__init__(self,args)
self.isAlive = True

def SvcDoRun(self):
import servicemanager

while self.isAlive:
servicemanager.LogInfoMsg("aservice - is alive and well")
time.sleep(10)
servicemanager.LogInfoMsg("aservice - Stopped")

def SvcStop(self):
import servicemanager

servicemanager.LogInfoMsg("aservice - Recieved stop signal")
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
self.isAlive = False

if __name__ == '__main__':
win32serviceutil.HandleCommandLine(aservice)


Note that if you're going to copy and paste this stuff, you may have some white space issues, since i'm just typing straight and editing here.


gtg

Sunday, April 17, 2005

Software design by Evolution - What Charles Darwin Had to say on programming.

Software design by Evolution - What Charles Darwin Had to say on programming.


There is the age old advice of sitting down to figure out every single thing about a system before attempting to build it. This works in lots of well structures that don't have changing requirements, and are usually cast in stone once the design is
approved. The problem is that in software systems, requirements keep changing, while delivery dates keep sliping.

It seems then that this method that works so well in a field like say, Civil Engineering and Construction is not doing so well in Software Development. A lot of other very intelligent people have talked a lot on the pros and cons of Detailed Initial Design, so I won't give it
much attention here. Instead i will talk about a method i've been using while coding, and has allowed me to challenge my clients with statements like, "This project can pretty much go any direction you want it to go.", "We can change anything, any how you want it". And just incase, you have not yet figured it out yet, i actually have been encouraging scope creep from my clients :). Should I add at this point that once faced with that possibility, they actually didn't take up the offer. (Uhh... don't try this with your clients... you may b biting off much more than you can chew ;) )

Anyway, the point is that it is possible to code flexibly, in such a way that features are not frowned upon, and the resulting code is still very well built and maintainable. I call this method 'Design By Evolution'. Actually, this is just a fancy name for Bottoms-Up programming, or what i usually love calling, the "UNIX" way, just for the simple reason that I first saw evidence of its existence in the way UNIX programs are written.


So what is Design by Evolution?
In a nutshell, it is a method of code design, where existing code naturally determines the next evolutionary step (which happens to be new features, etc). Also its most important feature or concept is that you NEVER design what you're not building immmidietly. In Agile development practices, i think this is called deferment or something like that.


Now, let me take it slightly deeper. Traditionally, if we want to design a program that accepts and parses various structure, read from a database, and the results stored into a file. We might want to start by building a super class that all the data
structures will inherit from, that will have methods for reading from the database, writing to file, etc. This is traditionally good OOP design. In 'Design By Evolution', I wont commit to the super class untill the need comes up. For instance, if in my first release, i will only be dealing with one type of data structure, Traditional design will be best served if i already have the super class built, at this first release, but design by evolution begs to leave this decision until it is unavoidable. So i would release my first version with just support for the single data type.

In a second version, if its justifiable to build a super class then i do, but if its not, i still hold off untill it becomes unavoidable. This may seem like concious bad design, but that's because I've already mentioned the constrasting traditional approach. In practice what this means, is to consider ONLY the problem that is before you, with NO THOUGHT of implementation detail on problems that have not yet arisen. My good friend Stan always says, "We'll cross that mountain when we get there".

What this does is many fold. First of all, it removes the overhead that comes with trying to figure out a solution so that you don't get stuck latter on. I say that this worrying is MOST of the time, premature. If you keep to good coding practices of modularity and small contained non-overlapping units, you'll be able to tear down and rebuild sections of code that need
to be torn down and rebuilt, without breaking other parts of the code. If indeed it ends up being a lot of work to do, i say that its more advantageous to do this work latter when it is needed, than earlier when its not needed.

The other aspect of Design by Evolution is the fact that its easier to build an EarthQuake resistant Airport (like Kansai Airport :D ), if you've actually been exposed to Earth Quakes, than if you learnt about Earth Quakes from School. This is because real life is soooo different from imagination. Relating this to programming and design, its usually easier to have a
code base without a feature, and then work the feature into the code base, within the design limitations afforded by the existing code base, than it is to design a feature complete code base and begin to build it.

This is because no amount of design can envisage every possible situation (except in the most trivial of cases), so there is almost always going to be a need to redesign and change initial design doing actual coding. Because of this reality, it is usually better to only design
and build what needs to be built at the time, and leave the rest for latter.

At this stage, It best i make this point. Design by Evolution is a very aggressive coding methodology, and is very non-forgiving of bad programming practices. If you make basic coding mistakes like tight-coupling of functionality, Design By Evolution will burn you. If you code without a rigorous testing methodology like Test Driven Development, Design by Evolution will burn you. If you leave unrefactored sections of code as you add to your existing code base, Design by Evolution will burn you. In fact, i could actually say that the foundations of Design by Evolution are 'loose-coupling', 'Refactoring', 'Testing', 'Testing' and 'Testing'.
If you mix 'loose-coupling', 'Refatoring' and aggressive testing, what you come up with are extremely stable reusable components. These can be libraries or smaller programs that can be glued together (which is the preferred 'UNIX' way), or both. Software that is built like this is very stable and flexible.

There is something that must be said about how the code evolves. When you defer design decision until the need arises, the existing code base imposes restrictions on what you can/should do or avoid. This is actually better since the code can be looked at to be choosing its own evolutionary path. If we had to make this decision before the code is built, we're bound to be a bit off the mark, but leaving this decision to be virtually made by the code itself, brings out the best combination of stable code and stable extension than would be possible otherwise.

Probably, the greatest advantage of this approach is the fact that there is no virtual closure on the code imposed by prior design. What do i mean? Well, when you first start by designing everything, you make some irreversible decisions/trade-offs, thus killing off some evolution paths that _may_ be desirable latter. On the other hand, leaving the code to evolve stage by
stage, while abiding by loose coupling, only kills of evolutionary paths for the individual parts, which could be easily reopened by reimplementing those parts. This way, we don't close code ourselves into a corner, and even if we do, we can just break down that corner without breaking the rest of our code.

[ADDENDUM: I added this following part, after suddenly realizing that Design By Evolution is not a word i came up with, but its been in existence in the Agile Programming Camp. I decided to take a closer look at all that Martin Fowler, and others have to say about it. Also in this time, I have started a 'Go Agile' initiative with my team of coders, and we're
aggressively studying the existing body of knowledge from the Agile Camp, with a view to integrate into our not at all agile methods. I'm actually hoping that one of them will read this material :)]


The other thing about Design By Evolution is that it can greatly aid Feature Driven Development. By letting your evolutionary blocks be feature driven, your client can see immidiet results, and your Time TO Market can be greatly reduced. In another
article I'll talk about Feature Driven Development.

As a conclusion, I'll just say this, "Agile Development is the future of an aggressively fast-paced market, where requirements can change in a whim, and by murphy's law, will change when you're least prepared to meet the change." To embrace Agile development might be the only way that software developers can hope to meet this reality head-on, but embracing
this methodologies is not for the lazy or the light hearted. You can ruin your business totally, if you don't take Agile development on its proper foundations of 'Loose Coupling', 'Refactoring', and 'Testing'.


[ Uhh... Mfon, i'm still owing you a Refactoring Tutorial :-w, and Farhan, I owe you some screenshots... dang!!! Ok... they're coming in the next week. This link is better now :) ]

Not blogged for a week or more!!!

Wow!!!

 

I’ve not been able to blog for a week or there about. For some crazy reasons, the most important being that my internet link just increased in SUCKERAGE density, in that period.  On a scale of 1 to 10, the link came in at a whopping 9.95 SUCKERS!!! Now that’s some suckerage. It was so bad that even trying to access my mails and the whole network would freeze up L

 

Anyway, on Friday, we sort of got the link upgraded to a 128k/128k link. That’s supposed to be good news right? WRONG!!!

 

I decided to test drive the link myself, by disconnecting all other machines on the network and going with just me, a single machine. I first went to http://performance.toast.net, for to verify the link speed on almost no load. I got a thruput of 116k on upload,  and hey… I can live with that. Problem came when I decided to do the downlink test. While I was downloading one of those images on the site, I tried to open http://www.linux.org, and boom, browser tells me, it can’t resolve DNS, I’m like WHAT?

 

So I decide to ping www.google.com, same thing, I take it close home, and ping my provider’s DNS machine (just one of them), and I get a connection timed out. OMG!!!

 

I’m like.. already? This is crazy!!! I ping up to my linux router, and everything works fine, I’m about to wonder what this is all about, since I can see that the downlink test is still going on. Just then the test finishes. I try to ping the DNS again, and hey… it works. I ping google again, it works…

 

At this point I realize that that simple test gulped my whole bandwidth… which means my provider platform is not handling large files well, and QOS issues are really bad L.

 

I realy don’t like this, as its going to mean I take out time and do some traffic shaping on my router, when I’m not really doing seriously demanding stuff to demand real shaping. (I’ll use iproute2’s tc for this, can be a bitch to configure from scratch to work properly)

 

Anyways, I’ll do it, if not on Monday, everyone is going to be screaming, hey Essien, you upgraded the bandwidth huh? IT STILL SUCKS!!!!!

 

Ohh.. the life of a network admin.

 

 

 

Tuesday, April 05, 2005

Autopackage Test Run

I AM IMPRESSED!!!

ok... i decided to take Autopackage for a test run over the weekend, and downloaded the Firefox package file from autopackage.org

I then removed firefox installation.

All i had to do was make the downloaded .package file executable with chmod +x, and i ran it at the prompt.

It went ahead, downloaded the base autopackage package, installed it, downloaded the gtk autopackage manager installed it, then it installed firefox for me. (I had been rather worried about this stage, since i had not found any link to downloading autopackage itself on the website)

All very smooth and painless. Tha's my definition of cool.

I seriously believe its can be the perfect answer to software installations on linux, and can serve as a good Desktop Linux tool.

I'm going to start planning work (only for now), on my perfect distro which is a combo of ArchLinux+E17+Autopackage+A_Couple_Of_Utilities_I_Will_write(tm)

Ahh... who wanted to know why i loved Opensource?

GO FIGURE!!!

Friday, April 01, 2005

pmutils update

Did some minor changes to pmutils today.

The previous version used to duplicate entries in the xml database when
‘pmsync’ was run multiple times. I had been lazy of fixing this since I
knew the obvious solution wasn’t the true solution. The obvious solution
was to make pmsync clobber all the xml databases and create them anew.
The unobvious and basically better solution, was to make it imposible to
enter an item with a duplicate id into any of the xml databases.

I had to do some refactoring, since I discovered I had the functionality
I needed already in db_core.py, just it was scattered around the place.
So I decided to once more put refactoring and test driven development to
test. I introduced get_info(), and get_all_info(), wrote the test
sections for them. I then modified list_info() and list_sub_info(), to
use the new get_info() and get_all_info().

Now, this is were I got tickled. All I had to do was make sure that the
TEST I had written for the old implementation worked properly for the
new implmentation, and boom. I didn’t even have to change anything in
any of the code that uses those guys… they just worked J

Ok nothing special, yeah I know, but if you’re not been using TDD and
Refactoring properly, and you actually start using it, coding becomes
soooo easy, it becomes fun once again.

If asked to summarize three important skills for a budding code slinger,
I’ll say:

1. learn advanced features of any language you’re using.

2. learn to hate unrefactored code, and refactor immidietly you see
such (point 1 above will help you with this)

3. learn to write and integrate Testing as a core part of your
coding. (This will make point 2 above easier to leave with)

ok… gtg.