Sunday, January 30, 2005

Weekend Reading

On weekends, i'm not supposed to be working 'officially'. Ok, sometimes,
I do actual work on my projects on the office, but most of those
weekends, i spend in these major forms.

1. I put work on my personal (and Open Source/Free Software) projects.
2. I scrounge around the internet for interesting and inspiring
articles. Usually from:
a. Paul Graham (http://www.paulgraham.com)
b. Phrack (http://www.phrack.org)
c. Fyodor (http://www.insecure.org)
d. The Linux Documentation Project (http://www.tldp.org)
e. Pragmatic Programmer (http://pragmaticprogrammer.com)
f. Well... Google ;)
g. There are really a lot I can't remember here.
3. Scrounge around Mailing list archives for interesting stuff
4. Get to study some of my working tools better by reading man pages,
looking for more advanced uses via Google. And trying out stuff for myself.

Here are some interesting ones I came across this week:

This one, I realy stumbled across munging along with Google, but it is
very promising:

http://www.acm.org/ubiquity/interviews/v5i21_aza.html


Here are other very inpspiring AND ofcourse, revealing stuff:

http://www.pragmaticprogrammer.com/articles/mind_the_gap.html
http://www.pragmaticprogrammer.com/ppllc/papers/1998_04.html
http://www.pragmaticprogrammer.com/ppllc/papers/1998_07.html
http://www.phrack.org/show.php?p=61&a=10
http://www.insecure.org/stf/hacker_crackdown.txt
http://www.insecure.org/stf/scoville_unix_as_literature.txt

njoi

Thursday, January 27, 2005

NEPA Strikes - AGAIN!!!

Today started out crazy :(

Actually it started out sometime yesterday. I'd gone to eat and left everyother person working. Ofcourse NEPA (the power company here in Nigeria), started killing us as usual with seriously fluctuating power. Well... to cut a long and annoying story short, the hard disk of my gateway/firewall just got busted.

When i came back and found it busted, i got pissed and went off to sleep. This morning, I had to fix it ofcourse.

Thank goodness that ArchLinux is the fastest distro in the world to install and configure. And thank goodness also for my configure scripts for various functionality which I keep offsite, the configuration was almost therapeutic.

The annoying thing was that 75% thru the installation, the whole power system trips at least 3 times :(

Men... this country has all the major makings of a frustrating residence. I wonder how we are even able to achieve what we achieve at times.

Well... at least we're back online.

Wednesday, January 26, 2005

Boring? Work update

Hehe… I want to beat my own record with this BlackBerry project. The idea is that if my boss can see that I just love working with embedded stuff and churn out results faster than he can get the QA team to react… he’ll line up more of those kind of projects for me.. and I’ll be finishing them up and going swimming during work hrs (I wish J )

 

Uh… yeah.. I’m aware that this plan can backfire badly… where I’m suddenly seen as a major producer and moved from my baby interesting project into the worst possible project interest wise… yeah.. I’m aware of the hazards.. but hey… maybe I can’t really help myself on this Black Berry project. Baah…

 

Anyways… today, I’ll delve into some graphic design project… to return to the BlackBerry App, ASAP.

 

That reminds me… I have a date with the ArchLinux community this weekend. The last major feature to add to PTSP, Incremental Rollback, _HAS_ to be rolled out this weekend, then I can get into a feature freeze and go thru the code to clean up loose ends.

 

Boy, this type of life can get interesting and intoxicating very fast.

 

Ok… lemme just finish my PM then I can …. Go… to… be….d…. zzzzzz….

 

 

Darn… must I get to work in the morning? L

 

Tuesday, January 25, 2005

Psalm 23 (Special Nigerian Version)

[I got this from a friend on Y! this morning…. Enjoy]

 

1. The Lord na my shephard, i dey kampe.

 

2. E make me sidon for where betta dey flow and come

put me next to stream make mai bodi thermacool.

 

3. E panel beat mai soul come spray am white, come dey

lead me dey go through express road of righteousness

sake of Hin name.

 

4. Walahi !, if I waka pass where arm robber, 419 and

juju people boku, come even join okada reach valley of

the shadow of death sef, mai bodi dey inside cloth.

Your rod and staff nko ? Na so dem dey like back bone

dey comfort me.

 

5. You don prepare Egusi and Pounded yam make I chop.

All mai enemies dey look waa waa. You rub me for head

wit vaseline intensive lotion. mai cup na River Niger

wey overflow hin bank.

 

6. True true, betta life and mercy go gum mai back

till I quench. And man pikin go tanda for God hou

 

Sniffing out code formation

Today (uhh… actually since yester-late-nite into wee hours on now), I’ve been working on a Black Berry Project at the office. My work is really diverse and priorities differ on projects, so I had started this project last year December and had to leave it at the initial release with the QA team to get onto something more ‘Billable’.

 

Well… finally the clients are getting ‘Giddy’ with expectation, so again I have to get back to it.

 

First a couple of Acid tests.

 

  1. Will I look at my code and go like… Ohhh…. Men… what the heck is this? (I passed this… everything just flowed, and I think 3 weeks is a long time span for this test)
  2. Will I start looking for ways to change my code because I was being plain what-i-don’t-know-to-call-it earlier? (I passed this test too… everything I did then, seemed right now, lets not think about 6 months from now yet…. I’ll still be maintaining it by then, so we’ll know)
  3. Will I view my own code in the light of QA requirements made by the QA team and go like… “Damn!!! How am I gonna fix this!” ? (Well again I passed this, as I instinctively knew where to look for stuff that was being requested and exactly where to add them. This was so cool that in just 1 day (actually less), I’ve fixed more than 50% of the QA requests and made another release to the QA team. Hehe… they won’t take me for granted next time :D )
  4. Will I… uhh… I can’t think of anymore acid tests J

 

 

Again, I learnt to appreciate a good sense of code smell. One of the bugs was an uncaught exception, which I didn’t handle in the first release. Well… I could have fixed that in less than 1 minute, but something told me… hey dude… why postpone that Exception Handling framework for this library? Just do it now… or you’ll be chasing yourself around by weekend probably.

 

So well… I went with my gut, developed the exception framework, which took me like 15 minutes to properly think out and integrate. In so doing, I also extended its reach to various parts of the code. Now, I can’t really quantify what this has done for me right now, but I think it has just saved me light years of work tmrw, so I can be royally lazy again and get to more interesting parts of code.

 

The last thing that made me really smile today, was a comment I had put above an Even Listener class. I had two components that where doing similar stuff that smelt a bit similar to me. In my first version, I didn’t want to fall culprit to premature optimization, so I just put a comment there that I suspected these two dudes could be replaced with one event listener if I gave it some more thought. Well… was I right… and today, without even so much thought… I just saw the answer before my eyes and implemented it… loosing like 25 or so lines of code into one small section with just 7 lines of code. Pretty cool huh?

 

I think leaving notes like this in your code is good. Infact in unix, the venerable Vi(m) will give the text TODO: special coloring, so you can easily spot it as you wade thru you code. This encourages you the coder to leave your self post-its in your own code.

 

Ahh…. I talk too much… I better go to bed, as I have to wake in the next 4 hrs time…

 

 uhh…. Work can suck wen it comes to hours.

Monday, January 24, 2005

PTSP Bugfix release v052

I got a notification on the Archlinux mailing list of a bug in PTSP (http://www.datavibe.net/~essiene/pacman).

When checking it out, it seems the user has an older version of the patch, but still then, I stumbled across 2 rather annoying bugs too.

Well... sum total, squashed them!!! and released a minor bug-fix release of the project.

Also, there is a remote possiblity that the bug the user experienced was related to these. Anyway, they're gone now... gone from the codiverse, exterminated, kaput! :)

Thnx GT, for bringing these to my notice.

(U mean other people apart from me actually use this stuff? :D Rather humbling I must say)

Sunday, January 23, 2005

From GAS to Flat

(For the curious, this post was sent from my commandline, using a combination of printf,cat and mail in one pipeline. nifty huh? linux rocks so does google so does blogger.com)

Recently, I have been dabbling in shellcode.

One frustration I had was getting at flat binaries while working with GAS. Well, while there may exist other solutions, I was sort of too lazy to find out, so i made this script, that uses objdump to print out hexcode of gas produced binaries.

As a side kick, it can also write this hexcode into a file hence producing a flat binary.

This script is part of a larger project i'm working on anyway, I guess I'll have a release ready sometime next week if work permits the time.

For the poetic, I embedded a small poem (should I say bad poem?) at the bottom of the program. It inspires me to always approach programming with the same joyfull fantasy I first had when I started, and to always strive to produce beautifull looking code (at least beautiful to me).

Enjoy :)





#! /bin/bash

# obj2hex, obj2bin

# Copyright (c) 2005 by Essien Ita Essien
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307,
# USA.
#


# Calling this program as obj2hex ouputs the hexcode representation of the binary file to stdout

# Calling this program as obj2bin creates a flat binary of the specified file into another specified file.


function usage()
{
programname=$1

[[ $programname == "obj2hex" ]] && echo "Usage: $programname " && exit
[[ $programname == "obj2bin" ]] && echo "Usage: $programname " && exit

}

programname=`basename $0`

[[ $programname == "obj2hex" ]] && [[ $# != 1 ]] && usage $programname
[[ $programname == "obj2bin" ]] && [[ $# != 2 ]] && usage $programname

filename=$1
outputfile=$2

all=`objdump -d $filename | grep '^ [0-9a-f]'| cut -f2`
b=

for a in $all;
do
b=$b`echo -n '\x'$a`
done;

[[ $programname == "obj2bin" ]] && printf "$b" > $outputfile && exit
echo "$b"


# There is an art in programming.
# A poem in every function,
# A tune in every loop,
# I hope is exists here too.
#
# -Essien Ita Essien

Saturday, January 22, 2005

Pacman Transaction Support Update

I just updated the Pacman Transaction Support Patch.

http://www.datavibe.net/~essiene/pacman

pacman has moved to pacman-2.9.5, so I had to port the patch from 2.9.4
to 2.9.5 . Ahhh... the beauty of good design. The port took me all of 5
minutes. And 1.X of those minutes was used to write a bash script
'migrate_patch' to help me reduce key presses :D

Actually, what migrate_patch does (its right now specific to this patch,
but when the need arises, I can easily extend it), is to help me manage
the files that I am actually changing.

I also added some comment tags in the source files I modified to add my
patch hooks, so I can easily find them. This could be a 'Secret Window'
into an 'Uber Secret' plan to totally reduce the number of key presses I
have to type.

Did I mention that I've learnt a lot from working on this patch? Well...
I've learnt that modularity isn't an overstatement at all. Infact, i've
come to the conclusion that unless you're modifying algorithms already
written down, the only DECENT way to extend an existing project is via
modules. You then have to only hook the modules into the project.

Before, I had a monolithic patch, so to speak, and the size was 148Kb.
When I refactored and modularised, and then added more features, the
size dropped to 41Kb. Amazing? You bet.

Pacman Transaction Support Project Screenshot

The screenshot is on my website at:
http://datavibe.net/~essiene/images/trnSS01.png

I was about to do a system upgrade this morning, so I can get the new
pacman version among other things. Ofcourse I use the patched pacman on
my system :). Here is a screenie of the first part of the process that
downloads, distills and then installs pacman.

The command line is:

#pacman -SutN jan-stable2

The strange flags there are:

-t : Enable transaction support for the -S option.
-N : Use this explicitly named transaction.

Every other option is as known already by pacman users.

Notice the:

distilling pacman...done.

Among the last lines at the bottom of the screenshot.

http://datavibe.net/~essiene/images/trnSS01.png

pyshellcode

For want of a better name for a shellcode that launches a python shell :)


Been researching into buffer overlows and vulnerabilities et al.

Beeing a hardcore python fanatic, I just had to try the following shellcode. I must say thanks to all the ppl with tutorials and materials over the web that I’ve read over and over again. Thank you GNU thnk you OpenSource thnk you Linux.


/*Begin Assembly Routine*/

BITS 32

xor eax,eax
jmp short string

start:
pop esi
mov byte [esi+15],al
mov [esi+16],esi
mov [esi+20],eax
mov byte al,0x0b
mov ebx,esi
lea ecx,[esi+16]
lea edx,[esi+20]
int 0x80
xor eax,eax
xor ebx,ebx
mov byte al,0x01
int 80

string:
call start
db '/usr/bin/python$XXXXNULL'

/*End Assembly Routine*/

Trying to use GAS, I ran into problems I can’t explain tho L

/*Begin problematic routine shellcode*/

jmp dummy
start:
popl %esi
xor %eax,%eax
movb %al,7(%esi)
movl %esi,8(%esi)
movl %eax,12(%esi)
movb $0xb,%al
movl %esi,%ebx
leal 8(%esi),%ecx
leal 12(%esi),%edx
int $0x80
xor %eax,%eax
xor %ebx,%ebx
movl $0x01,%eax
int $0x80
dummy:
call start
.ascii "/bin/sh"

/*End problematic routine shellcode*/

Dunno what I’m doing wrong in the problematic routine. Any one any ideas?

Friday, January 21, 2005

My Home Page

 

Finally got around to doing what all my friends, employers and co-workers have been wanting me to do for a long time now  - Put up a darned home page that makes it easy to get to the stuff I have been working on at home.

 

Its here at http://www.datavibe.net/~essiene

 

I’ve been lazy about this mostly because there was no really good web IDE for Linux for a while, and I had to fall back to my default editor, the venerable VI(m).

 

Recently, with the maturity of the Bluefish Project (http://bluefish.openoffice.nl/index2.html ), I’ve found a rather good web IDE. I only wish, I could use VI(m) as my editor while in Bluefish. :-s

 

Right now, I’m about to step out for a party… so hey… WHOOOOOOOOOOOOO!!!!!!

Thursday, January 20, 2005

The Fongus!! Whooped :P

Just for the record. I whooped ‘the fongus’ arse today on Y! Chess. He ran away squealing like a girl and feigned internet connection problems.

 

Moral Lesson: Children, Always eat your vegetables… or they might just eat you. :P

 

MUHAHAHAHA

E17 Mini Freeze

Whooooo!!!!

 

After years of waiting and trusting, Rasterman is still delivering as promised. Enlightenment R17 (The Linux Window Manager for the next generation of computer graphics looks and feel, I believe), has just reached a first Minor Freeze. According to Rasterman on his site (http://www.rasterman.com), he has decided to call a minor code freeze before they’ve gone too far, so the team can clean up bugs, tie up loose ends, before proceeding.

 

I’m really happy about this. I’m gonna have to create time this weekend to build this from CVS.

 

Thnx guys for the hard work.

Lazy Man's Ethernet Tutorials 1

This article and the next after it, are basically designed for my friend Esther Effiom, who is trying to merge what she’s learnt in school into real life practice. Well… I also had similar issues when I first started reading and then practicing networking. I found that the best thing is to have a working conceptual mental model that is more or less accurate, then to that model you’ll easily be able to add more information. These articles are written with that goal in mind. Not to be precise on every little technical detail, but more of a mid-level overview with enough low-level information to assemble into a mental picture. It will stay low-level enough to get the basics in, but high-level enough to easily assemble.


Lazy Man’s Ethernet Tutorials 1

In the course of these tutorial series, we’ll attempt to answer some basic questions about Ethernet.

1. What is Ethernet?
2. What is a packet and how does it carry information?
3. How does a system send information to another system on its subnet?
4. How does a system send information to another system on another subnet?
5. Network borders - Proxies, Packet Filters, Gateways and Routers.


1. What is Ethernet?

Ethernet is a nutshell is a broadcast mechanism that allows networked systems to talk to themselves. The keyword here is broadcast. Of course Ethernet is much more than this if you're trying to get extremely technical, but for the purpose of this tutorial, this definition will do all rite.

In Ethernet networks, all nodes have:
a. A network interface card also has a unique address called a MAC address.
b. A network interface card driver software, responsible for interpreting all the data that come in on the card, and passing it upwards to the Operating system's networking stack.
C. An IP Address which is used to identify the system at a higher level of communication than the MAC address can do. This may not be the reason, but I have observed that IP Addresses by their very design allows grouping of addresses into network groups, sub networks etc. This (in fact any kind of grouping), can not be done with the MAC addresses, so for any serious communication, a higher level of addressing is needed. But as we will see, when push comes to shove, we ALWAYS need to know the MAC address of the systems involved.
d. All systems will be directly or indirectly connected together either with a HUB or a SWITCH (Layer 2 switch, if you've heard this term before).

Keeping this in mind. Let us attempt to answer the next question.


2. What is a packet and how does it carry information?

First and foremost, the term packet in this document is generic, and refers to all incarnations of data at the various layers of the network stack. Usually, depending on which layer of the network data currently is at, it is given a different name. Hence it can be called a frame, a packet, a datagram, etc. In this tutorial, I’ll just call them all a packet. To make it easier to understand ( well, actually, to make it easier for me not to make a mistake :) ).

The packet is interesting because it has to be self sufficient on the network. Self sufficient in that any node receiving should know where it came from, and where it is going to. From what we've seen, Ethernet hosts have at least two addresses that are unique to it, the IP Address and the MAC address. This means that the packet will carry at least 4 addresses. IP addresses of both Source and Destination, and MAC addresses of both Source and Destination. For the purposes of illustration:

[src-mac-addr]
[dst-mac-addr]
[other-info ]
[src-ip-addr ]
[dst-ip-addr ]
[other-info ]
[Data Carried]



Well... forgive the diagram :)

What is going on here (in brief), is that before the packet leaves a network node, the IP address of its source and destination and the MAC address of its source and destination are prepended to it in the HEADER information area. This means that before the Data Carried is retrieved or sent over the network, the header area must be built (when sending) or interpreted/parsed (when receiving). We'll see how this works below.

Now onto the next question.



3. How does a system send information to another system on its subnet?

Firstly, what is a subnet? Well, if you have any experience with IP Addresses, you probably know that they look of the form X.X.X.X. Well, that's not actually all about them. These IP addresses have some kind of visual pattern, for instance, 192.168.1.1, 192.168.1.2, 192.168.3.4, 192.168.3.7, 10.0.1.1, 10.0.2.4, etc. By visual inspection, you can actually pick out that 192.168.1.1 and 192.168.1.2 differ only at the .1 and .2. 10.0.1.1 and 10.0.2.4 are similar up to 10.0. . AT this point, without delving into the mechanisms of how this is gotten, believe me when I say to you that there are two parts to every IP address. These two parts are:

a. The parts that looks alike for various machines.
b. The part that actually is unique for various machines.

Actually, this is not purely visual, as these comparisons are really done at binary level, but forget about that at least for right now. Just think simple in terms that (a) is the network address and (b) is the host address, and when you combine (a) and (b), you have a complete address for a host on a network. As more in-depth information, there is a number called the subnet mask, which actually mathematically separates these two numbers. But for now, you can forget about that in detail.

The main point here is that machines that have the same network part of their address for are said to be on the same subnet, and these are the only machines that can talk to themselves without help.

So anytime a machine wants to talk to another machine it checks if they are on the same subnet, if they are, communication can go on, if not, some help will be needed.

Lets paint a little picture.


[A-----B-----C]<----D----->[E-----F-----G]

A = 192.168.1.1 (11:22:33:44:55:66)
B = 192.168.1.2 (22:33:44:55:66:77)
C = 192.168.1.3 (33:44:55:66:77:88)

(FYI: the numbers with dots are the IP addresses, while the numbers with colons are the MAC addresses. To get this info on UNIX do 'ifconfig' on Windows do 'ipconfig/all')



Forgive this diagram ( :P ), but imagine with me the scenario. A,B and C are on a single network. This would mean from our point (1) above that they are connected via a single (or series of) hub(s) or switch(es).


For our explanation, assume A wants to send data to C. The first thing A does (at least in our context), is to verify that the systems are on the same subnet then to build the HEADER and prepend to the actual data to send.

Before sending the packet, A knows all its own addresses, but it probably has no information on C, but its IP address. But it has to relate the IP Address to a MAC address. To do this, A first checks its ARP table, which contains a mapping of known IP-address to MAC address pairs. If it doesn't find an entry for 192.168.1.3 there, A issues an ARP broadcast request.

ARP means Address Resolution Protocol, and in a nutshell what happens is that A sets loose a that everyone will identify as a broadcast packet (which means they can all accept it), the packet contains a single question, "Who owns IP Address 192.168.1.3". C, The system that owns that IP address, will reply with its own ARP Reply packet, saying "192.168.1.3 is at MAC 33:44:55:66:77:88", thus supplying its MAC address. Now A knows which MAC address belongs to C, and its IP Address. A also updates it ARP table, so if it has to send a packet to C again before its ARP table cache expires, it wont need to do a full ARP request again.


The final packet will loosely resemble this:

[11:22:33:44:55:66]
[33:44:55:66:77:88]
[.. other info ..]
[ 192.168.1.1 ]
[ 192.168.1.3 ]
[.. other info ..]
[.. Data Carried..]

This packet will now be broadcast on to the A-B-C network. What this means is that A,B and C will see the packet. The network interface cards on A,B and C will receive this packet and send it to the network interface card driver software. (Is it not odd that though A sends the packet, A also receives the packet? Well.... such is the broadcast world of Ethernet).

The driver software for each of the network nodes will proceed to examine the packet. A will discover that though the source MAC address is itself, the destination MAC address is another node. Under normal circumstances (uhh... there are some abnormal circumstances), A will ignore the packet, and will not send it further up the network stack for processing.

When B examines the packet, it will see that neither the source nor destination MAC addresses are itself. It will also ignore the packet.

C will examine the packet and see that the destination MAC address is indeed its own. It will then proceed to send the packet upwards the network stack.

For the purposes of this illustration, lets assume that there is a generated reply. To send the reply packet, C will examine the original packet, to get the source IP address on the packet. This source IP address will become the destination IP address of the recipient of the reply.

C then proceeds to do the same thing that A did above.

If you don't understand the preceding, stop at this point, and read it again slowly, pausing to make sure you get a mental picture of the scenario described. It's important, since every other networking information builds on the preceding. In the next Installment, we'll answer the remaining questions.

Obviously, there may be errors in this document caused by my own misinformation, oversimplification or plain omissions. So questions, suggestions, kudos, flames and 'are you nut's should be directed to essiene at datavibe dot net or essiene at gmail dot com


Just Because I Can

Okay,

 

I’m at work trying to meet a deadline, and I just thought, “Hey! I have a mail in interface to my blog. Why don’t I just send in a Good Ole Blog?”.

 

So here goes: “Good Ole Blog”!!!

 

:P

Wednesday, January 19, 2005

Webless Web Interactions

I think that somehow, WWW has almost downplayed the importance of email as an application. Well, coporate users know that email is probably the most important of all network applications, but some other people don’t feel this.

Take me for a quick example, I’m probably your typical techie, but somehow, never really caught on the idea of blogging, until recently. When I did, I encountered my age old reluctance of logging onto a website to fill in information. This is very pronounced since I don’t have a fast link to the internet.

Well.. once again email comes to the rescue, and now I don’t have to log in to a website to fill in my blog and make it available to all.

I think that for coporate users, these kinds of applications are very important. Those that allow you to do work, without changing much of your working habit/setup.

This is also why I strongly believe that webservices and XML are going to explode slowly at first, but surely, into some very nifty applications that no one, I mean ABSOLUTELY no one is even thinking about right now.

I’m kinda hoping I’ll be the first one to bring up that idea J

Hello World

"Hello World"

There, i've done it. Now no "Ghosts in the machine", will bug me. :)

For those that may have found my other blog site, i got frustrated b/cos there is no mail in interface. I'm moving to blogger.com because of the mail in interface, availability of neat API's and the simple fact that it is basically Google. Ahh... I just luv the word Google.

With the mail in interface, i should be blogging more regularly now. EGAD!!!