Archive for category code
A Brief Linux Sockets Connect Illustration
For a little while I’ve been playing with sockets in C now and have come up with the following succinct example of connecting. Note that the connection is fairly flexible with regards to protocol and transport type. It really is simply there to make the connection to somewhere else with as few questions asked. It will involve DNS and service lookups if you provide names instead of the network address and port. If you want to catch exceptions you need to clear the standard error variable and check it yourself after receiving a fail return value. It won’t, or at least shouldn’t, report any exceptions because for my current purposes failures are not really exceptions just dead ends on some of many paths. I tried to keep it simple too, have fun.
/* connect_to_socket
*
* connect to a socket using an initialised addrinfo structure,
*
* info is an initialised addrinfo structure
* sock is a pointer to the location to store the socket descriptor when opened
*
* returns 0 if successful
*/
int connect_to_socket(const struct addrinfo *info, int* sock) {
/* open socket */
*sock = socket(info->ai_family, info->ai_socktype, info->ai_protocol);
if (*sock < 1) return -1;
/* connect to the server*/
if (connect(*sock, info->ai_addr, info->ai_addrlen) == 0)
return 0;
/* clean up on failure */
close(*sock);
return -1;
}
/* connect_to_server
*
* connect to a server using the server name or adress and port or service name
*
* server is a string containing either the name or address of the server
* port is a string containing either the service name or port number to use
* sock is a pointer to the location to store the socket descriptor when opened
*
* returns 0 if successful
*/
int connect_to_server(const char* server, const char* port, int* sock) {
struct addrinfo hints, *info = NULL, *list = NULL;
int e = 0;
/* initialise the hints for retrieving the address details */
memset(&hints, 0, sizeof (hints));
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
hints.ai_flags = AI_PASSIVE;
hints.ai_protocol = 0;
/* get the address if the server to connect to */
e = getaddrinfo(server, port, &hints, &list);
if (e != 0) return -1;
/* connect to the first socket in the list */
for (info = list; info != NULL; info = info->ai_next)
if (connect_to_socket(info, sock) == 0)
break;
/* clean up and return */
if (list != NULL) freeaddrinfo(list);
if (sock > 0) return 0;
else return -1;
}
When there’s Nothing in your Sock
Normally I do most of my network programming in Java; life is just easier that way. But some times, *very* rarely, you need to carve up a little C. So I’ve not touched socket networking for a while and all its foibles. Hence a few problems with reading from a socket.
When socket programming with the ‘read’ function it does not always return the length read into the buffer. In fact if you are talking to an HTTP server a ‘GET /‘ will nearly always end with read returning 0. This is because the socket will have been closed on your last read. This of course begs the question, how do you know how much was read in the last read? For me this was simple; I just zero filled the buffer prior to reading.
ssize_t r_read(int sock, void* buf, size_t size)
{
ssize_t ret;
bzero(buf, size);
while (ret = read(sock, buf, size - 1), ret == -1 && errno == EINTR);
return ret;
}
Which does work because now its a simple matter of counting until the first 0 shows up, but I’d still like to know how to tell how many characters were read without having to do the count. This becomes more important when transferring binary files as you won’t know if the 0 is validly part of the file or not.
Looping Back through Netfilter
Using netfilter as a loopback device is surprisingly simple and can let you tap into that network conversation between components on a platform that tends to be hidden by the OS in its own loopback device. Here’s how:
iptables -t nat -A PREROUTING -s 192.168.122.2 -d 192.168.2.7 -p tcp --dport 80 -j DNAT --to 192.168.122.2:80 iptables -t nat -A POSTROUTING -s 192.168.122.2 -d 192.168.122.2 -p tcp --dport 80 -j SNAT --to 192.168.2.7
Well that’s the netfilter bit done! But of course there’s a bit more setup than this. Firstly you need to have your netfilter, soon to be loopback, host on the same subnet as the box that we are working with. Next you need to configure the client that makes the connection to the other component on the same box point to the netfilter host. Finally you need to work out which addresses and ports go where in the statements above when you execute them.
In the case above we are intercepting the communication between the HTTP client and server that are both hosted on the same box 192.168.122.2. The netfilter loopback is being hosted on 192.168.122.1. For my worked example here the client/server box is actually a virtual machine guest and the netfilter configuration is being placed on the virtual machine host. Note the third address I used here 192.168.2.7 is bound to another ethernet card in the virtual machine host. The third address is necessary, for no other reason than to help with the clarity of this example and to get the routing right.
Our first rule catches the inbound requests from our guest and redirects them back to the guest. Note that the rule is restricted to a specific initial destination, source, protocol and port so that no other hosts will be affected. In fact the rule will only catch the exact scenario that we want. However this is not the complete picture. If we were to try to browse from the client to 192.168.2.7 on port 80 and capture the chatter on 192.168.122.1 using tcpdump we would get the following:
16:54:37.800369 IP (tos 0x0, ttl 64, id 17896, offset 0, flags [DF], proto TCP (6), length 60) 192.168.122.2.41418 > 192.168.122.2.http: S, cksum 0x496b (correct), 901912667:901912667(0) win 5840E.. Here you see the first syn of the TCP three part handshake leaving the host and going back to the guest with the guest's IP address as the source! It is the packet going out of the host to the guest but the guest is not going to be able to route it back. This is where the second rule steps in. It is applied after the packet is routed by netfilter and it replaces the source address with 192.168.2.7. Now the client on the guest can communicate with the server on the guest and you can use tcpdump on the host to look at their conversation.
Apache Tomcat 6 and the HSM
Recently I’ve had some real fun with SafeNet HSM’s (Host Security Module or Hardware Security Module) and Tomcat 6 in the never ending process of PCIDSS compliance. People are probably going to line up to stone me after saying this but PCIDSS is a really great effort to make things safe and secure for all of us. In a lot of cases its the stick that drives organisations to adopt some good practices. One of which is the use of HSM’s for all things from the crypt.
Now this little rant won’t walk you through PCI compliance, rather its just about my experiences getting Safenet Eracom Orange or Blue HSM to work within Tomcat. Deployment is really easy:
- Install the latest version of Java,
- Install the Eracom Orange C runtime,
- Configure the Eracom Orange runtime to connect to the HSM,
- Configure the HSM with tokens, keys, certificates, users and whatever else you need,
- Install the Eracom Orange Java runtime,
- Install Apache Tomcat 6,
- Place the Eracom Orange ‘
jprov.jar‘ Tomcat’s server shared lib directory ‘CATALINE_HOME/lib‘, - Restart the Tomcat server if its already running or just start it if it is not,
- Deploy your webapp.
Apart from steps 7 and 8 simply follow the provider’s directions for all steps. The key in this thing is that the ‘jprov.jar‘ archive must be loaded when Tomcat starts. I imagine that it should be the same for other providers like for IBM’s toys.
Now for the development set up. Simply follow the same steps as above but install the SDK’s for the Eracom and include ‘jprov.jar’ as a library in your development environment. Typically your development environment will bundle the ‘jprov.jar‘ into the web application directory. It won’t do any harm there but to get it to work properly it will need to be in the Tomcat server’s common library directory as detailed above. In fact you don’t really need to install the SDK’s if you have an actual HSM to play with for development purposes, but the SDK’s do provide a software HSM simulator for those of us who don’t want the expense of buying one of these things for every developer on the floor. Plus the SDK’s will give you a good deal of documentation too.
So you are all set up and ready to build the next great crypt. Start by building a stand alone application to confirm that everything works before going any further. Lets get to the code:
/*
* Crypt.java
*
* A simple class to provide basic SDES encryption and decryption
*/
package crypt;
import java.io.IOException;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.Key;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.NoSuchProviderException;
import java.security.Security;
import java.security.UnrecoverableKeyException;
import java.security.cert.CertificateException;
import java.util.Properties;
import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.spec.IvParameterSpec;
import org.apache.commons.codec.DecoderException;
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.codec.binary.Hex;
import org.apache.commons.lang.StringUtils;
import org.apache.commons.logging.Log;
/**
*
* @author mc
*/
public class Crypt {
/**
* Creates a 3DES key cipher with the parameters in the config, valid
* modes are Cipher.ENCRYPT and Cipher.DECRYPT
*
* @param mode set the cipher to either encrypt or decrypt
* @param config the configuration parameters for this application
* @param log log to write debug output to
* @return a cipher that should be used then nulled to be thrown away
* @throws KeyStoreException
* @throws NoSuchProviderException
* @throws IOException
* @throws NoSuchAlgorithmException
* @throws CertificateException
* @throws UnrecoverableKeyException
* @throws NoSuchPaddingException
* @throws DecoderException
* @throws InvalidKeyException
* @throws InvalidAlgorithmParameterException
*/
private Cipher createCipher(int mode, Properties config, Log log)
throws
KeyStoreException,
NoSuchProviderException,
IOException,
NoSuchAlgorithmException,
CertificateException,
UnrecoverableKeyException,
NoSuchPaddingException,
DecoderException,
InvalidKeyException,
InvalidAlgorithmParameterException {
log.debug("Adding provider.");
// will gracefully not add provider if provider already loaded
Security.addProvider(
new au.com.eracom.crypto.provider.ERACOMProvider());
log.debug("Getting key store instance.");
KeyStore store = KeyStore.getInstance(
config.getProperty("crypt.keystore.type"),
config.getProperty("crypt.keystore.provider"));
log.debug("Loading key store.");
store.load(null, null);
log.debug("Retrieving key.");
Key key = store.getKey(
config.getProperty("crypt.keystore.entry"), null);
log.debug("Generating cipher.");
Cipher cipher = Cipher.getInstance(
config.getProperty("crypt.transformation"),
config.getProperty("crypt.provider"));
log.debug("Creating initialisation vector.");
IvParameterSpec i = new IvParameterSpec(Hex.decodeHex(
config.getProperty("crypt.vector").toCharArray()));
log.debug("Initialising cipher in the given mode.");
cipher.init(mode, key, i);
return cipher;
}
/**
* Takes a string and returns its equivalent Base64 encrypted string.
*
* @param data string to be encrypted
* @param config parameters for this application's encryption
* @param log log to write debug output to
* @return encrypted Base64 encoded form of the data string parameter
* @throws NoSuchAlgorithmException
* @throws NoSuchProviderException
* @throws NoSuchPaddingException
* @throws InvalidKeyException
* @throws IllegalBlockSizeException
* @throws BadPaddingException
* @throws InvalidAlgorithmParameterException
* @throws KeyStoreException
* @throws IOException
* @throws CertificateException
* @throws UnrecoverableKeyException
* @throws DecoderException
*/
public String b46Encrypt(String data, Properties config, Log log)
throws
NoSuchAlgorithmException,
NoSuchProviderException,
NoSuchPaddingException,
InvalidKeyException,
IllegalBlockSizeException,
BadPaddingException,
InvalidAlgorithmParameterException,
KeyStoreException,
IOException,
CertificateException,
UnrecoverableKeyException,
DecoderException {
log.debug("Encrypting data.");
byte[] encrypted =
createCipher(Cipher.ENCRYPT_MODE, config, log).doFinal(
data.getBytes());
log.debug("Base64 encoding data.");
String encoded = new String(Base64.encodeBase64(encrypted));
return encoded;
}
/**
* Takes a encrypted Base64 encoded string and returns its equivalent
* decrypted string.
*
* @param data string to be encrypted
* @param config parameters for this application's encryption
* @param log log to write debug output to
* @return encrypted Base64 encoded form of the data string parameter
* @throws NoSuchAlgorithmException
* @throws NoSuchProviderException
* @throws NoSuchPaddingException
* @throws InvalidKeyException
* @throws IllegalBlockSizeException
* @throws BadPaddingException
* @throws InvalidAlgorithmParameterException
* @throws KeyStoreException
* @throws IOException
* @throws CertificateException
* @throws UnrecoverableKeyException
* @throws DecoderException
*/
public String b64Decrypt(String data, Properties config, Log log)
throws
NoSuchAlgorithmException,
NoSuchProviderException,
NoSuchPaddingException,
InvalidKeyException,
IllegalBlockSizeException,
BadPaddingException,
InvalidAlgorithmParameterException,
KeyStoreException,
IOException,
CertificateException,
UnrecoverableKeyException,
DecoderException {
log.debug("Base64 decoding data.");
byte[] decoded = Base64.decodeBase64(data.getBytes());
log.debug("Decrypting data.");
byte[] decrypted =
createCipher(Cipher.DECRYPT_MODE, config, log).doFinal(decoded);
return new String(decrypted);
}
}
So that’s it, very straight forward but lets take time to look at one or two points of interest.
In a nutshell functionally this is a class that can be used to encrypt and decrypt data using stored 3DES keys. Note that the configuration information is all passed in via the properties parameter. Also note that the class uses base64 encoding because its easier to play with and store strings. Generally you would also want to replace unsafe base64 characters too if the storage unit required it.
On the encryption front there are a few things to note too. Take a look at the includes, as far as the crypt mechanisms it uses only the standard Java imports and practices. There is only one thing that is specifically from the Safenet SDK, the loading of the provider class in the following snippet:
// will gracefully not add provider if provider already loaded
Security.addProvider(
new au.com.eracom.crypto.provider.ERACOMProvider());
To complete the picture here is the main class and the configuration file to go with it. That’s the lot; three simple files and it should be all go to test your configuration.
/*
* main.java
*
* Test application for the Eracom encryption devices.
*/
package crypt;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Properties;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
/**
*
* @author mc
*/
public class Main {
/**
* @param args the command line arguments
*/
public static void main(String[] args) throws IOException {
Properties config = new Properties();
Log log = LogFactory.getLog(Main.class);
boolean encrypt = "encrypt".equalsIgnoreCase(args[0]);
config.loadFromXML(new FileInputStream(args[1]));
try {
if(encrypt)
System.out.println(
new Crypt().b46Encrypt(args[2], config, log));
else
System.out.println(
new Crypt().b64Decrypt(args[2], config, log));
} catch(Exception e) {
log.error(e.getLocalizedMessage(), e);
}
}
}
ERACOM au.com.eracom.crypto.provider.ERACOMProvider DESede/CBC/PKCS5Padding AAAAAAAA00000000 ERACOM CRYPTOKI KeyName
Once this is working it should be possible to use the same crypt class file in a Tomcat hosted web application with one slight adjustment. As the Tomcat service will load the provider you should not attempt to load the provider in your application. In fact if you try it will almost certainly fail as your application is unlikely, unless you have specifically configured it, will not have the privileges need to load classes. So just comment out the ‘Security.addProvider‘ call in the class.
So that’s it; all you needed to know to configure an HSM to start working with Tomcat. Simple wasn’t it? Now all you need to do is figure out how to use it to help you with PCIDSS and how to change the crypt file to suite your specific needs.
To Mail a File Elegantly – Python or Perl?
People fixate on particular languages and stir hot debates over the entrails of code compared in the cold hard light of a walk through but is the debate over which language is better worth even the energy of debate any more? Take for example a trivial task such as regularly sending a file via email, a simple administrative task. There was a day in which Java didn’t have Java mail and Perl didn’t have its elegant MIME::lite so the only way to so things was via bash and some mail client written in C. But today every language has mail client tools built into it, in fact so many of the modern languages like Python and PHP have so many solutions to the common problems worked out in them is there any point over personal preference?
Personally I still like to gut the compared implementations like a fish and have a look. Partly because these questions are fascinating to me, partly because I like to take the opportunity to wind up the religious proponents of each language, and partly because I prefer to gut code rather than fish. Your code might stink but never as much as fish guts.
There is one other reason that I like to perform comparative code autopsies and that is to get into the mind of the people it affects. So lets talk Python, the language of all things Gnome and Perl the language of the data miner. These two languages and their communities are very interesting. Our gnomes have only been, comparatively, around for a short period of time and tend to look at code for code’s sake. Yes python might be the language of GUIs and building blocks are important; but they’re not everything. On the other hand we have the gristled old data miners emerging from their day at the data face deep in their mine chiselling out useful information from the earthy data buried three fathoms deep in some odious data warehouse. They’re all about getting the task done, producing the goods for their team and finding the information that their users want. So there’s the stage a constructive pantheon of building block gnomes and the old data hounds seeking useful information.
Let’s set a comparison task now and see how their favourite tools stake up. But we have to be careful. If we choose a GUI task the miners will go on strike; alternatively if we select a data parsing task the gnomes will sit on their treasures and grump. Hence the task that I’ve chosen favours neither group, it’s stupidly simple and it is something any system administrator can relate to; creating a script to email a file. The rules of the game are:
- Each language must use its own mail tool set, no invoking mutt or any other such tool,
- For input the script must take all of the information: SMTP email server address, from address, to address, and fully qualified file name from the command line,
- A help text and minimal useful input validation along with error reporting must be included, but trivially,
- Only one file at a time is to be handled,
- Simple text in the body and subject must be hard coded in the script body with the exception of file name,
- The file must be in an attachment, not in-line Base64 and should be able to be of any type, not necessarily specified by MIME type,
- This task is stupidly simple, so so should the script be.
So to the breach! First up the Perl.
#!/usr/bin/perl
use MIME::Lite;
$address=$ARGV[0];
$from=$ARGV[1];
$to=$ARGV[2];
$path=$ARGV[3];
$usage='filemail.pl smtp from to path';
die "Usage: $usage\n" if $#ARGV != 3;
( $dir, $name ) = ($path=~m/(.*)[\\\/](.+)/) ? ( $1, $2 ) : ( undef, $path );
open FILE, $path or die "Could not find file '$path'";
close FILE;
$msg = MIME::Lite->new(
From => $from,
To => $to,
Cc => '',
Subject => "File '$name'.",
Data => "Please find file '$name' attached."
);
$msg->attach(
Type => 'application/octet-stream',
Path => $path,
Filename => $name,
Disposition => 'attachment'
);
MIME::Lite->send('smtp', $address, Timeout=>60);
$msg->send();
Next the Python.
#!/usr/bin/python
import os
import smtplib
import email
import email.mime.text
import email.mime.multipart
import optparse
# set up the command line parser and grab the args
parser = optparse.OptionParser(usage="""\
Send file attached to a MIME message.
Usage: %prog [options]
""")
parser.add_option('-a', '--address',
type='string', action='store', metavar='ADDRESS',
help="""Address of the SMTP server (required)""")
parser.add_option('-p', '--path',
type='string', action='store', metavar='PATH',
help="""Location of the file to send (required)""")
parser.add_option('-s', '--sender',
type='string', action='store', metavar='SENDER',
help='The value of the From: header (required)')
parser.add_option('-r', '--recipient',
type='string', action='append', metavar='RECIPIENT',
default=[], dest='recipients',
help='A To: header value (at least one required)')
opts, args = parser.parse_args()
# bomb with help if the required arguments are not present
if not opts.path \
or not opts.sender \
or not opts.recipients \
or not opts.address \
or not opts.path:
parser.print_help()
os.sys.exit(1)
if not os.path.isfile(opts.path):
print 'File \'' + opts.path + '\' not found.'
os.sys.exit(1)
# create the enclosing (msg) message
filename = opts.path.split('/')[-1]
msg = email.mime.multipart.MIMEMultipart()
msg['Subject'] = 'File \'' + filename + '\'.'
msg['To'] = ', '.join(opts.recipients)
msg['From'] = opts.sender
msg.preamble = 'Please find file \'' + filename + '\' attached.\n'
# message body
body = email.mime.multipart.MIMEMultipart('alternative')
body.attach(email.mime.text.MIMEText('Please find file \'' + filename + '\' attached.\n'))
body.attach(email.mime.text.MIMEText('Please find file \'' + filename + '\' attached.\n', 'html'))
msg.attach(body)
# attach the file
fp = open(opts.path, 'rb')
att = email.mime.base.MIMEBase('application', 'octet-stream')
att.set_payload(fp.read())
fp.close()
email.encoders.encode_base64(att)
att.add_header('Content-Disposition', 'attachment', filename=filename)
msg.attach(att)
# send the message
composed = msg.as_string()
s = smtplib.SMTP(opts.address)
s.sendmail(opts.sender, opts.recipients, composed)
s.quit()
Now for my taste there is no contest. I find myself in the company of wrinkled old miners covered in the dust of ages of data mining experience and dutiful determination. So why did I choose to use the tool of gnomes? Well several reasons really but first lets just make a couple of observations about the code. Neither good or bad, just observations. Then we’ll come back to the decision over choice and the ultimate answer to the ultimate question, is there really any difference of substance between the languages?
- For shear size Perl has the smaller amount of code.
- In detail Python is very revealing, it shows some of the nature and structure of the thing that it is building.
- Considering layout Perl provides us with more opportunities to lay things out for current and future reviewers.
- In terms of handiness Python provides a wonderful tool for handling standard command line parsing and help generation.
- Results from either implementation are near enough to identical where it counts.
Now the key here in this list is the last item, where it counts, in other words the results of running both of these scripts, is near enough to identical. The function for which these scripts was written has been achieved within the rules of the game. The only things that we are left debating are the non-functional aspects of the implementation, size, process description, navigation and flexibility. To me the elegance of the Perl solution simply trumps the Python implementation cold. But that’s personal preference based on ascetics. Python proponents will argue that the elegance of Perl is simply the result of a library which could quite easily have been created in Python. Which is true. But you see it isn’t. If it was it would have been. To expand; this is symptomatic of the perspective of the two different communities. Those playing with perls of wisdom know that scripting is all about getting things done, so they ask themselves what is the minimum amount of information that I can get away with providing to get the results that I want. Speed to an end. Focus on results. In the garden of code we find our gnomes building beautiful frameworks which can be used for all sorts of things, generalisation is the key, making it work for the world. Of course you can do mail with Python, but we would rather have a flexible solution for the masses than elegant solutions for the minorities.
To answer the question is there any point to using one language over another other than preference? No. However if you are sitting in a camp of gnomes armed to the teeth with pick axes then all hail the magnificent Python, but if you don’t want to get a shovel through your head after a day in the data mines polish your Perl. At the end of the day there is no real big difference between all of these languages, and if you are like me and sit in the camp with gnomes and miners just pick the one that means the least work for you. In my case it was just more expedient to deploy Python.
Python and pyinotify
A friend of mine posted a really good article over at blogspot about using pyinotify; after reading one of the comments there I started playing around with it. So if you want the original have a look at David Latham’s original post. I’ve simplified the code just a little to get to the essence of pyinotify. Incidentally if you’re working from Fedora like I am you might like to ‘yum install python-inotify*‘ and confirm that the module and documentation are installed. Here’s the base version.
#!/usr/bin/python
import os
import pyinotify
m = pyinotify.WatchManager()
notifier = pyinotify.Notifier(m, pyinotify.ProcessEvent())
m.add_watch('/tmp/', pyinotify.ALL_EVENTS, rec=True)
while True:
try:
notifier.process_events()
if notifier.check_events():
notifier.read_events()
except KeyboardInterrupt:
notifier.stop()
break
It works really well for this example but has one generally unwelcome side effect. It also blocks on the ‘notifier.check_events()‘ call. A blocking call is not really what you want in an event loop so it doesn’t really work too well for the core of the application. However this is overcome very simply by adding the three extra parameters defining the wait timing of the ‘pyinotify.Notifier(m, pyinotify.ProcessEvent(), 0, 0, 10)‘. In this case the block is turned to a 10 second wait courtesy of the last parameter as shown in the next code snippet below. In an application event loop you would use a value of 0 meaning that the call would not block at all.
#!/usr/bin/python
import os
import pyinotify
m = pyinotify.WatchManager()
notifier = pyinotify.Notifier(m, pyinotify.ProcessEvent(), 0, 0, 10)
m.add_watch('/tmp/', pyinotify.ALL_EVENTS, rec=True)
while True:
try:
notifier.process_events()
if notifier.check_events():
notifier.read_events()
except KeyboardInterrupt:
notifier.stop()
break
In this case the default event handler just prints XML formatted event lines to stdout. But there’s just one more unexpected side effect. You may find there are a few timing issues and how do you get the notification back to the main thread? But more on that later.
Simple Berkeley Sockets C Server Example
About Apache and Bloat
Every now and then I look at Apache and wonder about its size and complexity. Its a huge beastie now but absolutely brilliant too. I have quite a respect for it as do many around the world. But then I don’t really use very much of its functionality.
Sure its been re-factored into a modular form so that you only need to use the bits that you want but it still presents a mental preponderance thatmany would consider bloat. But bloat is a term used to describe software that has vast wads of unnessecary code that could be accomplished more simply. However Idon’t think that Apache has that. All of its code seems very purposeful, and as I indicated earlier it is reasonably easy to include or exclude as you will.
Introducing the Example
So this example is not here to show you the right way of doing things, but rather how simple the solution to you particular problem could be. In this case the example server simply serves up the time on the server machine to port 6543 for any client and then closes the stream. It can do this for many concurrent connections at once given the only complexity that is added to the server is client handling by forked process.
Now to the Code
/* -----------------------------------------------------------------------------
simple example of a server in C using very simple Berkeley sockets
This is just an example for looking at. Not really how you would actually
implement a server. For a start it does not take care of signals and isn't
capable of being implemented as a service.
----------------------------------------------------------------------------- */
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <strings.h>
#include <arpa/inet.h>
/* we're not going to use arguments for this example */
#define SERV_UDP_PORT 6543
#define SERV_TCP_PORT 6543
#define SERV_HOST_ADDR "192.168.2.200"
int main (int argc, char ** argv, char ** env) {
int socket_server, socket_client, child_pid;
struct sockaddr_in address_client, address_server;
socklen_t client_address_size;
FILE *stream_client;
time_t now;
struct tm *tm;
/* open a tcp socket to listen on */
if ( (socket_server = socket(AF_INET, SOCK_STREAM, 0) ) < 0)
err_dump("server: can't open stream socket");
/* bind to our local address ans start listening so that clients can find us */
bzero((void *) &address_server, (size_t)sizeof(address_server));
address_server.sin_family = AF_INET;
address_server.sin_addr.s_addr = htonl(INADDR_ANY);
address_server.sin_port = htons(SERV_TCP_PORT);
if (bind(
socket_server,
(struct sockaddr *) &address_server,
sizeof(address_server))
< 0)
err_dump("server: can't bind local address");
listen(socket_server, 5);
for ( ; ; ) {
/* wait for client connection and accpt it when it comes */
client_address_size = sizeof(address_client);
socket_client = accept(
socket_server,
(struct sockaddr *) &address_client,
&client_address_size);
if (socket_client < 0)
err_dump("server: accept error");
if ( (child_pid = fork() ) < 0)
err_dump("server: fork error");
/* specific client process handling starts here */
else if (child_pid == 0) {
/* client process has no need of the server socket */
close(socket_server);
/* generate a file stream from the client socket */
if ((stream_client = fdopen(socket_client, "w")) == NULL) {
perror("daytimed fdopen");
return 5;
}
/* write to the client stream as per normal then close it */
if ((now = time(NULL)) < 0) {
perror("daytimed time");
return 6;
}
tm = gmtime(&now);
fprintf(stream_client, "%.4i-%.2i-%.2iT%.2i:%.2i:%.2iZ\n",
tm->tm_year + 1900,
tm->tm_mon + 1,
tm->tm_mday,
tm->tm_hour,
tm->tm_min,
tm->tm_sec);
fclose(stream_client);
exit(EXIT_SUCCESS);
/* client process has left the building */
}
/* the server has no need to keep the client socket open */
close(socket_client);
}
return 0;
}
Do you have a Spring in your Wheel?
Spring is one of the frameworks of the moment but is it really a great framework or something entirely different? Most of the frameworks that I know of are fairly targeted and form a particular function. All of the good ones that I know of have a particular place in life; a frame work for doing some thing. Spring does not. But there’s gold in that there code…
Probably the most disappointing thing about Spring is its documentation. The entry barrier for integrating Spring with the rest of your tool set is quite high by its limited documentation. Yes there is a great volume of the stuff, but it appears to be limited to those who are already in ‘the know’. In other words if you really want to get a handle on Spring from scratch you have two choices: either buy a book or surf the web and hope.
But this is because we’ve been mislead. Struts is a framework, Spring is not. So a conclusive introduction to Struts is easy to write and easy to understand because it frames an idea. However as Spring does not frame a central idea and so writing an introduction to Spring is almost a lost cause from a framework point of view.
Spring is an SDK. A jolly brilliant SDK. When you look at the Spring development kit you find that it will work with what you have, doesn’t force a great monster of a framework on you and it riddled with simply brilliant concepts and patterns. As you start to delve into it you realise that the guys who put it together have brought together a great collection of conceptual gems. If you keep watching the SDK grow you will see more and more of them being added to the SDK as components for use in groups or by themselves.
Underlying all of these gems is a strong theme of patterns and approaches to design that pervades all corners of the SDK making it feel and look like a slick whole. These concepts include such things as the pervasive use of IOC accross the board to the modular adoption of specific patterns like MVC, which IS a framework, that are included in such a way as to let the designer include or exclude them in accordance with usefulness. In most ‘frameworks’ that I’ve seen its either all features or a difficult configuration to remove unwanted bloat.
So is Spring a case against bloat ware? No. Bloat ware is a term given to things that produce copious quantities of bulk which could easily be replaced by more stream lined solutions. When you see a product that makes you think ’surely that could be done with less’ you’re probably looking at bloat ware. Spring is not there for that purpose. You see almost everything it does belongs to the school of computing that is termed ‘middle ware’. Middle ware is almost by definition bloat.
However tempting to label Spring as bloat ware it does perform two very critical tasks often overlooked by those bloat ware vigilantes out there; it makes your software manageable. Everything that Spring does could be pared back to make it faster more efficient etc but you could barely dent it at the cost of making your software less easy to design, implement, install and operate. Carefully add Spring to your project and you are adding real value in terms of future proofing your product and saving yourself a ton of grief.
So if this is a SDK how does it compare to other SDK’s? To answer this question you need to think about perspective. If you look at the JDK, the standard C/C++ libraries, X windows xt or even Microsoft’s extensive .Net SDK you will find products that are built bottom up. That is products that have been built from a wide array of small useful components into a general smorgasbord of fancy components and knick knacks. Spring is built the other way around, it has been built from concepts to components. So when you look at Spring you see a large number of cohesive modules rather than the gravel and rocks of most other SDK’s. Effectively Spring takes the concepts and makes them real relying on other foundational tool kits to get the simple things in life done. Top down.
So Spring is not a framework, it is not a way of protecting you from bloat ware, it is an implementation of concepts, patterns and design built on the basics for you to cherry pick as you please and improve your product and its operation. Its an SDK like so many others; hardly a reinvention of an old wheel, but a new sporty set of spokes to use in building your own.
Simple PHP MVC
One of the much talked about patterns of the moment is the model-view-controller or model-view-presenter or any of the undoubtedly thousands of variants thereof. Here’s a deliberately really simple one to illustrate the pattern. If you can read it and understand it I hope its useful to you.
Interestingly a vague attempt at inversion of control, another design pattern which IMHO is more of a guide line, is in play here. It is done specifically to allow good component separation and useful test patterns. Hence as long as you have a class that fits the dependency obligations you can use it with the ones given here.
Now for the seriously crazy bit, the code on this page has not been tested to work. Its just here to give you some idea of what the model-view-controller pattern and how it could be implemented.
mvc.controller.php
Overview:
At the heart of each variant implementation of the model view controller pattern is the controller. A controller basically directs requests to the appropriate model or view processing functionality. In the case below it only makes the distinction between the two types and not the instances of functionality.
Some controllers combine a lot more but its not actually necessary and generally ends up making the controller very configuration and platform specific. By keeping configuration out of the controller we make it as general as it can be.
Notes:
Note that there is no exception handling here either. It is not wanted. Consider for a moment that if an exception was not caught, there would be no response to the client, which is a security requirement in many instances. However it there would be a log to trace to the point of failure because of the log entry call.
If exception handling were introduced it would compromise the generic nature of the controller as the controller would need to know what type of response to send to the client. For example if the error was thrown before the content of the request could be analysed and what data would you respond to the request with? HTML? XML? A media stream? A GIF? A JPG? The rule of thumb here is only respond if you know what language to talk in, if you don’t know the requested format you can’t even use clever configuration to determine how respond to a request.
Documented dependencies:
- class ‘Model’ : method ‘process_request’ : parameters ‘request to process’ : returns ‘new request for either model processing or view generation’
- class ‘View’ : method ‘process_request’ : parameters ‘request to process’ : returns ‘a view generated from the accumulated data in the request’
Undocumented dependencies:
- class ‘Log’ : method ‘writestr’ : parameters ’string to log’, ‘name of log level to record string at’ : returns ‘nothing’
- class ‘Log’ : method ‘writereq’ : parameters ‘request to log’, ‘name of log level to record string at’ : returns ‘nothing’
- class ‘Request’ : method ‘is_a_model_request’ : parameters ‘none’ : returns ‘true if the request is to be processed by the model’
Listing:
<?php
Log::getInstance()->writestr( "file:".__FILE__.":loaded", "debug" );
class Controller {
private $model;
private $view;
/*
* build this controller object
*/
public function __Construct ( $model, $view ) {
$this->model = $model;
$this->view = $view;
}
/*
* process a request
*/
public function process_request ( $request ) {
Log::getInstance()->writestr(
"class:".__CLASS__.":process_request", "debug" );
Log::getInstance()->writereq( $request, "debug" );
/*
* first process the model request
*/
$new_request = $this->model->process_request( $request );
/*
* if this request results in another model request
*/
if ( $new_request->is_a_model_request( ) )
return $this->process_request( $new_request );
/*
* otherwise generate view
*/
return $this->view->process_request( $new_request );
}
}
?>
mvc.model.php
Overview:
A model is responsible for the production of data for the view. Essentially it contains access points to the various methods of the underlying application called actions. In this case it simply has a list of actions that it locates and then executes the action sending the results back to the caller.
Notes:
As with the controller (see above) the model does little more than the basics leaving other classes that know what they are doing to handle the specifics of content. In line with this the model simply logs progress and fails silently.
Undocumented dependencies:
- class ‘Log’ : method ‘writestr’ : parameters ’string to log’, ‘name of log level to record string at’ : returns ‘nothing’
- class ‘Log’ : method ‘writereq’ : parameters ‘request to log’, ‘name of log level to record string at’ : returns ‘nothing’
- class ‘Actions’ : method ‘find_requested_action’ : parameters ‘request to locate action for’ : returns ‘an Action object’
- class ‘Action’ : method ‘process_request’ : parameters ‘request to process’ : returns ‘new request for either model processing or view generation’
Listing:
<?php
Log::getInstance()->writestr( "file:".__FILE__.":loaded", "debug" );
class Model {
private $actions;
/*
* build this model object
*/
public function __Construct ( $actions ) {
$this->actions = $actions;
}
/*
* process a request
*/
public function process_request ( $request ) {
Log::getInstance()->writestr(
"class:".__CLASS__.":process_request", "debug" );
Log::getInstance()->writereq( $request, "debug" );
/*
* find the action from the list of actions
*/
$action = $this->actions->find_requested_action( $request );
/*
* run the action and return the resulting request
*/
return $action->process_request( $request );
}
}
?>
mvc.view.php
Overview:
A view is called to format the data retrieved from the model into the format that the original client requested. Essentially it contains access points to the various view methods of the underlying application. In this case it simply has a list of views that it locates and then executes sending the results back to the caller.
Notes:
Now those sharp of eye will realise something; this looks pretty much like the model class above, and you’d be right. Apart from a a simple renaming of a few items the code is the same. In practice both the above model class and this view class would be implemented as a single class and simply configured differently. They are only provided in their separate formats here to highlight the model-view-controller form.
Undocumented dependencies:
- class ‘Log’ : method ‘writestr’ : parameters ’string to log’, ‘name of log level to record string at’ : returns ‘nothing’
- class ‘Log’ : method ‘writereq’ : parameters ‘request to log’, ‘name of log level to record string at’ : returns ‘nothing’
- class ‘Views’ : method ‘find_requested_view’ : parameters ‘request to locate view for’ : returns ‘an View object’
- class ‘View’ : method ‘process_request’ : parameters ‘request to process’ : returns ‘the final generated view’
Listing:
<?php
Log::getInstance()->writestr( "file:".__FILE__.":loaded", "debug" );
class View {
private $views;
/*
* build this view object
*/
public function __Construct ( $views ) {
$this->views = $views;
}
/*
* process a request
*/
public function process_request ( $request ) {
Log::getInstance()->writestr(
"class:".__CLASS__.":process_request", "debug" );
Log::getInstance()->writereq( $request, "debug" );
/*
* find the view from the list of views
*/
$view = $this->views->find_requested_view( $request );
/*
* process the request and return the resulting view
*/
return $view->process_request( $request );
}
}
?>
LDAP vs RDBMS is the war over?
Perhaps the question is better put, ‘Did the war even happen?’ What war you might well ask; well when different ways of looking at potentially the same data are viewed then there are bound to be zealots on either side fanning the flames of FUD on the opposition and extolling the vitues of their chosen creed. However although both RDBMS and LDAP are commonly used to host the same data for applications there appears to be little controversy.
Mostly, I think, there is no conflict because in general LDAP is not even an option for the vast masses of application developers out there. Especially as the bulk of application development appears to be done facing the Internet. Normally hosting environments provide a choice of RDBMS or RDBMS. LDAP is just too difficult a concept, and that’s probably the first reason that it has fallen out of contention for storing the masses of organisational data out there. Yet it is probably the best candidate that I know of for holding operational control of that data.
To illustrate this conceptual difficulty lets consider the contention between software developer and database architecht perspectives. Typically the developer does not really understand the various tools that the RDBMS offers to manage data. The database expert loaths the way that developers start implementing relational contraints in code instead of applying the appropriate design concepts of foriegn keys, triggers, stored procedures etc. I’ve seen this many times. Often I’ve seen the opposite where database folk have decided to embed code into their domain. To really get this right each party needs to understand the value that the other brings to the table. But I continually read laments from either side criticising the way that this or that should really be done over here and not there.
Which brings us into LDAP. This underused gem is typically used by the development community simply as a convienient pre-built authentication tool. Even when LDAP is used a lot of information is then stored in an RDBMS that it really does not do well; or at least as well as LDAP. I’ll spare you the history of why this is so and how it came to be but the fundamental difference between the two is that RDBMS generally is a collection of flat file tables that are related by loose rules; whereas the LDAP server is a tightly coupled hierarchy of objects (called the Directory Information Tree – DIT) similar in nature to that of other concepts like XML.
LDAP servers are generally built for high speed retrieval and mass replication focussed on the enterprise, where RDBMS is a general ledger store house, and pretty primative too. The kind of data then that you should store in the LDAP directory is structural data. Like say configuration data for a telephone system or contact information for a customer relationship management (CRM) type of application. Even the configuration and preferences information of Wordpress for this blog should really be stored in an LDAP server. But its not.
If you want to know if the data that you are looking at is suited to LDAP then consider.
- Is the data dynamic or relatively static?
- Does the data need to be distributed?
- Can the data be used by more than one application?
- Is the data multi-valued?
- Can your data or application take advantage of a hierarchical relationship?
- Do you need flexible security options?
- Do you need single sign-on?
- Do you need distributed or delegated administration capabilities?
If you can answer yes to some or all of these questions, then directories and directory-based applications would likely be useful to your application or project.
So why did I write this post? Basically it is a pointer into the world of LDAP for those who’ve not thought of it. A start. If you really want a better introduction go here. Have fun.