Posts Tagged code
A Brief Linux Sockets Connect Illustration
For a little while I’ve been playing with sockets in C now and have come up with the following succinct example of connecting. Note that the connection is fairly flexible with regards to protocol and transport type. It really is simply there to make the connection to somewhere else with as few questions asked. It will involve DNS and service lookups if you provide names instead of the network address and port. If you want to catch exceptions you need to clear the standard error variable and check it yourself after receiving a fail return value. It won’t, or at least shouldn’t, report any exceptions because for my current purposes failures are not really exceptions just dead ends on some of many paths. I tried to keep it simple too, have fun.
/* connect_to_socket
*
* connect to a socket using an initialised addrinfo structure,
*
* info is an initialised addrinfo structure
* sock is a pointer to the location to store the socket descriptor when opened
*
* returns 0 if successful
*/
int connect_to_socket(const struct addrinfo *info, int* sock) {
/* open socket */
*sock = socket(info->ai_family, info->ai_socktype, info->ai_protocol);
if (*sock < 1) return -1;
/* connect to the server*/
if (connect(*sock, info->ai_addr, info->ai_addrlen) == 0)
return 0;
/* clean up on failure */
close(*sock);
return -1;
}
/* connect_to_server
*
* connect to a server using the server name or adress and port or service name
*
* server is a string containing either the name or address of the server
* port is a string containing either the service name or port number to use
* sock is a pointer to the location to store the socket descriptor when opened
*
* returns 0 if successful
*/
int connect_to_server(const char* server, const char* port, int* sock) {
struct addrinfo hints, *info = NULL, *list = NULL;
int e = 0;
/* initialise the hints for retrieving the address details */
memset(&hints, 0, sizeof (hints));
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
hints.ai_flags = AI_PASSIVE;
hints.ai_protocol = 0;
/* get the address if the server to connect to */
e = getaddrinfo(server, port, &hints, &list);
if (e != 0) return -1;
/* connect to the first socket in the list */
for (info = list; info != NULL; info = info->ai_next)
if (connect_to_socket(info, sock) == 0)
break;
/* clean up and return */
if (list != NULL) freeaddrinfo(list);
if (sock > 0) return 0;
else return -1;
}
Apache Tomcat 6 and the HSM
Recently I’ve had some real fun with SafeNet HSM’s (Host Security Module or Hardware Security Module) and Tomcat 6 in the never ending process of PCIDSS compliance. People are probably going to line up to stone me after saying this but PCIDSS is a really great effort to make things safe and secure for all of us. In a lot of cases its the stick that drives organisations to adopt some good practices. One of which is the use of HSM’s for all things from the crypt.
Now this little rant won’t walk you through PCI compliance, rather its just about my experiences getting Safenet Eracom Orange or Blue HSM to work within Tomcat. Deployment is really easy:
- Install the latest version of Java,
- Install the Eracom Orange C runtime,
- Configure the Eracom Orange runtime to connect to the HSM,
- Configure the HSM with tokens, keys, certificates, users and whatever else you need,
- Install the Eracom Orange Java runtime,
- Install Apache Tomcat 6,
- Place the Eracom Orange ‘
jprov.jar‘ Tomcat’s server shared lib directory ‘CATALINE_HOME/lib‘, - Restart the Tomcat server if its already running or just start it if it is not,
- Deploy your webapp.
Apart from steps 7 and 8 simply follow the provider’s directions for all steps. The key in this thing is that the ‘jprov.jar‘ archive must be loaded when Tomcat starts. I imagine that it should be the same for other providers like for IBM’s toys.
Now for the development set up. Simply follow the same steps as above but install the SDK’s for the Eracom and include ‘jprov.jar’ as a library in your development environment. Typically your development environment will bundle the ‘jprov.jar‘ into the web application directory. It won’t do any harm there but to get it to work properly it will need to be in the Tomcat server’s common library directory as detailed above. In fact you don’t really need to install the SDK’s if you have an actual HSM to play with for development purposes, but the SDK’s do provide a software HSM simulator for those of us who don’t want the expense of buying one of these things for every developer on the floor. Plus the SDK’s will give you a good deal of documentation too.
So you are all set up and ready to build the next great crypt. Start by building a stand alone application to confirm that everything works before going any further. Lets get to the code:
/*
* Crypt.java
*
* A simple class to provide basic SDES encryption and decryption
*/
package crypt;
import java.io.IOException;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.Key;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.NoSuchProviderException;
import java.security.Security;
import java.security.UnrecoverableKeyException;
import java.security.cert.CertificateException;
import java.util.Properties;
import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.spec.IvParameterSpec;
import org.apache.commons.codec.DecoderException;
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.codec.binary.Hex;
import org.apache.commons.lang.StringUtils;
import org.apache.commons.logging.Log;
/**
*
* @author mc
*/
public class Crypt {
/**
* Creates a 3DES key cipher with the parameters in the config, valid
* modes are Cipher.ENCRYPT and Cipher.DECRYPT
*
* @param mode set the cipher to either encrypt or decrypt
* @param config the configuration parameters for this application
* @param log log to write debug output to
* @return a cipher that should be used then nulled to be thrown away
* @throws KeyStoreException
* @throws NoSuchProviderException
* @throws IOException
* @throws NoSuchAlgorithmException
* @throws CertificateException
* @throws UnrecoverableKeyException
* @throws NoSuchPaddingException
* @throws DecoderException
* @throws InvalidKeyException
* @throws InvalidAlgorithmParameterException
*/
private Cipher createCipher(int mode, Properties config, Log log)
throws
KeyStoreException,
NoSuchProviderException,
IOException,
NoSuchAlgorithmException,
CertificateException,
UnrecoverableKeyException,
NoSuchPaddingException,
DecoderException,
InvalidKeyException,
InvalidAlgorithmParameterException {
log.debug("Adding provider.");
// will gracefully not add provider if provider already loaded
Security.addProvider(
new au.com.eracom.crypto.provider.ERACOMProvider());
log.debug("Getting key store instance.");
KeyStore store = KeyStore.getInstance(
config.getProperty("crypt.keystore.type"),
config.getProperty("crypt.keystore.provider"));
log.debug("Loading key store.");
store.load(null, null);
log.debug("Retrieving key.");
Key key = store.getKey(
config.getProperty("crypt.keystore.entry"), null);
log.debug("Generating cipher.");
Cipher cipher = Cipher.getInstance(
config.getProperty("crypt.transformation"),
config.getProperty("crypt.provider"));
log.debug("Creating initialisation vector.");
IvParameterSpec i = new IvParameterSpec(Hex.decodeHex(
config.getProperty("crypt.vector").toCharArray()));
log.debug("Initialising cipher in the given mode.");
cipher.init(mode, key, i);
return cipher;
}
/**
* Takes a string and returns its equivalent Base64 encrypted string.
*
* @param data string to be encrypted
* @param config parameters for this application's encryption
* @param log log to write debug output to
* @return encrypted Base64 encoded form of the data string parameter
* @throws NoSuchAlgorithmException
* @throws NoSuchProviderException
* @throws NoSuchPaddingException
* @throws InvalidKeyException
* @throws IllegalBlockSizeException
* @throws BadPaddingException
* @throws InvalidAlgorithmParameterException
* @throws KeyStoreException
* @throws IOException
* @throws CertificateException
* @throws UnrecoverableKeyException
* @throws DecoderException
*/
public String b46Encrypt(String data, Properties config, Log log)
throws
NoSuchAlgorithmException,
NoSuchProviderException,
NoSuchPaddingException,
InvalidKeyException,
IllegalBlockSizeException,
BadPaddingException,
InvalidAlgorithmParameterException,
KeyStoreException,
IOException,
CertificateException,
UnrecoverableKeyException,
DecoderException {
log.debug("Encrypting data.");
byte[] encrypted =
createCipher(Cipher.ENCRYPT_MODE, config, log).doFinal(
data.getBytes());
log.debug("Base64 encoding data.");
String encoded = new String(Base64.encodeBase64(encrypted));
return encoded;
}
/**
* Takes a encrypted Base64 encoded string and returns its equivalent
* decrypted string.
*
* @param data string to be encrypted
* @param config parameters for this application's encryption
* @param log log to write debug output to
* @return encrypted Base64 encoded form of the data string parameter
* @throws NoSuchAlgorithmException
* @throws NoSuchProviderException
* @throws NoSuchPaddingException
* @throws InvalidKeyException
* @throws IllegalBlockSizeException
* @throws BadPaddingException
* @throws InvalidAlgorithmParameterException
* @throws KeyStoreException
* @throws IOException
* @throws CertificateException
* @throws UnrecoverableKeyException
* @throws DecoderException
*/
public String b64Decrypt(String data, Properties config, Log log)
throws
NoSuchAlgorithmException,
NoSuchProviderException,
NoSuchPaddingException,
InvalidKeyException,
IllegalBlockSizeException,
BadPaddingException,
InvalidAlgorithmParameterException,
KeyStoreException,
IOException,
CertificateException,
UnrecoverableKeyException,
DecoderException {
log.debug("Base64 decoding data.");
byte[] decoded = Base64.decodeBase64(data.getBytes());
log.debug("Decrypting data.");
byte[] decrypted =
createCipher(Cipher.DECRYPT_MODE, config, log).doFinal(decoded);
return new String(decrypted);
}
}
So that’s it, very straight forward but lets take time to look at one or two points of interest.
In a nutshell functionally this is a class that can be used to encrypt and decrypt data using stored 3DES keys. Note that the configuration information is all passed in via the properties parameter. Also note that the class uses base64 encoding because its easier to play with and store strings. Generally you would also want to replace unsafe base64 characters too if the storage unit required it.
On the encryption front there are a few things to note too. Take a look at the includes, as far as the crypt mechanisms it uses only the standard Java imports and practices. There is only one thing that is specifically from the Safenet SDK, the loading of the provider class in the following snippet:
// will gracefully not add provider if provider already loaded
Security.addProvider(
new au.com.eracom.crypto.provider.ERACOMProvider());
To complete the picture here is the main class and the configuration file to go with it. That’s the lot; three simple files and it should be all go to test your configuration.
/*
* main.java
*
* Test application for the Eracom encryption devices.
*/
package crypt;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Properties;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
/**
*
* @author mc
*/
public class Main {
/**
* @param args the command line arguments
*/
public static void main(String[] args) throws IOException {
Properties config = new Properties();
Log log = LogFactory.getLog(Main.class);
boolean encrypt = "encrypt".equalsIgnoreCase(args[0]);
config.loadFromXML(new FileInputStream(args[1]));
try {
if(encrypt)
System.out.println(
new Crypt().b46Encrypt(args[2], config, log));
else
System.out.println(
new Crypt().b64Decrypt(args[2], config, log));
} catch(Exception e) {
log.error(e.getLocalizedMessage(), e);
}
}
}
ERACOM au.com.eracom.crypto.provider.ERACOMProvider DESede/CBC/PKCS5Padding AAAAAAAA00000000 ERACOM CRYPTOKI KeyName
Once this is working it should be possible to use the same crypt class file in a Tomcat hosted web application with one slight adjustment. As the Tomcat service will load the provider you should not attempt to load the provider in your application. In fact if you try it will almost certainly fail as your application is unlikely, unless you have specifically configured it, will not have the privileges need to load classes. So just comment out the ‘Security.addProvider‘ call in the class.
So that’s it; all you needed to know to configure an HSM to start working with Tomcat. Simple wasn’t it? Now all you need to do is figure out how to use it to help you with PCIDSS and how to change the crypt file to suite your specific needs.
Simple Custom Event Processor for pyinotify
In my previous post on pyinotify I took a short look at the basic use of the pyinotify watcher using mostly default settings. Here I take a brief look at building a custom event processor. The event processor is called to retrieve the event data and, well, process it. The default processor simply prints the output to ‘stdout‘; mine will too. But it is just there to show how simple it is. Note that the processor actually processes the events in line so there’s no need for messy threading.
#!/usr/bin/python
import os
import pyinotify
from datetime import datetime
m = pyinotify.WatchManager()
notifier = pyinotify.Notifier(m, pyinotify.ProcessEvent(), 0, 0, 100)
m.add_watch('/tmp', pyinotify.ALL_EVENTS, rec=True)
while True:
print "START LOOP %s " % datetime.now()
try:
if notifier.check_events():
print "Result of notifier.check_events() = TRUE"
notifier.read_events()
print "Calling notifier.process_events()"
notifier.process_events()
except KeyboardInterrupt:
notifier.stop()
break
print "END LOOP %s " % datetime.now()
This is just the same code as we saw in the earlier post except that I’ve put the ‘notifier.process_events()‘ in its proper place so that it doesn’t get called in every loop execution. It is restricted by the ‘notifier.check_events()‘. To add a little info to the whole process I’ve included some print statements to illustrate the flow of execution. With this we get results that look like:
START LOOP 2009-05-07 23:02:25.025048 Result of notifier.check_events() = TRUE Calling notifier.process_events()END LOOP 2009-05-07 23:02:25.104546
Which is what you’d expect. However you will also get results that look like.
START LOOP 2009-05-07 23:02:18.808039 Result of notifier.check_events() = TRUE Calling notifier.process_events()END LOOP 2009-05-07 23:02:18.863855 START LOOP 2009-05-07 23:02:18.863912 Result of notifier.check_events() = TRUE Calling notifier.process_events() END LOOP 2009-05-07 23:02:18.864656 START LOOP 2009-05-07 23:02:18.864710 Result of notifier.check_events() = TRUE Calling notifier.process_events() END LOOP 2009-05-07 23:02:18.865415
So you can see that we get some reasonably unpredictable results.
Now let’s do something a little different and collect the event and print it ourselves a little differently. In this case we’re still collecting all events, we just wanted to mark up the output with something extra.
#!/usr/bin/python
import os
import pyinotify
from datetime import datetime
class e(pyinotify.ProcessEvent):
def process_default(self, event):
print(repr(event))
m = pyinotify.WatchManager()
notifier = pyinotify.Notifier(m, e(), 0, 0, 100)
m.add_watch('/tmp', pyinotify.ALL_EVENTS, rec=True)
while True:
print "START LOOP %s " % datetime.now()
try:
if notifier.check_events():
print "Result of notifier.check_events() = TRUE"
notifier.read_events()
print "Calling notifier.process_events()"
notifier.process_events()
except KeyboardInterrupt:
notifier.stop()
break
print "END LOOP %s " % datetime.now()
Easy huh? But there’s still a little more, there’s a problem! More on that later.
Simple Berkeley Sockets C Server Example
About Apache and Bloat
Every now and then I look at Apache and wonder about its size and complexity. Its a huge beastie now but absolutely brilliant too. I have quite a respect for it as do many around the world. But then I don’t really use very much of its functionality.
Sure its been re-factored into a modular form so that you only need to use the bits that you want but it still presents a mental preponderance thatmany would consider bloat. But bloat is a term used to describe software that has vast wads of unnessecary code that could be accomplished more simply. However Idon’t think that Apache has that. All of its code seems very purposeful, and as I indicated earlier it is reasonably easy to include or exclude as you will.
Introducing the Example
So this example is not here to show you the right way of doing things, but rather how simple the solution to you particular problem could be. In this case the example server simply serves up the time on the server machine to port 6543 for any client and then closes the stream. It can do this for many concurrent connections at once given the only complexity that is added to the server is client handling by forked process.
Now to the Code
/* -----------------------------------------------------------------------------
simple example of a server in C using very simple Berkeley sockets
This is just an example for looking at. Not really how you would actually
implement a server. For a start it does not take care of signals and isn't
capable of being implemented as a service.
----------------------------------------------------------------------------- */
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <strings.h>
#include <arpa/inet.h>
/* we're not going to use arguments for this example */
#define SERV_UDP_PORT 6543
#define SERV_TCP_PORT 6543
#define SERV_HOST_ADDR "192.168.2.200"
int main (int argc, char ** argv, char ** env) {
int socket_server, socket_client, child_pid;
struct sockaddr_in address_client, address_server;
socklen_t client_address_size;
FILE *stream_client;
time_t now;
struct tm *tm;
/* open a tcp socket to listen on */
if ( (socket_server = socket(AF_INET, SOCK_STREAM, 0) ) < 0)
err_dump("server: can't open stream socket");
/* bind to our local address ans start listening so that clients can find us */
bzero((void *) &address_server, (size_t)sizeof(address_server));
address_server.sin_family = AF_INET;
address_server.sin_addr.s_addr = htonl(INADDR_ANY);
address_server.sin_port = htons(SERV_TCP_PORT);
if (bind(
socket_server,
(struct sockaddr *) &address_server,
sizeof(address_server))
< 0)
err_dump("server: can't bind local address");
listen(socket_server, 5);
for ( ; ; ) {
/* wait for client connection and accpt it when it comes */
client_address_size = sizeof(address_client);
socket_client = accept(
socket_server,
(struct sockaddr *) &address_client,
&client_address_size);
if (socket_client < 0)
err_dump("server: accept error");
if ( (child_pid = fork() ) < 0)
err_dump("server: fork error");
/* specific client process handling starts here */
else if (child_pid == 0) {
/* client process has no need of the server socket */
close(socket_server);
/* generate a file stream from the client socket */
if ((stream_client = fdopen(socket_client, "w")) == NULL) {
perror("daytimed fdopen");
return 5;
}
/* write to the client stream as per normal then close it */
if ((now = time(NULL)) < 0) {
perror("daytimed time");
return 6;
}
tm = gmtime(&now);
fprintf(stream_client, "%.4i-%.2i-%.2iT%.2i:%.2i:%.2iZ\n",
tm->tm_year + 1900,
tm->tm_mon + 1,
tm->tm_mday,
tm->tm_hour,
tm->tm_min,
tm->tm_sec);
fclose(stream_client);
exit(EXIT_SUCCESS);
/* client process has left the building */
}
/* the server has no need to keep the client socket open */
close(socket_client);
}
return 0;
}
Batch Posting to a Website
Ok so this is mostly about me playing with the Google syntax highlighter. Its basically some code I rattled up to post data from a | seperated list of post_variable|post_value pairs. Read the code to find out more. Its not a real solution to anything if you want to run tests or batch stuff use Junit, Maven or Ant. But you can have fun with my limited toy if you like – but remember as always the code is GPL and will probably kill you so watch out!
/*
* Batch processor that takes | delimitied lines of fields in a file and posts
* them to an HTTP server printing out the message and response to std out.
*/
/*
* TODO:
* Could be extended to use a configuration file for inputs, logging and could
* bennefit from SSL and proxy configuration options.
*/
package batchpost;
import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.ProtocolException;
import java.net.URL;
import java.net.URLEncoder;
import java.util.logging.Level;
import java.util.logging.Logger;
/**
*
* @author mc
*/
public class batchpost {
/**
* General output device
*/
static private Logger log = Logger.getLogger(batchpost.class.getName());
/**
* main
*
* Main entry point.
*
* @param args the command line arguments
* args[0] - is the fully qualified name of the batch file to read
* args[1] - is the URL to post the transactions to
*/
public static void main(String[] args) {
/*
* TODO:
* I should validate arguments and do other things but I am skipping doing
* them because of time.
*/
processFile(args[0], args[1]);
}
/**
* processFile
*
* Takes a file apart one line at a time and hands processing off to a
* message processing method. Logs errors if the file is not found or there
* is an IO error while reading the file. Assumes that each line contains a
* complete message.
*
* @param url - path to the web server to send the messages to
* @param file - file to retrieve the message lines from
*/
static private void processFile(String url, String file) {
try {
String line;
BufferedReader br =
new BufferedReader(
new InputStreamReader(
new DataInputStream(
new FileInputStream(file))));
try {
while ((line = br.readLine()) != null) {
processMessage(url, line);
}
} catch (UnsupportedEncodingException ex) {
log.log(Level.SEVERE, "processFile - processing halted because it is " +
"not possible to encode the post string in UTF-8.", ex);
} catch (IOException ex) {
log.log(Level.SEVERE, "processFile - IO exception has halted " +
"processing of the file \"" + file + "\".", ex);
}
try {
br.close();
} catch (IOException ex) {
log.log(Level.SEVERE, "processFile - IO exception has interrupted " +
"closing of the file \"" + file + "\".", ex);
}
} catch (FileNotFoundException ex) {
log.log(Level.SEVERE, "processFile - unable to locate the file \"" +
file + "\".", ex);
}
}
/**
* processMessage
*
* Takes a line of | delimited message fields and posts them to a web server.
* Each line is composed of a list of field_name|field_value pairs such that
* the line looks like:
*
* name_1|value_1|name_2|value_2|...
*
* or
*
* name|bob charles|age|62|occupation|pro golfer
*
* The total number of | delimited fields must be a multiple of 2. The
* message fields must not contain a |.
*
* @param url - address of the web server to post this message to
* @param line - string containing a | delimited set of fields
* @throws java.io.UnsupportedEncodingException when the platform does not
* support utf-8
*/
static private void processMessage(String url, String line)
throws UnsupportedEncodingException {
String[] fields = line.split("\\|");
if (fields.length % 2 == 1) {
log.log(Level.SEVERE, "processMessage - unable to process the following" +
" message as it did not contain a multiple of 2 fields: message = " +
"\"" + line + "\"");
return;
}
String postData = createPostData(fields);
log.log(Level.INFO, "START OF DATA TO BE POSTED");
log.log(Level.INFO, postData);
log.log(Level.INFO, "END OF DATA TO BE POSTED");
String responseData;
responseData = doHTTPPost(url, postData);
log.log(Level.INFO, "START OF DATA RECEIVED FROM POST");
log.log(Level.INFO, responseData);
log.log(Level.INFO, "END OF DATA RECEIVED FROM POST");
}
/**
* createPostData
*
* Takes an array of field names and their values such that each two
* consecutive strings are a field name then value pair and prepares a POST
* data string for sending to a web server. Throws the encoding error as it
* is pointless to continue processing as this error is going to reccur for
* all post attempts if it takes place for one.
*
* @param fields - a string array of field name and value pairs.
* @return string containing the post data.
* @throws java.io.UnsupportedEncodingException
*/
static private String createPostData(String[] fields)
throws UnsupportedEncodingException {
StringBuffer buf = new StringBuffer();
String charEnc = "UTF-8";
try {
buf.append(URLEncoder.encode(fields[0], charEnc));
buf.append('=');
buf.append(URLEncoder.encode(fields[1], charEnc));
for (int i = 2; i < fields.length; i = i + 2) {
buf.append('&');
buf.append(URLEncoder.encode(fields[i], charEnc));
buf.append('=');
buf.append(URLEncoder.encode(fields[i + 1], charEnc));
}
return buf.toString();
} catch (UnsupportedEncodingException ex) {
log.log(Level.SEVERE, "createPostData - unable to encode the post data " +
"as " + charEnc + " is not available on this platform.", ex);
throw ex;
}
}
/**
* readAll
*
* Reads all of the input from the given stream and returns it in a byte array
* output stream, the stream contents can then be retrieved as a byte array or
* a string using either toByteArray or toString respectively.
*
* @param is - an open input stream to read the data from
*
* @return a byte array output stream that contains the data read from the is
* @throws IOException if there is an error reading from the input stream
*/
static private ByteArrayOutputStream readAll(InputStream is)
throws IOException {
ByteArrayOutputStream result = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int len = is.read(buf);
while (!(len < 0)) {
result.write(buf, 0, len);
len = is.read(buf);
}
return result;
}
/**
* doHTTPPost
*
* performs the https post operation of the given data to the given url
* returning the result in a byte array output stream, throws an exception if
* an error takes place while performing io operations, does not deal with
* proxy servers
*
* @param data - a post formatted data stream for sending to the url server
* @param url - the target to send the post to
*
* @return a data stream of the response from the server at the url
* @throws IOException on failure of any io operation
*/
static private String doHTTPPost(String url, String data) {
String result = "";
HttpURLConnection conn = null;
try {
// set up the url connection
conn = (HttpURLConnection) (new URL(url)).openConnection();
try {
// write out the data to the server
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.getOutputStream().write(data.getBytes());
System.out.println(conn.getResponseMessage());
// disconnect and recycle the connection, return the response data
result = readAll(conn.getInputStream()).toString();
} catch (ProtocolException ex) {
log.log(Level.SEVERE, "doHTTPPost - unable to marshal the required " +
"protocol to establish a connection to \"" + url + "\"", ex);
} catch (IOException ex) {
log.log(Level.SEVERE, "doHTTPPost - IO error while attempting to " +
"perform the data post operation.", ex);
} finally {
conn.disconnect();
}
} catch (MalformedURLException ex) {
log.log(Level.SEVERE, "doHTTPPost - the URL currently being attempted " +
"to connect with is malformed \"" + url + "\"", ex);
} catch (IOException ex) {
log.log(Level.SEVERE, "doHTTPPost - IO error while attempting to set up" +
" the data post operation.", ex);
} finally {
conn = null;
}
return result;
}
}
Driven to Reinvent
Its nice to have a standards based platform and in that vein C certainly has progressed a long way. But you still get the odd system that isn’t either set correctly up by default with my favorite extensions; namely all things GNU! Yes I admit to liking non-standard well coded extensions, in fact who would work without them these days except under a very specific set of requirements or academic interest. Things just would not get done so quickly without them.
So I got a Mac. Yep its sooo cool that when my 5 year old daughter walked in the room and saw it for the first time she exclaimed ‘Daddy! That’s sooo cool!’ Kind of an ego boost for someone who gave up on cool and cool things 20 years ago. However cool comes at a price, its got great development tools but I want cross platform development to run on my Linux boxen too. Long story short, it doesn’t have the standard GNU extensions so I had two choices, fart around with the environment or rewrite one or two functions myself.
So here’s the goods. Now because I don’t want people to just copy and paste my material without thinking about it (I don’t mind the copying I just like people to put in some effort and think) I’ve copied an earlier version of the functions which works for some situations but not for others. Its close tot he real deal but there are about 5 significant issues which would make it dangerous to use this code without doing something about them.
Here it is, have fun, have a play! (Note: the whole thing is under GPL 3, comments from GNU)
/*
ssize_t getdelim (char **lineptr, size_t *n, int delimiter, FILE *stream)
This function is like getline except that the character which tells it to stop
reading is not necessarily newline. The argument delimiter specifies the
delimiter character; getdelim keeps reading until it sees that character (or end
of file).
The text is stored in lineptr, including the delimiter character and a
terminating null. Like getline, getdelim makes lineptr bigger if it isn't big
enough.
getline is in fact implemented in terms of getdelim
*/
ssize_t getdelim( char **lineptr, size_t *n, int delimiter, FILE *stream )
{
/* setup the environment */
int c = getc( stream );
long i = 0;
char *a = NULL;
/* count the characters needed for the buffer */
while ( c != delimiter && c != EOF ) {
c = getc( stream );
i++;
}
/* test for and arrange the buffer size */
if ( i == 0 )
return -1;
if ( fseek( stream, -i, SEEK_CUR ) )
return -2;
if ( *n < i + 1 ) {
a = ( char * )realloc( *lineptr, i + 1 );
if ( a == NULL )
return -3;
*lineptr = a;
*n = i;
}
/* read the data into the buffer */
if ( fread( *lineptr, sizeof( char ), i, stream ) != i )
return -4;
( *lineptr )[i] = 0;
/* return the number of chars read */
return i;
}
/*
ssize_t getline (char **lineptr, size_t *n, FILE *stream)
This function reads an entire line from stream, storing the text (including the
newline and a terminating null character) in a buffer and storing the buffer
address in *lineptr.
Before calling getline, you should place in *lineptr the address of a buffer *n
bytes long, allocated with malloc. If this buffer is long enough to hold the
line, getline stores the line in this buffer. Otherwise, getline makes the
buffer bigger using realloc, storing the new buffer address back in *lineptr and
the increased size back in *n. See Unconstrained Allocation.
If you set *lineptr to a null pointer, and *n to zero, before the call, then
getline allocates the initial buffer for you by calling malloc.
In either case, when getline returns, *lineptr is a char * which points to the
text of the line.
When getline is successful, it returns the number of characters read (including
the newline, but not including the terminating null). This value enables you to
distinguish null characters that are part of the line from the null character
inserted as a terminator.
This function is a GNU extension, but it is the recommended way to read lines
from a stream. The alternative standard functions are unreliable.
If an error occurs or end of file is reached without any bytes read, getline
returns -1.
*/
ssize_t getline (char **lineptr, size_t *n, FILE *stream)
{
return getdelim ( lineptr, n, '\n', stream );
}
Generating Combinations
One of the tasks of numerical analysis is to search for or compare patterns. In the deep past this was quite a problem as it was easy to swamp the systems available with data making it more practical to employ a mathematician to perform some arcane analysis via algebra. However the manual way did not give the code opportunity to be tested against all possible combinations. Normally tests were conductd against boundaries and samples of statisitically significant combinations.
However in these days of Gig’s of memory and fast processors not to mention the terrabytes of disk space available cheaply it is possible to generate and test every possible combination in many cases. One of these is the classic combination space of mathematics. This following alogorithm is a crude base to progress from to generate such a number space.
// pattern generation
void genPat(size_t smpSize, size_t popSize, vector cmbSpace) {
t_combo v(smpSize); // entry vector
t_combo m(smpSize); // entry limit vector
int i, e, a; // index and place holders
// initialise the entry vector with an ascending order 1 to smpSize
// and the limit vector with the max value each entry should reach
for (i = 0; i < smpSize; i++)
{
v[i] = i + 1;
m[i] = popSize - smpSize + v[i];
}
// repeat until the first entry in the vector is less than its
// maximum value
while (v[0] < m[0])
{
// update the combination space
cmbSpace.push_back(v);
// set an index to the last element
e = smpSize - 1;
// find the first element from the right to be less than the limit
while (v[e] == m[e]) e--;
// add one to and store the value of the current element in a var
a = ++v[e];
// set each element to the right to one more than the current
while (e < smpSize - 1) v[++e] = ++a;
}
cmbSpace.push_back(v);
}
Playing with Valarray
In the past few weeks I’ve been looking at the C++ STL numeric container valarray. Its kind of like that forgotten ternary operator : ? in C (yes I know lots of people use it but most just use it for Shiek appeal) and there are quite a few ancient posts out there that have faded into the mists of time. Prime conjecture says its because valarray, like the ternary operator, is really a niche item seemingly randomly placed in a generalised library. So lets take a look at it.
My wanderings started with a database where I was trying to do numeric analysis. Not normally a bad thing when you have a very simple task, sum this row, add that column to that thingy over there. But even a simple task like a pivot table starts taking on a whole complex life of its own; and you can just forget brute force function sequence processing using sliding windows (generating a sequence from functions over subsequences of another). Believe it or not I got this working using MySQL but it was pure nightmare territory. Hence I started looking at coding it up by hand.
First thought was to use a library that someone else had written. But it soon became obvious that most of the BLAS libraries were written by boffins dedicated to their own arcane mathematical religious systems and not really understandable to someone with only a second year University course of linear algebra under his belt. Others were too simple and lacked the power of expression that was needed. What was really painfully obvious was that levels of abstraction were, in most of the examples out there quite an abstraction from the main goals of anyone using them, speed and comprehension. There’s a rule in security that, in my paraphrase, says ‘complexity is where the issues hide’. KISS
So on to the C and C++ libraries the search went at last there was something to use, valarray. Now valarray is NOT the fastest thing in code. I read lots of complaints on the Interweb about newbies who claim to make faster code with their hand coded C. They’d be right. Its blindingly simple to write faster code. However there are a couple of things to consider before throwing in the object-oriented towel and carving your own C. Firstly, and not very helpfully, valarray is an incomplete additive to a general purpose library. Its inventors were really looking to take advantage of vector processors, not something you see much of today – or so you think. So its intentions may not fit your needs. Next the valarray is slower than your carefully carved C but it is already there for you and it works faster than a vector; the C++ programmers back stop. Lastly coding to a standard helps because there are so many people who have been there before with it its probably, little though it might be, a lot better thought out than what you may come up with in the time that you have. Still its horses for courses so don’t be afraid to ditch it if its not right for you.
Here’s a little code to get stuck into:
#include <algorithm>
#include <iostream>
#include <valarray>
#include <vector>
using namespace std;
/*
* import a valarray from a whitespace seperated stream of numbers
*/
typedef istream_iterator<int> int_istrm_iter;
void importTextArray( istream & is, valarray<int> & va ) {
vector<int> v;
if (!is.good() || is.eof())
return;
copy( int_istrm_iter( is ), int_istrm_iter(), back_inserter( v ) );
va.resize( v.size(), sizeof( int ) );
copy( v.begin(), v.end(), &va[0] );
}
Hopefully you should be able to work out that it imports a file full of white space separated numbers into a vector and then sizes a valarray up for the data and copies it in. Naturally the first parameter is a file or stringbuffer or whatever that contains the data. Once you’ve done this you’ve got something to play with. So let’s play.
Of course a single dimension like valarray isn’t really that much use. What we want to do is be able to do mad things with matricies of n dimensions and repeatedly alter them till we’re happy. A little bit of what-if analysis if you please. There are a number of ways to do this and they involve the slice and gslice classes. You can work that out for yourself, its in most references on the topic. So lets do something a little more fun, lets stick two matricies together side by side.
/* * creates a new matrix by appending one matrix to the side of the other, note * that the three matrcies have the same row count r * * for example: * * matrix a matrix b matrix c * 01 02 03 04 01 02 03 01 02 03 04 01 02 03 * 05 06 07 08 + 04 05 06 = 05 06 07 08 04 05 06 * 09 10 11 12 07 08 09 09 10 11 12 07 08 09 */
void appendMatRows( const int height, const valarray<int> &first,
const valarray<int> &second, valarray<int> &result ) {
size_t firstwidth = first.size() / height;
size_t secondwidth = second.size() / height;
size_t resultwidth = firstwidth + secondwidth;
// make the result matrix the right size
result.resize( first.size() + second.size() );
// use a gslice to get a valarray for inserting the first matrix
size_t x1[] = { height, firstwidth }; // shape of extracted array
size_t s1[] = { resultwidth, 1 }; // parent row length and row item distance
valarray<size_t> xa1( x1, 2 );
valarray<size_t> sa1( s1, 2 );
result[( const gslice )gslice( 0, xa1, sa1 )] = first;
// then repeat with the second matrix
size_t x2[] = { height, secondwidth };
size_t s2[] = { resultwidth, 1 };
valarray<size_t> xa2( x2, 2 );
valarray<size_t> sa2( s2, 2 );
result[( const gslice )gslice( firstwidth, xa2, sa2 )] = second;
}
So that’s about it for now. I guess the thing about valarray for me is that there is a good base to work from which, at a little cost, saves me some time and pain in hand coding my own. It still needs a lot of basic work to do some of the things that are needed to play with but its pretty good for what my requirements are. Incidentally MySQL goes away for a while when performing these sorts of operations; whereas the equivalent coded version is uicker to develop, clearer to debug and over a thousand row matrix seems to process with out even bothering the CPU. That’s a great place to start.