Code Snips

One of the more fascinating areas of modern computing is that of parallel and concurrent (PAC) computing.  Of course it has only been in the last couple of years that I’ve had access to the equipment and environment in which to play.  However apart from the obvious droll exercises there was still the lack of a suitable fodder to use for demonstrating anything more than a contrived task.

So along came Sudoku.  Its a perfect example of a problem set that can be broken down in multiple ways.  More than that it is really simple and also flexible enough to investigate the intricacies of PAC.  Naturally it is a very small problem.  Most often the basic game takes place within an 81 (9×9) cell square board.  However it can be scaled up to any square number of cells.  So its a scalable problem too.

Now at this point the place to start is with the analysis of the game itself.  So for the rest of this article a simple method for solving trivial Sudoku puzzles will be developed.  For this series of articles only simple puzzles will be used to simplify the code used to investigate PAC.

Introducing the Game

Sudoku is played on a on a board made up of a grid of cells that can contain a number from 1 to 9.  For this example we will limit it to 9×9 or 81 cells but it is possible to scale up.  Logically then the board is also composed of 9 non overlapping squares of 3×3 cells each.  An empty board looks something like the following.

Starting with a set of known values in specific cells determine the values in the remaining cells currently of unknown value.  There are only two real rules:

• A number can only appear once in any given row or column, and,
• A number can only appear once in any given square.

Given these rules the empty grid has the following possibilities:

Which is all very simple.  Actually, as with many number games and puzzles, the whole puzzle is very simple.  Where it becomes a game is when it taxes the discipline and observation of the puzzler.  To assist with terminology as we get nested items there are three terms that I will attempt to be consistent with:

• Board shall mean the whole 9×9 board,
• Square shall mean a 3×3 square within the board, and
• Cell shall mean the smallest part of the board which can only contain a single value.

Puzzle Play

So the first thing that is provided is a grid with a number of values already provided on it.

Which we use to set the known cells in our grid of possibilities.

Now for each value we can remove all of the same possibilities in the square that it sits in excluding the known cell.  Observe that for each cell of a square where the ultimate value is yet to be determined they all have the same set of possibilities within that square.  This is not important for this article but may be referenced in a future article.

Next for each known value we can remove all of the same possibilities in the row and column that the known cell is in except of course for the known cell itself.

At this point a quick review of the possibilities reveals that there are some:

• Squares where a possibility only occurs in one cell, and
• Cells that only have one possibility.

This means that those are the know values for those cells and the process can start again: setting the cells, clearing the possibility from the squares, and clearing the possibility from the lines.  In the example this means that the following cells are now known (shown in yellowish).

Repeating the steps to reduce the possibilities in the the possibility grid we get the following.

If you look closely at the grid of possibilities while you apply the reduction rules you will see that more values become known before you have completely applied the reduction steps.  For example I observed the 5 in the centre bottom square before completing the reduction steps.

While these revelations are distracting, enticing the observer to pursue the new value, it is important to note that if you apply the steps sequentially looping through them all of these will be picked up in time.  The reason that this is important is to demonstrate that this whole process of reduction can be performed sequentially in a single threaded fashion before investigating the use of a PAC approach.

Closing the Loop

In time the loop will naturally halt because either all of the values have been discovered or because no further single values are found in an iteration of the reduction loop.  Because the identifying logic is very simple this will only satisfy very basic Sudoku puzzles.  It could be simply extended to include more complicated cases but for our example this would potentially introduce obscuring factors to the overall purpose for this investigation, which is to explore PAC.

For our example used in this post the loop eventually discovers all of the known values as shown in the probability grid.

Giving us the completed puzzle.

Next Time

In this article we simply looked at the mechanics of solving a simple Sudoku puzzle.  It gave us the basic procedure for solving the puzzle:

1. Identify values,
2. Reduce possibilities,
3. Loop until no new values are found in one iteration.

Also presented was the idea of using a possibilities table for the reduction process.  This will become important in helping keep all of the PAC threads singing from the same hymn sheet.

In the next article we will look at presenting the basic data structures and getting a single threaded, non-PAC, version of this process up and running.  With any luck it should prove my hand work above!

Recently I wrote about how to use OpenSSL to connect to a plain data server, this time I’m modifying the same code to perform encrypted connections.  Naturally this is more of an example for how to use the API than production ready code.  It’s main purpose is to show the very small difference between using the library as I did last time and how that example can be altered to create a basic SSL client.

The essential changes to the code below are the replacement of the connection function ‘connect_unencrypted(host_and_port)‘ with ‘connect_encrypted(host_and_port, store_path, store_type, &ctx, &ssl)‘ and the introduction of the SSL cleanup step ‘SSL_CTX_free(ctx)‘.  All other changes are purely cosmetic; which really shows how simple adding SSL to your application connections can be.  Externally you need to provide the root CA certificate for the connection to be verified by.  That’s it.

At this point I could warble through the connection function, but you should just read through it yourself and consult the SSL man pages.  Note that there is a dreadful buffer overflow possibility in this code and no real error handling, just a bit of logging.  This is to keep the example short and also because only you will know what valid handling should take place for each situation when you write your own code.

So take a look and enjoy.  To try this out yourself:

1. Make sure that you have Firefox, GCC and OpenSSL (development sources and libraries) installed.
2. Copy the following code to a file called ‘main.c‘ in a directory that you will be playing around in.
3. Compile the code using ‘gcc main.c -o sslclient -lssl‘ if you are on Linux or ‘gcc main.c -o sslclient -lssl-lcrypto‘ if you are on OSX.
4. Select an SSL (https) web site to connect to and find the Root CA’s certificate name in the site’s certificate.
5. Either export the appropriate root CA from Firefox or obtain it directly from the CA online in pem format and copy it to a file ‘certificate.pem‘ in the same directory as the ‘sslclient‘ file.
6. Run the following command:

'./sslclient servername:443 "GET / \r\n\r\n" certificate.pem f e'

/*
 * File:   main.cpp
 *
 * Licence: GPL2
 */

/* Standard headers */
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

/* OpenSSL headers */
#include <openssl/bio.h>
#include <openssl/ssl.h>
#include <openssl/err.h>

/**
 * Simple log function
 */
void slog(char* message) {
    fprintf(stdout, message);
}

/**
 * Print SSL error details
 */
void print_ssl_error(char* message, FILE* out) {

    fprintf(out, message);
    fprintf(out, "Error: %s\n", ERR_reason_error_string(ERR_get_error()));
    fprintf(out, "%s\n", ERR_error_string(ERR_get_error(), NULL));
    ERR_print_errors_fp(out);
}

/**
 * Print SSL error details with inserted content
 */
void print_ssl_error_2(char* message, char* content, FILE* out) {

    fprintf(out, message, content);
    fprintf(out, "Error: %s\n", ERR_reason_error_string(ERR_get_error()));
    fprintf(out, "%s\n", ERR_error_string(ERR_get_error(), NULL));
    ERR_print_errors_fp(out);
}

/**
 * Initialise OpenSSL
 */
void init_openssl() {

    /* call the standard SSL init functions */
    SSL_load_error_strings();
    SSL_library_init();
    ERR_load_BIO_strings();
    OpenSSL_add_all_algorithms();

    /* seed the random number system - only really nessecary for systems without '/dev/random' */
    /* RAND_add(?,?,?); need to work out a cryptographically significant way of generating the seed */
}

/**
 * Close an unencrypted connection gracefully
 */
int close_connection(BIO* bio) {

    int r = 0;

    r = BIO_free(bio);
    if (r == 0) {
        /* Error unable to free BIO */
    }

    return r;
}

/**
 * Connect to a host using an unencrypted stream
 */
BIO* connect_unencrypted(char* host_and_port) {

    BIO* bio = NULL;

    /* Create a new connection */
    bio = BIO_new_connect(host_and_port);
    if (bio == NULL) {

        print_ssl_error("Unable to create a new unencrypted BIO object.\n", stdout);
        return NULL;
    }

    /* Verify successful connection */
    if (BIO_do_connect(bio) != 1) {

        print_ssl_error("Unable to connect unencrypted.\n", stdout);
        close_connection(bio);
        return NULL;
    }

    return bio;
}

/**
 * Connect to a host using an encrypted stream
 */
BIO* connect_encrypted(char* host_and_port, char* store_path, char store_type, SSL_CTX** ctx, SSL** ssl) {

    BIO* bio = NULL;
    int r = 0;

    /* Set up the SSL pointers */
    *ctx = SSL_CTX_new(SSLv23_client_method());
    *ssl = NULL;

    /* Load the trust store from the pem location in argv[2] */
    if (store_type == 'f')
        r = SSL_CTX_load_verify_locations(*ctx, store_path, NULL);
    else
        r = SSL_CTX_load_verify_locations(*ctx, NULL, store_path);
    if (r == 0) {

        print_ssl_error_2("Unable to load the trust store from %s.\n", store_path, stdout);
        return NULL;
    }

    /* Setting up the BIO SSL object */
    bio = BIO_new_ssl_connect(*ctx);
    BIO_get_ssl(bio, ssl);
    if (!(*ssl)) {

        print_ssl_error("Unable to allocate SSL pointer.\n", stdout);
        return NULL;
    }
    SSL_set_mode(*ssl, SSL_MODE_AUTO_RETRY);

    /* Attempt to connect */
    BIO_set_conn_hostname(bio, host_and_port);

    /* Verify the connection opened and perform the handshake */
    if (BIO_do_connect(bio) < 1) {

        print_ssl_error_2("Unable to connect BIO.%s\n", host_and_port, stdout);
        return NULL;
    }

    if (SSL_get_verify_result(*ssl) != X509_V_OK) {

        print_ssl_error("Unable to verify connection result.\n", stdout);
    }

    return bio;
}

/**
 * Read a from a stream and handle restarts if nessecary
 */
ssize_t read_from_stream(BIO* bio, char* buffer, ssize_t length) {

    ssize_t r = -1;

    while (r < 0) {

        r = BIO_read(bio, buffer, length);
        if (r == 0) {

            print_ssl_error("Reached the end of the data stream.\n", stdout);
            continue;

        } else if (r < 0) {

            if (!BIO_should_retry(bio)) {

                print_ssl_error("BIO_read should retry test failed.\n", stdout);
                continue;
            }

            /* It would be prudent to check the reason for the retry and handle
             * it appropriately here */
        }

    };

    return r;
}

/**
 * Write to a stream and handle restarts if nessecary
 */
int write_to_stream(BIO* bio, char* buffer, ssize_t length) {

    ssize_t r = -1;

    while (r < 0) {

        r = BIO_write(bio, buffer, length);
        if (r <= 0) {

            if (!BIO_should_retry(bio)) {

                print_ssl_error("BIO_read should retry test failed.\n", stdout);
                continue;
            }

            /* It would be prudent to check the reason for the retry and handle
             * it appropriately here */
        }

    }

    return r;
}

/**
 * Main SSL demonstration code entry point
 */
int main(int argc, char** argv) {

    char* host_and_port = argv[1]; /* localhost:4422 */
    char* server_request = argv[2]; /* "GET / \r\n\r\n" */
    char* store_path = argv[3]; /* /home/user/projects/sslclient/certificate.pem */
    char store_type = argv[4][0]; /* f = file, anything else is a directory structure */
    char connection_type = argv[5][0]; /* e = encrypted, anything else is unencrypted */

    char buffer[4096];
    buffer[0] = 0;

    BIO* bio;
    SSL_CTX* ctx = NULL;
    SSL* ssl = NULL;

    /* initilise the OpenSSL library */
    init_openssl();

    /* encrypted link */
    if (connection_type == 'e') {

        if ((bio = connect_encrypted(host_and_port, store_path, store_type, &ctx, &ssl)) == NULL)
            return (EXIT_FAILURE);
    }
        /* unencrypted link */
    else if ((bio = connect_unencrypted(host_and_port)) == NULL)
        return (EXIT_FAILURE);

    write_to_stream(bio, server_request, strlen(server_request));
    read_from_stream(bio, buffer, 4096);
    printf("%s\r\n", buffer);

    if (close_connection(bio) == 0)
        return (EXIT_FAILURE);

    /* clean up the SSL context resources for the encrypted link */
    if (connection_type == 'e')
        SSL_CTX_free(ctx);

    return (EXIT_SUCCESS);
}

Strange as it seems but you can make non-SSL connections using the OpenSSL libraries.  The following code is a brief example.  I like it because it appears to simplify (arguably) the rather more involved Berkley sockets.  It still lets you get at the socket if you want to mangle it yourself.  But ultimately its kind of handy to have a library that does SSL and plain transfers without too much of a code change.

Here is a simple example of using the OpenSSL BIO interface to retrieve a root web page.  Naturally most of the exception handling is not included so don’t going using it for production purposes as is.  Not that it’d be a lot of use as anything other than an exaple of the OpenSSL BIO API usage.

Have fun.

/*
 * File:   main.cpp
 *
 * Licence: GPL2
 */

/* Standard headers */
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

/* OpenSSL headers */
#include <openssl/bio.h>
#include <openssl/ssl.h>
#include <openssl/err.h>

/**
 * Initialise OpenSSL
 */
void init_openssl() {

    SSL_load_error_strings();
    ERR_load_BIO_strings();
    OpenSSL_add_all_algorithms();
}

/**
 * Close a BIO connection gracefully
 */
int close_connection(BIO* bio) {

    int r = 0;

    r = BIO_free(bio);
    if (r == 0) {
        /* Error unable to free BIO */
    }

    return r;
}

/**
 * Connect to a host using an unencrypted stream
 */
BIO* connect_unencrypted(char* host_and_port) {

    BIO* bio = NULL;

    /* Create a new connection */
    bio = BIO_new_connect(host_and_port);
    if (bio == NULL) {

        printf("Error: %s\n", ERR_reason_error_string(ERR_get_error()));
        printf("%s\n", ERR_error_string(ERR_get_error(), NULL));

        return NULL;
    }

    /* Verify successful connection */
    if (BIO_do_connect(bio) != 1) {

        printf("Error: %s\n", ERR_reason_error_string(ERR_get_error()));
        printf("%s\n", ERR_error_string(ERR_get_error(), NULL));

        close_connection(bio);
        return NULL;
    }

    return bio;
}

/**
 * Read a from a stream and handle restarts if nessecary
 */
ssize_t read_from_stream(BIO* bio, char* buffer, ssize_t length) {

    ssize_t r = -1;

    while (r < 0) {

        r = BIO_read(bio, buffer, length);
        if (r == 0) {

            /* Handle closed connection */
            continue;

        } else if (r < 0) {

            if (!BIO_should_retry(bio)) {

                /* Handle failed read here */

                printf("Error: %s\n", ERR_reason_error_string(ERR_get_error()));
                printf("%s\n", ERR_error_string(ERR_get_error(), NULL));

                continue;
            }

            /* It would be prudent to check the reason for the retry and handle
             * it appropriately here */
        }

    };

    return r;
}

/**
 * Write to a stream and handle restarts if nessecary
 */
int write_to_stream(BIO* bio, char* buffer, ssize_t length) {

    ssize_t r = -1;

    while (r < 0) {

        r = BIO_write(bio, buffer, length);
        if (r <= 0) {

            if (!BIO_should_retry(bio)) {

                /* Handle failed write here */
                printf("Error: %s\n", ERR_reason_error_string(ERR_get_error()));
                printf("%s\n", ERR_error_string(ERR_get_error(), NULL));

                continue;
            }

            /* It would be prudent to check the reason for the retry and handle
             * it appropriately here */
        }

    }

    return r;
}

/**
 * Main SSL demonstration code entry point
 */
int main(int argc, char** argv) {

    BIO * bio;
    char *addr = "GET / HTTP/1.1\r\nHost: www.home.com\r\n\r\n";
    char buffer[4096];
    buffer[0] = 0;

    init_openssl();

    if ((bio = connect_unencrypted(argv[1])) == NULL)
        return (EXIT_FAILURE);

    write_to_stream(bio, addr, strlen(addr));
    read_from_stream(bio, buffer, 4096);
    printf("%s", buffer);

    if (close_connection(bio) == 0)
        return (EXIT_FAILURE);

    return (EXIT_SUCCESS);
}

For a little while I’ve been playing with sockets in C now and have come up with the following succinct example of connecting.  Note that the connection is fairly flexible with regards to protocol and transport type.  It really is simply there to make the connection to somewhere else with as few questions asked.  It will involve DNS and service lookups if you provide names instead of the network address and port.  If you want to catch exceptions you need to clear the standard error variable and check it yourself after receiving a fail return value.  It won’t, or at least shouldn’t, report any exceptions because for my current purposes failures are not really exceptions just dead ends on some of many paths.  I tried to keep it simple too, have fun.

/* connect_to_socket
 *
 * connect to a socket using an initialised addrinfo structure,
 *
 * info is an initialised addrinfo structure
 * sock is a pointer to the location to store the socket descriptor when opened
 *
 * returns 0 if successful
 */
int connect_to_socket(const struct addrinfo *info, int* sock) {

    /* open socket */
    *sock = socket(info->ai_family, info->ai_socktype, info->ai_protocol);
    if (*sock < 1) return -1;

    /* connect to the server*/
    if (connect(*sock, info->ai_addr, info->ai_addrlen) == 0)
        return 0;

    /* clean up on failure */
    close(*sock);
    return -1;
}

/* connect_to_server
 *
 * connect to a server using the server name or adress and port or service name
 *
 * server is a string containing either the name or address of the server
 * port is a string containing either the service name or port number to use
 * sock is a pointer to the location to store the socket descriptor when opened
 *
 *  returns 0 if successful
 */
int connect_to_server(const char* server, const char* port, int* sock) {

    struct addrinfo hints, *info = NULL, *list = NULL;
    int e = 0;

    /* initialise the hints for retrieving the address details */
    memset(&hints, 0, sizeof (hints));
    hints.ai_family = AF_UNSPEC;
    hints.ai_socktype = SOCK_STREAM;
    hints.ai_flags = AI_PASSIVE;
    hints.ai_protocol = 0;

    /* get the address if the server to connect to */
    e = getaddrinfo(server, port, &hints, &list);
    if (e != 0) return -1;

    /* connect to the first socket in the list */
    for (info = list; info != NULL; info = info->ai_next)
        if (connect_to_socket(info, sock) == 0)
            break;

    /* clean up and return */
    if (list != NULL) freeaddrinfo(list);
    if (sock > 0) return 0;
    else return -1;
}

Recently I’ve had some real fun with SafeNet HSM’s (Host Security Module or Hardware Security Module) and Tomcat 6 in the never ending process of PCIDSS compliance.  People are probably going to line up to stone me after saying this but PCIDSS is a really great effort to make things safe and secure for all of us.  In a lot of cases its the stick that drives organisations to adopt some good practices.  One of which is the use of HSM’s for all things from the crypt.

Now this little rant won’t walk you through PCI compliance, rather its just about my experiences getting Safenet Eracom Orange or Blue HSM to work within Tomcat.  Deployment is really easy:

1. Install the latest version of Java,
2. Install the Eracom Orange C runtime,
3. Configure the Eracom Orange runtime to connect to the HSM,
4. Configure the HSM with tokens, keys, certificates, users and whatever else you need,
5. Install the Eracom Orange Java runtime,
6. Install Apache Tomcat 6,
7. Place the Eracom Orange ‘jprov.jar‘ Tomcat’s server shared lib directory ‘CATALINE_HOME/lib‘,
8. Restart the Tomcat server if its already running or just start it if it is not,
9. Deploy your webapp.

Apart from steps 7 and 8 simply follow the provider’s directions for all steps.  The key in this thing is that the ‘jprov.jar‘ archive must be loaded when Tomcat starts.  I imagine that it should be the same for other providers like for IBM’s toys.

Now for the development set up.  Simply follow the same steps as above but install the SDK’s for the Eracom and include ‘jprov.jar’ as a library in your development environment.  Typically your development environment will bundle the ‘jprov.jar‘ into the web application directory.  It won’t do any harm there but to get it to work properly it will need to be in the Tomcat server’s common library directory as detailed above.  In fact you don’t really need to install the SDK’s if you have an actual HSM to play with for development purposes, but the SDK’s do provide a software HSM simulator for those of us who don’t want the expense of buying one of these things for every developer on the floor.  Plus the SDK’s will give you a good deal of documentation too.

So you are all set up and ready to build the next great crypt.  Start by building a stand alone application to confirm that everything works before going any further. Lets get to the code:

/*
 * Crypt.java
 *
 * A simple class to provide basic SDES encryption and decryption
 */
package crypt;

import java.io.IOException;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.Key;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.NoSuchProviderException;
import java.security.Security;
import java.security.UnrecoverableKeyException;
import java.security.cert.CertificateException;
import java.util.Properties;
import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.spec.IvParameterSpec;
import org.apache.commons.codec.DecoderException;
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.codec.binary.Hex;
import org.apache.commons.lang.StringUtils;
import org.apache.commons.logging.Log;

/**
 *
 * @author mc
 */
public class Crypt {

  /**
   * Creates a 3DES key cipher with the parameters in the config, valid
   * modes are Cipher.ENCRYPT and Cipher.DECRYPT
   *
   * @param mode set the cipher to either encrypt or decrypt
   * @param config the configuration parameters for this application
   * @param log log to write debug output to
   * @return a cipher that should be used then nulled to be thrown away
   * @throws KeyStoreException
   * @throws NoSuchProviderException
   * @throws IOException
   * @throws NoSuchAlgorithmException
   * @throws CertificateException
   * @throws UnrecoverableKeyException
   * @throws NoSuchPaddingException
   * @throws DecoderException
   * @throws InvalidKeyException
   * @throws InvalidAlgorithmParameterException
   */
  private Cipher createCipher(int mode, Properties config, Log log)
    throws
      KeyStoreException,
      NoSuchProviderException,
      IOException,
      NoSuchAlgorithmException,
      CertificateException,
      UnrecoverableKeyException,
      NoSuchPaddingException,
      DecoderException,
      InvalidKeyException,
      InvalidAlgorithmParameterException {

    log.debug("Adding provider.");
    // will gracefully not add provider if provider already loaded
    Security.addProvider(
        new au.com.eracom.crypto.provider.ERACOMProvider());

    log.debug("Getting key store instance.");
    KeyStore store = KeyStore.getInstance(
        config.getProperty("crypt.keystore.type"),
        config.getProperty("crypt.keystore.provider"));
    log.debug("Loading key store.");
    store.load(null, null);

    log.debug("Retrieving key.");
    Key key = store.getKey(
        config.getProperty("crypt.keystore.entry"), null);

    log.debug("Generating cipher.");
    Cipher cipher = Cipher.getInstance(
        config.getProperty("crypt.transformation"),
        config.getProperty("crypt.provider"));

    log.debug("Creating initialisation vector.");
    IvParameterSpec i = new IvParameterSpec(Hex.decodeHex(
        config.getProperty("crypt.vector").toCharArray()));

    log.debug("Initialising cipher in the given mode.");
    cipher.init(mode, key, i);

    return cipher;
  }

  /**
   * Takes a string and returns its equivalent Base64 encrypted string.
   *
   * @param data string to be encrypted
   * @param config parameters for this application's encryption
   * @param log log to write debug output to
   * @return encrypted Base64 encoded form of the data string parameter
   * @throws NoSuchAlgorithmException
   * @throws NoSuchProviderException
   * @throws NoSuchPaddingException
   * @throws InvalidKeyException
   * @throws IllegalBlockSizeException
   * @throws BadPaddingException
   * @throws InvalidAlgorithmParameterException
   * @throws KeyStoreException
   * @throws IOException
   * @throws CertificateException
   * @throws UnrecoverableKeyException
   * @throws DecoderException
   */
  public String b46Encrypt(String data, Properties config, Log log)
    throws
      NoSuchAlgorithmException,
      NoSuchProviderException,
      NoSuchPaddingException,
      InvalidKeyException,
      IllegalBlockSizeException,
      BadPaddingException,
      InvalidAlgorithmParameterException,
      KeyStoreException,
      IOException,
      CertificateException,
      UnrecoverableKeyException,
      DecoderException {

    log.debug("Encrypting data.");
    byte[] encrypted =
        createCipher(Cipher.ENCRYPT_MODE, config, log).doFinal(
            data.getBytes());

    log.debug("Base64 encoding data.");
    String encoded = new String(Base64.encodeBase64(encrypted));

    return encoded;
  }

  /**
   * Takes a encrypted Base64 encoded string and returns its equivalent
   * decrypted string.
   *
   * @param data string to be encrypted
   * @param config parameters for this application's encryption
   * @param log log to write debug output to
   * @return encrypted Base64 encoded form of the data string parameter
   * @throws NoSuchAlgorithmException
   * @throws NoSuchProviderException
   * @throws NoSuchPaddingException
   * @throws InvalidKeyException
   * @throws IllegalBlockSizeException
   * @throws BadPaddingException
   * @throws InvalidAlgorithmParameterException
   * @throws KeyStoreException
   * @throws IOException
   * @throws CertificateException
   * @throws UnrecoverableKeyException
   * @throws DecoderException
   */
  public String b64Decrypt(String data, Properties config, Log log)
    throws
      NoSuchAlgorithmException,
      NoSuchProviderException,
      NoSuchPaddingException,
      InvalidKeyException,
      IllegalBlockSizeException,
      BadPaddingException,
      InvalidAlgorithmParameterException,
      KeyStoreException,
      IOException,
      CertificateException,
      UnrecoverableKeyException,
      DecoderException {

    log.debug("Base64 decoding data.");
    byte[] decoded = Base64.decodeBase64(data.getBytes());

    log.debug("Decrypting data.");
    byte[] decrypted =
        createCipher(Cipher.DECRYPT_MODE, config, log).doFinal(decoded);

    return new String(decrypted);
  }

}

So that’s it, very straight forward but lets take time to look at one or two points of interest.

In a nutshell functionally this is a class that can be used to encrypt and decrypt data using stored 3DES keys. Note that the configuration information is all passed in via the properties parameter. Also note that the class uses base64 encoding because its easier to play with and store strings. Generally you would also want to replace unsafe base64 characters too if the storage unit required it.

On the encryption front there are a few things to note too. Take a look at the includes, as far as the crypt mechanisms it uses only the standard Java imports and practices. There is only one thing that is specifically from the Safenet SDK, the loading of the provider class in the following snippet:

    // will gracefully not add provider if provider already loaded
    Security.addProvider(
        new au.com.eracom.crypto.provider.ERACOMProvider());

To complete the picture here is the main class and the configuration file to go with it. That’s the lot; three simple files and it should be all go to test your configuration.

/*
 * main.java
 *
 * Test application for the Eracom encryption devices.
 */
package crypt;

import java.io.FileInputStream;
import java.io.IOException;
import java.util.Properties;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

/**
 *
 * @author mc
 */
public class Main {

  /**
   * @param args the command line arguments
   */
  public static void main(String[] args) throws IOException {

    Properties config = new Properties();
    Log log = LogFactory.getLog(Main.class);
    boolean encrypt = "encrypt".equalsIgnoreCase(args[0]);

    config.loadFromXML(new FileInputStream(args[1]));

    try {

      if(encrypt)
        System.out.println(
          new Crypt().b46Encrypt(args[2], config, log));
      else
        System.out.println(
          new Crypt().b64Decrypt(args[2], config, log)); 

    } catch(Exception e) {

      log.error(e.getLocalizedMessage(), e);
    }

  }

}

  
  ERACOM
  au.com.eracom.crypto.provider.ERACOMProvider
  DESede/CBC/PKCS5Padding
  AAAAAAAA00000000
  ERACOM
  CRYPTOKI
  KeyName

Once this is working it should be possible to use the same crypt class file in a Tomcat hosted web application with one slight adjustment. As the Tomcat service will load the provider you should not attempt to load the provider in your application. In fact if you try it will almost certainly fail as your application is unlikely, unless you have specifically configured it, will not have the privileges need to load classes. So just comment out the ‘Security.addProvider‘ call in the class.

So that’s it; all you needed to know to configure an HSM to start working with Tomcat. Simple wasn’t it? Now all you need to do is figure out how to use it to help you with PCIDSS and how to change the crypt file to suite your specific needs.

In my previous post on pyinotify I took a short look at the basic use of the pyinotify watcher using mostly default settings. Here I take a brief look at building a custom event processor. The event processor is called to retrieve the event data and, well, process it. The default processor simply prints the output to ‘stdout‘; mine will too. But it is just there to show how simple it is. Note that the processor actually processes the events in line so there’s no need for messy threading.

#!/usr/bin/python

import os
import pyinotify
from datetime import datetime

m = pyinotify.WatchManager()
notifier = pyinotify.Notifier(m, pyinotify.ProcessEvent(), 0,  0, 100)
m.add_watch('/tmp', pyinotify.ALL_EVENTS, rec=True)

while True:

    print "START LOOP %s " % datetime.now()

    try:
        if notifier.check_events():
            print "Result of notifier.check_events() = TRUE"
            notifier.read_events()
            print "Calling notifier.process_events()"
            notifier.process_events()

    except KeyboardInterrupt:
        notifier.stop()
        break

    print "END LOOP %s " % datetime.now()

This is just the same code as we saw in the earlier post except that I’ve put the ‘notifier.process_events()‘ in its proper place so that it doesn’t get called in every loop execution. It is restricted by the ‘notifier.check_events()‘. To add a little info to the whole process I’ve included some print statements to illustrate the flow of execution. With this we get results that look like:

START LOOP 2009-05-07 23:02:25.025048
Result of notifier.check_events() = TRUE
Calling notifier.process_events()



END LOOP 2009-05-07 23:02:25.104546

Which is what you’d expect. However you will also get results that look like.

START LOOP 2009-05-07 23:02:18.808039
Result of notifier.check_events() = TRUE
Calling notifier.process_events()

END LOOP 2009-05-07 23:02:18.863855
START LOOP 2009-05-07 23:02:18.863912
Result of notifier.check_events() = TRUE
Calling notifier.process_events()

END LOOP 2009-05-07 23:02:18.864656
START LOOP 2009-05-07 23:02:18.864710
Result of notifier.check_events() = TRUE
Calling notifier.process_events()

END LOOP 2009-05-07 23:02:18.865415

So you can see that we get some reasonably unpredictable results.

Now let’s do something a little different and collect the event and print it ourselves a little differently. In this case we’re still collecting all events, we just wanted to mark up the output with something extra.

#!/usr/bin/python

import os
import pyinotify
from datetime import datetime

class e(pyinotify.ProcessEvent):
    def process_default(self, event):
        print(repr(event))

m = pyinotify.WatchManager()
notifier = pyinotify.Notifier(m, e(), 0,  0, 100)
m.add_watch('/tmp', pyinotify.ALL_EVENTS, rec=True)

while True:

    print "START LOOP %s " % datetime.now()

    try:
        if notifier.check_events():
            print "Result of notifier.check_events() = TRUE"
            notifier.read_events()
            print "Calling notifier.process_events()"
            notifier.process_events()

    except KeyboardInterrupt:
        notifier.stop()
        break

    print "END LOOP %s " % datetime.now()

Easy huh? But there’s still a little more, there’s a problem! More on that later.

About Apache and Bloat

Every now and then I look at Apache and wonder about its size and complexity. Its a huge beastie now but absolutely brilliant too. I have quite a respect for it as do many around the world. But then I don’t really use very much of its functionality.

Sure its been re-factored into a modular form so that you only need to use the bits that you want but it still presents a mental preponderance thatmany would consider bloat.  But bloat is a term used to describe software that has vast wads of unnessecary code that could be accomplished more simply.  However Idon’t think that Apache has that.  All of its code seems very purposeful, and as I indicated earlier it is reasonably easy to include or exclude as you will.

Introducing the Example

So this example is not here to show you the right way of doing things, but rather how simple the solution to you particular problem could be.  In this case the example server simply serves up the time on the server machine to port 6543 for any client and then closes the stream.  It can do this for many concurrent connections at once given the only complexity that is added to the server is client handling by forked process.

Now to the Code

/* -----------------------------------------------------------------------------

	simple example of a server in C using very simple Berkeley sockets 

	This is just an example for looking at.  Not really how you would actually
	implement a server.  For a start it does not take care of signals and isn't
	capable of being implemented as a service.

----------------------------------------------------------------------------- */

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <strings.h>
#include <arpa/inet.h>

/* we're not going to use arguments for this example */

#define SERV_UDP_PORT 6543
#define SERV_TCP_PORT 6543
#define SERV_HOST_ADDR "192.168.2.200"

int main (int argc, char ** argv, char ** env) {

	int socket_server, socket_client, child_pid;
	struct sockaddr_in address_client, address_server;
	socklen_t client_address_size;
	FILE *stream_client;
	time_t now;
	struct tm *tm;

	/* open a tcp socket to listen on */

	if ( (socket_server = socket(AF_INET, SOCK_STREAM, 0) ) < 0)
		err_dump("server: can't open stream socket");

	/* bind to our local address  ans start listening so that clients can find us */

	bzero((void *) &address_server, (size_t)sizeof(address_server));

	address_server.sin_family = AF_INET;
	address_server.sin_addr.s_addr = htonl(INADDR_ANY);
	address_server.sin_port = htons(SERV_TCP_PORT);

	if (bind(
		socket_server,
		(struct sockaddr *) &address_server,
		sizeof(address_server))
			< 0)
		err_dump("server: can't bind local address");

	listen(socket_server, 5);

	for ( ; ; ) {

		/* wait for client connection and accpt it when it comes */

		client_address_size = sizeof(address_client);
		socket_client = accept(
				socket_server,
				(struct sockaddr *) &address_client,
				&client_address_size);

		if (socket_client < 0)
			err_dump("server: accept error");

		if ( (child_pid = fork() ) < 0)
			err_dump("server: fork error");

		/* specific client process handling starts here */

		else if (child_pid == 0) {

			/* client process has no need of the server socket */

			close(socket_server);

			/* generate a file stream from the client socket */

			if ((stream_client = fdopen(socket_client, "w")) == NULL) {
				perror("daytimed fdopen");
				return 5;
			}

			/* write to the client stream as per normal then close it */

        		if ((now = time(NULL)) < 0) {
            			perror("daytimed time");

            			return 6;
        		}

        		tm = gmtime(&now);
        		fprintf(stream_client, "%.4i-%.2i-%.2iT%.2i:%.2i:%.2iZ\n",
           			 tm->tm_year + 1900,
           			 tm->tm_mon + 1,
            			tm->tm_mday,
           			 tm->tm_hour,
           			 tm->tm_min,
            			tm->tm_sec);

        		fclose(stream_client);

			exit(EXIT_SUCCESS);

			/* client process has left the building */
		}

		/* the server has no need to keep the client socket open */

		close(socket_client);
	}

	return 0;
}

Ok so this is mostly about me playing with the Google syntax highlighter.  Its basically some code I rattled up to post data from a | seperated list of post_variable|post_value pairs.  Read the code to find out more.  Its not a real solution to anything if you want to run tests or batch stuff use Junit, Maven or Ant.  But you can have fun with my limited toy if you like – but remember as always the code is GPL and will probably kill you so watch out!

/*
 * Batch processor that takes | delimitied lines of fields in a file and posts
 * them to an HTTP server printing out the message and response to std out.
 */

/*
 * TODO:
 * Could be extended to use a configuration file for inputs, logging and could
 * bennefit from SSL and proxy configuration options.
 */
package batchpost;

import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;

import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.ProtocolException;
import java.net.URL;
import java.net.URLEncoder;

import java.util.logging.Level;
import java.util.logging.Logger;

/**
 *
 * @author mc
 */
public class batchpost {

  /**
   * General output device
   */
  static private Logger log = Logger.getLogger(batchpost.class.getName());

  /**
   * main
   *
   * Main entry point.
   *
   * @param args the command line arguments
   *   args[0] - is the fully qualified name of the batch file to read
   *   args[1] - is the URL to post the transactions to
   */
  public static void main(String[] args) {

    /*
     * TODO:
     * I should validate arguments  and do other things but I am skipping doing
     * them because of time.
     */

    processFile(args[0], args[1]);
  }

  /**
   * processFile
   *
   * Takes a file apart one line at a time and hands processing off to a
   * message processing method.  Logs errors if the file is not found or there
   * is an IO error while reading the file.  Assumes that each line contains a
   * complete message.
   *
   * @param url - path to the web server to send the messages to
   * @param file - file to retrieve the message lines from
   */
  static private void processFile(String url, String file) {

    try {

      String line;
      BufferedReader br =
          new BufferedReader(
          new InputStreamReader(
          new DataInputStream(
          new FileInputStream(file))));

      try {

        while ((line = br.readLine()) != null) {
          processMessage(url, line);
        }

      } catch (UnsupportedEncodingException ex) {

        log.log(Level.SEVERE, "processFile - processing halted because it is " +
            "not possible to encode  the post string in UTF-8.", ex);
      } catch (IOException ex) {

        log.log(Level.SEVERE, "processFile - IO exception has halted " +
            "processing of the file \"" + file + "\".", ex);
      }

      try {

        br.close();

      } catch (IOException ex) {

        log.log(Level.SEVERE, "processFile - IO exception has interrupted " +
            "closing of the file \"" + file + "\".", ex);
      }
    } catch (FileNotFoundException ex) {

      log.log(Level.SEVERE, "processFile - unable to locate the file \"" +
          file + "\".", ex);
    }
  }

  /**
   * processMessage
   *
   * Takes a line of | delimited message fields and posts them to a web server.
   * Each line is composed of a list of field_name|field_value pairs such that
   * the line looks like:
   *
   *    name_1|value_1|name_2|value_2|...
   *
   *    or
   *
   *    name|bob charles|age|62|occupation|pro golfer
   *
   * The total number of | delimited fields must be a multiple of 2.  The
   * message fields must not contain a |.
   *
   * @param url - address of the web server to post this message to
   * @param line - string containing a | delimited set of fields
   * @throws java.io.UnsupportedEncodingException when the platform does not
   *    support utf-8
   */
  static private void processMessage(String url, String line)
      throws UnsupportedEncodingException {

    String[] fields = line.split("\\|");
    if (fields.length % 2 == 1) {
      log.log(Level.SEVERE, "processMessage - unable to process the following" +
          " message as it did not contain a multiple of 2 fields: message = " +
          "\"" + line + "\"");
      return;
    }
    String postData = createPostData(fields);

    log.log(Level.INFO, "START OF DATA TO BE POSTED");
    log.log(Level.INFO, postData);
    log.log(Level.INFO, "END OF DATA TO BE POSTED");

    String responseData;
    responseData = doHTTPPost(url, postData);

    log.log(Level.INFO, "START OF DATA RECEIVED FROM POST");
    log.log(Level.INFO, responseData);
    log.log(Level.INFO, "END OF DATA RECEIVED FROM POST");
  }

  /**
   * createPostData
   *
   * Takes an array of field names and their values such that each two
   * consecutive strings are a field name then value pair and prepares a POST
   * data string for sending to a web server.  Throws the encoding error as it
   * is pointless to continue processing as this error is going to reccur for
   * all post attempts if it takes place for one.
   *
   * @param fields - a string array of field name and value pairs.
   * @return string containing the post data.
   * @throws java.io.UnsupportedEncodingException
   */
  static private String createPostData(String[] fields)
      throws UnsupportedEncodingException {

    StringBuffer buf = new StringBuffer();
    String charEnc = "UTF-8";

    try {

      buf.append(URLEncoder.encode(fields[0], charEnc));
      buf.append('=');
      buf.append(URLEncoder.encode(fields[1], charEnc));

      for (int i = 2; i < fields.length; i = i + 2) {

        buf.append('&');
        buf.append(URLEncoder.encode(fields[i], charEnc));
        buf.append('=');
        buf.append(URLEncoder.encode(fields[i + 1], charEnc));
      }

      return buf.toString();
    } catch (UnsupportedEncodingException ex) {

      log.log(Level.SEVERE, "createPostData - unable to encode the post data " +
          "as " + charEnc + " is not available on this platform.", ex);
      throw ex;
    }

  }

  /**
   * readAll
   *
   * Reads all of the input from the given stream and returns it in a byte array
   * output stream, the stream contents can then be retrieved as a byte array or
   * a string using either toByteArray or toString respectively.
   *
   * @param is - an open input stream to read the data from
   *
   * @return a byte array output stream that contains the data read from the is
   * @throws IOException if there is an error reading from the input stream
   */
  static private ByteArrayOutputStream readAll(InputStream is)
      throws IOException {

    ByteArrayOutputStream result = new ByteArrayOutputStream();
    byte[] buf = new byte[1024];

    int len = is.read(buf);

    while (!(len < 0)) {

      result.write(buf, 0, len);
      len = is.read(buf);
    }

    return result;
  }

  /**
   * doHTTPPost
   *
   * performs the https post operation of the given data to the given url
   * returning the result in a byte array output stream, throws an exception if
   * an error takes place while performing io operations, does not deal with
   * proxy servers
   *
   * @param data - a post formatted data stream for sending to the url server
   * @param url - the target to send the post to
   *
   * @return a data stream of the response from the server at the url
   * @throws IOException on failure of any io operation
   */
  static private String doHTTPPost(String url, String data) {

    String result = "";
    HttpURLConnection conn = null;

    try {

      // set up the url connection
      conn = (HttpURLConnection) (new URL(url)).openConnection();
      try {

        // write out the data to the server
        conn.setDoOutput(true);
        conn.setRequestMethod("POST");
        conn.getOutputStream().write(data.getBytes());
        System.out.println(conn.getResponseMessage());

        // disconnect and recycle the connection, return the response data
        result = readAll(conn.getInputStream()).toString();
      } catch (ProtocolException ex) {

        log.log(Level.SEVERE, "doHTTPPost - unable to marshal the required " +
            "protocol to establish a connection to \"" + url + "\"", ex);
      } catch (IOException ex) {

        log.log(Level.SEVERE, "doHTTPPost - IO error while attempting to " +
            "perform the data post operation.", ex);
      } finally {

        conn.disconnect();
      }

    } catch (MalformedURLException ex) {

      log.log(Level.SEVERE, "doHTTPPost - the URL currently being attempted " +
          "to connect with is malformed \"" + url + "\"", ex);
    } catch (IOException ex) {

      log.log(Level.SEVERE, "doHTTPPost - IO error while attempting to set up" +
          " the data post operation.", ex);
    } finally {

      conn = null;
    }

    return result;
  }

}

Its nice to have a standards based platform and in that vein C certainly has progressed a long way. But you still get the odd system that isn’t either set correctly up by default with my favorite extensions; namely all things GNU! Yes I admit to liking non-standard well coded extensions, in fact who would work without them these days except under a very specific set of requirements or academic interest. Things just would not get done so quickly without them.

So I got a Mac. Yep its sooo cool that when my 5 year old daughter walked in the room and saw it for the first time she exclaimed ‘Daddy! That’s sooo cool!’ Kind of an ego boost for someone who gave up on cool and cool things 20 years ago. However cool comes at a price, its got great development tools but I want cross platform development to run on my Linux boxen too. Long story short, it doesn’t have the standard GNU extensions so I had two choices, fart around with the environment or rewrite one or two functions myself.

So here’s the goods. Now because I don’t want people to just copy and paste my material without thinking about it (I don’t mind the copying I just like people to put in some effort and think) I’ve copied an earlier version of the functions which works for some situations but not for others. Its close tot he real deal but there are about 5 significant issues which would make it dangerous to use this code without doing something about them.

Here it is, have fun, have a play! (Note: the whole thing is under GPL 3, comments from GNU)

/*

ssize_t getdelim (char **lineptr, size_t *n, int delimiter, FILE *stream)  

This function is like getline except that the character which tells it to stop 

reading is not necessarily newline. The argument delimiter specifies the 

delimiter character; getdelim keeps reading until it sees that character (or end 

of file).

The text is stored in lineptr, including the delimiter character and a 

terminating null. Like getline, getdelim makes lineptr bigger if it isn't big 

enough.

getline is in fact implemented in terms of getdelim

*/

ssize_t getdelim( char **lineptr, size_t *n, int delimiter, FILE *stream ) 

{

	/* setup the environment */

	int c = getc( stream );

	long i = 0;

	char *a = NULL;

	/* count the characters needed for the buffer */

	while ( c != delimiter && c != EOF ) {

		c = getc( stream );

		i++;

	}

	/* test for and arrange the buffer size */

	if ( i == 0 ) 

		return -1;

	if ( fseek( stream, -i, SEEK_CUR ) ) 

		return -2;

	if ( *n < i + 1 ) {

		a = ( char * )realloc( *lineptr, i + 1 );

		if ( a == NULL )

			return -3;

		*lineptr = a;

		*n = i;

	}

	/* read the data into the buffer */

	if ( fread( *lineptr, sizeof( char ), i, stream ) != i )

		return -4;

	( *lineptr )[i] = 0;

	/* return the number of chars read */

	return i;

}

/*

ssize_t getline (char **lineptr, size_t *n, FILE *stream)

This function reads an entire line from stream, storing the text (including the 

newline and a terminating null character) in a buffer and storing the buffer 

address in *lineptr.

Before calling getline, you should place in *lineptr the address of a buffer *n 

bytes long, allocated with malloc. If this buffer is long enough to hold the 

line, getline stores the line in this buffer. Otherwise, getline makes the 

buffer bigger using realloc, storing the new buffer address back in *lineptr and 

the increased size back in *n. See Unconstrained Allocation.

If you set *lineptr to a null pointer, and *n to zero, before the call, then 

getline allocates the initial buffer for you by calling malloc.

In either case, when getline returns, *lineptr is a char * which points to the 

text of the line.

When getline is successful, it returns the number of characters read (including 

the newline, but not including the terminating null). This value enables you to 

distinguish null characters that are part of the line from the null character 

inserted as a terminator.

This function is a GNU extension, but it is the recommended way to read lines 

from a stream. The alternative standard functions are unreliable.

If an error occurs or end of file is reached without any bytes read, getline 

returns -1. 

*/

ssize_t getline (char **lineptr, size_t *n, FILE *stream)

{

	return getdelim ( lineptr, n, '\n', stream );

}

One of the tasks of numerical analysis is to search for or compare patterns. In the deep past this was quite a problem as it was easy to swamp the systems available with data making it more practical to employ a mathematician to perform some arcane analysis via algebra. However the manual way did not give the code opportunity to be tested against all possible combinations. Normally tests were conductd against boundaries and samples of statisitically significant combinations.

However in these days of Gig’s of memory and fast processors not to mention the terrabytes of disk space available cheaply it is possible to generate and test every possible combination in many cases. One of these is the classic combination space of mathematics. This following alogorithm is a crude base to progress from to generate such a number space.

// pattern generation
void genPat(size_t smpSize, size_t popSize, vector cmbSpace) {
  t_combo v(smpSize); // entry vector
  t_combo m(smpSize); // entry limit vector

  int i, e, a; // index and place holders

  // initialise the entry vector with an ascending order 1 to smpSize
  // and the limit vector with the max value each entry should reach
  for (i = 0; i < smpSize; i++)
  {
    v[i] = i + 1;
    m[i] = popSize - smpSize + v[i];
  }

  // repeat until the first entry in the vector is less than its
  // maximum value
  while (v[0] < m[0])
  {
    // update the combination space
    cmbSpace.push_back(v);

    // set an index to the last element
    e = smpSize - 1;

    // find the first element from the right to be less than the limit
    while (v[e] == m[e]) e--;

    // add one to and store the value of the current element in a var
    a = ++v[e];

    // set each element to the right to one more than the current
    while (e < smpSize - 1) v[++e] = ++a;
  }

  cmbSpace.push_back(v);
}