A Journeyman's Journal

Setup NFS4 & LiveMigration with OpenNebula 2.2.1 on Ubuntu 11.04 Natty

2011-09-14T11:56:00.002-04:00

About 2 months ago we setup a experimental internal cloud environment for one of our client using pure commodity servers and open source technology (OpenNebula 2.2.1 on Ubuntu 11.04 Enterprise Server). Since then the proof-of-concept environment has been working pretty well for our client hosting internal QA, Development, as well as some IT virtualized appliances. However the initial setup was based on SSH transport which works well in the small environment but struggles with large VMs. The main disadvantages of SSH based configuration are:

Performance with large VM image - With ssh transport each VM image will have to be transferred to the cloud node from the repository by the controller, and then retrieved back from the node (if save flag is on). Obviously this process takes a lot of bandwidth with large images.
No live migration - Live migration requires shared VM image storage between cloud nodes. Live migration is the key feature allowing any cloud operator to maintain the no downtime illusion and elasticity.
Vulnerable to data loss - If you are like our client who is running a cloud on top of commodity servers or even PCs, then most likely your cloud node server does not have RAID disks or much of other hardware redundancies. Since ssh based solution transfers the image to the node then it lets the node to run the VM using local hard disk, thus if you suffer a hard drive failure while running the VM you might loose all or part of your VM data.

To solve these problems OpenNebula and Linux community have already provided many solutions, in this post I would like to introduce the most straightforward alternative solution with NFSv4. With NFS a server can export a part of it's own file system to client machines whom in turn can then mount the exported file system locally and use it just as any other local file system. Originally we expected the change to be quite simple and straightforward however it turned out there were a few interesting hurdles some OpenNebula specific some Ubuntu. I hope this post can help you implementing similar solution with little effort.

First: NFS Server

Since this NFS server will be your centralized storage server therefore a little bit extra investment in this server would not hurt. We recommend to at least have 2 physical NICs, dual power supply, and RAID 1/5/10 disks.

(In our environment we chose software RAID 5 with 3 disks and 1 hot spare)

Install NFS4 Server

sudo apt-get install nfs-kernel-server

In /etc/default/nfs-common turn on IDMAPD:

sudo vi /etc/default/nfs-common

# NFSv4 specific
NEED_IDMAPD=yes
NEED_GSSD=no # no is default

Export /storage folder:

sudo vi /etc/exports
[nfs client or your network]/storage (rw,async,no_root_squash,insecure,no_subtree_check,anonuid=1001,anongid=1001)

Here in /etc/exports we use async flag to improve NFS performance and also map default uid and gid to oneadmin/cloud (1001/1001).

# Refresh export table

sudo exportfs -r

Create oneadmin and cloud group

sudo groupadd --gid 1001 cloud

useradd --uid 1001 -g cloud oneadmin

Now you are done with the server setup.

Second: NFS Client (OpenNebula controller and nodes)

Install NFS Client

sudo apt-get install nfs-common

In/etc/default/nfs-common turn on IDMAPD:

# NFSv4 specific
NEED_IDMAPD=yes
NEED_GSSD=no # no is default

# Add mount
sudo vi/etc/fstab
[nfs server]:/storage /storage nfs4 rw,_netdev,rsize=1048576,wsize=1048576,auto 0 0

Restart the OS:
sudo reboot

Now lets test the NFS mount:
#NFS performance test
dd if=/dev/zero of=/storage/100mb bs=131072 count=800

Shutdown OpenNebula and all VMs then move /srv/cloud/one/var to NFS mount:

mv /srv/cloud/one/var /storage/one/var

ln -s /storage/one/var /srv/cloud/one/var

(Also merge the var content on cloud nodes to the same directory if you have been running under ssh mode)

Turn Off Dynamic Ownership for QEMU on Nodes

sudo vi /etc/libvirtd/qemu.conf

# The user ID for QEMU processes run by the system instance
user = "root"

# The group ID for QEMU processes run by the system instance
group = "root"
dynamic_ownership = 0

Otherwise you might experience "unable to set user and group to '102:105' error" when migrating VMs

Fix LiveMigration Bug

Right now if you try to livemigrate with this setup you will get "Cannot access CA certificate '/etc/pki/CA/cacert.pem'" error since QEMU by default is configured to use TLS. With OpenNebula since you have already configured bi-directional passwordless ssh access between controller and nodes the easiest fix it to ask qemu to use ssh instead of TLS.

vi /srv/cloud/one/var/remotes/vmm/kvmrc

export LIBVIRT_URI=qemu+ssh:///system
export QEMU_PROTOCOL=qemu+ssh

Disable AppArmor for Libvirtd on Nodes

By default Ubuntu AppArmor will prevent libvirtd from access nfs mount, you can add the rw permission for the directory or simply disble apparmor for libvirtd. I chose to disable it:

sudo ln-s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/

apparmor_parser-R /etc/apparmor.d/usr.sbin.libvirtd

At this point you should have a working NFS based OpenNebula setup. Now uncomment the tm_nfs section in oned.conf then add your hosts back use the new NFS transport:

onehost create im_kvm vmm_kvm tm_nfs

You should be able to use all the features OpenNebula provides now including live migration and should see a significant performance improvement for VM provisioning. Have fun and enjoy :-)

Griffon Validation Plugin v0.8.1 Release

2011-09-10T11:39:00.001-04:00

Just released v0.8.1 for Griffon Validation Plugin, a minor bug fix release to address the multi-error rendering issue recently discovered by Frank Shaw. If you are using several constraint especially custom constraints on your model, it is highly recommended to upgrade to this release immediately since the error rendering engine currently does not handle multi-error scenario correctly. To upgrade please use the following command:

griffon install-plugin validation

Griffon Validation Plugin 0.8 Released

2011-08-19T14:24:00.003-04:00

I am happy to announce the release of Griffon Validation plugin v0.8. In this release the following enhancement and new features were introduced:

1. Upgraded to match Griffon core 0.9.3 release

For v0.8 release you will need Griffon 0.9.3+ to run it.

2. Upgraded to i18n plugin 0.4 release

Thanks to the new multi-bundle support in i18n plugin now when using validation plugin you no longer need to manually create default error messages. The validation plugin is now shipped with these default messages and will automatically fall back to it if no other localized messages are found.

3. Real time validation support

Now validation plugin can automatically trigger validation for your model beans if you include the realTime flag in your annotation.

@Validatable(realTime=true)

class MyModel{

  ....

}

The real time validation feature is implemented relying on the property change support therefore any property value change, for example triggered by bind{}, will invoke the validation logic associated with that particular property. The actual validation is performed in a separate thread using GriffonApplication.execAsync() before it switches back to EDT for error rendering. Currently this flag only works with Griffon MVC model beans if you apply it to a regular POGO object the flag will be ignored.

As usually you can upgrade your validation by simply issue the following command:

griffon install-plugin validation

And please refer to the plugin Wiki for detail information of the plugin.

The End is the Beginning

2011-07-11T16:05:00.004-04:00

I have decided today to terminate Hydra Cache project. What is Hydra Cache anyway?

About 2 and half years ago, a few of my friends and I, inspired by our experience with one of our client at that time and Amazon's Dynamo design, started an open source project to build a distributed elastic cache called Hydra Cache. However after the inception of this project I personally was involved in a number of startup business and open source projects, thus spreading myself too thin and even till today still only have the Hydra Cache project in a pre-release 1.0RC1 stage (although really close to be release ready).

Of course this slow development of a promising project only had myself to blame. When I was searching for the reason why I want to terminate the project despite it being so close to release, I had many answers such as competent alternative open source options (JBoss InfiniSpan) now available, but I think the most important reason is that I have lost my drive and obsession with this idea a long time ago. All the things I did after that point for one reason or another were not really helping the project but rather just me dragging my feet to admit the fate of the project. Nevertheless this project has taught me many lessons, both technical and non-technical, however most importantly after this experience I realized the best way to do an outstanding job with open source project is to do it with your passion and have fun. The moment you realize the flame of your passion is gone and you are no longer having fun with it, you should stop dragging it alone. Since we don't get to be paid for open source contribution, so if you are not having fun then why even bother :)

Install OpenNebula 2.2.1 on Ubuntu 11.04 Enterprise Server

2011-06-29T14:41:00.008-04:00

Recently I was in a process of setting up an open source cloud computing environment. The experience I gained through this project was very interesting to say the least. The goal for me is to create a ultra reliable cloud computing environment with the potential to handle not only IaaS requirement but also PaaS as well as future SaaS goals. On top of that this environment should be able to utilize commodity servers or even desktop grade machine to form the cloud. After much research and experiment, eventually I have decided to go with OpenNebula due to its rich feature set and well thought-out integration with different hypervisors as well as public cloud vendor.

In this post, I would like to share my findings while setting up OpenNebula 2.2.1 on Ubuntu 11.04 Enterprise Server to maybe make your life a bit easier if you decided to go down a similar path. The hypervisor technology we decided to use is KVM however OpenNebula works with Xen and VMWare as well.

(In this guide I will focus on how not why. To find out more about why certain steps were performed please refer to OpenNebula document)

Cloud Controller Setup

1. Getting OpenNebula 2.2.1

I basically just downloaded the tar ball from http://downloads.dsa-research.org/opennebula/

2. Requisite Software Installation

sudo apt-get install ruby sqlite3 libxmlrpc-c3-0 openssl ssh

sudo apt-get install ruby-dev rubygems rake make libxml-parser-ruby libxml2 libxslt1.1 libxml-ruby libxslt-ruby libnokogiri-ruby1.8

3. Setup Directory Structure

$ tree /srv
/srv/
|
`-- cloud
  |-- one
  `-- images

4. Create OpenNebula User and Group Account

groupadd cloud
useradd -d /srv/cloud/one -g cloud -m oneadmin
sudo passwd oneadmin
$ id oneadmin
uid=1001(oneadmin) gid=1001(cloud) groups=1001(cloud)

5. Prepare for Build

sudo apt-get install libsqlite3-dev libxmlrpc-c3-dev scons g++ ruby libopenssl-ruby libssl-dev ruby-dev make rake rubygems libxml-parser-ruby1.8 libxslt1-dev

6. Fix bug #265 (Optional)

Currently OpenNebula automatically deletes VM image that fails to start for whatever reason. This behavior might not be desirable for some installation including ours. This is currently filed as a bug/improvement for 3.0 release, but you can apply the temporary patch following the instruction on this page

http://dev.opennebula.org/issues/265

7. Build

~$ wget
~$ tar xzf
~$ cd one-2.0
~/one-2.0$ scons -j2
....
scons: done building targets.
~/one-2.0$ ./install.sh -d /srv/cloud/one

8. Add OpenNebula environment variables

Edit ~/.profile and add the following lines (or .bashrc if you want to use bash)

export PATH=$PATH:/var/lib/gems/1.8/bin:/srv/cloud/one/bin
export ONE_LOCATION=/srv/cloud/one
export ONE_XMLRPC=http://localhost:2633/RPC2
export ONE_AUTH=/srv/cloud/one/.one/one_auth

9. Create the user's OpenNebula config directory:

~$ mkdir ~/.one

10. Configure Authentication File

# Add this one liner to one_auth file match oneadmin user's password
~$ vim /srv/cloud/one/.one/one_auth

oneadmin:oneadmin

11. Prepare to install Sunstone (WebUI for OpenNebula)

sudo gem install json sinatra thin rack sequel

* Ubuntu does not add gem to your path so add the following directory to your path /var/lib/gems/1.8/bin

At this point your controller server is pretty much setup now. Next we will create a cloud node server.

Cloud Node Setup

1. Verify CPU Virtualization Support

Since KVM relies on CPU virtualization support before we install KVM we need to make sure it will work with your hardware.

egrep '(vmx|svm)' /proc/cpuinfo

If there is any match, it means you CPU supports KVM otherwise you can try VMWare hypervisor or Xen.

2. Install KVM

If you have already selected KVM during Ubuntu server installation then you really don't have to do anything here, otherwise:

sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder virt-manager virtinst bridge-utils

3. Remove default NAT bridge

By default if you selected KVM during Ubuntu installation it will also install the default NAT bridge. But usually this NAT bridge is not needed for server type setup and compromise performance, so I will remove the bridge here.

# virsh net-destroy default
# virsh net-undefine default
# service libvirt-bin restart
# ifconfig

4. Setup bridge for VM network

In a server type environment you want your VM to be accessible from anyone on the network, to achieve that you need to bridge the network from the host. Modify the /etc/network/interfaces file.

DHCP

auto eth1
iface eth1 inet manual

auto virbr1
iface virbr1 inet dhcp
hostname 
bridge_ports eth1
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off

Static IP

auto eth1
iface eth1 inet manual

auto virbr1
iface virbr1 inet static
   address 10.128.129.43
   network 10.128.129.0
   netmask 255.255.255.0
   broadcast 10.128.129.255
   gateway 10.128.129.1
   bridge_ports eth1
   bridge_fd 9
   bridge_hello 2
   bridge_maxage 12
   bridge_stp off

5. Setup Software required by OpenNebula

sudo apt-get install ruby ssh

6. Create Directory Structure

$ tree /srv
/srv/
|
`-- cloud
  |-- one

7. Create OpenNebula User and Group (with the same G/UID)

groupadd --gid 1001 cloud
useradd --uid 1001 -g cloud -G libvirtd -d /srv/cloud/one oneadmin
sudo passwd oneadmin – setup the password to oneadmin
chown -R oneadmin:cloud cloud

* Make sure the gid and uid match the ones on the controller

8. Setup password-less SSH access from controller to node for oneadmin user

ssh-keygen
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ cat ~/.ssh/config
Host *
StrictHostKeyChecking no

* If you are using SSH transport driver with OpenNebula, you need to make sure there is bi-directional password-less ssh access from controller to node and vice versa.

Finally now you are done setting up a basic cloud environment. Try start OpenNebula on the controller:

~/one/bin/one start

You should be able to add the host by using onehost command:

onehost create im_kvm vmm_kvm tm_ssh

Hope you have found this guide helpful and having fun with OpenNebula :) In future post I will try to cover how to run OpenNebula on distributed reliable file system MoosFS to achieve true elasticity without using expensive SAN solution.

How To Configure izPack with Installer plugin in Griffon

2011-06-20T10:48:00.006-04:00

Recently I used Installer plugin for Griffon in one of my open source project. Overall it was very easy to install and have it creating a simple installer for your Griffon application, however when it comes to customizing the izPack configuration I did not find any good documentation. Thanks to the usual helpfulness of Andres Almiray :) and going through some source code, I managed to customize the installer to meet my requirement and would like to share my findings here.

Once you install the installer plugin through:

griffon install-plugin installer

A set of izPack configuration templates will also be installed with the plugin. The template works pretty well for any Griffon example application but probably does not make too much sense for anything else. The easiest way to provide your own customization is by creating your custom template and override the default ones by hooking into Griffon build time event notification. I will show you how to do that here step-by-step.

First: Create your installer source directory to store your configuration and resources such as icons. In this tutorial we will create a folder structure as the following:

/src/installer/izpack/resources

Second: Create your own izPack configuration. Copy the default configuration to the resources folder you just created. The default configuration files can be found under ~/.griffon/projects/installer/izpack/resources.

Third: Create event handler to listen on packaging event and override the default configurations with yours. Open _Events.groovy file under /scripts (Create it if it does not exist yet). Add the following lines:

eventPreparePackageEnd = {installers ->
 ant.copy( todir: "${projectWorkDir}/installer/izpack/resources", overwrite: true ) {
     fileset( dir: "${basedir}/src/installer/izpack/resources", includes: "**" )
 }

 ant.replace( dir: "${projectWorkDir}/installer/izpack/resources" ) {
     replacefilter(token: "@app.name@", value: griffonAppName)
     replacefilter(token: "@app.version@", value: griffonAppVersion)
 }
}

Now run the izPack packaging command

griffon package izpack

You should see the customized installer based on your configuration being generated. Hope you have found this tutorial helpful and again you are always welcome to provide your feedback and comment here.

JNDI Warrior v0.3 Released

2011-06-20T10:07:00.003-04:00

Just released JNDI Warrior 0.3 yesterday. In this release the following improvement and new features have been implemented:

Better UI look and feel to give it more mature look rather than the original demo-like appearance
Better ability to manage classpath per JNDI connection session
A izPack powered cross-platform installer to make it easy to install the application on any platform
New feature implemented to create, edit, and execute Groovy script from the application directly making prototyping and debugging with JNDI even easier.

Two built-in variables are available currently within the script.

context - IntitialContext of connected JNDI provider
out - a PrintWriter instance which allows you to print directly to script output area

Here is a screenshot of the 0.3 release:

Griffon Validation Plugin 0.7.1 Released

2011-05-13T15:49:00.003-04:00

This is bug fix minor release for the Griffon Validation plugin. The main issue fixed in this release was the problem when combining popup error rendering style in a JDialog. To upgrade your installed plugin use the following command:

griffon install-plugin validation

Griffon Airbag Plugin 0.1 Release

2011-05-05T15:34:00.002-04:00

In one of my recent open source project I created a mechanism in Griffon framework to capture any uncaught exception raised in an application regardless what threading model it is using. The idea eventually caught some attention on Griffon dev maillist and absorbed into the 0.9.2 core. The actual implementation of this mechanism now in Griffon core is definitely more efficient and reliable than my original meta method wrapping based attempt :)

The rest of the original implementation, mostly a Swing dialog, designed to provide a simple and user friendly way to inform any unexpected exception has been packaged as a standard Griffon plugin called Airbag. I have just released the 0.1 version of it which includes pretty much a direct port of the code from my other project.

To Install

griffon install-plugin airbag

How to use?

Once installed Airbag dialog can be easily initiated by using the following event handler in your main controller.

   void onUncaughtExceptionThrown(ex) {
      doLater {
          try {
              def errorDialog = new AirBagErrorDialog(view.mainWindow,
                      "Uncaught Exception Occured",
                      ex)

              errorDialog.show()
          } catch (Exception e) {
              e.printStackTrace(System.err)
          }
      }
  }

Screen Shot

Griffon Validation Plugin 0.7 Released

2011-05-03T19:02:00.013-04:00

Version 0.7 of Griffon Validation Plugin has just been released to the plugin repository. This is a major release on Validation plugin road map which included some significant internal refactoring as well as some key additional features. In this post, I would like to provide a walk-through of these major improvements.

1. Removed runtime enhancement favoring AST based compile-time enhancement (backward compatibility breaking change!)

From the inception of this plugin, all model instances are automatically enhanced during runtime to have the validation capability injected. Since v0.4 an additional AST transformation annotation @Validatable has been added so non-model classes can be annotated to have the same validation capability injected. However since these two approaches work quite differently; one performs meta class based enhancement during runtime, and the other AST based approach performs byte code level enhancement during compile-time. After much discussion we have realized that firstly the on-going effort to make sure two different approaches produce identical result is getting more and more complicated as new feature being introduced to the enhancement logic. Secondly the default runtime enhancement is less flexible and less efficient comparing to the AST based approach. Thus a hard decision was made to drop the support for the auto runtime enhancement for model instances. In other words after upgrading to 0.7 you will need to add @Validatable to all model classes that you want validation feature to be injected.

2. Error Renderers

One of the common challenge we face when building UI using any GUI framework is how to effectively and easily notify the user about errors. Ideally a validation framework should not just help developer define constraints and validation logic but also handle the presentation of the error message automatically with little coding involved. With this vision in mind, also thanks to many useful input and ideas provided by Andres during our discussion, this mechanism called Error Renderer was created in this release as my first attempt to address this challenge with some Groovy/Griffon awesomeness.

Error Renderer can be declared easily by using the additional synthetic attribute 'errorRenderer' introduced in this release. See the following example:

textField(text: bind(target: model, 'email'), errorRenderer:'for: email, styles: [highlight, popup]')

In the above example, two error renderers were declared for the textField widget for the 'email' field in the model. Basically what it means is that if any error was detected for the email field in the model two types of error renderer will be activated to display the error(s). The styles portion of the configuration is optional. If no renderer style is defined, by default highlight renderer will be used. Currently three types of error renderer styles are implemented, I will go through them quickly here.

I. Highlight Error Renderer

This renderer basically change the background color of the component to pink. Mostly it is used for text based input fields. Here is a screen shot of the rendering result.

II. Popup Error Renderer

This renderer display the error message associated with the error using a tooltip-like popup box. Here is a screen shot of the rendering result.

III. On With Error Renderer

This is an invisible renderer that does not render anything itself but switch the component visible attribute on when the error is detected. It is commonly used to display initially invisible custom component when error occurs. This renderer is used in combination of the new errorIcon widget also introduced in this release. Here is a screen shot of it used with errorIcon.

3. New Error Related Widgets

A few simple widgets were also added to this release to reduce the effort required when working errors.

I. Error Messages Widgets

This is basically the old wine in a new bottle. This widget is essentially identical to the ErrorMessagePanel class existed since v0.2 however it is now implemented as a widget to make it easier to use. Usage:

errorMessages(constraints: NORTH, errors: bind(source: model, 'errors'))

Screen shot:

II. Error Icon Widgets

As mentioned before this widget is mainly used in combination with the onWithError renderer. This icon widget is initially invisible and will only be turned on by the onWithError renderer. Usage:

errorIcon(errorRenderer:'for: creditCard, styles: [onWithError]')

Screen shot:

4. Limitation

Currently the error renderer only works with the model instance within a MVC group. Future work is need to support plain POGO annotated with @Validatable.

How to support custom artifact type in Griffon 2nd revision

2011-02-28T15:17:00.005-05:00

Last year June I wrote a blog post about how to support custom artifact type in Griffon framework. Now Griffon 0.9.2 release is just around the corner. In this release some major refactoring was done with the Artifact API, that's why I am updating this guide to match the new API.

Out-of-box Griffon supports 4 different types of artifacts: Model, View, Controller (MVC) plus Service, these are the major building blocks of any Griffon application. Just like it's cousin Grails, in Griffon plugins and addons can also introduce new artifact types, however the process is fairly different from what Grails employs. As part of Griffon Validation Plugin I implemented the support for a new custom artifact type - Constraint, and I would like to share some of my learning here so it would be a little bit easier if you are planning to do something similar.

Step 1 - Handle your artifact

To support a new artifact type you have to tell the Griffon core about the artifact type first. You can achieve this by implementing your own ArtifactClass and ArtifactHandler, for most of the common cases extending from DefaultGriffonClass and ArtifactHandlerAdapter should be enough. Here is what it looks like for the Constraint artifact type:

public interface GriffonConstraintClass extends GriffonClass {
/** "constraint" */
String TYPE = "constraint";
/** "Constraint" */
String TRAILING = "Constraint";
}

public class ConstraintArtifactHandler extends ArtifactHandlerAdapter {
public ConstraintArtifactHandler(GriffonApplication app) {
   super(app, GriffonConstraintClass.TYPE, GriffonConstraintClass.TRAILING);
}

protected GriffonClass newGriffonClassInstance(Class clazz) {
   return new DefaultGriffonConstraintClass(getApp(), clazz);
}
}

public class DefaultGriffonConstraintClass extends DefaultGriffonClass implements GriffonConstraintClass {
public DefaultGriffonConstraintClass(GriffonApplication app, Class clazz) {
   super(app, clazz, GriffonConstraintClass.TYPE, GriffonConstraintClass.TRAILING);
}
}

One note of caution on the artifact class and it's handler:
I ran into some problem when implementing them in Groovy initially, and had to change all implementation to Java instead. Thanks to Andres for the investigation and input on this discovery.

Step 2 - Register your artifact

As shown above, if you have experience working with Grails artefact support, you will notice the handler implementation is almost identical in Griffon, however things starting to differ from this point forward. Now you have the handler implemented, next thing is to register it with Griffon core. This is best achieved during initialization phase in your addon. Open the [PluginName]GriffonAddon.groovy file add the following callback if its not already there:


def addonInit = {app ->
....
app.artifactManager.registerArtifactHandler(new ConstraintArtifactHandler(app))
....
}

Step 3 - Find your artifacts

Now we have the new artifact type registered, next step is to tell Griffon where to find the artifacts. Ever wonder how Griffon knows to look under griffon-app/models for model classes? This is what we are going to do in this step. This is also where Griffon custom artifact support truly differs from Grails. Instead of handling it at run-time, Griffon chooses to handle this at build time, so to achieve this you need to tap into the Griffon event model. Open the _Events.groovy script and implement the following event listener:


eventCollectArtifacts = { artifactsInfo ->
if(!artifactsInfo.find{ it.type == 'constraint' }) {
   artifactsInfo << [type: 'constraint', path: 'constraints', suffix: 'Constraint']          }  }

This event listener will tell Griffon to look for anything under griffon-app/constraints folder with suffix 'Constraint' and register them as constraint artifacts.

Step 4 - Measure your artifacts

Now we pretty much have the basic bolts and nuts in place, its time to make our newly found artifact type to be more integrated with Griffon as any other first class artifact types do. One of the nice feature of Griffon is the stats command, it gives you an instant overview of how big your app is in terms of how many files and Line of Code per type including artifacts. Won't it be nice to have it also display the metrics about our own custom artifacts? fortunately its actually pretty easy to achieve in Griffon, similar to the previous step we will add another listener to the _Events script.


eventStatsStart = { pathToInfo ->
if(!pathToInfo.find{ it.path == 'constraints'} ) {
   pathToInfo << [name: 'Constraints', path: 'constraints', filetype: ['.groovy','.java']]    } }

Griffon event model is a very powerful concept, usually when I am not sure how to do something funky in Griffon this is the first place I look.

Step 5 - Automate your artifacts

One of big selling point of the next generation Rails-like RIA framework is the ability to create any artifact simply by using one of the built-in command, for example grails create-controller or griffon create-mvc. To make our new artifact type a true first-class citizen of Griffon, of course we need all the bells and whistles. To add a new command to Griffon, you need to create a new Groovy script under scripts folder:

CreateConstraint.groovy


includeTargets << griffonScript("_GriffonInit")
includeTargets << griffonScript("_GriffonCreateArtifacts")

target('default': "Creates a new constraint") {
    depends(checkVersion, parseArguments)

    promptForName(type: "Constraint")

    def name = argsMap["params"][0]
    createArtifact(name: name, suffix: "Constraint", type: "Constraint", path: "griffon-app/constraints")
 createUnitTest(name: name, suffix: "Constraint")
}

Like other convention-over-configuration framework, Griffon relies heavily on simple naming conventions, so in the script make sure you naming everything consistent to avoid unnecessary complexity. This script will create artifact for the type of Constriant and related unit test case, as you can see it will be a simple matter to create integration test case if need be.

Now with the command in place, you can finally provide the template for the new artifact being created. Again naming convention is being used to determine where to find template file, for our example the template file should be placed under src/templates/artifacts and named Constraint.groovy:


@artifact.package@class @artifact.name@ {

def validate(propertyValue, bean, parameter) {
   // insert your custom constraint logic
}

}

Phew, now finally we are done, hope I did not miss anything :) This is a long post and as you can see a lot of plumbing; this is exactly why currently there are some discussion going on within Griffon dev mail list to provide declaration based custom artifact support either using Grails style or Groovy AST transformation, so stay tuned for future updates on this topic.

Griffon Validation Plugin 0.6 Released

2011-02-21T15:27:00.003-05:00

Griffon Validation Plugin v0.6 was released today. In this release the validation plugin's artifact handling has been rewritten to be compatible withe the up-coming Griffon 0.9.2 release. Also in this release validation plugin now simulates inheritance effect for the constraints you define. Since the constraints are defined using static fields following Grails convention, no real inheritance can be implemented however now with 0.6 release you class will basically copy the parent class' constraints, and additionally you can also override the parent constraint. See the following example:


@Validatable
class ServerParameter {
   @Bindable String serverName
   @Bindable int port
   @Bindable String displayName

   def beforeValidation = {
       setDisplayName "${serverName}:${port}"
   }

   static constraints = {
       serverName(nullable: false, blank: false)
       port(range: 0..65535)
       displayName(nullable: false, blank: false)
   }
}

@Validatable
class ProtocolSpecificServerParameter extends ServerParameter{
   @Bindable String protocol

   def beforeValidation = {
       setDisplayName "${protocol}://${serverName}:${port}"
   }

   static constraints = {
       protocol(blank: false, nullable: false)
   }
}

In the above example, the ProtocolSpecificServerParameter will not only inherent ServerParameter's serverName and port fields but also their associated constraints. The only resitration you need to be aware of is if the parent constraint generates error for a certain condition then the overriding child constraint has to generate error as well. In other words, validation plugin does not allow error hiding by using constraint override in the child class, similar to the method exception handling during inheritance within Java.

JNDI Warrior v0.2 Released

2011-02-20T16:49:00.003-05:00

JNDI Warrior v0.2 was released today. In this version you can now save your JNDI connection settings including it's classpath for future reuse. As well a simple sorting capability was added to the JNDI tree browser to help navigating through big JNDI tree commonly see in large enterprise. You can download the latest binary from here.

You are always welcome to use our Trac to submit any bug report or feature request.

Enhanced connection dialog:

Sample pom.xml For Groovy Project with Spock

2011-01-27T16:58:00.004-05:00

Had to setup a new Groovy project using Maven today with Spock as its testing framework. Ran into a few glitch; thought to share it here with a sample pom.xml as a day-log.






   4.0.0

   sample-group
   sample
   General Purpose Groovy Project
   jar
   1.0-SNAPSHOT

   
       0.4-groovy-1.7
   

   
       
           
               org.codehaus.gmaven
               gmaven-plugin
               1.3
               
                   
                       
                           1.7
                       
                       
                           generateStubs
                           compile
                           generateTestStubs
                           testCompile
                       
                   
               
           

           
               org.spockframework
               spock-maven
               ${spock.version}
               
                   
                       
                           find-specs
                       
                   
               
           
       
   

   
       
           org.codehaus.gmaven.runtime
           gmaven-runtime-1.7
           1.3
       

       
           org.spockframework
           spock-core
           ${spock.version}
           test

Few things I ran into:

Maven 3 currently does not work with the Spock plugin
Spock plugin version 0.5-groovy-1.7 does not seem to pick up unit spec test correctly, so I had to fall back to 0.4 to make it work
In order to mix both Java and Groovy code you need to enable the stub generate execution phase in GMaven plugin

JNDI Warrior v0.1 Released

2011-01-04T15:42:00.007-05:00

It started as a simple demonstration application to showcase Griffon framework’s capability to deliver rich Swing based client application with unprecedented speed. However the demo itself turned out to be quite a handy little tool especially for Java EE developers, I personally had used it a couple of times to debug some odd JNDI lookup and classpath issues in the last a couple of weeks. So I have decided to officially host it on SourceForge.net as an open source project - JNDI Warrior.

Its a simple Swing based client utility application which allows you to connect to a given JNDI provider typically found with any Java EE application server. It has three JNDI templates built-in (Weblogic, JBoss AS, and Glassfish) to help you filling some of the blanks, but in theory it should work like any Java based JNDI client with any provider by manually entering your JNDI settings. You can download this little application from here.

You are always welcome to use project Trac to submit any bug report or feature request.

Happy new year! and enjoy :)

Here are a few screen shots:

What Do You Want From Your Team

2010-12-16T15:16:00.001-05:00

I have been working as an independent consultant on-and-off for the last 6 years, and because of that I got to work with many different companies and management styles. Also being a consultant giving me an unique and almost neutral third-party perspective whenever I work with a company. One interesting observation I made in these years was despite all the time, energy, and training spent in many companies, a lot of the managers don't seem to know what exactly they want from their employees. That sounds too harsh, you might say. Or is it? Let me give you an example. One of a recent email I witnessed from one of the manager in my client's company stated that they want all the employees to start their day and be ready on company IM no later than 9:30am. Ok, it appears like a perfectly reasonable request when you first look at it, however the only problem is the team had an implicit flexible working hour arrangement for months already if not years. So now let us see what exactly does this manager achieved by doing this:

People are going to start their working day earlier however grumpy they are. They might need to negotiate with their spouse to re-arrange their pick up and drop off schedule for their kids. Some might need to switch from public transit to car or vice versa since they now have to join the rest of the city in a mad rush to work every morning.
People who starts early are for sure gonna leave early, so the total number of productive hours per day you get from each employee is not going to change. The only thing this changes is how crowded the subway is when you employee leaves the office.
People might start asking, especially the talented ones in your team, why they rejected that generous offer from a competitor last month, so they can enjoy the daily rush hour?

You might ask “Is it really this bad? All they are asking is for people to be on time.” I admit I might have stretched the reaction a bit, however I can tell you in my experience I have seen number of teams dismembers themselves because something small like this eventually snowballed out of proportion. What I am trying to bring out here is not “Don't ask you employee to come early” but rather as a manager before you ask employee to do anything think about what exactly you want to get from your team. As Archimedes famously said, “Give me a lever and I can move the world.” In our little work society mostly governed by Behavioral Economics, everyone reacts to incentives therefore like some economist would like to say “Give me a proper incentive scheme and I can get anyone to do anything”. This may sound arrogant or even devious, however we should not underestimate the power of incentives in our micro-society at work. In my personal opinion everything a manager/leader does is either an incentive which motivates people towards a common goal or negative-incentive which motivates people to work against the common goal.

Lets take a closer look at this particular manager and the problem he is facing. In reality the problem that he is trying to tackle is loss of productivity as well as lacking predictability in project schedule. He is under a lot of stress delivering a major project for the company, but his team had a bad track records of under delivering and schedule overrun in a form of either massive OT or last minute show stopper or both. Thus what he really wants from his team is improved productivity, predictability, and better quality of work. Now if we also take a look at how the team works:

Team works strictly within a waterfall process with no iteration or even intermediate milestone to serve as check points to prevent last minute breakdown
Most of the developers in the team does not practice any form of unit test but rather code by vague design and imagination
There is very limited daily/weekly status meetings of any sort so everyone is working in their own black box with little knowledge of overall progress until its too late
Architect and designer do not code in this team, and all developers are either contractor or outsourced therefore there is an ever-increasing knowledge gap between people who creates design and people who implements them

Its not hard to find many much better incentives that could have worked in team's favor than the 9:30am start time.

Use Flat Local Folder as Custom Repository for Grails

2010-11-15T15:38:00.002-05:00

Currently I am introducing Grails as one of the new web application frameworks to my client in their existing environment, however their environment currently does not rely on Maven or Ivy repository manager to distribute and share their internal libraries but rather checked into version control under one directory flat. To make Grails recognize this flat file folder as one of the dependency repository, we had to configure the BuildConfig script with a custom Ivy resolver. I thought someone else might find its useful if you are dealing with legacy build environment.


// create a custom file based resolver
def libResolver = new org.apache.ivy.plugins.resolver.FileSystemResolver()

// specify common lib directory
def commonLibDir = new File("c:\commonLib")

// specify patterns
libResolver.addArtifactPattern(commonLibDir + "/[artifact]")
libResolver.addIvyPattern(commonLibDir + "/[artifact]")

libResolver.name = "my-repository"

grails.project.dependency.resolution = {
   // inherit Grails' default dependencies
   inherits("global") {
       // uncomment to disable ehcache
       // excludes 'ehcache'
   }
   log "warn" // log level of Ivy resolver, either 'error', 'warn', 'info', 'debug' or 'verbose'
   repositories {
       grailsPlugins()
       grailsHome()
       grailsCentral()
     
       resolver libResolver
   }
   dependencies {
       // here the groupid and version are basically ignored since our
       // flat lib folder does not maintain such information
       compile 'yourcompany:artifact.jar:0.0.0'
   }
}

Q&A on Refactoring and Architecture in Agile

2010-08-05T16:29:00.003-04:00

Recently I answered a list of survey questions, on refactoring and evolving architecture in Agile project, from Lianping Chen - a doctoral researcher from Lero, University of Limerick. Here I want to share it with the community and feel free to share your comment and experiences.

1. Have you seen/experienced cases where software architecture emerges successfully from continuous small refactoring?

Yes

2. What are the main reasons that make the emerging of software architecture from continuous small refactoring success, according to your observations/experiences?

Agility in the engineering practice is definitely a must supplemented by the agility in the process as a secondary but important aspect. I also find having a light up-front design to get everybody to agree on a very high level architecture has also been very useful in guiding the refactory. Here when I talk about agility in engineering I refer to implementing the full set of agile practices: unit test, high coverage, continuous integration, refactory, and ideally also TDD and PP, since in my experience all these practice compliment each other, strengthen each other while eliminating the pitfalls that otherwise might be introduced by other practices. For example refactoring without high unit test coverage is more like hacking than engineering since you have no idea whether certain change you make will break other part of the system, and unit test without refactory will definitely introduce rigidity in architecture compounded with light up-front design could contribute to severe and rapid architectural deterioration thus compromise in quality and productivity.

3. Have you seen/experienced cases where architecture failed to emerge from continuous small refactoring?

Yes

4. What are the main reasons that cause such unsuccessful emerging of software architecture from continuous small refactoring, according to your observations/experiences?

It happens actually quite often when you have a team that only practices the process side of the agile methodology, or pick and choose some aspects of the engineering discipline for example only implement refactory but not unit test.

5. Could you please describe some cases/examples of such unsuccessful emerging of software architecture from continuous small refactoring?

...

If it is convenient, could you please also provide the following information?

1. How many years of experience do you have in using agile methods?

7 years

2. Which agile methods (e.g., Scrum, XP) have you used?

Both, and I prefer XP due to the fact many teams in the field that chooses to implement Scrum are the ones who choose to only implement the easy process of Agile but deliberately choose to ignore the hard engineering aspects of Agile

3. Which application domains (e.g., automobiles, telecommunications, finance, medical devices, web-based socio-technical systems, and etc.) you have been working in?

Telco, Online Gaming, and Government

4. What is your definition/understanding of software architecture? (You can refer to a definition you like or give your own understanding.)

I like Ralph Johnson's definition mentioned in XP mail listing:

In most successful software projects, the expert developers working on that project have a shared understanding of the system design. This shared understanding is called ‘architecture.’

In short architecture is a group consensus among expert developers.

One last thing I would like to mention is that initially I always believed any team can adopt Agile practice and be effective, however after almost 8 years of trying to convince and help people adopt Agile I realized that not every team or every one can adopt Agile. To successfully adopt Agile a team has to be:

Very experienced (PP lower the requirement of experience a bit since it helps disperse knowledge and train less experienced developer a lot more efficiently)
Having an open, blame-free, and trusting environment
Deep and thorough understanding on not just the practice of Agile but also the reason behind Agile practice among team members (An Agile coach is required if local expertise lacks in this area)
Having management backing and overall Agile adoption even outside of the development context

The most successful Agile adoption in my experience are the ones with all the above quality and combined with top-down support and mandate with bottom-up organic initiatives.

Hope this help a bit, and I am quite interested in your research and findings so once its published I would like to have a copy if possible.

Griffon Validation Plugin 0.5 Released

2010-07-22T17:06:00.005-04:00

I am happy to announce v0.5 release of the Griffon Validation plugin. This release was mainly designed to upgrade Validation plugin to be compatible with Griffon 0.9 core, plus some minor internal clean up. To upgrade your application to this version you need to first upgrade your Griffon installation to 0.9 by:

Download Griffon 0.9 from http://griffon.codehaus.org/Download
Run "griffon upgrade" under your application folder

Then run "griffon install-plugin validation" to upgrade the plugin

Other than the core upgrade, another somewhat important change in this release is now Validation plugin officially depends on i18n plugin, so once you upgrade to v0.5 Griffon will automatically install i18n (currently 0.2) into your application if its not already installed yet.

For more information on GValidation plugin please check out the Wiki page, and as usual please feel free to contact me for any bugs or feature requests.

Griffon Validation Plugin 0.4 Release

2010-06-22T16:03:00.004-04:00

Just released GValidation plugin 0.4. In this post I would like to highlight some of enhancement and new feature introduced in this release.

1. @Validatable AST Transformation

In the previous version user can enhance regular Groovy class in their Griffon application manually using:

ValidationEnhancer.enhance(object)

With 0.4 release now this manual process can be replace by the convenient @Validatable annotation thanks to Groovy AST transformation. Any class now annotated with @Validatable will be automatically synthesized with validation capability.


import net.sourceforge.gvalidation.annotation.Validatable

@Validatable
class AnnotatedModel {
   String id
   String email = " "

   static constraints = {
     id(nullable: false)
     email(email: true)
   }
}

Other than the fact that annotated class is enhanced during build-time instead of run-time as for the models and manually enhanced objects, they behave identically from user's point-of-view.

2. Bindable errors

In this release the dynamic errors field in models and Validatable objects has been enhanced to be Bindable, hence standard SwingBuilder binding can be utilized to tie errors automatically to a component. This is particularly handy when building error notification component to display error message for the user. For example with this new capability now you can simple bind the error to the build-in ErrorMessagePanel without manually synchronizing it after each validation.


container(new ErrorMessagePanel(messageSource),
                id: 'errorMessagePanel', constraints: NORTH,
                errors: bind(source: model, 'errors'))

3. beforeValidation callback

Inspired by Rails before_validation callback, now GValidation provides a similar pre-validation callback to give model developer a chance to manipulate data right before validation. Here is an example how this kind of callback can be used:

class ServerModel {
 @Bindable String serverName
 @Bindable int port
 @Bindable String stringForm

 def beforeValidation = {
     setStringForm "${serverName}:${port}"
 }

 ….
}

For more information please check out the GValidation Wiki Page. As usual your feedback and comment are always welcome.

How to support custom artifact type in Griffon plugin

2010-06-02T14:51:00.006-04:00

A newer revised version of this guide (compatible with Griffon 0.9.2+) is now available here.

Out-of-box Griffon supports 4 different types of artifacts: Model, View, Controller (MVC) plus Service, these are the major building blocks of Griffon. Just like it's cousin Grails, in Griffon plugins and addons can also introduce new artifact types, however the process is fairly different from what Grails employs. In the recent 0.3 release of the GValidation Plugin a new custom artifact type has been introduced - Constraint, and I would like to share some of my learning here so it would be a little bit easier if you are planning to do something similar.

Step 1 - Handle your artifact

To support a new artifact type you have to tell the Griffon core about the artifact type first. You can achieve this by implementing your own ArtifactHandler, for most of the common cases extending from ArtifactHandlerAdapter will be enough otherwise you can implement the interface manually. Here is what it looks like for the Constraint artifact type:


class ConstraintArtifactHandler extends ArtifactHandlerAdapter {
   public static final String TYPE = "constraint"
   public static final String TRAILING = "Constraint"

   ConstraintArtifactHandler() {
       super(TYPE)
   }
}

Step 2 - Register your artifact

As shown above, if you have experience working with Grails artefact support, you will notice the handler implementation is almost identical in Griffon, however things starting to differ from this point forward. Now you have the handler implemented, next thing is to register it with Griffon core. This is best achieved during initialization phase in your addon. Open the [PluginName]GriffonAddon.groovy file add the following callback if its not already there:


def addonInit = {app ->
   ....
   app.artifactManager.registerArtifactHandler(new ConstraintArtifactHandler())
   ....
}


eventCollectArtifacts = { artifactsInfo ->
   if(!artifactsInfo.find{ it.type == 'constraint' }) {
       artifactsInfo << [type: 'constraint', path: 'constraints', suffix: 'Constraint']          }  }


eventStatsStart = { pathToInfo ->
   if(!pathToInfo.find{ it.path == 'constraints'} ) {
       pathToInfo << [name: 'Constraints', path: 'constraints', filetype:['.groovy','.java']]          }  }


target('default': "Creates a new constraint") {
   depends(checkVersion, parseArguments)

   promptForName(type: "Constraint")

   def name = argsMap["params"][0]
   createArtifact(name: name, suffix: "Constraint", type: "Constraint", path: "griffon-app/constraints")
   createUnitTest(name: name, suffix: "Constraint")
}


@artifact.package@class @artifact.name@ {

   def validate(propertyValue, bean, parameter) {
       // insert your custom constraint logic
   }

}

Phew, now finally we are done. This is a long post and as you can see a lot of plumbing; this is exactly why currently there are some discussion going on within Griffon dev mail list to provide declaration based custom artifact support either using Grails style or Groovy AST transformation, so stay tuned for future updates on this topic.

I also need to thank Andres Almiray for pointing me to the right direction when I first started looking into this topic.

Griffon Validation Plugin 0.3 Release

2010-05-26T12:43:00.008-04:00

I am happy to announce the 0.3 release of GValidation plugin - a Griffon validation plugin designed to provide Grails like constraints and validation support. In this post I would like to introduce a few new features implemented in this release.

1.Custom Constraint Support

This is probably the biggest improvement introduced in this release. Inspired by Grails' Custom Constraint plugin, once upgraded GValidation will introduce a new type of artifact to your Griffon application - Constraint.

In previous version GValidation allows you to define a custom validator using a closure just like in Grails:

// Simple custom validator
even( validator: {
 return (it % 2) == 0
})

However this kind of closure based simple validator is hard to reuse therefore you will have to rewrite them every single time, a major inconvenience and a violation of the DRY principle. In version 0.3 now you can create a top level reusable custom constraint by using the following script:

griffon create-constraint [package].[constraint-name]

A Groovy class will be created under griffon-app/constraints folder to allow you to define your custom validation logic, a typical custom constraint looks like this:

class MagicConstraint {

   /**
   * Generated message
   *
   * @param propertyValue value of the property to be validated
   * @param bean object owner of the property
   * @param parameter configuration parameter for the constraint
   * @return true if validation passes otherwise false
   */
  def validate(propertyValue, bean, parameter) {
      if (!parameter)
          return true

      return propertyValue == 42
  }

}

Once created a custom constraint pretty much behaves exactly like a built-in constraint, you can easily invoke them in your model by following the simple naming convention, with the above example you can apply the constraint on any field in your model by using the following declaration:

class DemoModel{
  ….

  @Bindable int magicNumber

  static constraints = {
      ….
      magicNumber(magic: true)
      ….
}

2. Selective Validation

Originally proposed by Andres Almiray, GValidation now offers capability to perform validation on only a selected number of fields in the model instead of all. Here is a typical single field validation usage scenario:

model.validate('name')
...
if(model.hasErrors()){
  // notify user
  ...
}

You can also perform selective validation on a list of fields:

model.validate(['name', 'email'])
...
if(model.hasErrors()){
  // notify user
  ...
}

3. Default Catch-All Error Message Code

This is a minor enhancement however a great time saver for someone who has to perform a large number of validation in their app. In previous version validation plugin only generates model specific error message code:

[modelclass].[field].[validator].message

Now for every built-in and custom validator the plugin will also generate a default error message code additionally:

default.[validator].message

You can retrieve the error code and default error code from the Error object using the following fields respectively:


error.errorCode
error.defaultErrorCode

For a complete guide on the plugin, please check out the Wiki page.

Quote of the Day

2010-04-19T14:25:00.003-04:00

Most teams purporting to be doing agile software development are not applying the level of technical rigor necessary to succeed at it. Most "agile" team have actually only adopted Scrum's project-management practice and have failed to effectively adopt "the hard disciplines" like test-driven development, refactoring, pair programming, simple design, and continuous integration.

- Formal Versus Agile: Survival of the Fittest - IEEE Computer 2009 September

Use controller action via listeners in Griffon view (Updated)

2010-04-16T15:25:00.004-04:00

I have been asked recently a couple of times about how to use listeners in Griffon view for event handling such as mousePressed or focusLost. Apparently neither Swing builder nor Griffon documented this aspect in details, and it has been a bit confusing for some folks to figure it out since the usage is slightly different from regular actions.

In Griffon view you can supply a predefined action to a component such as JButton to invoke a controller action in a breeze. The following code snippet demonstrate how it is done in Griffon:


actions {
   action(id: "quitAction",
           name: messageSource.getMessage('menuItem.exit.caption'),
           mnemonic: "x",
           closure: controller.quit)
}

...

menuBar {
       menu(messageSource.getMessage('menu.file.caption')) {
           menuItem(quitAction)
       }
}

...

From Andres Almiray:

The main difference between all examples is that the first assigns an instance of javax.swing.Action to the node's value. The builder knows that it can configure a button/menuItem/etc out of an Action so it does it.

When it comes to listener and event handling, one would expect a similar pattern applies; for example according to Swing builder documentation mousePressed event exists on JComponent level so one would hope the following code can bind a controller action to an mousePressed event.


actions {
    action(id: "nodeSelectionAction",
           closure: controller.selectNode)
}

list(id:'nodeList', model: model.serverListModel, eventPressed: nodeSelectionAction)

From Andres Almiray:

The second example does not work because it attempts to assign an instance of javax.swing.Action as the value of the menuPressed property, which has to be a MouseListener. That's why it breaks (although the error text may be misleading).

Unfortunately you will get No such property: mousePressed for class: javax.swing.JList error message. Instead the correct way to do this is by using the closure instead of an Action since mousePressed expects a listener. Groovy closure declared here will be converted to a listener automatically:


list(id:'nodeList', model: model.serverListModel, mousePressed: controller.selectNode)

From Andres Almiray:

The third example works because it assigns a closure as the value of the mousePressed property. Groovy will generate under the covers a proxy of a MouseListener that uses that closure as the implementation of said proxy's mousePressed method.

Hope this post will save some investigation time for someone who is new to Griffon and Swing builder.

Go Griffon!

GValidation 0.2 release

2010-04-06T10:39:00.005-04:00

After I released the very first version of the Griffon Validation plugin, I have received quite a few positive feedback, and thank everybody for all the feedback. Based on some initial bug report I have fixed some dependency problem with the initial version now the plugin is shipped with all dependencies it requires so you don't need to manually add jars into your project. Currently GValidation depends on:

```
Apache Commons Lang 2.5
```
```
Apache Commons Validator 1.3.1
```
```
Jakarta ORO 2.0.8
```

In the 0.2 release I have also fixed some inconsistency in some of the validator implementation, now all validator except nullable and blank will ignore both null and blank value. In other words, unless you provide nullable or blank validator the validation logic will assume all fields are optional by default.

To upgrade your application to the new plugin version please uninstall validation plugin first:

griffon uninstall-plugin validation
griffon install-plugin validation

I have also updated the Wiki with a quick tutorial showing how validation can be utilized in a simple but somewhat telling example. The same tutorial demo application binary can also be downloaded from the project files page.

Grails like Validation plugin for Griffon

2010-04-01T16:51:00.008-04:00

Recently I am working on a Griffon powered Swing application in one of my open source project Hydra Cache. While really love the convention over configuration setup and the grooviness of swing builder, soon I realized one tool I rely on frequently in Grails is missing in Griffon. Griffon does not tie the model to database via GORM as Grails does, which makes perfect sense since when building rich client application in many cases your model layer is not (or some will argue should not) directly tie to the database, after all that's why the 3-tier architecture was invented in the first place. That's when I start thinking if the convenient declaration based constraint and validation support in Grails can live without GORM but be applied on simple POGO model classes in Griffon. After writing validation listeners on the view a couple of times, my laziness pushed me to create a plugin to do just that. After a few positive and encouraging email exchange on dev@griffon.codehaus.org mail list and a few late night coding session, the first release of GValidation plugin is now ready for your evaluation. The plugin is implemented without dependency on Spring framework and purely implemented in Groovy.

To install the plugin use:

griffon install-plugin validation

Once the plugin is installed the following field and method will be automatically injected into your model classes:


Errors errors
boolean hasErrors()
boolean validation()

Here error field is a Groovy clone of Spring Validation Errors class in order to retain certain API consistency when compared with Grails validation support. Now in a typical scenario you can achieve the generic validation by first declare your constraints in the model:


class PersonModel {
@Bindable String name
@Bindable String email
@Bindable String blog

static constraints = {
  name(blank: false)
  email(blank: false, email: true)
  blog(url: true)
}
}

Then perform the validation, usually in a controller action:



def doSomething = {evt = null ->
if (!model.validate()) {
  doLater {
      // display error messages
  }
} else {
  doLater {
      // clear error messages
  }

  doOutside {
      // do something interesting
  }
}
}

You can also check and manipulate the errors generated by validation method using the errors object directly:



model.validate()
// do something else
..
if(model.hasErrors()){
model.errors.each{error->
    // do something with the error
}
}

The only validators that are not ported from Grails are:

scale
unique

Since they are largely there to influence database schema generation rather than performing actual validation. A new validator inetAddress is introduced to ensure the field is a valid host name or IP address. For a complete list of the built-in validator shipped with this plugin please see the GValidation Wiki.

Just like Grails constraints, you can also implement your custom validator using Groovy closure:



// Simple custom validator
even( validator: {
return (it % 2) == 0
})


// Custom validator with access to the object under validation
password1( validator: {
val, obj ->
 obj.properties['password2'] == val
})


// Custom validator with custom error
magicNumber( validator: {
val, obj ->
def result = checkMagicNumber()
if(!result)
   obj.errors.rejectValue('magicNumber', 'customErrorCode')
return result
})

And last but not least once installed, GValidation allows you to enhance any POGO object in your application to have validation support, just simply use the ValidationEnhancer with your object:



import net.sourceforge.gvalidation.ValidationEnhancer

ValidationEnhancer.enhance(pogo)

pogo.validate()

poso.errors.each{
  println it
}

For more information please check out the Wiki page, and feel free to send me your feedback or suggestion. For bug report please use the Bug Tracker.

Use Grails (GORM) with legacy Oracle 9i database

2010-03-29T16:57:00.002-04:00

I am currently consulting on a project with a client who is using the legacy Oracle 9i database with JDK 1.4.2. I have recommended Groovy and Grails for their new enhancement project, however to add some grooviness to this kind of legacy platform introduce some pretty unique challenges. In this post I will attempt to compile a list of hurdles we went through to get Grails working perfectly with the legacy environment.

Grails version

First thing you need to know is that only Grails 1.0.x is compatible with JDK 1.4 so only limited grooviness is available but major improvement nevertheless from our client's existing environment.

Hibernate Dialect

Some of the meta API that Hibernate relies on to pick the right database dialect is not available with 9i so in your DataSource.groovy use the following configuration to specify a dialect manually.


dataSource  {
    ....
    driverClassName = "oracle.jdbc.OracleDriver"
    dialect = "org.hibernate.dialect.Oracle9iDialect"
    ....
}

Table Level Mapping

A few thing needs to be configured on pretty much every one of your domain class to work with existing data schema. First map the table to an existing table (most of the legacy table naming convention will not be same as what hibernate expect unless you are extremely lucky ;-), and secondly turn off the optimistic locking so hibernate will not look for the version column on the table. This can be achieved by adding this static mapping field on our GORM domain class.


static  mapping = {
        table "LEGACYORACLETABLE"
        version false
}

ID Generation Strategy

Most likely the legacy database might have some pretty custom ID generation strategy. To have the flexibility to map your domain class id to the legacy column you need to define the id field manually. Normally GORM/Hibernate automatically map the id field for you (usually using bigint or number of some sort) but manually defining it allows you to map it to pretty much any kind of data type.


String id

static  mapping = {
        id column: 'ENTITYID', generator: 'assigned',  type: 'string', sqlType: 'varchar2'
}

We have found keeping the id field for every model class with the default name "id" makes the scaffolding and view generation working for most of the scenarios even with the legacy database, which is a big time saver for us.

For auto increment id with sequence you can rely on the hibernate default sequence, just make sure to create the sequence "hibernate_sequence" before you start the application. Or you can specify your generator using the following mapping:


Long id

static mapping  = {
        id column: 'ENTITYID', generator: 'sequence',  params:[sequence:'custom_seq'],  sqlType: 'integer'
}

Install plugins with Grails 1.0.x

The Grails plugin repository has grown tremendously in size, and the out-dated install-plugin script used by Grails 1.0.x does not seems to be able to handle the large list, besides it tries to update the entire plugin list each time which is very time consuming. What we found the best way to install plugin with Grails 1.0.x is by pin down the plugin yourself first, and then just install the plugin using the URL directly. Just make sure the right plugin version is used since not all the plugin works with Grails 1.0.x. All Grails plugins can be found here: http://svn.codehaus.org/grails-plugins/

For example to install Testing plugin (part of Grails core since 1.1) using the following command:
grails install-plugin http://svn.codehaus.org/grails-plugins/grails-testing/trunk/grails-testing-0.5.zip

After this you should be well on your way to unleash some grooviness with your perfectly aged environment. Time to impress the heck out of your boss and revitalize your work environment :D

Stub static method on domain class in Grails explained

2010-02-10T15:54:00.003-05:00

Since Grails 1.1 the unit test capability has been greatly improved for pretty much anything you can imagine in Grails from domain class to controller even taglibs can be easily tested in plain old junit with the help of GrailsUnitTestCase and many of it's cousins such as ControllerUnitTestCase specialized in different aspects of the framework.

However the fantastic mocking framework is not without its boundary. The mockDomain method provided by GrailsUnitTestCase has its limit, it does not provide the capability to fully mock GORM dynamic finders as well as criteria and executeQuery methods. Luckily the test framework also provide the ability to mock any method regular or static using mockFor() method (details) however it comes short when you have static query method created on domain class. Imagine the following example:


class Foo{
        ...

        static def listAllAwesomeFoo(keyword, params) {
                Foo.executeQuery('''select f from Foo as f
                            inner join f.tags as t
                            where f.awesome = true and t.keyword = ?''',
                        [keyword],
                        params)
        }
}

class FooService {
        def doSomethingUseful(){
                def awesomeFoos = Foo.listAllAwesomeFoo('something', [max:10])
                ....
        }
}

Now to test the FooService you need the ability to stub or mock the Foo.listAllAwesomeFoo() method and since its a static method there is really no easy way to do it even with Grails test framework; or is there? This is where Groovy MOP can save you a lot of grief and time. To stub this kind of static method all you need to do in Groovy is add the following code before your test code:


Foo.metaClass.static.listAllAwesomeFoo = {keyword, params -> [foo1, foo2] }

Thats it. Now you can happily test all your service logic without worrying about stubbing the whole database to run the HQL. Later on you can always isolate and use integration test with in-memory database to test the HQL separately.

Create Open Flash Chart in Grails without using plugins

2010-02-03T15:26:00.008-05:00

If you are like me tired of outdated Grails plugins and verbose API in Java for Open Flash Chart 2, here is a short demonstration of generating OFC2 using just naked Grooviness. OFC2 charts are driven by JSON data feed, and with Grails' built in JSON converter and support you really don't need another API or plugin to use OFC2. Just download the flash binary and javascript from http://teethgrinder.co.uk/open-flash-chart-2/ and create a controller in your Grails application. Now you have two choices, you can either use the JSON builder in your controller or just plain maps to generate JSON data. I chose plain maps since its easier to manipulate and examine in unit tests, but the JSON builder will work just as well with even more expressive DSL. Here is an example of a simple chart:


def chartData = [
    "elements": [ ["type": "line", "values": [ 1, 2, 1] ] ],  
    "title": [ "text": "Wed Feb 03 2010" ]
]
render chartData as JSON

Now you are free to use the latest OFC binary without worrying about upgrading the plugin or mocking the complex API in your test. The syntax is extremely similar to the JSON output so you can basically learn from checking out the examples on OFC site. Here is a more complicated example with tooltips, on-load animation and custom look-and-feel.


[
    'elements': [
        ["font-size": 10, "text": "Daily Info", 
         "tip": '$#val# 
 on #x_label#', "type": "line", "width": 4,
         "dot-style": ["type": "dot", "dot-size": 5, "colour": "#DFC329", "tip": '$#val# 
 on #x_label#'],
         'values': values, 'colour': '#8F9CFF', "on-show": ["type": "drop", "delay": 0.5]
        ]
    ],
    "is_decimal_separator_comma": 0,
    "is_fixed_num_decimals_forced": 0,
    "is_thousand_separator_disabled": 0,
    "num_decimals": 2,
    'title': ['text': "Sample Chart"],
    'x_axis': ['steps': 10, labels: ['steps':10, 'visible-steps': 5, 'labels': labels], "colour": "#0067A6", "grid-colour": "#FFCAA8"],
    'y_axis': ['min': 0, 'max': yMax, 'steps': calculateYAxisSteps(yMax), "colour": "#0067A6", "grid-colour": "#DEFFA8"],
    'bg_colour': '#FFFFFF',
    "tooltip": [
        "colour": "#5CC0FF", "background": "#FFFFFF", "title": "{font-size: 14px; color: #5C6FFF; font-weight: bold}", 
        "body": "{font-size: 11px; font-weight: bold; color: #000000;}"
    ]
]

Grails JQuery plugin 1.4 does not work in WAR mode

2010-01-29T11:53:00.003-05:00

Haven't blogged for a while. The arrival of my baby daughter has kept my life quite busy and full of joy, also my sleep deprivation level at a level hasn't been seen since university time.

First day after my paternity leave, back at office upgraded a current Grails project to the fantastic Grails 1.2 and getting ready to release for the iteration. However as soon as I start running the application in WAR mode, I was getting strange exceptions:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'JQueryTagLib': Initialization of bean failed; nested exception is org.springframework.beans.ConversionNotSupportedException: Failed to convert property value

After a bit of digging I realize its caused by the JQuery plugin after I upgraded to 1.4 version. Apparently in JQueryTagLib a typed non-private field "GroovyClassLoader classLoader" was the source of the problem. Because its not private Groovy compiler automatically generates getter and setter for this field, and when Spring tries to inject dependency for this field a class conversion problem happens since the field is typed as strictly GroovyClassLoader, since the native class loader in a WAR environment usually will not be a GroovyClassLoader.

I have contributed my finding to the JQuery plugin project, and the fix is incorporated in the 1.4.1 release, but for whatever reason if your project suffers from the same issue and can not upgrade for the moment the easiest fix would be just simply change the classLoader field to private in JQueryTagLib class under your plugin directory.

Lessons learned from Impact pitch competition

2009-11-24T13:14:00.000-05:00

By this time everyone can probably guess that GoEyeball did not win the pitch competition at Impact National 2009 conference. Otherwise you probably would have seen me flashing the giant check of 10K ;-) (although not in cash but in service with KPMG). Today I am writing this blog to share some of the experience and lessons learned from this competition, so I guess we can score a few wins from this loss.

#1 Pitch an idea especially a new one that involves innovation in technology in 5 mins elevator pitch is not a easy task

We actually received feedbacks in conversations after the pitch that a lot of the audience, including the judge, did not understand or realize that we have invented a new technology to provide a service that is not available from our competitors

#2 You need to be clear on what you are asking for

After the competition, my business partner and I regrouped and did some retrospective on the performance and realized we were not clear on the message. Here is what we believe a better format for our pitch could be:

Who we are?
What problem do we solve?
Why will we be successful?
What do we need? Funding? Mentor?
What is in there for you? Exit strategy for investors.

We did not win the top prize, but we still had a great time at Impact. This is only our first appearance for public pitching; we will continue working on it, but most importantly we will continue to have fun after all that was why I quite my full time job in the first place - to have FUN. So far it has been a journey of every bit of fun I could have ever imagined :-D

GoEyeball is one of the finalists for Impact National 2009

2009-11-19T12:59:00.002-05:00

I am very proud to announce that GoEyeball.com is one of the 4 finalists at Impact National Conference 2009 pitch competition hosted from 3:30-5:00pm on Saturday Nov 21 at:

Westin Harbour Castle

1 Harbour Square

Toronto, ON M5J 1A6

We are also an Exhibition Sponsor for the conference with a booth in exhibition hall on Friday afternoon to showcase our technology and answer any question from users, potential investors, and any one who is interested in our idea. If you are attending the conference or know some one attending please come check out our booth and show your support at the competition.

Will keep everyone updated with the result on this blog.

JointSource is open for business in a new office

2009-11-02T11:56:00.004-05:00

We are now open for business in our new office near Yonge and Eglinton (3 mins walk from the subway station). Our new address:

120 Eglinton Ave East, Suite 1000
Toronto, ON Canada M4P1E2

Phone: 416.322.2931 Fax: 416.322.2877

Email: nzhu@jointsource.com
Website: www.jointsource.com

View Larger Map

My New Business Venture

2009-10-26T14:00:00.006-04:00

Last week after a long talk with my wife and with her blessing I made one of the most important decisions I have ever made, other than marrying the woman I love :-), to quit my individual consulting career and start working full-time to develop my own software business specializes in what I call “UnSourcing”. I have been doing independent consulting on and off for the last 6-7 years, the outsource experience I had in Siemens, Rogers, and many other firms convinced me something I have always suspected

“With a highly capable team equipped with a proper Agile process, it should be able to build high quality custom software with much less TCO comparing to a typical off-shore outsourcing vendor“

The reasons are simple. The outsource model works great in manufacturing because of the revolutionary work done by Federick Winslow Taylor (Scientific Management) and Henry Ford (Assembly Line) and many others like them throughout the 20th century. The manufacturing process is virtually optimized to the limit; the only way left to reduce cost is by simply reduce the cost of labor. Furthermore because the process is so streamlined and well defined outsourcing it to a different country literally has no major impact to neither the quality nor productivity of the process at all. In software industry despite how much we have been trying to fit the manufacturing model, we are not manufacturing software and coding is not an act of manufacture or construction but rather an on-going design process.

“Source code itself is a major part of the design. The construction of the software is done by the compiler and machines cheaply and flawlessly, but not by the developers.”

Since coding is more a design activity than manufacturing, more of a craft than engineering, that’s why a master programmer can be up to 28 times more productive than a not-so-experienced counterpart. With that in mind, its not difficult to understand before we go for off-shore outsourcing there are actually huge amount of optimization we can do in other parts of the system to lower the cost. The recent Agile movement in the industry is a perfect example of how much waste we can cut out before we resort to the labor cost.

I am starting my business with my business partner in the hope of being able to help companies who have suffered from outsource mess to actually deliver high quality software and solutions with true business value and most importantly with much less TCO. In the next a few months I will record our journey here in my blog, and of course we appreciate anyone who would like to share their experience, advices, and leads with us if you would like to see this somewhat quality obsessed software shop to survive this largely manufacture minded industry out there :)

DuctTape Guy or Craftman?

2009-09-30T11:11:00.000-04:00

I just finish reading The DuctTape Programmer from Joel Spolsky and Uncle Bob's response this morning, and I have to say what a contrast in their views of how to define a good programmer. I completely agree with Joel as a programmer for business in a real world especially if you are running or working for a startup company where you believe there are only a few months window to achieve what you set out to achieve before somebody else eats your lunch, over engineering or pre-mature optimization over business functionality is definitely stupid and sometime even lethal. If you can achieve the same functionalities with a simpler design or a simpler tool or both then stick with the simpler version. Remember

"Simplicity is the ultimate sophistication." -- Leonardo da Vinci

Once you have been trying to make everything you bulid as simple as possible but not any simpler, you will probably realize its actually not a "simple" goal to achieve sometimes even turning out way more difficult than using the complicated alternative. However like Uncle Bob I strongly disagree with Joel's constant bash on Unit Testing, since unit testing is not really a technology or even a technique of choice but next step in the evolution of programming. Its not like a decision about multi-threading vs. single-threading or even using a certain design pattern or not, either way you can implement the functionality so there is no arguement here ship the single thread implementation. When it comes down to unit testing there is no alternative, different technology or framework for implementing your test cases but not the principle itself. On top of that when you use unit test in a TDD way it actually has not much to do with testing, but instead a brand new way (compare to the triditional non-TDD way at least) to materialize your design, drive your implemetation, and a core enabler for many other useful Agile practices such as Refactoring.

I believe when you are building a product, just like building a company, carrying some debt (in this case the technical debt) is healthy and beneficiary to the growth, but the balance is essential here don't declare technical bankrupcy before you can even ship something. By the way, even if you are working in a startup, don't put too much emphasize to the First-Mover Adavantage since its just a marketing myth. The recent success in Google and Apple's iPod are the perfect examples of why the first-mover advantage is highly overrated. For startup I truly believe the vision, innovation and quality are the key building blocks for any enduring success.

Technical mess is a one way ticket to Technical Bankrupcy

2009-09-22T11:09:00.001-04:00

I just finished reading Uncle Bob's new post A Mess is not a Technical Debt. Lack of understanding of this key differentiation is so common in the industry. I have witnessed many projects and businesses creating a mess thinking that is a way to produce something under tight budget and timeline. They did not realize this debt (if you consider debt way beyond your affordability is still a debt) is so overwhelming, it basically turns out to be a one way express ticket to technical bankrupcy. It's an especially popular view among startups. Many startup entrepreneurs are very cautious when it comes taking on finicial debt however reckless when it comes to technical debt. When the debt accumulates to a certain turning point then it is no longer debt but a suicide. I consider a suboptimal algorithm is a technical debt or even unit test coverage from 90% to 80% is a debt however unwise, but 0% unit test coverage is not a debt but a bandrupcy, a train wreck in slow motion resulted many slow and painful deaths I witnessed in my almost a decade long career.

One thing still perplex me is I have also witnessed a few companies enjoyed some pretty decent finicial success while creating a big technical mess...

Install PHP 5.3 on CentOS

2009-08-24T16:37:00.001-04:00

A couple days ago I compiled and installed PHP 5.3 with MySQL and SOAP support on a CentOS box since its not available in Yum repo. Here is some notes and problem I encountered. My configuration:

./configure --with-mysql=shared --enable-pdo --with-pdo-mysql=shared --with-mcrypt=shared --with-mhash=shared --with-gd=shared --enable-soap=shared --with-config-file-path=/etc/ --with-jpeg-dir=/usr/lib/ --with-apxs2 -–with-mysqli --enable-zip -–enable-shared

The only problem I ran into other than installing all the dependencies is libxml2 library that PHP depends on is currently not available in Yum repo you will have to download it and install manually.

wget ftp://xmlsoft.org/libxml2/LATEST_LIBXML2
tar -xf LATEST_LIBXML2
cd libxml2-x.x.x
./configure
make
sudo make install

Redirect AntBuilder output in Groovy

2009-08-19T11:21:00.004-04:00

In Groovy you can use the following code to redirect default AntBuilder output to a log file:


def ant = new AntBuilder()
def logOutput = new PrintStream(logFile)

// redirect std out and error output
System.out = logOutput
System.err = logOutput

// redirect ant output
ant.project.getBuildListeners().each{   
     it.setOutputPrintStream(logOutput)
}

The specified module could not be found

2009-07-28T11:43:00.001-04:00

The title of this blog is an error message I got while installing PHP 5 and Aapache 2 on a Windows XP workstation. Due to project need at work I needed to setup a local PHP environment at work. I have to confess I haven't touched PHP in probably more than 7 years and first thing I noticed was "O boy! It has come a long way". The windows installer for PHP is very well designed and implemented, almost everything was a breeze until I spot this error message in the apache error log:

PHP Warning: PHP Startup: Unable to load dynamic library 'c:\\php\\ext\\php_mysql.dll' - The specified module could not be found.\r\n in Unknown on line 0

For some unknown reason certain php modules could not be loaded properly. So like anyone who is new to a certain environment I suspected that I did something wrong during the installation process and immediately started investigation and researching on google. After following many different threads and trying various fixes like adding dll folders to the $PATH, moving php.ini, checking phpinfo(); output, or even reinstall php engine but all efforts to no avail, I ended my day in discouragement and frustration.

This morning when I was fully charged on coffee and ready to take another crack at this problem, miraculously everything is working all of sudden. Right away I remembered this advice I got from a very wise friend years ago "Take a break and reboot your machine usually is the best solution if you are stuck with a problem, especially if the problem is on a windows machine". Sign~~~ how am I gonna get my wasted 3 hours of life back.

Grails TransactionManager Config

2009-05-29T16:32:00.000-04:00

Today I was asked by one of my colleague "why grails does not ship with a default transaction manager?" I was kind of confused since I am very certain that Grails ships with a HibernateTransactionManager since Gorm is built on top of Hibernate 3. After a little bit discussion I realized the culprit is this message generated by Jetty:

INFO: No Transaction manager found - if your webapp requires one, please configure one.

Since Jetty does not recognize the Spring transaction manager therefore this message is generated, so to verify that Grails does use transaction manager fire up a Grails shell console and type the following:

println ctx.transactionManager

You should see something like this in the output:

org.springframework.orm.hibernate3.HibernateTransactionManager@12f40eb
===> null

So rest for sure all those transactional = true declaration in your service class are not just for show ;-)

Performance Tuning Daylog for Hydra Cache M4

2009-04-17T17:32:00.002-04:00

After almost 7 months (wow can't believe its been more than half year already) of hard work finally Hydra Cache project is reaching its Milestone 4 - Feature Completion, and very close to its public debut v1.0 RC1. You can find the official Project Wiki Site and the new Maven Project Portal for the code snapshot, javadoc, and various reports. Hydra Cache was built so far without performance tuning in mind. Yes you heard it right, performance tuning was not our focus at all throughout the entire development for the first 4 milestones. Our focus so far are on code quality, testability, architecture, and maintainability. We intentionally picked techniques that make our code succinct and readable over high performance ones, for example we used:

HashCodeBuilder.reflectionHashCode(this);

in hashcode() method instead of hand crafted bits shifting magic. Why? because its shorter and more readable. But.. but its using reflection! I know, but trust me you can write much slower code (a few order of magnitude slower) just a few blocks away from here with a simple log.debug(...) so why focus on this one, plus the more you delay the optimization the more options you will have to throw at a performance problem even if this one actually becomes a real bottleneck.

* Thanks to my team mate David Dossot to constantly reminding me at the beginning of the project to hold off the nerd inside of me that craves for high octane performance and some fancy hash code algorithms.

But now to reach RC1, one of the task is to actually optimize the performance since one of the goal for Hydra Cache is to provide highly scalable and fast caching system. First thing first, lets build a performance test tool kit so we can have a consistent and controled environment to benchmark Hydra. I picked Apache JMeter an open source performance testing tool as our testing framework since I have worked with JMeter before and plus of course its open source. Two JMeter Samplers were created for GET and PUT operation respectively, and a throughput test plan designed to stress the server in an artificial manner so we can measure the throughput and the cpu usage under stress. Lets fire it up and see what we got here. Since this is a not a real performance test to actually measure the throughput but rather a test designed to identify performance bottleneck, therefore it was being executed on my laptop (Intel Centrino Duo Core 2Ghz with 2Gb of RAM running on Sun JDK 1.6.0_13). 4 local Hydra servers were launched with 64MB of heap memory configured each, and the Hydra space configuration for each server node is identical with the following parameter: N=3, W=2, R=1. What this means is each PUT operation has to be sucessfully multi-casted and propogated to at least 1 (W) servers before the PUT operation is considered to be succesful and eventually up to 3 (N including the initial server that received request) servers will store the same replica in the space. However the GET operation will be performed by a single node only without multi-server consolidation. This is a typical configuration to offer reliable writes and redundency but optimized reads for better performance. Here is the performance with 10 test threads and 1000 writes/reads each.

GET Throughput Chart

PUT Throughput Chart

49k read/minute (800 read/sec) and 9k write/minute (150 write/sec) this is not too bad considering till now performance optimization is not even attempted nor considered during coding.

* I am going to stress here again this is not a throughput test but rather a test specially designed to pin-point performance bottleneck, so do not take the number here as the maximum throughput number for Hydra Cache

Now to find out what's hogging all the CPU cycle we need to do some profiling. Thanks to the fantastic EjTechnologies JProfiler and their much appreciated support on open source effort; here is the profiling result.

As you can see the top two hotspots are the toString() and hashCode() methods on Identity class. Aha~ remember the reflection based builder I have shown you at the beginning of the post? It was used in both of the these methods. But before you are going to tell me "I told you so", I need to explain that the same reflection builder was used in almost every model class, and a premature optimization would result in wasted energy on hand writing these methods for a lot of classes that is no way near the hotspots. Alright enough stalling lets roll up the sleeve and create a kick ass super fast implementation then, but hold on a second I remember the Identity class is a pretty special one. What? It is Immutable! Really? That changes the whole picture. If its immutable instead of computing the hashCode and toString every time we can just simply cache it and eliminate the cpu cost almost entirely. This shows another positive aspect of delaying optimization since the more you delay it the smaller you can pin-point the scope therefore more customized the solution can be tailor which usually means more effeciency. If we had applied a general optimization strategy up-front, most likely it would have resulted in a broad-stroke hand-crafted hashCode and toString for every class strategy, and we would have probably missed this caching opportunity since we were busy hand coding 50 different hashCode and toString. A few lines of code... taa daa ... done. Lets profile it again.

Alright much better now the top 5 hotspots each only contribute to 1% of the cpu time. Lets also fire of the JMeter to see the improvement in action.

GET Throughput Chart

PUT Throughput Chart

36k reads/minute and 12k writes/minute. Hmm... much better write throughput and slightly lower read throughput. Its a bit strange but considering the small test sample size and all the other craps running on my laptop this kind of fluctuation is in the margin, and since I am not really measuring the throughput I dont really care too much. Run all unit test cases, nothing is broken, feeling good and check all the changes in. Now its time to start the next iteration. Took a second look at the hotspot list and some numbers immediately caught my attention - the number of invocation for the top 5 hotspots, they seem to be a bit higher than my expectation. So whats causing all these invocations? Scan through the code and the references, ahaa the debug statement is the culprit this time. Although the debug logging is turned off to mimic production environment, however the statement is still getting executed. We are using Log4j so a regular debug statement looks like this:

log.debug("Something useful: " + object.getValue());

This statement will not produce a log but the string concatenation still happens and getValue() method still gets executed. A common approach to address this kind of performance penalty is to add a Code Guard like this:

if(log.isDebugEnabled())
log.debug("Something useful: " + object.getValue());

This way when the debug is turned off none of the code will actually get executed. Now once again delaying the optimization gave me the opportunity to identify and guard only the expensive debug statement without enforcing everyone to follow this practice throughout the project which is much cheaper and just as efficient. Once again a few lines of code I am done, lets profile it again.

Great! 4 out of 5 top hotspots were demoted to less than 1% cpu time. Alright time to look at it from a different angle now. Lets look at the aggregated cpu time per methods.

Bingo! There is a big cpu hog - JGroupSpace.findSubstancesForLocalNode(). Hmm... make sense this is a pretty expensive computation and it happens every time a PUT or GET operation is invoked. After looking at the code I realize there is already a layer of abstraction introduced to on top of that but it currently does not have the capability to compute the substances for the local node thats why everyone is calling this method to compute everything from scratch at the space level. It makes sense to pull it up into the intermediate layer since we can cache the computation result there when the space is stable and only recompute when member node leave or join the space, and even better the intermediate layer is already getting the notification for this kind of membership change for other functions anyway. Alright sold! thats the plan. Start with some new test cases and refactoring, a few iteration later, done! All test cases pass, a miracle :) Quickly check them in. Profile it again ...

MultiplexMessageReceiver.receive() dropped from 25ms for 4k invokation to 4ms, not too shaby at all. Now lets put everything together and see it in action. Fire up the JMeter!

GET Throughput Chart

PUT Throughput Chart

A whooping 81k reads/minute and 11k writes/minute, so overall reads are 2-3 times faster now and writes are about 20-30% faster. Not too bad for an afternoon session heh? Fire off the integration test cases and nothing is broken, great mission accomplished. My next step is to observe and profile the memory usage pattern to first of all make sure there is no memory leak and secondly optimize the garbage collection behavior. Maybe its also time to put Hydra in a real performance testing lab to find out the actual throughput ceiling.

Stay tuned and Hydra Cache RC1 will be available for download this summer.

Implement Harmony Oriented Programming in Hydra Cache: A real world project

2009-03-25T15:26:00.006-04:00

In this post I would like to share some experience and lesson we learned when applying concepts and principals inspired by the Harmony Oriented Programming (HOP) in Hydra Cache project. Hydra Cache is an open source project created to implement Amazon’s Dynamo system in Java in order to provide highly reliable and scalable distributed in-memory cache. The system itself is essentially like a giant hash table which gets partitioned and distributed to a number of servers (we call this partition aware server cluster a Hydra Space) and with all servers working together to provide a unified and transparent storage for the client code. One of the core objectives we are tackling in this project is to provide reliable in-memory storage by partitioning, distributing and coordinating all GET and PUT operations to multiple servers hence offering redundancy and enhancing reliability. Let’s look at the following parameters used to describe Dynamo:

T – Total number of servers in the space.
N – Total number of replica that eventually will receive the replication
W – Minimum number of replicas to acknowledge for a successful write operation
R – Minimum number of replicas to consolidate for every successful read operation

To achieve reliability each PUT operation invoked on any of the node will be replicated to minimum W number of servers and eventually N number of servers, for example if I store my shopping cart data to the cache when W=3, N=6 then in theory the same data will be at least replicated to 3 servers before the PUT is considered to be successful, and eventually 6 servers. Similarly each GET operation will require minimum R number of servers to cooperate and consolidate the data they have in order to perform a successful reliable GET. Usually a system like Dynamo or Hydra Cache will be configured with the following characteristics T > N > max(W,R) and W >= R to make sense, and when you have W+R > N then you have yourself a basic quorum system which guarantees strong consistency. On the other hand, when you have W+R <= N configuration the system will produce weak consistency (in this case eventual consistency) since it’s not guaranteed to read most recent version written; however unsafe it might sound this is actually a widely deployed configuration when the performance penalty of strong consistency is not desirable when eventual consistency especially during failure can be tolerated, and moreover according to the CAP theorem only two of the three properties Consistency, Availability, Tolerance to network Partition can be achieved at the same time. It is common for large internet based service to sacrifice consistency for high availability and tolerance.

* In Hydra Cache the consistent hashing algorithm and partition-aware client help minimize the inconsistency window when weak/eventual consistency is configured meaning although still constraint by CAP theorem Hydra Cache delivers consistent data most of the time when there is no node or network failure.

While designing the system, initially I experiment driving design directly from this model and found the implementation sort of rigid and lacking of metaphor. Personally I always found powerful metaphor the best help you can get when designing a system, so whenever I code the first thing I do is to search for a metaphor with good fit for the problem at hand. Struggling with the design one speech at the OOPSLA caught my attention. The speech it self is about Harmony Oriented Programming. Although I was not so interested at the programming level, but fascinated about how the idea of basic Harmony Orientation principals and practice can be leveraged as a powerful ally for modeling the reliability framework in Hydra Cache project.

Harmony Orientation Principals

Balance – For a HOP system the ultimate goal is to achieve balance (harmony) as a whole. No single member of the system should selfishly perform at the cost of its neighbors. This principal pretty much describes the essence of the Hydra Cache since none of the node should out perform or under perform much comparing to their peers and as whole (Hydra Space) should always maintain the equilibrium even when node/connection failure happens or new node is introduced.
Code Fragmentation – In HOP the dominant decomposition of a program is a code fragmentation with minimal encapsulation; in a simplest case a fragmentation could be just a single line of statement. For Hydra Cache this principal perfectly maps to the two core operations the Hydra Space supports GET and PUT both can be simplified to almost a single statement from the client’s point-of-view.
Code Positioning - The code positioning principle states that every part of a program is assigned to one and more locations in a virtual space. Related parts of a program are positioned close to each other to form a specific context. In Hydra Space all code (node) are positioned in a consistent hash ring with related nodes positioned close to each other based on their hash value.
Information Sharing - The information sharing principle suggests that all data is readable (or can be queried) by any part of the program. This is the core feature that Hydra Cache offers - to be able to query any data stored anywhere in the space from pretty much anywhere.
Information Diffusion - The information diffusion principle states that data or a description of the data (meta data) generated by any part of the program is diffused throughout the virtual program space. Data has an associated intensity that decreases the further it is diffused. The concept of information diffusion is expressed in HOP using the concept of substance, which I will illustrate further; this concept maps nicely to the N/W/R values we discussed in the beginning of this post.

Enough principals and dry ideas, now let’s see how HOP (more accurately Harmony Oriented Design - HOD) is implemented in Hydra Cache. In Hydra Cache each server node is considered as a code fragmentation holder that resides in a HOP space, see the following illustration for a 4 server space deployment.

* In this diagram Hydra space is illustrated as a 2 dimensional space however in implementation the space is structured on a Consistent Hash Ring (a.k.a a Continuum) using a well uniformly distributed consistent hashing algorithm inspired by Ketama project, therefore the distance between server nodes as well as the radius of the substances are measured using the difference between each node’s hash value.

As you can see in this diagram Server A emits a substance to include both Server B and D hence any code fragment resides on Server A will diffuse its information to both B and D. In other words, every time when a PUT operation is invoked on A the same operation will be invoked on both B and D. Now it becomes obvious that the size of the substance is the N value, and on top of that Hydra Cache also allow server admin to configure the W and R parameter to fine tune the consistency level. Server A is not the only one that emits substance but rather all servers in the space do, here is a bit messy diagram showing all 4 substances.

Let’s see it in action. Based on the above example the Hydra space would be configured with N=2. To optimize read you can configure W = 2 and R = 1 which means every PUT operation will be only considered successful when the data gets diffused to at least one server (either D or C) in the substance, and eventually reaches both servers (most likely D and C but could be B if one of them crashes during the diffusion). For reads the server will just simply return the local copy since R is 1. As we have demonstrated here these parameters can be a very flexible and powerful tool to tune the space to focus and balance on different priorities: consistency, reliability, fault-tolerance, read vs. write, and performance.

HOD offered an almost perfect metaphor for solving and modeling our problem domain which in return made it extremely effective to communicate our design in human language and code. In our version 1.0 release the Hydra space is implemented through reliable multi-cast using JGroups. If you are interested to know more about Hydra Cache please visit our website http://www.hydracache.org.

References:

GoEyeball.com

2009-02-12T16:47:00.002-05:00

Last year me and my family moved to a new home mid-town Toronto, and among endlessly packing and unpacking we also found ourselves wrapped with a big shopping list; some electronics need to be upgraded and new furniture need to be purchased. At that time because it was almost the Christmas timeframe we did not buy anything right away since we were betting on that some of those items on our wishlist will go on sale for sure during Christmas time. What we end up doing for the next month and so was continuously going back to 6-7 different sites checking the price for roughly 10 different items on my list almost everyday hoping some of them would be on sale. After about a week my wife complained that she spent so much time browsing the web looking for deals everyday, and it would be really cool if a website could allow her to create a robot monitoring those pages for her so she can be freed from this repetitive work and just get notified when the price has dropped. My initial reaction was that's a good idea but there gotta be a site out there does this, but after some research I realized there is actually not a single site that does exactly this.

I found some sites do crowl the web and record prices for different products but they only supports a narrow range of sites so lets say if you want to monitor something on Walmart.ca then you are out-of-luck since they don't support the site. And some other ones only allow users or their sponsors to advertise deals manually so if what you want is not in the deal list then you still have to go back everyday to check. Most of the retail site do not provide this kind of price change notification service since the main purpose of sale and discount is get the customer to browse through the site thus triggering impulsive purchase since the profit margin for the on sale items are slim, its against their best interest to provide service like this. Eventually I decided to create a website www.GoEyeball.com to do one and only one simply thing:

You can create an Eyeball (robot) for a product (on the detail page) on pretty much any site and getting notified when the price drops below your threshold.

We launched the beta version of the web site this week. Its completely free of charge and free of advertisement. Come check it out and let us know what you think. In the next a few posts I will share some of the creative ways our users have figured out how to use the site, and you are always more than welcome to let us know your findings and experience.

Exception vs. Error Code - round 2

2009-01-07T15:07:00.000-05:00

This post is a follow up of my previous post on the same topic Exception vs. Error Code - round 1. Here I would like to discuss what Uncle Bob and other Object Mentors' view on this topic which was explained in the their book Clean Code.

First of all their view is definitely pro exception, and the two major reasons given against returning error code were:

a. Returning error code is a subtle violation of Command-Query-Separation (CQS) principle. (page 46)

Take a look at the following code sniplet:


if(connection.close() == IO_ERROR){
    handleIoError(connection);
}

The close() method is clearly a command but by returning the error code this method offers both query and command style invocation thus violates the CQS principle.

b. Returning error code clutter the caller code and since its easy to miss handling certain type of error code therefore it is also error prone. (page 104)

This point is consistent with Kent Beck's view which I have discussed in the previous post.

As Unble Bob mentioned programming is sort of like martial art. Different master will form their own style and some one can choose to practice certain style exclusively or a mix of styles, but when certain practice is criticized by more than one master then you better take it seriously and rethink your practice because there is probably something wrong with that.

Reflection with Generics

2008-12-10T16:18:00.007-05:00

I ran into some pretty interesting reflection problem today, and thought to jog down some notes here. In Java Generic, concept introduced with Java 5, is implemented pretty much as a compiler trick which performs the type check then generates the same byte code as the non-generic code, and inserts casts for the generic code automatically. The following two code samples will generate identical byte code by the compiler.


void doSomething(List ids)

vs.


void doSomething(List<Long> ids)

* That's also why code like: list instanceof List<Long> does not work in Java since this information is not available at runtime

So now lets imagine you try to use the reflection to find the method with a List<Double> at runtime, you will actually be able to find the method doSomething(List<Long> ids) and you can even invoke the method without any problem since at the byte code level this method is really taking a non-generic List. Now the problem arise, in the method since you already specify the generic parameter thus you probably will not check the element of the list again but rather use foreach loop as shown in the following code sample:


for(Long id : ids){
    ...
}

This code works fine without reflection since the compiler will force the type check, but while working with reflection you can basically pass in any List at runtime, hence you will get a class cast exception at the start of this loop. Fortunately the generic type information is not all gone at the runtime. They do get compiled into meta information for the class so using reflection you can retrieve this kind of information at the runtime, although I have to say I am not a big fan how this generic related type API was designed in the reflection package, its very inconvenient to use to say the best. If you are interested IBM Developer Works has an excellent article on this topic.

In summary, keep in mind generic is just a compile time transformation and be very careful when using reflection with generic.

Implement Flash socket policy file server

2008-12-03T14:34:00.001-05:00

Recently I had to implement a Flash socket policy file server. Here are a few things I discovered while creating the server.

Why socket policy file server?

Since Flash player 9 you can no longer use the HTTP based crossdomain.xml file to specify cross-domain policy for socket based server as part of the on-going security enhancement from Adobe. Therefore a specialized socket file server (default on 843 for master policy) needs to be created to return the policy file(s) for the socket based client. See Policy file changes in Flash Player 9 and Flash Player 10 for more details.

Whats the expectatoin?

The basic idea behind this file server is fairly simple. The server will wait for the policy file request from the Flash player sent directly by the Flash player. The policy request is just a string of followed by a NULL byte (00), upon receiving the request the server will return the content of the policy file.

Whats the catch?

1. String chararacter set

In Adobe's document there was no mentioning of the encoding charset for the returning string, and based on my experiment I found the Flash player did not like the UTF-8 encoding in Java. Probably because Java DataOutputStream uses a modified UTF-8 encoding. At the end I found that ISO-8859-1 worked out pretty well with the Flash player, and since your policy file should not contain any unicode character anyway I think this is a pretty good approach.

2. Write timing

One interesting thing I found is although Adobe mentioned in their documentation the request handling should be in the following sequence:

Receive request
Verify request
Write policy content
Close socket

I found the Flash player is happy even if you just simply write policy content as soon as the socket is accepted. Although if you are writing your own file server you probably should stick with Adobe's spec not only because it makes sense but also because the undocumented Flash player behavior might change without notice in the future release.

3. Close your socket

In Adobe's documentation it mentioned that the Flash player will close the socket as soon as it received the policy file, but based on my experiment it seems that Flash player performs less consistent if the server does not close the socket right after the content is written.

Hope this information will help you save some time while implementing your own policy file server.

More reference:

Exception vs. Error Code - round 1

2008-11-20T11:41:00.002-05:00

Many times I have heard this argument in my career that throwing exception is too expensive therefore high performance code should always favor returning error code over exception. While it was probably true on early-day JVM, I am still not a big fan of this. First of all in my opinion the biggest drawback of extensive usage of error code is that you lose the most powerful tool that allows you to highlight your main flow from the exceptional ones. Since the main flow is probably the most valuable, most executed and most read code you should always thrive to make the main flow as easy to identify as possible. As Kent Beck suggested in his Implementation Pattern book:

Expressing all paths equally would result in a bowl of worms, with flags set here and used there and return values with special meanings. Answering the basic question, "What statements are executed?" becomes an exercise in a combination of archaeology and logic. Pick the main flow. Express it clearly. Use exceptions to express other paths.

Now consider the following code sample written using error code:


def input = input()
if(input.errorCode == IO_ERROR)
    return new Result(IO_ERROR, ...)
// could be more error types here

def processResult = process(input)
if(processResult.errorCode == IO_ERROR)
    return new Result(IO_ERROR, ...)
// could be more error types here

def output = output()
if(output.errorCode == IO_ERROR)
     return new Result(IO_ERROR, ...) // could be more error types here
else
    return new Result(SUCCESS)

If we can rewrite it using Exception, it will look like this:


input()
process()
output()

* The code sample is in Groovy so the error code version is already somewhat shorter than the Java version. Although the Exception version will stay pretty much the same in both Groovy and Java.

As you can see it is obvious using Exception greatly simplify the main flow, and make it so much easier to read hence cheaper to maintain and easier to improve. But how about the performance argument that we always hear. Well... modern JVM is highly optimized for exception handling, although optimization strategy varies between vendors, some strategy focus on optimizing the normal path, others focus on the exceptional paths such as EDO, and the newest ones are adaptive at the runtime and smart enough to pick the right strategy for the specific scenario. In a simple test it took 30ms to throw 10,000 exceptions on my 1.6_07 32 bit JDK with Hotspot 10. Also remember cleaner code makes it much easier to pin-point performance bottle-neck and make improvement. In most cases, 90% of the performance slow down is usually caused by 3% of the code and in my experience the culprit was never the exception handling, and if it is then you have a much bigger design issue at hand than just merely exception handling.

What if someone argues that their main flow actually triggers the exception path way too often and it is causing significant performance degredation, for example during a DOS attack. My suggestion? Simple! If thats the case, your code is telling you this exception path is actually not exceptional but rather part of the main flow. For example if you server is expected to withstand a DOS attack then you can't really treat certain corrupted packets or pre-maturely dropped connection as exceptional cases anymore. In other words, if exception handling is causing you performance problem, you better rethink how the system is designed in the first place instead of simply replacing it with error code.

Last but not least, I would like to make it clear that I am NOT suggesting here that you should use exception as your first choice to control your flow, you should always use sequence, message, condition, iteration, and exception (in this order) to control your flow. Use exception to handle only the exceptional cases but don't simply dismiss it because some out-dated misunderstood performance concerns.

Oopsla 2008: Designed as Designer

2008-10-22T01:29:00.002-04:00

Eassy by Richard P. Gabriel, it is as controversial as usual but also the most intriguing presentation of the day at least from my point of view. The essay is almost a follow up of Fred Brook's speech on Oopsla 2007 around the central argument of that conceptual integrity arises not (simply) from one mind or small number of agreeing resonant minds, but from sometimes hidden co-authors and the thing designed itself. A few points I took away from this presentation as well as the interesting Q&A session:

"First to the market wins" is merely a myth
Worse is Better - ship the rough product earlier to allow user to contribute and generate the real requirement. Perfection is the enemy of good.
People are often judged by their reward instead of their skills, and that's why CEO in successful (sometimes even failing) company are usually rewarded the most even when they have little impact on the success.
Software is implemented by the compiler and machine, every single line of code is a practice of design
The first draft of design is usually just a collaboration enabler so others can contribute with certain degree of conceptual integrity
By refusing fully knowing the world when you design you open the door for new insights and let the product itself to lead you to the truth
Facts are not truth. Simply because you customer is describing a certain feature to you does not mean that is what they really want or need.

Oopsla 2008: Comunity-Based Innovation: from Sports Equipment to Software

2008-10-22T01:01:00.000-04:00

This presentation was given by Sonali Shah of University of Washington Business School. The speaker shared many of her research findings and insights into how community-based innovation is reshaping the business world today from sports equipment industry to cloth design and manufacturing to software. One really interesting point raised by this research as a by-product was that it found the most committed and innovative long-term open source developers all have a full time day-job but found their day-job is either not challenge enough or too restritive or both for them to innovate therefore they would rather spend their spare time to work on something else. At this age of uncertainty when many economists predict that the only way for North America to maintain the competitive edge is innovation, its puzzling to see that many of the companies out there fail to recognize that.

See Clay Shirky's excellent presentation at TED from a different angle on this topic, and my post "What can we learn from the Open Source community" in 2006 on how I think the companies can harvest this kind of passion and innovation.

Oopsla 2008: Social Programming A Pyramid, and possibly other lar

2008-10-22T00:36:00.000-04:00

Very interesting presentation given by Mark Lehner on how ancient Egyptian managed to organize almost humanly impossible effort building those gigantic pyramids both physically and socially for the most part of their 3000 years of history. The speaker also drawn some, although sometime a bit far fetched, analogies from this type of massively labor-intensive undertaking to how software itself and project is being organized nowadays. One really interesting point I took from this presentation was that although the social organization contains highly modularized units, the final product of eternity (Pyramids) does not necessarily resemble any strict modularity but rather simply and straight replication. Which says something does it? Or maybe we are all trying too hard to find similarity, the proof or disproof of what we do and the way we are doing it.

Want Groovy?

2008-09-29T16:44:00.002-04:00

Groovy as a dynamic JVM powered language (JSR-241) has been gaining a lot of momentum and attention recently, especially when the Grails - a web application development framework built using Groovy with design similar to Rails. Some people even started to consider Groovy as a better version of Java, which I don't personally agree but I do think that Groovy is better suited for many tasks typicall performed by Java triditionally, such as building DSL, creating dynamic framework, and more.

Despite of all the grooviness about Groovy, it is still pretty difficult for any organization to adopt this relately young technology due to a classical chicken-and-egg delimma. Before formal adoption in a coporation settings, most of us would like to try the language out, but without formal adoption in a real project its almost impossible to really evaluate and learn the language, so what do we do? One effective way I found to introduce Groovy into your organization is starting with writing your unit tests in Groovy first. Because its just the test code usually there is less red tapes on it and since it will never be deployed into a production environment typically its a lot easier to get approval for trying it out. With the help of Maven and Groovy plugin its actually quite easy to add some grooviness to your project.

Step 1 - Add GMaven plugin to your pom.xml


          
              org.codehaus.groovy.maven
              gmaven-plugin
              
                  
                      
                          generateTestStubs
                          testCompile
                      
                      
                          
                              
                                  
                                      ${pom.basedir}/src/test/java
                                  
                                  
                                      **/*.groovy

This definition basically tells Maven to use GMaven plugin to compile all *.groovy files under your standard test/java directory, which esentially allows you to write unit tests in both Java and Groovy.

Step 2 - Add Groovy runtime to your test classpath


          
              org.codehaus.groovy.maven.runtime
              gmaven-runtime-default
              1.0-rc-3
              test

By add this dependency to your dependencies will add Groovy runtime to your project and Eclipse classpath if you are using eclipse:eclipse plugin.

Step 3 - Install Groovy Eclipse plugin (Optional)

If you are using Eclipse, you might find its useful to install the Groovy plugin for your IDE although this plugin still has a few rough edges, it allows you to run your Groovy powered unit tests using GUnit which I found is a productivity boost.

Groovy! Now you are free to writing your unit and integration tests in both Groovy or Java, hence free to try out the language and feature at your own pace. Have fun.

New Open Source Project - Hydra Cache

2008-09-26T11:25:00.001-04:00

Inspired by our recent experience and events, a few of my friends and I started a new open source project called Hydra aiming to provide the community a open source implementation of Amazon Dynamo in Java. The project design is based on the published papers and algorithms in public domain only and mainly Werner's paper on Amazon Dynamo. Currently the project is in its design and prototype stage. If you are interested in this project, check out our Wiki at HydraCache.org and if you are interested to contribute as a developer please contact the project admins at Hydra Project Page.

Consistent Hash based Distributed Data Storage

2008-09-16T00:14:00.003-04:00

Challenge-

In an ultra large enterprise application, such as an online e-commerce or online gaming site, the site is dealing with millions of users and thousands of transactions every second. To handle this kind of traffic the number of servers, routers, databases, and storage hardware makes hardware or network failure a norm instead of an exception. Despite of the constant hardware failure in your system, your customer will not tolerate the slightest down time; the more successful your system is the more important it becomes to your client, and less happy they are when they experience an outage.

Solution -

To solve this challenge we need a highly available, decentralized, and high performance data storage service shielding the application from the harsh reality and complexity. No exception will be thrown when hardware or network failure occurs and the application code can always safely assume that the data storage service is available to write and read at any given point of time. This data storage service also needs to be evolutionarily scalable since down time is not acceptable, thus adding new node and storage capacity should not require shutting down the entire system, and it should only have limited impact on the service and it's client. A bonus side effect of this solution is that the distributed data storage can also act as a distributed cache system to reduce the hit to the persistent data storage such as a relational database.

Design -

- Consistent Hash

In a large online application, the type of data that require this kind of high availability are usually data that can be identified by a primary key and stored as binary content, for example user session(session id), shopping cart(cart id), preferences(user id), and etc,. Due to this nature, a Consistent Hash based distributed storage solution was proposed. Consistent Hash algorithm was initially introduced in Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web by David Karger in 1997. The key benefit of Consistent Hashing is that hashing any given key will always return the same value even when new slots are added or removed. The principle of this design can be illustrated using the following diagram. Imagine a virtual clock that represents an integer value from -2³¹ to 2³¹-1, and each server (A, B and C) in the storage cluster has a hash value assigned, hence for each given key (K₁ and K₂) can only land somewhere between these server nodes on the clock. The algorithm will search the clock clock-wise and pick the first server it encounters as the storage server for the given key, and because the hashing algorithm is consistent therefore any future put or get operation is guaranteed to be performed on the same node. Moreover the consistent hash algorithm also minimiz the impact for adding and removing node to its neighboring nodes instead of the entire cluster.

Challenge 1: Non-Uniformed Distribution

The first problem we need to solve is that the server hash value are most likely not uniformly distributed on the clock, as the result the server utilization will be skewed which is hardly an ideal situation. To solve this problem we are planning to borrow the idea discussed in Werner's paper on Amazon Dynamo by creating virtual nodes for the servers, and when you have enough virtual nodes created on the clock a close to uniformed distribution can be achieved.

Challenge 2: Availability and Replication

To provide high availability, the stored data need to be replicated to multiple servers. Based on the algorithm we are employing, in the case of a server failure any data stored on this specific server will automatically become the subjacent server's responsibility as shown in the following diagram.

Therefore our replication strategy is quite simple that every node will replicate the data it stores to it's immediate subjacent neighboring server. You can also include more than two servers in the replication group for even higher availability, although in our project I believe paired availability server group will provide desired availability without introducing too much complexity.

Open Source Alternative -

Open source Consistent Hash based distributed cache Memcached is built based on the similar design but without the high availability replication capability. It expects the application code being able to recover and restore the cache when a node becomes unavailable, which is usually an acceptable alternative for replication based availability at the cost of performance penalty during outage and increased code complexity. I usually recommend Memcached over custom in-house distributed cache system, since its proven, free, and a lot less work; what more can you ask :-)

Further Improvement -

Although its currently not in the plan but down the road there might be a need to implement Vector Clock based object versioning and merging capability to support simultaneous writes on multiple nodes, which is crucial for maintaining data consistency during partial system failure. Currently we are simply planning to employ "last-write-win" strategy to resolve the conflict.

Related Readings -

Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web

Amazon Dynamo

Tim's Blog on Consistent Hash

Enable USB in VMWare under Ubuntu 8.04 - Hardy Heron

2008-09-04T10:34:00.001-04:00

Just figured out how to enable USD controllers in VMWare windows instance under Ubuntu 8.04 - Hardy Heron, thought it might be useful to share it here in my blog.

If you just upgraded to Hardy Heron, you will notice that the USB are no longer working for your VMWare instance, its because that the Ubuntu development team removed /proc/bus/usb mount, and thats what VMWare depends on for detecting USB devices. To re-enable this is actually quite simple, just modify the device mount script "sudo vim /etc/init.d/mountdevsubfs.sh" and uncomment the following part:

#
# Magic to make /proc/bus/usb work
#
mkdir -p /dev/bus/usb/.usbfs
domount usbfs "" /dev/bus/usb/.usbfs -obusmode=0700,devmode=0600,listmode=0644
ln -s .usbfs/devices /dev/bus/usb/devices
mount --rbind /dev/bus/usb /proc/bus/usb

Then restart the mount "sudo /etc/init.d/mountdevsubfs.sh start", now you should be able to add USB device to VMWare instance again. Have fun!

Overlay DHTML popup on top of a Flash movie

2008-08-31T23:03:00.006-04:00

Just solved a tricky overlay problem, at least a pretty tricky problem for someone like me who do not work with Flash a lot, and thought to jog down some notes here in my day log.

I ran into a overlay problem today when using RichFaces Calendar component with Open Flash Chart, for some reason the pop-up date picker overlay generated by the Calendar component kept staying behind the Flash movie used by Open Flash Chart. Firstly I tried playing with z-index with both of the calendar and flash div had no luck. After some research found out that apparent this is caused by how browser treats Window and Windowless flash differently. By default flash has its window mode set to "window", and as result the browser will always render it on top level in its own window, therefore no matter how you set the z-index the overlay will not be able to go over on top of the flash movie. To fix this issue you need to change the flash wmode parameter to either "opaque" or "transparent" by adding a new parameter in the object tag:

<param name="wmode" value="transparent"></param>

or wmode attribute to the embed tag:

<embed src="..." wmode="transparent"></embed>

To have the Flash rendered in the Windowless mode, which will allow the browser to render Flash as a regular internal layer hence can be covered by a DHTML overlay.

Note: Adding wmode only to the object tag will work for IE but not Firefox.

Java a OS?

2008-08-12T10:43:00.002-04:00

For the last several weeks I had opportunities working on three vastly different projects, one built using old school Java + JDBC, one with JSF + Hibernate + Spring, the other one using Groovy + Grails. The experience of joggling among these three quite different projects every day (one with my day job, the other two with my night job - open source) actually reminded me a very interesting question that was asked during the 2008 Java One during James Goslin's talk. One of developer from the audience asked him a question like "Now Java has grown to a tremendous size with multiple languages supported, it feels more like an OS than a language" (can't really remember the exact question). James' answer was surprising straightforward, he said something like "Yes, that's what was intended from the very beginning" (again can't really remember the exact answer :-)

My experience in the past weeks showed me how much more productive and natural you can get while using the same set of API and Libraries (OS) but a different language. Even the best framework in Java can not stack up with the productivity you gain from Grails, based on my personal estimate the JSF/Hibernate/Spring stack is probably 3-4 times more productive than the simple Java approach. However productive it is comparing to the naked Java approach, the Groovy/Grails combo out-performed it about another 3-4 times (sometimes even more for the CRUD operations), mostly due to the opinionated framework, dynamic methods generation, code generation, and various ready to go plug-ins. If you look at the end result of the generated code by Grails, its pretty much as good as a Java application can be - a standard Spring + Hibernate architecture with full JEE compatibility enhanced with any plug-in you choose to deploy, built-in Ajax support, and fully unit testable with mock objects plus automated integration testability. So if you ask me now if I will ever start building another Java web application without using Groovy/Grails, I can hardly come up with a scenario that I will go with Java alone. Just as though you can implement the most sophisticated web 2.x site right now using C and CGI API alone but at what cost?

With JVM being a general purpose virtual machine or portable OS, we can predict many special purpose languages and DSL will be developed down the road in our never ending quest for the silver bullet. Even today you can already see the trend, people are now using Groovy or Jython for scripting, Grails or JRuby + Rrails for web development, Scala for concurrency or library building, and with of course a few touches of old school Java here and there some times. While for sure an exciting landscape for the Java community is ahead of us, it also reminds me a bit of the time when Java was still a new born baby, people were excited about Java and using it as an easy language to glue C and C++ code together through JNI, now as a full grown adult its time for Java to pass the torch. An exciting time indeed :-)

Project Vital Sign Charting Spreadsheet

2008-07-17T22:00:00.001-04:00

Recently I read the article in Thoughtworks Anthology called Project Vital Sign written by Stelios Pantazopoulos. In this article the author proposed a few type of charts that an Agile team can produce, usually by the Team Lead or Iteration Manager, as Information Radiators to improve communication among team members as well as with stake holders.

After reading the article, I suddenly recalled a conversation I had a while back when working with a PM who is relatively new to the Agile landscape. She asked me a question that at the time I thought the answer was obvious, she asked that how can you find out if an Agile project is in trouble or on schedule under budget. At the time my answer was, if you attend every kickoff and retrospective meeting then you can pretty much tell from the story board. She left with a puzzled look on her face, apparently what I thought was obvious was not obvious at all to some folks on the team. If this is the case for a PM who works pretty much everyday with the developers in the trench, then you can imagine the disconnection and difficulty a less technical senior manager would face when he/she tries to find out the status of an Agile project. One of the contributing factors* to Scrum's rapid adoption rate in larger corporation is the Burn-Down chart it produces which clearly communicates project status to anyone who would like to know.

But thanks to Stelios' article now we can generate several very useful charts for any Agile project for both the developers as well as any one who is interested to know the project status including the senior managers. I am planning to use some of these charts in my next project, and created spreadsheet template based on the suggestion in the article. I have uploaded this template, please feel free to modify and use it in your project, and let me know if it turns out useful at all for you.

* Other factors are Long Iteration (less agility) and a nice title for the PM (Scrum Master) plus certification program, and of course as always better marketing ;-)

Cygwin and Maven problem

2008-07-16T21:11:00.001-04:00

I ran into a pretty nasty problem while running Maven under Cygwin yesterday. Why? since all workstation at my current client's site are Windows based. The problem happens when I replaced Windows CMD with Cygwin Bash by adding Autorun key under 'HKEY_CURRENT_USER\Software\Microsoft\Command Processor' in the registry. After that Maven stopped working, took me a while to pinpoint the problem, although all other Java applications run just fine including Eclipse, Groovy, and Tomcat.

Finally I gave up replacing the Windows CMD with Bash. At the end, I decided to run all my console using an open source program http://sourceforge.net/projects/console which allows you to have multiple tabs of Cygwin console running on your Windows machine plus some eye candies. This setup worked out pretty good for me so far.

SEDA

2008-07-15T21:30:00.001-04:00

Recently I had a few discussion with different developers regarding to server architecture, and to my surprise that few really understand what SEDA is and is not, so I decided to jog down some of my thoughts here and hopefully can clear the things a bit. SEDA - Staged Event-Driven Architecture - the de facto industry golden standard for implementing scalable server. SEDA was firstly introduced in 2001 by Matt Welsh, David Culler, and Eric Brewer, see the original paper. The common misunderstanding about SEDA is many developers believe "SEDA is a high performance architecture", it is not, actually based on my experience implementing the SEDA model usually means sacrificing 10-20% performance. The main problem that SEDA addresses is graceful degradation in server scalability not the performance. What graceful degradation (also known as well conditioning) means is when your server experience an extremely high burst, for example 100x or more volume than the average traffic. While it is certain that the server will not be able to handle the burst but instead of becoming non-responsive or simply crash, ideally we would like to have the server performance degrade gracefully for instance maintain the quality of service for existing clients but rejecting all new clients with a user-friendly message, that's where SEDA comes into the picture. Here is some comparison between common server architectures and the problem while handling this kind of burst.

1. Thread-based Concurrency (The vanilla one thread per client model)

This model does not scale very well* and will not be able to handle the burst and eventually become completely unresponsive or crash the server due to resource exhaustion.

* When I use the word 'scale' here I mean scale to tens of thousand of sockets or more. This simple minded model can scale pretty well on modern operating system (Linux kernel 2.6+ and Windows NT 4+) and shown superior performance with relatively small number of threads (a few thousands), so if you server is never expected to handle tens or even hundreds thousands of sockets this is actually a pretty good architecture.

2. Bounded Thread Pool

To solve the over commit problem with the model #1, thread pool implementation was introduced and widely used, and since the thread pool has a max number of threads configured therefore it is impossible for the server to create unlimited number of threads which leads to the resource exhaustion problem in model #1. But this model can introduce great deal of unfairness during the saturation since when all the threads are busy from the pool all requests will be queued up, thus the server service quality degrades rapidly as soon as it starts reaching the limit of max thread pool size. This degradation is especially fatal for stateless request-and-response based protocol such as HTTP.

3. Event Driven Concurrency (Async Non-Blocking IO)

Event driven server design relies on non-blocking IO to processing each task as a Finite State Machine (FSM). The thread only works on a task when receiving an event from the scheduler informing there certain operation, read or write, is available to be performed. This kind of design usually is implemented by a single thread. Although with some additional complexity in programming, this model scale fairly well with even millions of tasks and maintaining consistent throughput. Although massively more scalable, this model still does not address the fundamental problem during durst, when reaching the saturation point the task processing latency increases exponentially, so this model just simply postpones the problem instead of solving it.

4. Staged Event Driven Architecture (SEDA)

To address the problem in straight event driven model, SEDA introduced a new concept Stage. Instead of processing each task individually, SEDA breaks the process procedure to multiple stages, and for each stage a dedicated scheduler, event queue, and thread pool are implemented. The main benefit of this architecture is because of the multiple stages design you end up having multiple response measuring point for implementing request shedding. For example a SEDA based web server can implement shedding logic at the second stage when normally a dynamic page (JSP/PHP/ASP) would be executed, but while experiencing a burst the second stage event can be reroute to an alternative queue where simple static but user friendly content can be returned to signify that the server is overloaded hence providing some user friendly feedback while protecting the server from resource exhaustion at the same time. Of course SEDA also provides some additional benefit such as dynamic resource allocation and easier code modularity, but nevertheless the biggest benefit is no doubt the graceful degradation capability.

Some final notes on SEDA:

In my experience, I found SEDA model actually behave the best when there is not too many stages implemented, usually 2-4 stages.

Interestingly enough, Enterprise Service Bus (ESB) architecture actually resembles SEDA model at a much larger and higher level, but because of the resemblance ESB architecture has also shown excellent massive concurrency and graceful degradation capability. ESB is a good architecture of choice for well conditioned massive concurrent enterprise integration system, if you do it right of course ;-)

Set up Felix OSGi container with Maven and Spring in 20 mins

2008-07-10T14:29:00.002-04:00

Recently I had a chance to try out the Apache Felix an open source implementation of OSGi R4 Service Platform and found there is not a lot of document regarding how to setup your development environment for OSGi, that's why I decided to record some of my finding and learning experience here, and hopefully will shed some light on this issue.

My goal, when I started this exercise, is firstly to use Maven 2 as the build management tool so I can setup an OSGi project just like any other Java project plus easy integration with any continuous integration tools out there. Second, I wanted to setup Spring as the micro container to manage all the wiring and all the neat aspect oriented programming stuff. Personally at the beginning I did not know exactly how well Felix and Spring will mix together, but roughly I had the idea to use OSGi for service level dynamic module management and Spring for lower level wiring. Ok, enough intro lets do some coding.

a. Download Felix binary from Apache Felix

b. Start Felix by running 'java -jar bin/felix.jar', and just type any name for the profile name
Note: running java -jar in bin folder directly will not work

c. Type ps in Felix shell will show you a list of bundles that are already loaded. Now type shutdown to stop Felix.

d. After some research and trial-and-error, I found the Maven spring-osgi-bundle-archetype is the best fit I can find as a starting place for my little project. Type 'mvn archetype:generate' and pick 32 for spring-osgi-bundle-archetype.
Note: I am using maven 1.0.9

e. Run 'mvn clean install' will actually produce a valid OSGi bundle already without any coding. Try it out.

f. Restart Felix and in the shell type 'install file:path-to-your-bundle-file' should install the bundle you just created, yes its that simple, use 'ps' to check it out. Now you can start or stop the bundle.

g. Lets implement the Activator tutorial from Felix with our newly setup Maven project. See code sample and explanation here. Now if you do another 'mvn clean install' and run 'update #bundel-number' in Felix and expect to see something, you won't. Why? Because we haven't configure the Activator class as a bundle activator. To do that you need to first remove the auto generated Felix plug-in version number 1.0.0 from your pom.xml, since the old 1.0.0 plug-in does not support this configuration. After removing the version number, you need to add


your.package.Activator

under plug-in configuration.

h. Now run 'mvn clean install' again, then type 'update #bundle-number' in Felix shell, now whenever you start or stop the bundle you will see the log message got printed on the screen.

As you probably already noticed that the Maven project has already Spring configured for you, so you are pretty much all set at this point to start developing a OSGi application. Alright hopefully this setup did not take you more than 20 mins, unless you have a super slow internet connection ;-) Last but not least, you can also integrate Felix inside your Eclipse IDE for debugging and profiling purposes, see http://felix.apache.org/site/integration-of-felix-inside-eclipse.html for more details.

Have fun with OSGi and feel free to let me know your experience trying this setup out.

Interview Phantom Read

2008-07-09T13:37:00.000-04:00

I have been conducting both technical and management interviews for quite a few years now, and occasionally had to sit in a few of these interview as well. Recently just a few weeks ago, when I was sitting in the meeting room and conducting an interview with three other interviewers, one interviewer asked a question "if you encounter ..... this type of scenario, what would you do?"and I could not stop but start thinking that this type of question is useless at best and usually misleading. The reason is simple since the scenario in the question is hypothetical, thus the interviewee can fabricate an answer without worrying about any kind of constraints existed in the reality. I am not saying everybody will lie under this kind of circumstances, but the problem is you can't verify whether it is a lie or not, since the whole thing is fabricated. In most cases, I found that the interviewee will give you a perfect answer, or a solution that they would like to perform if they are working in an ideal world, I call this kind of answer - Phantom Read. If you buy into this kind of answer you get, you will probably end up hiring the person that the interviewee would like to be in the ideal world but not the actual person sitting in the room, in other words the Phantom.

So what is a good question then? A good question should always be based on the actual experience, sometimes a mere description of what they did could be the best answer you will need. Usually you can comfortably lead to this kind of question by simply asking about the past project experience, and then ask "As you mentioned ..... could you also tell us about what you did when ..... happened?" A follow up question like "If you get to do this all over again, what would you do differently to improve ..... " can provide further insight into your candidate's thinking process and self learning capability from their success or failure.

Porting Container Managed Authentication (CMA) to Spring Security 2

2008-06-23T11:41:00.005-04:00

Last week I ported one of my open source project from Container Managed Authentication (CMA) on Tomcat 6 to Spring Security 2, and decided to record some of my finding here. Spring Security 2 offers a wide range of support for different types of authentication mechanism and also allows you to centralize all security related configuration in your Spring context xml file, plus like its predecessor Acegi it allows you to fully customize every step of authentication process which is way more flexible comparing to CMA in my opinion. Enough introduction, here is the steps I performed for the porting.

Step 1 - Add Spring Security dependency in your POM:



    org.springframework.security
    spring-security-core
    2.0.2

If you are not using Maven, you need to download Spring Security library manually and add it to your project build path.

Step 2 - Remove CMA security configuration in your web.xml:

Including all security-constraint, login-config, and security-role elements in your web.xml file

Step 3 - Remove Tomcat security realm configuration:

My realm configuration was defined in META-INF/context.xml file in the WAR, but it could also be defined in server level conf/context.xml file.

note: till now your CMA configuration has been completely removed

Step 4 - Add Spring Security filter chain in your web.xml:



    springSecurityFilterChain
    
        org.springframework.web.filter.DelegatingFilterProxy
    



    springSecurityFilterChain
    /*

note: don't worry about where the chain is, for now ;-)

Step 5 - Create Spring Security context xml file


    context="http://www.springframework.org/schema/context" 
    xsi="http://www.w3.org/2001/XMLSchema-instance" 
    aop="http://www.springframework.org/schema/aop" 
    tx="http://www.springframework.org/schema/tx" 
    security="http://www.springframework.org/schema/security" 

schemalocation="http://www.springframework.org/schema/beans          http://www.springframework.org/schema/beans/spring-beans-2.5.xsd            http://www.springframework.org/schema/context            http://www.springframework.org/schema/context/spring-context-2.5.xsd        http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd        http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.5.xsd        http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security-2.0.xsd">


 
 




class="org.springframework.security.ui.webapp.AuthenticationProcessingFilter">

As you probably noticed already, two spring beans referred in this xml file have not been mentioned yet - authenticationService and passwordEncoderService. AuthenticationService is a custom class that implements UserDetailsService interface which is responsible for loading the user details for authentication purposes. You can use Hibernate if you have mapping setup for user and role entities already or lighter weight iBATIS or even raw JDBC call for the implementation. PasswordEncoderService class, implements PasswordEncoder interface, was created to help Spring Security compare encoded password, although Spring Security comes with some build-in encoders but I could not find a match for Tomcat's MD5+Hex (Base 16) style encoding therefore provided my own implementation, see Tomcat documentation and source code for details. Both beans are declared using annotation and autowiring.

Step 6 - Change your login form submit target

Change your login form submit from j_security_check to j_spring_security_check so it can be processed by Spring Security instead

Now you should be able to login as usual without changing anything in your database through Spring Security, but some of your pages might not be rendering correctly. That is because some of the method calls on HttpServletRequest do not return correct value anymore, such as getRemoteUser(), since default Tomcat HttpServletRequest implementation is not aware of Spring Security SecurityContext therefore you need to provide a wrapper class that can correctly translate these calls to return the right value. Luckily Spring Security has these wrappers build-in already, all you need to do is add an extra filter in the filter chain.

Final Step - Add securityContextHolderAwareRequestFilter


class="org.springframework.security.wrapper.SecurityContextHolderAwareRequestFilter">

Now the porting is finally done. Hopefully through this example you can see the power and flexbility of Spring Security 2, with much simplified configuration comparing to Acegi it is indeed a well designed and robust security framework deserve much consideration .

Be aware the implication when using Spring JMS template with JBoss Messaging

2008-06-12T10:44:00.006-04:00

This is a daylog note that I keep for myself and share with anyone that is interested to know. We all know the convenience of using various of template classes that Spring framework provides, but seldom we pay much attention to the implication and implicit design choices made by Spring, and recently while working with JBoss Messaging I almost missed something critical due to this kind of implication.

JmsTemplate implementation for JMS 1.1 made a crucial design choice of relying on application container to providing JMS connection pooling and caching. According to the API document:

The ConnectionFactory used with this template should return pooled Connections (or a single shared Connection) as well as pooled Sessions and MessageProducers. Otherwise, performance of ad-hoc JMS operations is going to suffer.

Spring framework assumed that most of the application server will provide a pooled JMS connection, but apparently JBoss does not think so and even openly declared that they do not provide any support if you running into any problem using Spring JmsTemplate, in this JBoss Messaging Wiki page they mentioned:

The Spring JMSTemplate code employs several anti-patterns, like creating a new connection, session, producer just to send a message, then closing them again.
...
This not only results in very poor performance, but can also make you run out of operating system resources such as threads and file handles, since some of the connection resources are released asynchronously.
....
Please note that JBoss / Red Hat will not support people using the Spring JMSTemplate with JBoss Messaging apart from the one acceptable use case for the reasons outlined above.

That being said, I am not sure that JMSTemplate actually employs anti patterns, as serious as they mentioned, in my opinion its a rather a design choice that JBoss messaging decided not to support. The recommended solution for small scale JMS usage is to decorate your connection factory with SingleConnectionFactory, this factory always returns a shared single connection wrapper and ignores the close method, which will work for most of the low volume scenario. On top of that, the new JBoss Messaging clustered connection factories, when used with clustered destination, allows seamless fail-over using a single connection which
makes this a pretty reliable solution in a clustered JMS environment. For high volume scenario, my suggestion is to use JCA managed connection factory within JBoss application server or implement your own JMS connection factory bean with connection caching or pooling outside the container (check out open JCA container implementation - Jencks) to eliminate this problem, if you are determined to work with JMSTemplate implementation.

Other references:

http://activemq.apache.org/jmstemplate-gotchas.html

How to hire cheaper talent and retain them

2008-06-03T19:26:00.007-04:00

If you are wondering where you can find cheaper talent, check out Mr. Fowler's new hypothesis:

http://martinfowler.com/bliki/CheaperTalentHypothesis.html

That's great now you got "cheaper" talent, but you will also realize that they are actually quite picky and keep jumping ships. Don't worry, the following post reveals the secret of retaining these "cheaper" talent (thanks to David for introducing this post).

http://blog.pmarca.com/2007/07/the-pmarca-gu-1.html

People are not asset, the right people are!

TinyMCE the best replacement for Tomohawk inputHtml

2008-06-02T20:45:00.005-04:00

Recently I upgrade one of my web application from JSF 1.1 to 1.2 core, and this specific application was built with MyFaces Core + Tomahawk + Facelet + Richfaces + Ajax4JSF. After the upgrade everything pretty much worked out-of-box, the only part that broke was the Tomahawk inputHtml component due to a known issue Tomahawk issue 1088, but since Tomahawk currently does not officially support JSF 1.2 there is really not much we can expect.

After this little setback, I did some research and found that TinyMCE seems to be a nice rich text editor implementation and based on the MyFaces Wiki documentation it turned out was a quite strightforward integration. Although I did end up had to do a little bit enhancement on top of the suggested implementation, the reason for the enhancement was because I had more than one text area on the page but I did not want to turn all of them into rich text editor. The only change I had to do was using 'editor_selector' instead of 'mode' while initializing the TinyMCE script.


tinyMCE.init({
mode : "textareas",
theme : "advanced",
editor_selector : "mceEditor",
width : "640",
height : "480"
});

After that all you need to do is just set your textArea style class to 'mceEditor', which will automatically convert your textArea to a nice rich text editor.

My environment info:

MyFaces - 1.2.2
Tomahawk - 1.1.6
Facelets - 1.1.11
Richfaces - 3.2.1.GA

Hibernate 3 session createCriteria returns duplicated entities

2008-05-31T17:40:00.003-04:00

Just ran into a minor problem with Hibernate 3, and thought might worth recording the solution and notes here.

Problem statement:

When using Hibernate Session.createCriteria(Class persistentClass) method to fetch a list of entities based on the given class type, the return result will contain duplicated instances if the entity has many-to-many eager-fetch relationship, see HB-520 for more details on this issue.

Solution:

For my specific problem, the solution is pretty simple all I had to do is add a transformer

setResultTransformer(CriteriaSpecification.DISTINCT_ROOT_ENTITY)

this way Hibernate will remove all duplicated entities from the return list. Although this simple solution will not work for more complicated scenario, when you have pagination or other requirement, in which a custom query might be used to provide more control and flexibility.

Code syntax highlighter for Blogger

2008-05-31T16:06:00.002-04:00

In my previous post, in order to display some xml code nicely on Blogger I did some research and finally found a nice little javascript/css library called SyntaxHighlighter which does pretty awesome job for most of the mainstream programming languages. On top of that it is also extremely easy to setup and does require any server side programming capability from your hosting company. It also has a one neat feature to remove the annoying "br" line breakers that Blogger automatically generates for you.

Give a try, it will make it a lot easier and prettier for you to post some example code on your blog.

Integrating Spring Security 2 with Active Directory

2008-05-29T17:45:00.014-04:00

Recently I worked on getting newly released Spring Security (formerly known as Acegi Security) to work with Microsoft Active Directory LDAP server. Although the configuration for Spring Security has massively improved comparing to the early days of Acegi, however since Active Directory has its own format plus some bugs in the early release (I am using 2.0.1 right now since thats the latest one in public Maven repository) therefore integration is not as straightforward as I expected. Thats why I decided to record the finding here in my Daylog.

Firstly we need to setup http security:

here it was configured to protect everything under protected folder using Basic authentication and also forcing the HTTPS protocol.

Secondly, we need to connect to the LDAP server:

Then, we need to let Spring Security know where to search for the users:

in my case the search base is "ou=Offices" but based on your LDAP setting it might be different. The strange looking "(sAMAccountName={0})" is the Active Directory specific syntax for matching the user name.

Last but definitely not least, we need to setup our authentication provider:


  class="org.springframework.security.providers.ldap.LdapAuthenticationProvider"
  autowire="default">
  
  
       class="org.springframework.security.providers.ldap.authenticator.BindAuthenticator">
    
    
           class="org.springframework.security.ldap.search.FilterBasedLdapUserSearch">
      
             value="(sAMAccountName={0})">
      
     
    
   
  
  
       class="org.springframework.security.ldap.populator.DefaultLdapAuthoritiesPopulator">

If you are familiar with Spring 2.x configuration you probably will start asking why all of sudden I switched from name space based security configuration to manual bean based approach. The reason is that a known bug (SEC-836 - its fixed in 2.0.2 release which is currently not yet available in Maven) in Spring Security prevent the group search from scanning the sub-tree
, therefore if your group tree has multiple levels the search will not return the right result.

Last note, all your roles defined in your directory will be returned in upper case with "ROLE_" prefix appended. This configuration was created and tested with Spring 2.5.2 and Spring Security 2.0.1.

The Death of a Start-up

2008-04-12T21:37:00.005-04:00

As a company grows and becomes more complex, it begins to trip over its own success - too many new people , too many new customers, too many new orders, too many new products. When was once great fun becomes an unwieldly ball of disorganized stuff. Lack of planning, lack of accounting, lack of system, and lack of hiring constraints create friction. Problems surface - with customer, with cash flow, with schedules.

In response, someone (often a board member) says, "It's time to grow up. This places needs some professional management." The company begins to hire MBAs and seasoned executives from blue-chip companies. Processes, procedures, checklists, and all the rest begin to sprout up like weeds. What was once an egalitarian environment gets replaced with a hierarchy. Chains of command appear for the first time. Reporting relationships become clear, and an executive class with special perks begins to appear. "We" and "they" segmentations appear - just like in a real company.

The professional managers finally rein in the mess. They create order out of chaos, but they also kill the entrepreneurial spirit. Members of the founding team begins to grumble, "This isn't fun anymore. I used to be able to just get things done. Now I have to fill out these stupid forms and follow these stupid rules. Worst of all, I have to spend a horrendous amount of time in useless meetings." The creative magic begins to wane as some of the most innovative people leave, disgusted by the burgeoning bureaucracy and hierarchy. The exciting start-up transforms into just another company, with nothing special to recommend it. The cancer of mediocrity begins to grow in earnest.

--- quoted from "Good to Great"

Although sound unrealistic, but if you have experienced of seeing a start-up company tripping over its own success and eventually getting bog down, even killed sometimes, by the exact bureaucracy and hierarchy created to save them, you know how accurately Jim Collins illustrated the scenario here. I am always amazed to see different start-up company with different style of leadership in completely different industry repeating the same story over and over again. I have personally witnessed two of such melt-down, and perhaps witnessing another one with my present job. It used to puzzle me, so "How can a somewhat successfully start-up company transform into not just another company but a good successful company or even maybe a great company without losing the innovative culture?". If you have been haunted by the same question, or working for a company that is going through the transformation stage, or founding a start-up company, check out the Good to Great by Jim Collins (if you haven't), although most of the book is about how a good company can transform itself to a great company, there are also a lot of insight and systematic thinking that I personally found priceless for any entrepreneur.

Just remember the only purpose of bureaucracy and hierarchy is to compensate the incompetence, and whatever you believe they are not the magic bullet to transform the company.

Spring 2.5.1 auto wire annotation has problem working with inner class

2008-03-22T21:40:00.005-04:00

This afternoon I ran into a problem with the new auto-wiring annotation provided by Spring 2.5.1. It seems that the @Service or @Component does not pick an annotated class if it contains an inner class. For example:

A.java


@Service
public class A{
    private class B{
       ...
    }
}

With this kind of structure, for some reason Spring annotation scanner will just simply ignore the class A in this example. For now the workaround I found is instead of using inner class, use a default scope class. For example:

A.java


@Service
public class A{
    ...
}

class B{
    ...
}

Or fall back to the old style xml bean definition for any of your service bean that contains inner class.

Currently the Spring issue tracking system is down, as soon as it comes back online I will see if this is a known issue, if not will create a issue for this problem.

Why programming?

2007-12-22T22:22:00.000-05:00

Why is programming fun? What delights may its practitioner expect as his reward?

First is the sheer joy of making things. As the child delights in his mud pie, so the adult enjoys building things, especially things of his own design. I think this delight must be an image of God's delight in making things, a delight shown in the distinctness and newness of each leaf and each snowflake.

Second is the pleasure of making things that are useful to other people. Deep within, we want others to use our work and to find it helpful. In this respect the programming system is not essentially different from the child's first clay pencil holder "for Daddy's office"

Third is the fascination of fashioning complex puzzle-like objects of interlocking moving parts and watching them work in subtle cycles, playing out the consequences of principles built in from the beginning. The programmed computer has all the fascination of the pinball machine or the jukebox mechanism, carried to the ultimate.

Fourth is the joy of always learning, which springs from nonrepeating nature of the task. In one way or another the problem is ever new, and its solver learns something: sometimes practical, sometimes theoretical, and sometimes both.

Finally, there is the delight of working in such a tractable medium. The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures.

-- Chapter 1: The Tar Pit (from The Mythical Man-Month by Federik P. Brooks, Jr.)

This is where I found out my reasoning for sticking around in this industry and doing what I am doing. Its a delightful and inspiring read from a software developer's point of view, however it is also an excellent guide on how to motivate a software development team. If you can satisfy your team members from all four aspects mentioned in this post, literally your developer will even work for you for free, just look at what's going on in the open source community. Check out this post "What can we learn from the Open Source community?" for more info on how to motivate a software team.

Oopsla - Making Object-Orientation Work Better (David Lorge Parnas)

2007-11-13T21:36:00.000-05:00

Its inspiring seeing David's presentation on this topic despite my personal opinion difference.

Object Orientation is a principle not an attribute of language (you can create non-OO program using OO language and vice versa)
Outdated documentation is worse than no-documentation at all
Program without documentation (documentation can be in many form) is not maintainable
Finish coding without writing documentation should not be considered complete

Oopsla - Panel "No Silver Bullet" Reloaded

2007-11-13T21:09:00.000-05:00

This is the best panel I attended in this year's Oopsla. Its nice to see Federick and David on the panel, and of course the big bad weir-wolf by Martin Fowler.

Federick's definition of inherent complexity vs. incidental complexity 20 years ago is still the one of the main driving forces of many recent process model and framework development aiming at minimizing incidental complexity hence focusing on the inherent complexity
Human incompetence is unavoidable
There is no silver bullet, but maybe a couple of lead bullets will do the job
There is no silver bullet, but we might have created something although complex but powerful enough to kill or seriously injure the weir-wolf, just like the creation of modern chemistry from the pursuit of ancient alchemist's dream of turning stone to gold
Embrace the inherent complexity

Oopsla - Research: Language Design

2007-11-11T02:21:00.000-05:00

This is the only academic research talk I attended in this year Oopsla which turned out quite interesting.

Software based transactional memory is approximately about 50% as efficient as hardware based solutions.
StreamFlex provides a Java based soft real-time streaming (no buffer & no dropping) platform
Annotation might be a good choice for constructing object meta-model (knowledge level model)
Its always a good idea to make your meta-model generic

Oopsla - Actor-Network Theory: Nothing to do with TCP/IP or distributed objects (Brian Marick)

2007-11-11T02:03:00.000-05:00

This is one of the my favorite talks I attended in this year Oopsla, and it is so interesting that I was busy enjoying the talk and did not take too many notes :-)

Break your mental model (to be successful in this rapid changing industry you need to constantly challenge your past experience and your way of thinking - your mental model)
Conduct retrospective of your personal career development every 5 years
Read books and topics outside of our industry to help you breaking the mental model
To produce quality software programmer and tester need to form a unity of team effort
Tester's job is not a destroyer but rather helping programmer to beef up the product together
Agile daily stand-up meeting should not be a shrunk version of weekly status report. Daily stand-up should be purely user story oriented since thats the only measurement of real progress in an agile project

Oopsla - Creating Passionate User (Kathy Sierra)

2007-11-10T01:55:00.000-05:00

This presentation is, like its name suggests, about creating passionate users which is not only essential to the success of the creation of any useful software (especially important in agile environment), but also equally important from the sales and marketing perspective for creating consumer loyalty as well as utilizing the power of word of mouth.

User can't be passionate if they suck (at using your product or doing whatever they do)
Human can acquire high definition experience once they are good at something, for example watching tennis matches are way more exciting if you actually play tennis and understand the difficulty and the tactics that the players are facing in the match
HD experience and passion threshold model also apply to personal career development
When designing software, remember its not about the software (tools) but the goals (what user wants to achieve through using the tools). Make user good at achieving the goals, not just using the tools.
Constantly ask yourself "What can I do to make my user kick ass?"
Set up milestones and development path for the user, so they know how they can get better and measure the progress too. Learn from the video game model. Again this approach can also be applied to career development and management
For startups you don't need to out spend your competitor in marketing or sales, but just out teach your competitor hence obtain consumer recognition and the power of word of mouth
One step further, with highly satisfied and ass-kicking users it is possible to create tribal community among your users to further cultivate recognition and loyalty, for example Apple 1997 "Think Different" campaign

2007 Oopsla Montreal Notes

2007-11-04T22:06:00.000-05:00

As I promised :-) in the next several blog entries I am going to share the session notes I took at 2007 Oopsla conference in Montreal. My notes are not meant to be detailed summary of what was presented at the session, but rather a bunch of sparks and ideas that I collected with very strong personal bias, please keep this in mind. At mean time, I highly recommend you checking out the content of the presentation before you start reading these notes.

Enjoy.

Extreme Programming with Ordinary Talent

2007-09-09T23:34:00.000-04:00

Recently one of my friends asked me how come I haven't published any article on my blog in about 2 months. The answer is again, almost sound like a broken record, the last two months has been pretty hectic, not in a bad way though. The project team I took over several months ago, transformed from zero unit-test coverage to above 70% within 2 releases, development iteration cut from months to 2 weeks, team has started digging their self out of bug fixes and starting to deliver business values, big-bang integration test turned into daily continous integration tests (fully automated with Maven2 and Bamboo), critical production defects became a rarity, and most importantly I stopped getting emergency phone calls at the middle of the night, therefore finally now I have the time and energy to write this blog.

In this blog, I want to talk about some common misunderstanding about Extreme Programming which is by the way my favorite Agile process and the one that my team currently following. One of the most common misunderstanding is that XP can only be successfully adopted by highly skilled and experienced development teams. This kind of thinking is easy to understand since if you look at a successful XP team, usually you will see a high caliber team consists of a group of skilled developer, and furthermore XP principles require team member to be highly self-managed and multi-talented since a typical XP developer will assume multiple roles in a triditional process, such as designer, business analyist, QA, programmer, project manager and whatever is required to be effective. However what you see in a sucessful XP team is simply the result of praticing XP rather than the prerequisite. In Extreme Programming Explained – Embrace Change (2^nd Edition) Kent Beck defined:

XP is a path of improvement to excellence for people coming together to develop software.

As you can see, XP is a process to develop excellence in both software product and the people who develop it. Many of the XP principle and pratices were actually designed specifically allowing people with ordinary talent to be able to develop non-oridinary software product and grow along the way. I am just going to list some key practices that I think are particularly helpful:

Close communication: Help clear ambiguities so you don't need someone who is exceptionally skilled in communiction to serve as a bridge between technical and business persons, but also allow everyone on the team to practice their interpersonal skills and develop their communication skills along the way.
Automcated tests: Help catch defects and shorten debug cycle therefore the team does not need to rely on super skilled developers, and also allow less skilled and experienced developers to grow in a relatively stress free and safe environment.
Pair Programming: Similar effects as the point above but also allow the more skilled developers to lead by examples and spread the knowledge more effectively among different team members, therefore almost provide a shortcut to seniority for the greener folks.
Incremental and Evolutionary Design Process: Help producing good design in a cost effective way, and also eliminate the need of having some genius architect or designer to come up with a flawless design up front (despite that it might not be even possible). At mean time, this practice also provide a perfect opportunity for every one on the team to learn from their own and other's mistakes and successes instead of just being a mindless coding machine.

Of course you can find this spirit in almost every XP principle and practice, since XP is a process more about people than software itself. As Kent put it perfectly in Extreme Programming Explained:

Its (XP's) reliance on the close collaboration of actively engaged individuals with ordinary talent.

Quotes of the Day

2007-07-31T20:36:00.000-04:00

The cheapest, fastest and most reliable components of a computer system are those that aren't there.

-- Gordon Bell

Everything should be made as simple as possible, but not simpler.

-- Albert Einstein

Pragmatic Programmer

2007-06-10T23:17:00.000-04:00

I just finished reading Pragmatic Programmer recently. Its been on my to-read list for about a year and half now, never got a chance or the urge to read it. My recent job change gave me a chance to take public transit to work instead of driving which all of sudden freed up about 2 hours everyday exclusive reading time (vs. staring at total strangers otherwise), so I just started plowing through all my to-read list. Initially even before I picked up Pragmatic Programmer, I was still not very thrilled or feeling the urge to read it since just by going through the index I thought I knew pretty much all the topics in this book already, and the only reason I picked this book was because I need a light weight book for my business trip to Vancouver, and little did I know how much I would enjoy that trip just because the lightweightness I picked. Pragmatic Programmer is not only a book that was written by veteran and sucessful programmers who had been there and done that, but also written with great sense of humor to make it a very enjoyable ride through it. For beginners it is the only book I found so far that covers such a wide range of topics from algorithm to leadership, from requirement definition to software development process, and from database design to work ethic. It covers almost every topic you need to know as a professional and soon-to-be pragmatic programmer with vivid example and real world stories that vastly expand your experience, I consider it as a short-cut to seniority. Or if you are a seasoned veteran in the trench, or a already pragmatic programmer, I think it is still a must-read since the most valueable asset I gained from this book is not the methods or theories, but the stories and metaphors such as the Tracer Bullets, Broken Windows, and Boiled Frog theses metaphors are particularly powerful when used consistently within your team or even your company to describe a complex situation. For example, you can either spend half an hour to describe the idea of fast retaining prototyping strategy with short and highly iterative development process model to your team member and project manager, or just mention two words Tracer Bullets. I found these stories and metaphors improve communication drastically if applied properly, kind of similar to the domain language mentioned in Domain Driven Design, and similarly it can too form an ubiquitous language from a more technical and process point-of-view. Again a highly recommend must-read from me, pick up one from Amazon if you have not already.

Talk about software maintenance

2007-04-12T16:09:00.000-04:00

IEEE Transaction on Software Engineering Dec 2006 issue published a very interesting study conducted by researchers at Carnegie Mellon University intended to explore and seek the answer to how software developers approach typical software maintenance tasks and if there is any behaviour pattern that can be observed as cues to help improving existing theory and toolset to make developers more efficient at performing such tasks.

The whole experiment in a nutshell consists of the following building blocks:

A group of above-average Java developers as test targets
A simple paint application written in Java with around hundreds of line of source code
A set of consistent pre-prepared typical maintenance tasks (including defect fixes and minor feature enhancement) for each developer
A video recording program running at the background to record how the developers perform their task
A artificial periodic interruption mechanism simulates real-life interruptions that happens in any software organization

Based on this study the researcher concluded that the following behaviour pattern was observed during the experiment:

Search – Developer explores cues in environment to choose a sufficiently relevant node to start comprehending
Relate – Developer explores cues in environment to decide whether to navigate a dependency, return to a previously visited node, or stop relating. If the node is relevant the developer collects it.
Collect – Developer uses some form of memory, external or otherwise, to remember what was found

For each task every developer went through a search-relate-collect cycle till they collect enough information to attempt a solution implementation, and if they encounter further uncertainty during the implementation they will repeat the cycle to collect more information that is necessary to help them moving forward. Along with the behaviour pattern, the research study also suggested some improvements for existing toolset and IDE on how to help developers to perform the search, relate, and collect actions more efficiently, but what really caught my eyes is the fact that since this research was completely conducted based on the traditional way of software development, without the agile elements, it becomes a perfect study material for finding out how agile approach can eliminate or at least minimize some of the difficulties that were observed in this study as well as what we should look-out in an agile project to ensure high software maintainability.

Any successful commercial or open-source system goes through a relatively rapid green field development stage, and a much longer sometimes more challenging brown field maintenance phase. Customer satisfaction (for commercial software) and community endorsement (for open-source software) are both highly dependent on how well and easily the system is maintained. According to this research, to improve software maintainability it pretty much boils down to helping developers search, relate, and collect information in hundreds of thousand or million lines of source code. Let’s talk about each aspect in more details and how agile approach helps in many different ways:

I. Helping Developers Search More Effectively

A. It was identified during the study that method naming played a significant role in helping or preventing developers from searching and understanding code effectively, thus Continuous Refactory, Short Method, Self-descriptive Method Name can really help you on this.

B. In the report, researchers also noticed the fact of having an effective way of communicating the intent and purpose of the code really help developers to pinpoint the optimal starting point of the their search effort as well as reduce the scope of the search they have to perform, and the researcher suggested that well-written and up-to-date document is a sound solution for this purpose, but in agile realm I think a well-written test suite will be a even better choice since they don’t just describe the intent and purpose but also coded and compiled against them so they can be verified automatically and kept up-to-date.

II. Helping Developers Relate Information More Effectively

A. Once developers find the clues they need to comprehend them and identify whether they are relevant to the tasks on hand or not, and it becomes obvious that the more readable the source code is the more effective the developer could be relating information. Agile principle Keep It Simple and Stupid, Just Enough Design, and Continuous Refactory can greatly enhance the readability of your code thus make this task a lot easier than it has to be.

III. Helping Developers Collect Information More Effectively

A. Collecting information seems all about how to store, categorize, share and maybe annotate the information that was allocated in the previous step, and it sounds like a job for the IDE to handle which is probably true just like what the researchers suggested in the report, our tool vendors do have a lot of room to improve and innovate; however if you look at this issue from a different perspective there might be just enough things that agile methodology can do to make this task easy enough so the existing IDE or your limited brain memory can easily handle it without major technological or biological advance. Now imagine a typical scenario that after hours of searching and relating finally you managed to comprehend a extremely complicated algorithm, a lot of information need to be collected and maybe stored somewhere in a document since it is so complex that you just don’t trust your own memory, and also you maybe would like to share it with your colleagues. Now instead of doing that how about directly apply your what you have learnt to the source code level, and use Refactory to reduce the complexity and increase the readability, therefore basically combining the documentation with your source code in one unified form along with the help from what I described in the previous two steps you might just be able to completely remove the need to collect information on paper, since the code is so simple and self-descriptive all you need to do is just search and relate, the rest is just plain in sight – no need to collect (at least not in a document).

How good is enough

2007-03-01T13:50:00.001-05:00

This is the title of a recent column on JDJ magazine by Nigel Cheshire, and almost in the same month ACM published its own issue of special edition on software quality where number of papers shed their thoughts on the same question “How good is enough”; a month apart, in IEEE Software journal the coauthor of Testing Extreme Programming Lisa Crispin published her article “Driving Software Quality: How Test-Driven Development Impacts Software Quality” related to this topic as well. Why all of sudden all this attention to this seemingly straightforward question? The answer is simple because most of us do not know the answer.

How many times as a manager you have seen project produced higher than normal defect counts in the QA stage after many late night pizza deliveries and weekend team lunches and finally finished implementation on time. Just when you thought your job is done, it actually just got started. The bug fixing and rework stage actually turn out to be even longer and more painful than the implementation. Many projects eventually get bogged down and lead to significant schedule and budget overrun even after they managed to initially achieve the scheduled milestone for construction phase. What went wrong? Why do we keep repeating this self destructive pattern over and over again? Nigel is his article proposed a very prominent viewpoint in which he points out that one of the major contributing factor to this problem is the over-simplified concentration on using project schedule as the dominate metric to measure the project progress. Traditionally due to the relative easiness for measuring project schedule, it is widely adopted in the industry as a, and some times the only, metric for software project. However when you use the schedule alone without the help from other quality related metrics, it does not really tell you anything other than how many days have been passed since you project started. Without the context information from quality-aware metrics when a developer tells you that “I finished the implementation and checked in all the source code on time” what it really means is “I really have no idea if this code will work or not” so this could be well the end of the construction or the start of it. Nigel in his article comes to the conclusion that “What gets measured gets managed”, but he did not get down to the details on how you suppose to measure the true quality, cost, and progress.

In Agile process model, we throw away the pure construction milestones because we understand the pure construction without test case verification base on the user requirement does not reflect the progress at all, that’s why Lisa in her article on IEEE Software journal focused on how Test-Driven Development and Design can give you a hand on this. In her article, she basically comes to the plain ugly truth that “Anything not covered in a test isn’t going to be there” so the only way to guarantee you deliver what your customer has asked for is by creating test cases for every single requirement, and only by satisfying the test cases you move the project forward. This is especially crucial in Agile project, since you have to constantly make “How good is enough” decisions, no only on overall quality alone but for almost everything else such as your just good enough design and just in time implementation. With the help of test driven requirement it makes our life much easier to answer the question, but writing testable or quantifiable requirement is not just about some extra description in our story card or requirement document, however it requires a fundamental mindset shift from the traditional way of presenting and organizing requirement. But if you are interested in this topic Nail Maiden in his back-to-back article on requirement quantification “Improve Your Requirements: Quantify Them” and “My Requirements? Well, That Depends” provided some very interesting idea and material for further reading, and while you exploring all these new and exciting technique and theory in the search of the answer of this million-dollar question just don’t forget this century old wisdom:

Perfection is the enemy of the good.

Gustave Flaubert
French realist novelist (1821 - 1880)

Quote of the Day and Happy Chinese New Year Everybody

2007-02-19T11:15:00.000-05:00

Sometimes through heroism you can make something work. However, understanding why it worked, abstracting it, making it a primitive is the key to getting to the next order of magnitude of scale.

-- Robert Calderbank
Princeton

Congrats to Java finally cracking into hard real-time environment

2007-02-13T16:22:00.000-05:00

David F. Bacon from IBM research published an article Real Time Garbage Collection on recent ACM Queue magazine. In which he introduced project Metronome at IBM that successfully delivered a JVM implementation that removed three major road blocks that’s been keeping people away from using Java in any hard real-time environment (even with the real-time JVM specification extension):

Undeterministic garbage collection
JIT compilation
Dynamic class loading

The Metronome project has been adopted in the new Navy’s DDG-1000 destroyer. This is pretty significant breakthrough since now people will have a choice to facilitate safer language features such as garbage collection in not just soft real-time but hard real-time system like the weapon control system on a DDG.

It’s an exciting time again for programmers

2007-02-13T14:27:00.000-05:00

About a couple of months ago, during the farewell lunch of mine at my previous work place one of my colleague asked me the question “What is the most exciting thing you think will happen to our field (as programmers) in the next 5-10 years?” To my surprise, I did not even pause for long (considering I am usually not very good at this kind of prediction) before I giving out my answer “The increasing parallelism in the hardware”. Now several months have passed, not years, you can already see the movement almost everywhere from hardware to software, from industry to academy. Just when you think Moore’s Law no longer applies to the modern processor design any more, it seems morphed into a different dimension, although the processor speed increase slowed down, the number of independent cores is currently increasing at a dazzling speed. Intel just releases its showcase an 80-core processor prototype. Now just imagine the challenge for software engineers to utilize this kind of concurrency and calculation power. Presently even with some of the natively thread-friendly language such as Java or C#, programmer still had hard-time to build robust and highly parallel software to utilize 8-16(16-32 cores) CPUs in some high-end servers; now imagine 5 years from now not only the high-end servers but almost all the computers even the pocket size ones will have multiple cores. The reality is calling for a new mechanism to deal with this unprecedented new challenge and that’s why Sun is working on Fortress (designed to work with tens of thousand processors, petabytes of memory) and IBM is working on its own X10 language (designed to help people moving from Java platform) to harness the power of this growing beast. Recently an excellent article Unlocking Concurrency was published on ACM Queue magazine, a very interesting read if you are looking for a peek into the future, but nevertheless this is probably the most exciting time for programmers since Object Oriented Programming was invented in 60-70’s.

Talk about requirement engineering (RE)

2007-02-09T17:01:00.000-05:00

Recently I read a so far the most comprehensive research paper on how requirement engineering improvement benefits other aspects of both upstream and downstream activities in software development life cycle. This paper "An Empirical Study of the Complex Relationships between Requirements Engineering Processes" by D. Damian and J. Chisan in IEEE Transaction on Software Engineering July 2006 Issue presents the empirical findings from a case study conducted at Australia Center for Unisys Software (ACUS) during a RE improvement initiative aimed to move its RE practice up from CMMI level 1 to CMMI level 2. Before the initiative ACUS performs its RE process by simply gather its requirement in a primitive one to two sentence feature description list, due to this vague definition mechanism ACCUS suffers multiple RE related problems such as feature creeps, incorrect implementation, inaccurate estimation, and eventually lead to budget and schedule overrun as well as low customer satisfaction. This research is exceptionally interesting to me since one of my recent project suffers from exactly the same RE challenge which resulting to a somewhat project failure although there are many other contributing factors, the RE problem is definitely one of the major factors. ACUS's initiative is fairly straightforward without adopting any rigid RE methodology but instead an optimized process roughly based on RUP which contains the following elements:

· Feature Decomposition

· Requirement Tracability

· Group Analysis Session

· Cross Functional Team

· Structured Requirement

· Testing According to Requirement

According to the research, to the researcher's surprises the Group Analysis Meeting and Cross Functional Team were rated the top contributors among engineers, and on the other hand Feature Decomposition and Requirement Tracability occupied the top 2 places among managers. This conclusion is consistent with the widely accepted belief that Agile practice such as Group (Paired) Analysis Meeting and Cross Functional Team benefit mostly to the engineering team result in quality and productivity improvement, and formal techniques such as Feature Decomposition and Requirement Tracability mostly benefit managerial effort and overall controllability.

Some history on the software development process model

2007-01-07T23:12:00.000-05:00

People always have different ideas or mentalities towards software development process model. Veteran folks ask “I remember software development used to be as easy as 1-2-3, and just follow the steps things eventually get done one way or the other. We never had to worry about iterations, methodologies, or the process, so what good is this iterative/agile approach?”, and on the other hand greener folks would ask “Why aren't we following a iterative approach, and by the way what is this waterfall model you guys are talking about?”. More than 10 years after the iterative approach was introduced to the industry, you will still be amazed that just how many of the practitioners, including both developers and managers, can not tell the difference between the spiral model and the iterative model, and also just the number of the organizations that still following the decades old waterfall model or even worse still in the darkage. That is why I am writing this short article on the software development process model history hoping to help explain two questions: Why are we here? and How did we get here?

Waterfall Model

Introduction:

Winston W. Royce proposed the initial waterfall model concept in 1970 recognizing the cost of discovering defects late in the development cycle, adopted a logical stepwise process, including steps from Requirements -> Design -> Coding and unit test -> System Integration -> Operation and Maintenance, progress through this set of fixed steps to produce software system. Comparably the waterfall model is considerably better than its predecessor, the two-phase “code and fix” model, where by developer directly jumps into coding right away and then keeps fixing defects until it couldn't be fixed anymore. The water fall model is widely adopted in the industry during 70s-80s including some most influential players in the field employed by NASA and US Department of Defense.

Weakness:

The most criticized problem of the waterfall model is ironically also the single fact that contributes to its success - the fixation of the steps. The waterfall model promotes fixed steps as well as barriers created between these steps, for example in a pure waterfall model the requirements are “frozen” after the requirement definition stage throughout the entire life-cycle. Although this practice sounds logical from the software engineer's point-of-view, it is very counter-intuitive from the business perspective. In many cases with pure waterfall model, over time the development team becomes so disengaged from the real world requirement on which the project was originally based on due to the nature of the ever changing world of business. Contrary to widely accepted belief, waterfall model is exceptionally ineffective when used to implement large project. Because of the broad scope of the requirement in a large project, it is extremely difficult to get all the requirement captured and defined correctly in one shot without venturing into the construction stage, but because lack of feedback loops in the waterfall model often the development teams are forced to march through construction with an ill-defined requirement which in result inevitably leads to nothing deliverable and chaos, or even sometimes the team manages to deliver something but it is not what the customer wanted.

Spiral Model

Introduction:

In Barry Boehm's article A Spiral Model of Software Development and Enhancement published in 1988, he recommended a different approach for guiding the software development process. In his spiral model, Boehm acknowledged the weakness in the waterfall model when applied to large, complex, and high-risk project the upfront requirement analysis stage is usually too superficial, without getting into design and implementation details, to fully define the requirements which becomes a common pitfall for many waterfall driven projects. Thus Boehm recommended a more risk-driven and incremental development method in which after requirement definition and concept validation stage one or more prototypes are built to assist in early confirmation of the understanding of the requirement of the system as well as mitigate any major technical risks followed by a structured water-fall like process to produce the software system. The major advantage of this process model is it provides multiple feedback opportunities early on to verify the requirement and mitigate risks before entering the construction phase so it is more likely to produce the final product on time as well as to produce what the customer truly wants. The spiral model became the role model of software development process for the whole 90's, I still remember in my text book on Software Engineering when I took the course in 97 it was still considered the state of art process model.

Weakness:

Criticisms to this rigorous approach are mostly focused at its inability to perform in a more dynamic and competitive environment due to two main weakness: 1) It is very expensive and time-consuming making two or three non-deliverable prototypes to just confirm the requirement and mitigate risk 2) It lacks of effective mechanism to handle requirement change after it enters the construction phase. Despite the criticisms, spiral model does provide us a valid but expensive way to systematically develop a system at any size therefore even today it is still a very popular choice for large complex but rigid projects at US Army(the famous Future Combat System is a good example for the spiral model) and NASA.

Iterative Model

Introduction:

In 1995 Philippe Kruchten described a new approach, one that combines the best of two worlds from waterfall model as well as spiral model, called Iterative approach. In the traditional software development model as time progress the development process moves forward through a series of sequential steps, but in iterative model the software itself is broken down into smaller pieces and one or more of these pieces gets developed in each iteration. The whole project will typically go through multiple iterations till the whole system is implemented, and every each one of these iterations is like a mini-waterfall life-cycle has multiple phases. In addition, each iterations is like a spiral model designed to mitigate risks for that specific iteration. More recently accompanied by the advancement in engineering practice, such as refactory and automation, some developers have pushed iterative model to its extreme with shorten iterations as well as simplified life-cycle model to harvest more productivity out of the iterative model for certain kind of projects, and this highly optimized iterative model is often referred as Agile process model.

Advantages:

The main advantages that iterative model brings into the industry is basically allowing us to revisit various activities, such as requirement definition and design, multiple times during different iterations in a project, hence allowing us to correct and refine them along the way to deliver what the customers really want in a more timely and predictable fashion. Additional to this, the iterative model also enable us to design and build something even without fully understanding the requirement since in each iteration we only work on the parts that we have already understood, and furthermore, not like the spiral model, in the iterative model every iteration is a complete mini-lifecycle and produces a incremental but executable product vs. a mere prototype in the spiral model. A true incremental delivery ability which is the key in today's highly dynamic and competitive world.

Must read book list for any serious Java developer

2006-12-16T13:48:00.000-05:00

Last week one of my friend who was planning to shift his career focus from something else to Java platform asked me to recommend some must read books to advance his understanding and coding ability in Java, and I came up with this list. Although by no means if you haven't read all the books does not mean you are not a serious Java developer, especially for people who collected his/her fundamental coding knowledge on platform other than Java before he/she started exploring the Java world, but this is a list at least from my opinion definitely worth having them on your bookshelf.

Thinking in Java – Entry level book perfect as a reference book as well as be used to establish a sound and solid understanding on Java language and its core API

Effective Java Programming Language Guide – A book teaches you how to be more efficient and disciplined on Java platform

Design Pattern in Java – A Java version of the classic GoF pattern book a must read for any developer if you have not already
Refactory Improving the Design of Existing Code – A must read for any developer who want to be able to stop of the constant struggle between creating the design as well as being enslaved by your design

Once you finish all these books I believe you will obtain both a solid understanding of the platform and most importantly a set of good habits for programming on Java platform or in this matter even any other Object Oriented platform, because I believe what Kent Beck said about a great programmer is nothing but just a good programmer with very good habits.

Comments

2006-12-05T05:19:00.000-05:00

DO NOT write comments to comment what the code does; use short method and self-descriptive method name to explain it.

DO NOT write comments to comment when the code should be used; use assertion to not only describe it but force it too.

DO NOT write comments to comment how the code should be used; use design pattern to communicate it.

Write comments to comment why the code is written the way it is.

Thinking from Memory Defragmentation to Architecture Reengineering

2006-11-20T21:14:00.000-05:00

Recently I don't know if its just my luck or the whole industry is planning to upgrade their architecture, almost every friend I talked to and even every interview I had asked me the same question “How to drastically improve your architecture without trashing the existing one”, and my recent research and study in a seemingly irrelevant field “Memory Management” provided me some unexpected insight into this topic.

Despite the apparent disparity between these two topic, there are some shocking similarity under the skin for these two. Lets take a look at a classic problem in Memory Management domain the Memory Fragmentation problem. In short, Memory Fragmentation is a memory state, usually caused by some long-running process, in which blocks of allocated memory scattered between blocks of free memory instead of one continuous extent. Memory Fragmentation causes the performance of memory allocation to decrease or even prevent program from allocating memory if the free memory is too fragmented. Memory Defragmentation is a reverse process that we use to fix the fragmentation problem, and the standard approach is pretty drastic: we can shut everything down (or at least freeze everything), free all memory, and restart (or resume) the whole program.

Now how about the architecture side? Although you usually will not have fragmentation problem in your architecture (notice I said usually), but without careful design, relentless refactory, and proper migration path software architecture tend to get outdated pretty quickly (what I call Architecture Deterioration), if you had experience integrating with some legacy systems out there you will know what I mean, and that's why we have the Architecture Reengineering to basically reverse the architecture deterioration to incorporate better technology and better approach before the architecture becomes another “legacy system”. The standard approach of Architecture Reengineering could be pretty drastic too: we can stop releasing new features or updates for the product or solution where the architecture was used for 6-12 months, redesign and implement the architecture and make sure it fulfill all the requirement that the predecessor does, then resume the new feature and release cycle.

Do you see the resemblance now? As you can see that both of the standard approaches to these two problems are probably not good enough for most of the situations we as software engineers commonly encounter. The standard Memory Defragmentation approach will result at least a long noticeable freeze which will lead to not only performance lose but even some unacceptable consequences in a mission-critical hard real-time system. On the other hand, the standard Architecture Reengineering approach won't even work in most of cases simply because freezing feature release cycle for 6-12 months will simply be committing suicide in the business world. So what is the better way to attack this kind of problem? Simple, instead of try to fix the whole thing in one shot, take an incremental approach fix the system in bits and pieces. First, lets take a look at how people perform Memory Defragmentation in an incremental fashion. In a normal C-based system, this can only be done with the classic programmer trick

Any problem in computer science can be solved with another layer of indirection. -- David Wheeler

If programs are not allowed to create direct pointer to the memory address, but only pointers to pointers that are pointing to memory, defragmentation process can move memory and only update the single pointer in the system, hence incrementally defragment the memory space.

Following the same line of thinking, adding a another layer of indirection should be able to help us reengineering existing architecture. Here I want to introduce a design pattern that I often use to help me to perform incremental Architecture Reengineering and eventually achieve what I call Continuous Evolving Architecture. This design pattern consists of two basic steps:

First – You need to encapsulate the part of the system you want to reengineer to level-4 which is the service level (or just wrap the whole system if you are planning to do an overhaul on the entire architecture). When I mentioned service here, I did not mean you need to implement a set of Web-Service interfaces and use SOAP as the protocol, but a rather more conceptual idea of services.

Second – You need to implement a classic Message BUS as a mediator and proxy to expose the services that you just created for your existing system through pure message exchange. If you are familiar with the Enterprise Service BUS concept, you can imagine this as a small scale implementation without all the Web-Service and BPEL stuff.

Now you have a layer of indirection that allow you to implement new features that might heavily rely on the existing system but without the need of talking to the existing system directly or even knowing the existence of the existing components. At the same time, it gives you the freedom to change any existing part of the system according to your own time line and budget without worrying about affecting the rest of the system, and hence a path to reegineer your existing architecture incrementally and safely.

Two common mistakes that people makes when adopting Scrum for the first time

2006-11-02T20:57:00.000-05:00

I came back from an almost a-month-long vacation about 2 days ago fully refreshed and with somewhat terrible jet lag as well, but the worst thing is since I currently don't really have a full time job yet therefore there is no motivation whatsoever for me to adjust my biological clock this time that's why I think my timezone right now is still somewhere around Honolulu.

In this first post of November, I want to talk about two common but sometime fatal mistakes that people usually make when they attempt to adopt Scrum methodology for the first time in their organization.

Adopt Scrum as everything but also the only thing

A lot of people adopt Scrum without realizing that Scrum is not a complete package. As stated on Control Chaos website:

Scrum is an agile, lightweight process that can be used to manage and control software and product development using iterative, incremental practices. Wrapping existing engineering practices, including Extreme Programming and RUP, Scrum generates the benefits of agile development with the advantages of a simple implementation.

As you can see, Scrum by itself is not complete because it is only a management process including a set of principals, disciplines, and artifacts that can be used to manage a agile and iterative project, but it does not contain any engineering practices thats why it must be co-implemented with other compatible engineering practices, such as XP and RUP, in order to be effective. For people who forgets about this or misunderstands what Scrum is all about, can turn a project into total chaos instead of controlling it, since Scrum alone does not address the following major engineering concerns:

Design – How and when the architecture is drawn and the major design choices are made?
Risk Management – How and when the technical risk can be mitigated?
Quality Control – How, when and who does the job to guarantee the product meeting customer's requirement as well as the quality expectation?

I can hardly imagine any project that fails to address these concerns delivering high quality product on-time without blowing off budget.

Chicken-head syndrome

I called it chicken-head syndrome when you see a Scrum project where the “chickens” are making decisions or even creating and dispatching ad-hoc tasks to the “pigs”. This usually happens with inexperienced Scrum master who allows “chicken” to give orders during the daily Scrum meeting instead of being a mere observer, since in the most of organizations at least some “chickens” are usually higher up on the organizational chart. A mild symptom of this syndrome is seeing “chicken” jump in during the daily meeting or initiating too many discussions or meetings with the developers (pigs). In a Scrum project a good Scrum master should act as a umbrella that shields the team from unnecessary communications from the outside world (personally I prefer 0 external communication during an active Sprint) hence allowing the developers to be productive and focus on completing their backlog items and constructive communications with the product owner thats why an iteration in Scrum is called Sprint not socializing or mingling. Projects suffer from this syndrome usually result in drop of productivity which eventually leads to schedule overrun, budget overrun, or both.

Does agile process need architecture upfront?

2006-10-06T19:39:00.000-04:00

September turned out was almost the busiest month of the whole 2006 so far. It was the end of my last contract, and as usual the deadline of some critical milestones of my current project, plus I just moved to my new apartment last week and still no internet yet, I am writing this post in a Starbucks. Now put that all together with a couple of camping and cottage trips (because of the tight schedule I did not get to take vacation in the whole summer, this is some kind of compensation for that before temperature drops below zero again) thats the reason why not a single post in September from me.

Now since the craziness is slowly cooling down a bit, I think its time to attempt to talk about something that has been repeatedly popping up and becoming the center of many discussions I had in the last two months and so. There are many faces of this topic, such as “Do we need a architect?” or “Do we need to design this now?” or even the “How come component A looks just like component B, and by the way what are they for?”, but it all boils down to one fundamental question “Does agile process need an architecture upfront”. Before I could think through this question, based on my agile nature my first instinct was to answer “of course not, that's what agile is all about – no big and expensive architecture upfront and by relying on many best practices such as TDD(Test Driven Design/Development), Refactory, and Paired Programming to guarantee the architectural integrity and taking an evolving approach to eventually arrive at the best architecture”. Before I can even feel proud and good about my hardcore Agileness, many discussions and debates I had in last two months led me to re-examine the reality; although there is no official number to show the project success rate after applying Agile approach but many study suggested the failure rate is probably around 30% or even higher, so why that many projects are still failing. You can argue that “People do not know how to apply Agile approach” or even somewhat degrading excuse like “They don't know what they are doing”, you can see just how many of this kind of excuses flying around everywhere now-a-days, however if we take a more constructive approach then you will quickly realize that there must be something we as the Agile community is not doing right, and one of these things that seems to confuse a hell out of most of the people is about the necessity of architecture in Agile approach. People I met are either completely lost on this if they just moved from a traditional model to an Agile method, or following certain practices but without any rationale, or even creating architecture for Agile project but just too ashamed to tell any of his/her friend that almost felt like they are some kind of traitors of Agile thinking. After a couple of days of meditation in the middle of nowhere in the Killarny provincial park, I am writing this post and hopefully can give some answers to all those confusion, pride, guilt, and even betrayal :-)

Before I can just answer “yes” or “no” to this question, I recalled something I learned when I was 10 years old “Other than Black and White, there is also something in this world called Gray”, but to make the answer not complete useless we need to find a precise definition for the gray, something more than “The answer is yes and no”. Now lets take a look at what is Software Architecture? In the traditional sense Software Architecture is defined as

The software architecture of a system comprises its software components, their external properties, and their relationships with one another. The term also refers to documentation of a system's software architecture. Documenting software architecture facilitates communication between stakeholders, documents early decisions about high-level design, and allows reuse of design components and patterns between projects. (by Bass, Len, Paul Clements, and Rick Kazman in Software Architecture In Practice (p. 23-28) and quoted by Wikipedia)

This definition is just abstract and almost useless like other textbook definitions such as the IEEE definition, and pointed out loud and clear by Ralph Jahnson in extreme programming mail-list in 2003 that:

There is no highest level concept of a system. Customers have a different concept than developers. Customers do not care at all about the structure of significant components. So, perhaps an architecture is the highest level concept that developers have of a system in its environment.

According to Johnson in his post, architecture is just a social contract among the developers, regardless it is on the paper or just in their mind, and most importantly it has almost nothing to do with other major stakeholders since they don't really care (As a matter of fact in the reality some stakeholders don't even necessarily want the project to succeed.). So far our question is almost answered, the answer is “Software architecture is whatever your developers agree upon, a merely enough consensus among them before they can move on to the construction stage”, in other words even in Agile you still need some Architecture upfront but whether its zero documentation or cutting down a forest its all developers' call, moreover in Agile you should never document architecture for users or any other stakeholders. Now some of you might wonder again this answer sounds like a good idea and very agile but it does not seems very helpful or providing any guidance, so how much really do I have to create for the architecture before I can proceed without too much worry? To answer this in short, you should create just enough architecture based on your project size and complexity (This is another reason why my favorite methodology is RUP, because it scales as long as you capture the spirit, see Per Runeson and Peter Greberg's Extreme Programming and Rational Unified Process – Contrasts or Synonyms? for more information on this topic), but if you are just like most of the people I talked to still have no concrete ideas about just how much is enough, I want to recommend a very interesting research paper I read recently to help you understand what is considered and proven to be important in Agile process. Dr. Diane Kelly published her research paper “A Study of Design Characteristics in Evolving Software Using Stability as a Criterion” in IEEE Transactions on Software Engineering Vol. 32, NO. 5 May 2006. The original idea of this paper has nothing to do with Agile methodology but it was the target project she picked for conducting this research captured my attention as soon as I realize this is a perfect study that provides us some very rare insight into what really matters when it comes to software architecture by closely examine a large, complex, and actively maintained scientific software system and its 30 years of active history. The target project used in her study is a system that:

Simulates thermal hydraulic subsystems in Canadian designed nuclear generating station. This software was first released in 1975 and is still actively maintained and in use. Awards were given to its developers from peers in the Canadian nuclear industry. That, its long life, and its use for safety-related, operational, and design issues in the international nuclear community point to the success of this software.

Clearly this is a very successful software system, an ideal role model for almost all the software systems out there we are trying to build, but I can just imagine that 9 out of the 10 companies I have worked for if they were about to build a system like this, no one in the planning meeting would even brave enough to just mention the word Agile or XP, maybe... just maybe someone will whisper the name of Agile Unified Process (If you are interested in how Agile performs in large scale project here is another fascinating article Agile Software Testing in a Large-Scale Project published on IEEE Software July/August 2006 issue). OK now just lets take a look at what they did 30 years ago:

Developers of scientific software are generally domain experts and not software engineers. Applications require deep domain knowledge and have extensive mathematical content.....Software engineering practices are generally not used...However, from the author's observations, there was a common culture of disciplined software practices in all the groups....Often little or no documentation is produced.

Doesn't that sound familiar to you? And remember this was done 30 years ago when systemic software design, architecture, and even the development life cycle theory is still in its cradle. With all these characteristics we observed from the system and now we also know how it was constructed, its time to show you the best part of this study that is to reveal the secret of the major design related factors that contributed to the success of this software system. The author listed the following major contributing factors:

The use of common blocks was the establishment of a common vocabulary made up of the variables names associated with the common blocks...Some of the research described in this paper reveals that the common blocks provided a kind of road map into the software code and a stable basis for long-term maintainability...this finding suggests that the developers relay on the common block structure more than the procedural portion of the code (My interpretation: Domain Object Model, Responsibility Model, and Key Building Blocks)
Rather than attempt to foresee and design for predicted changes, create a design that allows the software to evolve in response to unforeseeable changes. This may point, for example, to the architectural style typical of scientific software being appropriate and supportive of the changes that must be absorbed over the long term by this type of software.(My interpretation: Simplicity, Just Good Enough Design, Just In Time Implementation, and Refactory)
The software in this study has a high percentage of variable whose semantics and names relate to entities in the application and user domains. (My interpretation: Proper Object-Oriented Design and Programming - even on non-object oriented platform)
First, initial attention to naming convention is key. Second, maintaining those convention both over time and throughout all parts of the software provides a form of stable documentation. (Coding/Naming Convention and Self Descriptive Coding Practice)

Hopefully these factors I mentioned above can serve you as a starting point for your own Agile Architecture, and as well hoping after reading this post you can be as excited as I am right now in this ever intriguing but promising software engineering world, like Herb Shulter said in the September issue of 2006 Dr. Dobbs “It is a wonderful time to be a software engineer” !

What can we learn from the Open Source community?

2006-08-31T16:24:00.000-04:00

The most common complains I heard from the developers during my consulting career are as such that “The job is boring”, “The job is too stressful”, or “The manager does not know what he/she is doing”, and if I hear that a lot from the team then it means the team is probably demoralized which in most cases inevitably leads to a project failure. At the same time on the other side, the common complains that coming from the managers usually consist “The developers are not self-motivated/managed”, “The developers are too picky”, or “The developers are too greedy” as you can simply see both sides are not happy with each other, and in most cases that I have observed eventually the trust between managers and developers will be completely broken resulting in further demoralization which ultimately contributes to the failure of the project or can even bring down a company.

Now lets take a look at some of the characteristics of a successful open source team. In my opinion almost every successful and popular open source project has the following characteristics: firstly the team delivers high quality software that largely satisfy end-users' requirement, secondly it is consist of a group of highly skilled and self motivated developers, and finally (this is the sweetest part for the corporation) they don't get to paid to do all of that, but rather doing it for FREE. Now it is obvious this kind of developers are the perfect kind of employees, a dream come true if I may say for many of the corporations out there, so my next question will be how can a company acquire this kind of employees? If I am running a recruiting company, I will probably tell you that I can find someone like that for you, but since I don't I am going to tell you the truth. It all starts from building the company culture. In my opinion, two major contributing factor formed the solid foundation of any successful open source team and their formidable power of devotion.

The Learning Edge

Recent research in Psychology shows that when a person is in a constant and comfortable learning zone without too much pressure, then he/she has a much higher chance to get into a state called “Flow”. Flow, as described in The Psychology of Optimal Experience by Mihaly Csikszentmihalyi is a state of altered consciousness in which our ability to concentrate and perform is enormously enhanced. The research also suggests that within this state a great level of self-satisfaction and appreciation are being generated for the individual, in other words in this state a person can be more easily satisfied without too much stimulation from the external environment. The open source project perfectly reflects this theory, most of the team members are not being paid to work but still they devote a large chunk of their time and energy to do something, why? Because they can achieve satisfaction this way without any reward in any materialized form. We all know from Maslow's Hierarchy of Human Needs that once the basic needs are met the effectiveness of putting more investment such as increasing salary or benefit suffers from the law of Marginal Efficiency of Investment, hence it can not be used effectively as a motivation tool forever, sooner or later the company has to consult to other means to motivate people and of course the Learning Edge is the cheapest and proven the most effective way of doing that. For more information on the Learning Edge management philosophy check out the “The Learning Edge” article that is published on June 2006 ACM Communications journal.

Direct Feedback from the End Users

This is something else that's usually missing in many corporations but playing an very important role in open source community that also contributes to the high level of self-satisfaction and motivation demonstrated by some of the most successful open source team in the field. A good manager knows to openly express his/her appreciation to the team to increase the morale and the team spirit, but just like salary this kind of appreciation from a single manager or even a couple of them also suffers from the law of Marginal Efficiency of Investment and eventually become ineffective or too expensive to be effective, however if you open the communication channel between the developers and the end users directly then you actually open the door to a infinite appreciation program (if the software quality is good and also serves the end user's requirement) as well as an equally infinite monitoring and inspection program (if the software suffers from quality issues or fails to meet the end user's requirement) although special care need to be taken in consideration to remove the distraction and any other negative effects as side effects this direct channel can create, but overall I think the benefit greatly outweighs the shortcomings.

To summarize, to be competitive as a technology company at this post-dot-com era you can not only rely on the traditional management toolkits and philosophies but also need to learn from the open source community to first of all maintain a reasonable salary level as well as talents and most importantly to survive and strive.

Deadlock problem in MySQL JConnector 3.1

2006-08-21T11:45:00.000-04:00

Recently I found a very interesting but also annoying database deadlock problem in the my current project. For the performance reason we uses different storage engine in MySQL for different type of tables, for example InnoDB engine for OLTP type tables, MyISAM engine when high data throughput as well as certain level of data inconsistency can coexist, MEMORY engine for transient data or tables that have extreme performance requirement.

Just a couple of weeks ago, during system stability testing we discovered that occasionally deadlocks were found in MySQL JConnector 3.1 driver when we have large number of InnoDB and MyISAM tables coexist in the same database. According to MySQL bug base, this problem is scheduled to be fixed in JConnector 5, unfortunately due to many different reasons both technical and political we do not have the luxury to upgrade to JConnector 5 at least for now in this project, but we also disparately need this mixed engine deployment feature from MySQL to meet our performance requirement. It became obvious a couple weeks ago, I had to fixed this problem for this project to be successfully released. After downloading JConnector source code, by using the combination of simple JVM dump and JProfiler finally I pinpointed the problem in two classes com.mysql.jdbc.Connection and com.mysql.jdbc.ServerPreparedStatement. In the original Connection class prepareStatement method implementation, where PreparedStatement gets created, it only uses a simple synchronized keyword at method level as the semaphore. Within this method Connection class performs many call to ServerPreparedStatement to open, close, and check state of the statement. Based on my observation and investigation, realClose method on ServerPreparedStatement seems is the one that causes all of the problems. Looked at the realClose method, quickly I realized the problem is that in this method ServerPreparedStatement makes callback to Connection class which causes a classic cross-reference scenario (the breeding bed of the notorious deadlock problem). After that I also examined how ServerPreparedStatement uses its semaphore, it appears to me it is obvious that the orginal implementation of the locking mechanism in this part of the code is rather naïve and not well thought-through. The original implementation of ServerPreparedStatement uses three different semaphores in realClose method in the order of statement.this, connection.this, connection.mutex, and now if you look at the Connection.prepareStatement implementation as we mentioned before at first it seems only uses connection.this as the semaphore, but with a little bit digging I realized that before calling prepareStatement method the current thread has already acquired connection.mutex as a semaphore which means Connection.prepareStatement locks in connection.mutex then connection.this sequence, but ServerPreparedStatement.realClose locks in connection.this then connection.mutex sequence. Now everything is clear, it is a classic locking sequence problem, and the solution is easy that is to make the sequence of semaphore acquisition the same in both Connection.prepareStatement and ServerPreparedStatement.realClose methods by adding explicit locking order:

synchronized (getMutex()) {
synchronized (this) {

to Connection.prepareStatement implementation and switching the locking sequence:

synchronized (this.connection.getMutex()) {
synchronized (this.connection) {

in ServerPreparedStatement.realClose implementation.

Although this fix did fixed all the problem we had in our project, but according to MySQL official bug base that this deadlock problem might have a much broader base of causes and impact, therefore it can be only address in version 5, so before you apply this fix make sure you fully understand the implication and the causes of your particular problem, also I strongly recommend you running a thorough stability test and profiling session after applying this fix, nevertheless I am hoping this post will shed some light to this problem as well as its solution and hopefully can also provide some help.