Saturday, 7 November 2009

Using groovy to generate java sources in maven

In this article I'll briefly cover how to use Groovy to generate java sources as part of a maven build.

There are times when you need to generate Java source which is either repetitive, based on external data or both. Now there are tools out there for some tasks like this, but for most simple tasks running a groovy script is more than enough.

Now for the purposes of this article, I'm going to presume we need an enum which represents the cards in a Standard deck of cards, one entry per card.

First we need to create our groovy script. Now the best place to put this is in the src/main/script directory of your project. Here's the script for this (writing groovy is not in scope for this article) which goes into the file src/main/script/CardDeckGen.groovy:
class CardDeckGen {

    public generate( project, String packageName, String className ) {

        // Where to write the classes
        File targetDirectory = new File( project.build.directory + '/generated-sources/groovy' )

        // The directory to write the source to
        File packageDir = new File( targetDirectory, packageName.replace( '.', '/' ) )

        // Now to create our enum
        def out = []
        out<<'package '+packageName+';\n'
        out<<'public enum '+className+' {\n'

        // We have four suits
        def suit = [ 'D', 'S', 'H', 'C']

        // Each suit has A, 2-9, T, J, Q & K
        def rank = [ 'A', '2', '3', '4', '5', '6', '7', '8', '9', 'T', 'J', 'Q', 'K' ]

        // For each suit, write an entry for each card
        suit.eachWithIndex {
            s, i -> def indent = i==0 ? '    ' : ',\n    '
            rank.each {
                r -> out<< indent + s + r
                indent = ','
            }
        }

        // Mark the end of the enum list
        out<<';\n'

        // Finish the enum class
        out<<'}\n'

        // Convert the array into a string
        StringBuilder sb = new StringBuilder()
        out.each { sb.append(it) }

        // Now write the source, ensuring the directory exists first
        packageDir.mkdirs()
        new File( packageDir, className + ".java" ).write( sb.toString() );
    }

}
Next we need to get maven to run this script, so in pom.xml we first need to make sure we are using the correct version of the jdk - 1.5 or 1.6 would be enough:
<plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <source>1.6</source>
        <target>1.6</target>
    </configuration>
</plugin>
Next groovy, here we add the gmaven-plugin which runs a small code snippet calling our class:
<plugin>
    <groupId>org.codehaus.groovy.maven</groupId>
    <artifactId>gmaven-plugin</artifactId>
    <executions>
        <execution>
            <phase>generate-resources</phase>
            <goals>
                <goal>execute</goal>
            </goals>
            <configuration>
                <scriptpath>
                    <element>${basedir}/src/main/script</element>
                </scriptpath>
                <source>
                    CardDeckGen cdg = new CardDeckGen()
                    cdg.generate( project, 'uk.org.retep.example', 'CardDeck' )
                </source>
            </configuration>
        </execution>
    </executions>
</plugin>
Now that's enough to generate our enum, however maven doesn't know about it so it wont compile. To do that we need to add it so the compiler picks it up:
<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>add-groovy-sources</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>add-source</goal>
            </goals>
            <configuration>
                <sources>
                    <source>${project.build.directory}/generated-sources/groovy</source>
                </sources>
            </configuration>
        </execution>
    </executions>
Now you can run maven.mvn clean install
If all goes well you'll get a maven artifact containing an enum:
package uk.org.retep.example;
public enum CardDeck {
    DA,D2,D3,D4,D5,D6,D7,D8,D9,DT,DJ,DQ,DK,
    SA,S2,S3,S4,S5,S6,S7,S8,S9,ST,SJ,SQ,SK,
    HA,H2,H3,H4,H5,H6,H7,H8,H9,HT,HJ,HQ,HK,
    CA,C2,C3,C4,C5,C6,C7,C8,C9,CT,CJ,CQ,CK;
}
This was a simple example but quite a powerful one. I've used this technique for generating more complex versions of Card's to classes based on data contained within RFC's (STRINGPREP etc).

For help on writing Groovy stripts take a look at the Groovy Getting Started Guide.

Releasing to Kenai via Maven

Ever since the website feature appeared on Kenai I've been trying to get releases to work using it to host the maven repository. Although it works, it's always had the odd issue where whilst uploading artifacts to the repository, the connection would timeout causing the release build to fail.

For small projects this isn't really a problem, but for some of mine this is an issue to either the number of artifacts present (retepXMPP currently has over a hundred) we can't have the release build to fail part way through leaving both the mercurial and maven repositories in an inconsistent state.

The retepMicroKernel project is further complicated by the fact that it has native code - so a release has to be performed in parts with the native code built on the respective platforms.

Fortunately there is a solution to this - perform the release build into a staging repository then once it's complete merge that repository with the public one. This also means we can perform a release and if something horrible goes wrong we just redo the release without the public seeing the failed release.

Fabrizio Giudici also found this solution with an additional benefit. The maven release process makes several commits into the repository, but as mercurial is a distributed scm we can use that to our advantage - if the release build fails it doesn't cause issues remotely with having to remove tags etc (which I've done before).

So here's what I did when releasing version 9.11 of retepTools this week, first I followed Fabrizio's suggestions for configuring the root pom.xml file.

Then I created a temporary directory and inside that clone the public mercurial repo then make a clone of the clone:
# The maven staging repository
mkdir -p ~/tmp/stagingrepo

# The staging mercurial repository - this is a clone of the public repository
cd ~/tmp
hg clone https://hg.kenai.com/hg/reteptools~hg reteptools-stage

# The release mercurial repository - this is the one we will run maven against
# it is a clone of the staging repository
hg clone reteptools-stage reteptools
Now we run the release process:
cd ~/tmp/reteptools
mvn --batch-mode \
       -Dstaging.hg.repo.url=~/tmp/reteptools \
       -Dstaging.mvn.repo.url=file:///Users/peter/tmp/stagingrepo \
       -DreleaseVersion=9.11 \
       -DdevelopmentVersion=9.12-SNAPSHOT \
       -Dtag=retepTools-9.11 \
       release:prepare release:perform
Maven will now perform the release, making commits into the ~/tmp/reteptools repository and pushing them to ~/tmp/reteptools-stage.

Once we are happy that the release is clean we make it public. First mercurial - we need to push the changes back to kenai:
cd ~/tmp/reteptools-stage
hg push
Next the artifacts:
# A temporary directory for maven - it will not run without this directory
mkdir -p tmpdir
mvn org.codehaus.mojo:wagon-maven-plugin:1.0-beta-2:merge-maven-repos \
  -Dwagon.source=file:///Users/peter/tmp/stagingrepo/ \
  -Dwagon.target=dav:https://kenai.com/website/reteptools/maven/releases/ \
  -Dwagon.targetId=retep.releases \
  -Djava.io.tmpdir=target

[Edited to fix a couple of errors in the maven commands]

Tuesday, 11 August 2009

Forking projects

Just a quick one this (apart from trying out the ScribeFire FireFox plugin to see how it does for posting to the blog)... today whilst searching Google for something totally different I found a project on SourceForge called gnujpdf which is a fork of my old retepPDF project.

I've not tried it, but it's good to see someone resurrecting old code (although it does appear to have been dormant for about 2 years now).
This project is a new version of a project called 'retepPDF' by Peter Mount. The project hadn't been touched in a long time and I wanted to make some considerable changes to it, so I started a new project and renamed the package. I haven't been able to contact Peter yet, but all credit to him for the hard work of figuring out Adobe's wacky file format.
That's nice - actually Adobe's format is pretty well engineered, some of the better specification documentation I've ever seen so it was easy.

The one thing that got me was this:
This library started out as RetepPDF, by a guy named Peter T. Mount, whom I have yet to speak with. When researching existing projects to fill the need for the PDF functionality, I found RetepPdf on sourceforge and tried to get in touch with Peter, but never had any luck. The package hadn't been touched in a while, and I wanted to improve it, so I opened a new project on sourceforge.
Hmmm, unless it was an obsolete email address (not likely since 2000) it's probably been a spam filter as I always reply to this sort of thing.

I don't mind people forking my code, it actually shows to me it's being used. It's just nice to know of the forks as I'd probably either help out or see how differently people do things or find bugs etc.

I'm online in various forms, either email (peter at retep dot org dot uk), jabber (peter at retep dot org), msn (retep207 at hotmail dot com) the two blogs and various forums dotted around so it's not like I'm living on a different planet...

Tuesday, 19 May 2009

Performing Maven Releases on Kenai with Hudson

This article follows on from my previous article where I covered how to get a maven repository working on Kenai.com as they currently do not provide a maven repository and describes how to get Hudson to perform maven releases into that repository automatically.

Configuring Hudson for releasing your project

Maven configuration

The first part is to prepare your project’s pom.xml defining both the scm and distributionManagement elements:

For scm we define the url to point to the remote source repository. Unfortunately you have to put a username in here and you cannot define a connection entry:

<scm>
        <developerConnection>scm:hg:ssh://petermount@hg.kenai.com/reteptools~hg</developerConnection>
        <url>https://kenai.com/hg/reteptools~hg</url>
    </scm>

Next the distributionManagement element. Here we define it to point to the local copy of our repository that we created in the earlier article:

<distributionManagement>
        <repository>
            <uniqueVersion>false</uniqueVersion>
            <id>retep.releases</id>
            <name>Retep.org repository</name>
            <url>file:/Users/peter/repository/reteptools~maven/releases</url>
        </repository>
    </distributionManagement>

Hudson configuration

Next for Hudson. I’m running Hudson 1.306 but this should work for earlier versions. You’ll need to install following plugins via Hudson’s plugin manager:

Once the plugins are installed and hudson restarted you need to create the job that will perform the release:

Under Source Code Management select Mercurial and enter the repository url on kenai.

Next under Build enter the following for Goals and options:

Here I define maven.repo.local to a unique one so that my local development and release repositories are separate to prevent accidentally mixing the two.

Next we need to configure the job to manage the subversion repository. Under Build Environment check M2 Extra Build Steps and add the following two build steps:

In Steps to run before mvn build add the following script which will update the local repository:

cd /Users/peter/repository/reteptools~maven/releases
  svn update

Then in Steps to run after mvn build add the following which will add the final artifacts and commit them:

cd /Users/peter/repository/reteptools~maven/releases
  svn add * --force
  svn commit -m "Release $BUILD_TAG $BUILD_ID"
Finally check Maven release build. The default goals and options should suffice.

Performing an actual release

Unless this is the first time you’ve run the job, enter your job and select Workspace followed by Wipe Out Workspace then confirm. The purpose of this step is to ensure everything is clean and hudson will perform a fresh clone from the repository.

Next select Perform Maven Release. Check Specify release version(s) to confirm that the version numbers match what you want:

When you are happy then select Schedule Maven Release Build.

If all goes well then you should find that Hudson performs the release - although depending on how fast your network connection is you may want to go and make a cup of Coffee...

Example output

If you watch the Console Output, you should see something like the following:

First release:prepare makes a clone and runs a build and all tests:

Started by user peter
$ hg clone ssh://petermount@hg.kenai.com/reteptools~hg /Users/peter/.hudson/jobs/release_retepTools/workspace
requesting all changes
adding changesets
adding manifests
adding file changes
added 244 changesets with 1491 changes to 1013 files (+1 heads)
updating working directory
426 files updated, 0 files merged, 0 files removed, 0 files unresolved
Parsing POMs
[workspace] $ /bin/sh -xe /tmp/hudson44025.sh
+ cd /Users/peter/repository/reteptools~maven/releases
+ svn update
At revision 7.
[workspace] $ java -cp /Users/peter/.hudson/plugins/maven-plugin/WEB-INF/lib/maven-agent-1.306.jar:/usr/local/apache/apache-maven-2.0.9/boot/classworlds-1.1.jar hudson.maven.agent.Main /usr/local/apache/apache-maven-2.0.9/ /Users/peter/.hudson/war/WEB-INF/lib/remoting-1.306.jar /Users/peter/.hudson/plugins/maven-plugin/WEB-INF/lib/maven-interceptor-1.306.jar 65499
channel started
Executing Maven:  -B -f /Users/peter/.hudson/jobs/release_retepTools/workspace/pom.xml -Dproject.dev.uk.org.retep:retepTools-generator=9.6-SNAPSHOT -Dproject.rel.uk.org.retep:retepTools-generator=9.5 -Dproject.dev.uk.org.retep.tools:retepTools=9.6-SNAPSHOT -Dproject.rel.uk.org.retep:retepTools=9.5 -Dproject.dev.uk.org.retep:retepTools=9.6-SNAPSHOT -Dproject.rel.uk.org.retep.tools:retepTools=9.5 -Dresume=false release:prepare release:perform
[INFO] Scanning for projects...
[INFO] Reactor build order: 
[INFO]   retepTools
[INFO]   retepTools Source Generator
[INFO]   retepTools Core
[INFO] Searching repository for plugin with prefix: ‘release’.
[INFO] ------------------------------------------------------------------------
[INFO] Building retepTools
[INFO]    task-segment: [release:prepare, release:perform] (aggregator-style)
[INFO] ------------------------------------------------------------------------
[INFO] [release:prepare]
[INFO] Verifying that there are no local modifications...
[INFO] EXECUTING: hg status
[INFO] [release.properties:unknown]
[INFO] Checking dependencies and plugins for snapshots ...
[INFO] Transforming ‘retepTools’...
[INFO] Transforming ‘retepTools Source Generator’...
[INFO] Transforming ‘retepTools Core’...
[INFO] Updating retepTools-generator to 9.5
[INFO] Not generating release POMs
[INFO] Executing goals ‘clean verify’...
[INFO] Executing: mvn clean verify --no-plugin-updates --batch-mode
[INFO] Scanning for projects...
        [INFO] Reactor build order: 
        [INFO]   retepTools
        [INFO]   retepTools Source Generator
        [INFO]   retepTools Core

next it updates the pom’s to the new release and commits them in tagging as it does so:

[INFO] 
        [INFO] 
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO] ------------------------------------------------------------------------
        [INFO] retepTools ............................................ SUCCESS [2.486s]
        [INFO] retepTools Source Generator ........................... SUCCESS [4.260s]
        [INFO] retepTools Core ....................................... SUCCESS [2:47.495s]
        [INFO] ------------------------------------------------------------------------
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD SUCCESSFUL
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 2 minutes 54 seconds
        [INFO] Finished at: Tue May 19 19:53:00 BST 2009
        [INFO] Final Memory: 37M/80M
        [INFO] ------------------------------------------------------------------------
        [INFO] Checking in modified POMs...
[INFO] EXECUTING: hg commit --message “[maven-release-plugin] prepare release retepTools-9.5” /Users/peter/.hudson/jobs/release_retepTools/workspace/pom.xml /Users/peter/.hudson/jobs/release_retepTools/workspace/generator/pom.xml /Users/peter/.hudson/jobs/release_retepTools/workspace/core/pom.xml
[INFO] EXECUTING: hg push ssh://petermount@hg.kenai.com/reteptools~hg
[INFO] Tagging release with the label retepTools-9.5...
[INFO] EXECUTING: hg tag --message “[maven-release-plugin]  copy for tag retepTools-9.5” retepTools-9.5
[INFO] EXECUTING: hg push ssh://petermount@hg.kenai.com/reteptools~hg
[INFO] EXECUTING: hg locate
[INFO] Transforming ‘retepTools’...
[INFO] Transforming ‘retepTools Source Generator’...
[INFO] Transforming ‘retepTools Core’...
[INFO] Updating retepTools-generator to 9.6-SNAPSHOT
[INFO] Not removing release POMs
[INFO] Checking in modified POMs...
[INFO] EXECUTING: hg commit --message “[maven-release-plugin] prepare for next development iteration” /Users/peter/.hudson/jobs/release_retepTools/workspace/pom.xml /Users/peter/.hudson/jobs/release_retepTools/workspace/generator/pom.xml /Users/peter/.hudson/jobs/release_retepTools/workspace/core/pom.xml
[INFO] EXECUTING: hg push ssh://petermount@hg.kenai.com/reteptools~hg
[INFO] Release preparation complete.

Then it starts release:perform. Here it makes a fresh clone but this time for the newly created tag and compiles it again:

[INFO] [release:perform]
[INFO] Checking out the project to perform the release ...
[INFO] Removing /Users/peter/.hudson/jobs/release_retepTools/workspace/target/checkout
[INFO] EXECUTING: hg clone -r retepTools-9.5 ssh://petermount@hg.kenai.com/reteptools~hg /Users/peter/.hudson/jobs/release_retepTools/workspace/target/checkout
[INFO] EXECUTING: hg locate

...

        [INFO] Uploading project information for retepTools 9.5
        [INFO] Retrieving previous metadata from retep.releases
        [INFO] Uploading repository metadata for: 'artifact uk.org.retep:retepTools'
        Uploading: file:/Users/peter/repository/reteptools~maven/releases/uk/org/retep/retepTools/9.5/retepTools-9.5-sources.jar
        783K uploaded
        Uploading: file:/Users/peter/repository/reteptools~maven/releases/uk/org/retep/retepTools/9.5/retepTools-9.5-javadoc.jar
        2383K uploaded
        [INFO] 
        [INFO] 
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO] ------------------------------------------------------------------------
        [INFO] retepTools ............................................ SUCCESS [4.546s]
        [INFO] retepTools Source Generator ........................... SUCCESS [5.987s]
        [INFO] retepTools Core ....................................... SUCCESS [2:47.913s]
        [INFO] ------------------------------------------------------------------------
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD SUCCESSFUL
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 2 minutes 58 seconds
        [INFO] Finished at: Tue May 19 19:58:10 BST 2009
        [INFO] Final Memory: 40M/80M
        [INFO] ------------------------------------------------------------------------
        [INFO] Cleaning up after release...
[HUDSON] Archiving /Users/peter/.hudson/jobs/release_retepTools/workspace/pom.xml to /Users/peter/.hudson/jobs/release_retepTools/modules/uk.org.retep.tools$retepTools/builds/2009-05-19_19-50-00/archive/uk.org.retep.tools/retepTools/9.5-SNAPSHOT/pom.xml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8 minutes 9 seconds
[INFO] Finished at: Tue May 19 19:58:10 BST 2009
[INFO] Final Memory: 11M/27M
[INFO] ------------------------------------------------------------------------
channel stopped

finally it adds and commits the new artifacts into subversion. Once this step completes your release is publicly available.

[workspace] $ /bin/sh -xe /tmp/hudson44036.sh
+ cd /Users/peter/repository/reteptools~maven/releases
+ svn add uk --force
A         uk/org/retep/maven-metadata.xml
A         uk/org/retep/maven-metadata.xml.md5
A         uk/org/retep/maven-metadata.xml.sha1
A         uk/org/retep/retepTools/9.5
A  (bin)  uk/org/retep/retepTools/9.5/retepTools-9.5-javadoc.jar
A         uk/org/retep/retepTools/9.5/retepTools-9.5-javadoc.jar.md5
A         uk/org/retep/retepTools/9.5/retepTools-9.5-javadoc.jar.sha1
A  (bin)  uk/org/retep/retepTools/9.5/retepTools-9.5-sources.jar
A         uk/org/retep/retepTools/9.5/retepTools-9.5-sources.jar.md5
A         uk/org/retep/retepTools/9.5/retepTools-9.5-sources.jar.sha1
A  (bin)  uk/org/retep/retepTools/9.5/retepTools-9.5.jar
A         uk/org/retep/retepTools/9.5/retepTools-9.5.jar.md5
A         uk/org/retep/retepTools/9.5/retepTools-9.5.jar.sha1
A         uk/org/retep/retepTools/9.5/retepTools-9.5.pom
A         uk/org/retep/retepTools/9.5/retepTools-9.5.pom.md5
A         uk/org/retep/retepTools/9.5/retepTools-9.5.pom.sha1
...
+ svn commit -m 'Release hudson-release_retepTools-15 2009-05-19_19-47-45'
Adding         releases/uk/org/retep/maven-metadata.xml
Adding         releases/uk/org/retep/maven-metadata.xml.md5
Adding         releases/uk/org/retep/maven-metadata.xml.sha1
Adding         releases/uk/org/retep/retepTools/9.5
Adding  (bin)  releases/uk/org/retep/retepTools/9.5/retepTools-9.5-javadoc.jar
Adding         releases/uk/org/retep/retepTools/9.5/retepTools-9.5-javadoc.jar.md5
Adding         releases/uk/org/retep/retepTools/9.5/retepTools-9.5-javadoc.jar.sha1
Adding  (bin)  releases/uk/org/retep/retepTools/9.5/retepTools-9.5-sources.jar
Adding         releases/uk/org/retep/retepTools/9.5/retepTools-9.5-sources.jar.md5
Adding         releases/uk/org/retep/retepTools/9.5/retepTools-9.5-sources.jar.sha1
Adding  (bin)  releases/uk/org/retep/retepTools/9.5/retepTools-9.5.jar
Adding         releases/uk/org/retep/retepTools/9.5/retepTools-9.5.jar.md5
Adding         releases/uk/org/retep/retepTools/9.5/retepTools-9.5.jar.sha1
Adding         releases/uk/org/retep/retepTools/9.5/retepTools-9.5.pom
Adding         releases/uk/org/retep/retepTools/9.5/retepTools-9.5.pom.md5
Adding         releases/uk/org/retep/retepTools/9.5/retepTools-9.5.pom.sha1
Sending        releases/uk/org/retep/retepTools/maven-metadata.xml
Sending        releases/uk/org/retep/retepTools/maven-metadata.xml.md5
Sending        releases/uk/org/retep/retepTools/maven-metadata.xml.sha1
...
Adding         releases/uk/org/retep/tools/retepTools/maven-metadata.xml
Adding         releases/uk/org/retep/tools/retepTools/maven-metadata.xml.md5
Adding         releases/uk/org/retep/tools/retepTools/maven-metadata.xml.sha1
Transmitting file data .......................................
Committed revision 8.
Finished: SUCCESS

That’s it, the release process is now fully automated and can be repeated with a couple of clicks.

Friday, 24 April 2009

Setting up a Maven repository on Kenai

Over at Project Kenai one minor issue is that it doesn’t currently support maven repositories. Although it is on their list of features to implement it’s probably not going to be available for a while.

So what if your project needs a public maven repository on Kenai?

Well the trick is to:

    add an additional subversion repository to your project which will host your repository.
  • Then on your local machine check out that new repository and tell maven to deploy to that directory instead of a remote repository.
  • Manually add any new files that have been added by maven
  • commit the changes

To use the repository simply use the subversion repositories public url as a repository definition.

Here’s the gory details on what to add to your pom’s. For all of these examples you will need to substitute reteptools~maven with the name of your subversion repository. I’m using a releases subdirectory within the repo but that is optional.

In your pom.xml you need to add a distributionManagement element pointing to your local checked out copy of the repository. If your project has child modules in it you can put this into the parent pom as it will be inherited.

    <distributionManagement>
        <repository>
            <uniqueVersion>false</uniqueVersion>
            <id>retep.releases</id>
            <name>Retep.org repository</name>
            <url>file:/Users/peter/repository/reteptools~maven/releases</url>
        </repository>
    </distributionManagement>

Next if your pom’s need to refer to your public repository you need to reference it with a repository and/or pluginRepository definition. In this case you need to use http://kenai.com/svn/ and your project~repository name as the url. Here’s what you would use to access my repository on kenai:

        <repository>
            <id>retep.releases</id>
            <name>retep.org Maven 2 Repository</name>
            <url>http://kenai.com/svn/reteptools~maven/releases/</url>
        </repository>

        <pluginRepository>
            <id>retep.releases</id>
            <name>retep.org Maven 2 Repository</name>
            <url>http://kenai.com/svn/reteptools~maven/releases/</url>
        </pluginRepository>

Now you have done this, you can build as normal. However when it comes to deploying you first run:

 mvn deploy

to deploy to your repository. Because of the distributionManagement element it will deploy to your checked out copy.

Next change directory into your repository and add any new files:

 cd /Users/peter/repository/reteptools~maven/releases
 svn status
 svn add ...add any files shown as unknown...
 svn commit -m “brief commit message”

That’s all there is to it.

At some point I’m going to see if it’s possible to automate the commit process into subversion, but for now this works.

Update:

2009-04-24 12:14: ODFToolkit (which uses a subset of kenai and was partly why I started looking into this issue) doesn’t have subversion available just Mercurial so we had the question - would this work with mercurial.

Well it looks like it is possible, just base your repository path with the template: http://kenai.com/hg/projectName~repositoryName/raw-file/tip/

This appears to serve the files cleanly.

Thursday, 9 April 2009

ssh-askpass on OSX 10.5

I’ve been playing with NetBeans 6.7M3 and the latest Mercurial plugin and found that I couldn’t push to a remote repository via ssh. All netbeans would return was:

Mercurial Push
--------------
INFO Pushing To: ssh://pmount@lego.office.gameaccount.com/hg/ga4Partner ...
ERROR Command failed:
Command: [/usr/local/bin/hg, outgoing, -v, --template=rev:{rev}\nauth:{author}\ndesc:{desc}\ndate:{date|hgdate}\nid:{node|short}\n\nendCS:\n, --repository, /Users/peter/dev/gameaccount/maven/ga4Partner, ssh://pmount@lego.office.gameaccount.com/hg/ga4Partner]
Output: [running ssh pmount@lego.office.gameaccount.com “hg -R hg/ga4Partner serve --stdio”, remote: ssh_askpass: exec(/usr/libexec/ssh-askpass): No such file or directory, remote: Host key verification failed., abort: no suitable response from remote hg!]
INFO: End of Mercurial Push

Here the remote server is trying to prompt for the login password but fails because OSX 10.5 does not have the ssk-askpass command.

After some searching on the net (ok Google who else) I found Mercurial Push from IntelliJ where someone had a similar position with IntelliJ. By creating the following script mercurial in NB6.7M3 works flawlessly:

sabrina:~ peter$ sudo vi /usr/libexec/ssh-askpass
#! /bin/sh  
  
#  
# An SSH_ASKPASS command for MacOS X  
#  
# Author: Joseph Mocker, Sun Microsystems  
  
#  
# To use this script:  
#     setenv SSH_ASKPASS "macos-askpass"  
#     setenv DISPLAY ":0"  
#  
  
TITLE=${MACOS_ASKPASS_TITLE:-"SSH"}  
  
DIALOG="display dialog \"$@\" default answer \"\" with title \"$TITLE\""  
DIALOG="$DIALOG with icon caution with hidden answer"  
  
result=`osascript -e 'tell application "Finder"' -e "activate"  -e "$DIALOG" -e 'end tell'`  
  
if [ "$result" = "" ]; then  
    exit 1  
else  
    echo "$result" | sed -e 's/^text returned://' -e 's/, button returned:.*$//'  
    exit 0  
fi  

sabrina:~ peter$ sudo chmod +x /usr/libexec/ssh-askpass

Tuesday, 24 March 2009

Moving to Kenai

Well having tried various PHP content management systems on retep.org I’ve just about had enough. I was trying Liferay as well (and that was looking good), but in the end I was spending way too much time on getting things working rather than getting down to coding I’ve decided to move back to using one of the “forges” out there.


The other reason was cost - running a dedicated server was just costing too much each month.


Anyhow I’ve going to move everything over to Project Kenai provided by Sun. It’s supports Mercurial, Forums a Wiki and more importantly Jira.


RetepTools is now over and it’s project page is now http://kenai.com/projects/reteptools, and Jira is now up to date with what was in Mantis.


I’ll be moving the MicroKernel over later tonight and I’ll be deciding on how to migrate the other projects over with the intention of shutting down the old site at the weekend. The one thing I have left to sort is getting a working Maven repository working. I’m not the only one using that site and want Maven so hopefully it wont be long (the main requestors are the ODFToolkit team).


Anyhow one this process is complete then this blog will be moving to become the main retep.org homepage.


Thursday, 12 March 2009

retepTools Concurrency Support

retepTools Concurrency Support


The retepTools library provides additional concurrency support to that provided by the java.util.concurrency package of JDK 1.5 or later.


The core component of the concurrency support provided by retepTools is the locking. By utilising a set of annotations, it is possible to mark a method so that the entire body of that method is bound to the scope of that Lock.


For example, we have a bean with a property. The properties value can be read by any number of Threads, but it can be set by only one at a time. In the normal java.util.concurrent.lock way you would write something like this:


import java.util.concurrent.locks.ReadWriteLock;

import java.util.concurrent.locks.ReentrantReadWriteLock;


public class MyBean

{

  private final ReadWriteLock lock = new ReentrantReadWriteLock();

  private int value;


  public int getValue()

  {

    lock.readLock().lock();

    try

    {

      return value;

    }

    finally

    {

      lock.readLock().unlock();

    }

  }


  public void setValue( int newValue )

  {

    lock.writeLock().lock();

    try

    {

      value = newValue;

    }

    finally

    {

      lock.writeLock().unlock();

    }

  }

}


Now imaging doing that on an object with dozens of properties... a lot of boiler plate code which is prone to typing errors. retepTools removes this boiler plate code by using a set of thee annotations: @Lock, @ReadLock and @WriteLock.


All three annotations follow a simple contract:


    * your object must implement a corresponding method for each annotation.

    * the associated method is named as it’s annotation, i.e. @ReadLock expects readLock(), etc.

    * that method must be declared either private or “protected final”.

    * the Lock object returned by those methods must also be final - specifically it must be the same object returned for every invocation of that method on that instance.


For convenience the class uk.org.retep.util.collections.ConcurrencySupport implements this contract for objects using @ReadLock and @WriteLock.


So here’s the above code rewritten to use the annotations:


import java.util.concurrent.locks.Lock;

import java.util.concurrent.locks.ReadWriteLock;

import java.util.concurrent.locks.ReentrantReadWriteLock;

import net.jcip.annotations.ThreadSafe;

import uk.org.retep.annotations.ReadLock;

import uk.org.retep.annotations.WriteLock;


@ThreadSafe

public class MyBean

{

  private final ReadWriteLock lock = new ReentrantReadWriteLock();

  private int value;


  protected final Lock readLock()

  {

    return lock.readLock();

  }


  protected final Lock writeLock()

  {

    return lock.writeLock();

  }


  @ReadLock

  public int getValue()

  {

      return value;

  }


  @WriteLock

  public void setValue( int newValue )

  {

      value = newValue;

  }

}


Now thats alot cleaner, less error prone and yet easier to see what the business logic is rather than having it obscured by the locking mechanism.


retepTools also provides several concrete base classes to make this even easier. The main one is uk.org.retep.util.concurrent.ReadWriteConcurrencySupport which provides concrete implementations of readLock() and writeLock(). So the above example can be made even simpler by extending ReadWriteConcurrencySupport:


import java.util.concurrent.locks.ReadWriteLock;

import java.util.concurrent.locks.ReentrantReadWriteLock;

import net.jcip.annotations.ThreadSafe;

import uk.org.retep.annotations.ReadLock;

import uk.org.retep.annotations.WriteLock;

import uk.org.retep.util.concurrent.ReadWriteConcurrencySupport;


@ThreadSafe

public class MyBean extends ConcurrencySupport

{

  private int value;


  @ReadLock

  public int getValue()

  {

      return value;

  }


  @WriteLock

  public void setValue( int newValue )

  {

      value = newValue;

  }

}


Now thats alot cleaner, less error prone and yet easier to see what the business logic is rather than having it obscured by the locking mechanism.


Footnotes


  1. I originally started to write this article back in February however since retepTools 9.2 a new annotation processor was added which performs additional sanity checks.
  2. Project Coin is currently taking proposals for small changes to the Java Language. One of those proposals was Automatic Resource Management and during the debate about that proposal Lock’s were brought up. Now locks are out of scope for that proposal and there is an additional one which now looks like it’s not going to be accepted. I’ll be writing another article about that one later.


Implementing custom types within JAXB

With JAXB it does a pretty good job in mapping between XML and Java Objects. In normal use this is fine but what happens when you need to use a custom data type for an attribute? Well JAXB supports this in binding so this article will describe this.


The originating issue that’s caused this article to be written cropped up at work earlier today where we had an issue with currency values. Over time the various programmers had naturally used a Java double to represent them but problems started to appear when some testers started to use amounts equivalent to €1.2 Billion and things started to fall apart.


Now obviously it’s bad practice to use double for monetary values (we’ve all done it at some point) due to the fact that specific values cannot be represented accurately (eg 39.995 is represented as 39.9949989[1]), it gets worse when dealing with large values. With large amounts you start to loose precision at the milli and penny levels, so adding 1¢ to the amount will fail because the double cannot handle it because of rounding errors.


So the solution is to use BigDecimal as it has arbitrary precision. With JAXB this is not a problem as it supports BigDecimal when defining an attribute with type xsd:decimal - as long as you own the xsd then it’s simply a case of changing it from xsd:double. The problem is when you don’t own the xsd, i.e. it’s an external specification. In this case you cannot change it as you would break other implementations. This is exactly where JAXB bindings[2] come into play.


Changing the object used in an attribute


In this first example taken from my xmpp project we have a series of xsd files and a single xjb file containing the bindings.


From client.xsd which defines the “jabber:client” namespace[3]:


<xs:element name='message'>

     <xs:complexType>

        <xs:sequence>

          <xs:choice minOccurs='0' maxOccurs='unbounded'>

            <xs:element ref='subject'/>

            <xs:element ref='body'/>

            <xs:element ref='thread'/>

          </xs:choice>

          <xs:any namespace='##other' minOccurs='0' maxOccurs='unbounded'/>

          <xs:element ref='error' minOccurs='0'/>

        </xs:sequence>

        <xs:attribute name='from' type='xs:string' use='optional'/>

        <xs:attribute name='id' type='xs:NMTOKEN' use='optional'/>

        <xs:attribute name='to' type='xs:string' use='optional'/>

       <xs:attribute ref='xml:lang' use='optional'/>

     </xs:complexType>

  </xs:element>


Now here the from and to attributes represent not just a string but a Jabber IDentifier JID which within the project is represented by the class uk.org.retep.xmpp.JID. This class ensures it’s both syntactically correct and provides access to the various components within the JID.


For JAXB to use this JID class instead of String we can either add the bindings into the original xsd or into a separate xjb file. I prefer the latter as it keeps the originals untouched so that if someone updates them then the bindings are not lost:


So to add the bindings for JID into the above message element we have the following bindings.xjb file:


<?xml version='1.0' encoding='UTF-8'?>

<bindings

    xmlns=”http://java.sun.com/xml/ns/jaxb”

    xmlns:xsd=”http://www.w3.org/2001/XMLSchema”

    xmlns:retep=”http://retep.org/xml/ns/retepTools”

    version=”2.0”>

   <bindings schemaLocation="client.xsd" node="/xsd:schema/xsd:element[@name='message']/xsd:complexType/xsd:attribute[@name='from']">

        <property>

            <baseType>

                <javaType name="uk.org.retep.xmpp.JID" parseMethod="uk.org.retep.xmpp.JID.parse" printMethod="uk.org.retep.xmpp.JID.print" />

            </baseType>

        </property>

    </bindings>

    <bindings schemaLocation="client.xsd" node="/xsd:schema/xsd:element[@name='message']/xsd:complexType/xsd:attribute[@name='to']">

        <property>

            <baseType>

                <javaType name="uk.org.retep.xmpp.JID" parseMethod="uk.org.retep.xmpp.JID.parse" printMethod="uk.org.retep.xmpp.JID.print" />

            </baseType>

        </property>

    </bindings>

</bindings>


Here we define the attributes we want to change using an XPath expression and then the object we want to bind to that attribute. We also provide two methods to parse the attribute from a String to the object and to print it (from the object to a String). In this case those two methods are provided by the JID class itself but they don’t have to be.


Implementing the parse and print methods


The main things to watch out with the parse and print methods is that they have to be public static and they must handle null values. If the attribute is not present or should be a default value then they will be passed nulls:


From JID.java (abridged):


    public static JID parse( final String jid )

    {

        return jid == null ? null : new JID( jid );

    }


    public static String print( final JID jid )

    {

        return jid == null ? null : jid.toString();

    }


Back to the currency issue


Now with the currency issue which initially started off this rambling, there’s one more complication. In the specific DB the currency values are not actually stored as floats but in a custom SQL Type which I’ll call Money to protect the innocent. On the Java side there’s a similar object which uses a BigDecimal for the storage but also handles things like null’s etc. Anyhow in this case all that is required is to add a parse and print method into that class and add the equivalent bindings to the xjb file.


References

  1. Round and Round on the Daily WTF 
  2. Customising JAXB Bindings on sun.com
  3. jabber-client.xsd on xmpp.org

Sunday, 15 February 2009

The Generic Messaging API

A common use case is the routing of messages from one component within a system to another. The standard API for this type of messaging within Java is the Java Messaging Service or JMS - but that can be a bit overkill for small embedded systems or if you are implementing another existing messaging protocol like Jabber. Originating from the retepXMPP project, retepTools 9.2 introduces a core messaging API which can form the basis of many simple messaging systems.


Introduction


The Messaging API is relatively simple comprising of four main classes in the uk.org.retep.util.messaging package:


  • MessagingService - a class that forms the entry point into a messaging system for messages
  • Message - an interface defining the message that can be handled by the service
  • Component - an interface defining a component that can accept messages
  • Router - an interface defining a component that can route messages to another Router or component.


The API is totally generic so the above classes have no concept of the inplementation of an address (where messages are sent to) or the route required to send them between Routers - this is left entirely to the implementing code. The only requirement is that the implementing classes obey the contract between hashCode and equals.


The API also provides some concrete implementations of the interfaces. These implementations are themselves generic, but they are also fully concurrent allowing the implemented messaging system to operate in a fully multi-threaded environment.


The rest of this article will show how to implement a simple messaging system where a group of interacting components can pass messages comprising of simple POJO’s between themselves.


Concepts


Addressing


The first concept within the Messaging API is addressing. An address is a POJO that defines the Component that the message should be sent to. It also defines where the message was sent from if a response is required. This POJO can be any Java object. For this example we will be using java.lang.String but for more complex systems you would implement a class specifically for that purpose.


For example, in Jabber an address is known as a JID (Jabber ID) and has the form node@domain/resource (where node and resource are optional). In retepXMPP the address is implemented by a JID class which holds those three components individually allowing the messages to be routed by domain first and then by node and finally by resource.


Component


The second concept is a Component, represented by the uk.org.retep.util.messaging.Component interface. It represents a POJO that can accept a Message for some form of processing.


Route


The third concept is a Route. A route is a POJO used by a Router to determine where to send messages. Usually a Route implementation would hold the part of an address relevant to the specific Router using it. You could think of the global Domain Name System (DNS). For DNS you could have an address of “retep.org”. Then sending a messsage to that address would invove the core router using the Route “org” to send the message to the router handling “org”. That router wouild then send the message to the end Component using the Route “retep”.


Router


The final concept is a Router. A Router, represented by the uk.org.retep.util.messaging.Router interface is a specialised Component that can contain other Components. It deals with forwarding Messages to one or more Components by using Route’s.


A Simple example


To show how simple it is to setup a MessagingService, we’ll create a simple system comprising of three Components that will interact with each other when an event occurs. We’ll call these components A, B and C (yes imaginative naming here).


Address


For this example our address is a java.lang.String.


Message


The first step is to create the message. This will hold an int describing the event.


package example;


import uk.org.retep.util.messaging.message.AbstractMessage;


public class Event

        extends AbstractMessage<String>

{


    private int event;


    public int getEvent()

    {

        return event;

    }


    public void setEvent( int event )

    {

        this.event = event;

    }

}


AbstractMessage is a concrete implementation of the Message interface implementing all of it’s methods. The <String> generic defines the type of the address, in this case java.lang.String.


Components


Now for our three components. Here component A will simply send a message to B when it’s setValue() method is called. B will log that event and forward the message to C. C will then log the event itself and reply back to the originator (in this case A) with the negative value. A will then log the returned value.


package example;


import net.jcip.annotations.ThreadSafe;

import uk.org.retep.util.messaging.MessageException;

import uk.org.retep.util.messaging.component.AbstractComponent;


@ThreadSafe

public class A

        extends AbstractComponent<String, Event>

{


    public void setValue( int value )

    {

        System.out.printf( "A.setValue( %d )\n", value );

        

        Event event = new Event();

        event.setFrom( "A" );

        event.setTo( "B" );

        event.setEvent( value );

        

        try

        {

            send( event );

        }

        catch( MessageException ex )

        {

            ex.printStackTrace();

        }    }


    @Override

    public void consume( Event message )

            throws MessageException

    {

        System.out.printf( "A received event from %s: %d\n",

                           message.getFrom(),

                           message.getEvent() );

    }

}


You’ll notice that the setValue method creates a new Event, sets the from and to addresses, the value of the event and then sends the message via the send() method. Once the Component is attached to the MessagingSystem then that message will be sent on to its destination.


@ThreadSafe

public class B

        extends AbstractComponent<String, Event>

{


    @Override

    public void consume( Event event )

            throws MessageException

    {

        System.out.printf( "B received event from %s: %s\n",

                           event.getFrom(),

                           event.getEvent() );


        Event newEvent = new Event();

        newEvent.setFrom( event.getFrom() );

        newEvent.setTo( "C" );

        newEvent.setEvent( event.getEvent() );


        send( newEvent );

    }

}


@ThreadSafe

public class C

        extends AbstractComponent<String, Event>

{


    @Override

    public void consume( Event event )

            throws MessageException

    {

        System.out.printf( "C received event from %s: %s\n",

                           event.getFrom(),

                           event.getEvent() );


        Event newEvent = new Event();

        newEvent.setFrom( "C" );

        newEvent.setTo( event.getFrom() );

        newEvent.setEvent( -event.getEvent() );


        send( newEvent );

    }

}


Router and Route


Now every messaging service has a core router which accepts every message as it first enters the system. For this example we’ll use the address as the route as it’s a single keyword:


package example;


import net.jcip.annotations.ThreadSafe;

import uk.org.retep.util.messaging.router.AbstractRouter;


@ThreadSafe

public class SimpleRouter

        extends AbstractRouter<String, String>

{


    @Override

    protected String getRoute( String to )

    {

        return to;

    }

}


The Generics here define the types for the Address and the Route respectively. Now as we are using String’s then both are set to String. Also our router implements the getRoute() method which should form the Route for this instance from the to address. In this case it’s just the address, but usually you would have some other logic here.


Deploy and Run


Now all there’s left is to deploy the system and send a message:


package example;


import uk.org.retep.util.messaging.MessagingService;

import uk.org.retep.util.messaging.Router;


public class Main

{


    private Main()

    {

    }


    public static void main( String... args )

            throws Exception

    {

        Router<String, String> router = new SimpleRouter();


        MessagingService<String, String> messagingService = new MessagingService<String, String>(

                router );


        A a = new A();

        messagingService.addRoute( "A", a );

        messagingService.addRoute( "B", new B() );

        messagingService.addRoute( "C", new C() );


        messagingService.startService();


        a.setValue( 42 );

    }

}


Here we create the core Router and then the MessagingService using that router. We then create the A component and add it to the service. We then do the same for B and C. Finally we start the service, and then set the value of A to 42. When run we should then see the following as the event propagates through the system:


A.setValue( 42 )

B received event from A: 42

C received event from A: 42

A received event from C: -42


Note: C says it’s received the event from A and not B because the event’s from address is A and not B - this is from the statement newEvent.setTo( event.getFrom() ); in B.


Now with this example it’s pretty simple. We could have started the service before adding the components if we wanted to - components can be added or removed at will.


Also this example is bad in the sense that we are adding all components directly to the MessagingService. This would be correct for components routed by the core route, but say we had a router called D containing a component called D/A. The component D/A would be attached to router D and D would be connected to the messaging service.