The RubyGems packaging system has several advantages for java-hosted components. It goes places that Maven won’t go, like all the way into production and managing non-trivial executables from the command line or inittab. In my first post on rjack I make some further case for this approach and describe some considerations in packaging ruby/java components as gems. In this post I’ll describe some of the complexities with building such gems, and workarounds to common problems.

Gems with jars

The rjack project is best understood with an install of jruby, optimally hashdot, and for example, the jetty gem:

% jgem install jetty
% jetty-service

This will afford the opportunity to look at how the gem is built, as the full source including the build mechanism is included. The top two levels of the gem directory structure are shown below:

|-- History.txt
|-- Manifest.txt
|-- README.txt
|-- Rakefile
|-- assembly.xml
|-- bin/
|   `-- jetty-service*
|-- lib/
|   |-- jetty/
|   `-- jetty.rb
|-- pom.xml
|-- src/
|   `-- main/
|-- test/
|   `-- test_jetty.rb
`-- webapps/
    |-- test/
    `-- test.war

Note the presence of both a Rakefile, and the Maven pom.xml and assembly.xml. Other projects that produce gems including jars manage the jar files as if they were source, but this becomes a maintenance liability in any of the following scenarios:

Hoe, Rake, Maven: The unholy union

The jetty gem Rakefile (see below) attempts the rather perilous integration of Maven into Rake, with Hoe for gem building and rubyforge publishing support. Here you will find subtle and not-so-subtle workarounds for a variety of issues encountered, including:

  1. In standard gem build tradition, the gem version is actually obtained by loading ruby source. This has the advantage of the version being specified in one (fewer) place. However, there is an unfortunate incompatibility with JRuby and Rake both currently defining Object::import (JRUBY-3171). There is a simple workaround: Include an alternative ruby include that just defines version constants without any java imports. In this case its lib/jetty/base.rb, which defines VERSION and JETTY_VERSION in the Jetty module.

  2. Hoe requires a Manifest.txt file on disk a priori, apparently to protect us from making mistakes in what is included in the gem. The problem with this is that jar names to be included may frequently change, and only maven knows the full dependency details. The workaround is to generate the Manifest.txt file with a :manifest task, but this task must be run and completed in its own process before any Hoe tasks utilize it as part of task definition. In practice we need to remember to update the manifest when jar’s change, less than ideal.

  3. In the jetty gem, the full list of JARS (l28) are mapped from the jetty/base.rb included VERSIONs. In other gems the assembly brings in arbitrary dependencies of mixed versions, it becomes necessary to Dir.glob( “..assembly/*.jar” ) for these and the :manifest task acquires a “mvn package” task dependency.

  4. We know to build the assembly via “mvn package” if it doesn’t yet exist, or if the pom.xml or assembly.xml is newer. The assembly is made a dependency of the :gem and :test tasks (l84). However, for gems packing java source, the dependencies are extended to all java source files as well.

  5. Hoe wants to use a graphics tool called “dot” for rdoc generation. I don’t. Disabled via a crufty ENV[‘NOTDOT’] setter.

These might appear to constitute a rant, but I’m well aware of the fact that all of this software is free. I’m just fishing for a better solution in a relatively new problem space.

Gem/jar dependencies

For a more complicated example, consider a dependency graph between two gems both containing java source:

gem jar dependencies

There is no dependency loops and everything works surprisingly well once gem and maven releases have been made to the respective repositories. However its complicated, manual, and error prone when implementing new versions of both gems in parallel, and needing to share changes with other developers. We must abandon Maven’s mvn release:prepare functionality for tagging a release version and incrementing to the next snapshot. Instead the following steps are typical:

  1. For gem-a, jrake clean to remove older jar versions
  2. Update versions numbers in the pom.xml, lib/gem-a/base.rb, and a new version to History.txt
  3. jrake manifest to update Manifest.text with new jars.
  4. jrake test gem to build, test and package the gem.
  5. mvn install to make a.jar available for gem-b java build.
  6. jgem install pkg/gem-a-VERSION.gem to make gem-a available for gem-b
  7. For gem-b, repeat steps 1-4, etc.

Also consider that much of the above will need to be repeated if gem-a undergoes changes that gem-b is dependent on. One saving feature is that from a Maven perspective, gem-a and gem-b still look like “normal” java jar projects, if gem-a and gem-b java changes can be made in parallel with typical maven means, including use of parent pom declaring gem-a and gem-b as modules. But in general, its clear that a better integrated tool, offering full build automation for gems with jars is needed. An evaluation of alternatives will be saved for a subsequent post.

Rakefile for jetty gem

require 'rubygems'

ENV['NODOT'] = "no thank you"
require 'hoe'

$LOAD_PATH << './lib'
require 'jetty/base'

JARS = %w{ jetty jetty-util jetty-rewrite-handler }.map do |n|
  "#{n}-#{ Jetty::JETTY_VERSION }.jar"
JARS << "servlet-api-#{ Jetty::SERVLET_API_VERSION }-#{ Jetty::JETTY_VERSION }.jar"
JARS << 'gravitext-testservlets-1.0.jar'
JAR_FILES = { |jar| "lib/jetty/#{jar}" }

desc "Update the Manifest with actual jars"
task :manifest do
  out = 'Manifest.txt', 'w' )
    out.write <<END
    out.puts JAR_FILES

ASSEMBLY = "target/gravitext-testservlets-1.0-bin.dir"

file 'webapps/test.war' => [ 'webapps/test/index.html',
                             'webapps/test/WEB-INF/web.xml' ] do
  sh( 'jar cvf webapps/test.war ' +
      '-C webapps/test index.html -C webapps/test WEB-INF/web.xml' )

file ASSEMBLY => [ 'pom.xml', 'assembly.xml' ] do
  sh( 'mvn package' )

JARS.each do |jar|
  file "lib/jetty/#{jar}" => [ ASSEMBLY ] do
    cp_r( File.join( ASSEMBLY, jar ), 'lib/jetty' )

[ :gem, :test ].each { |t| task t => JAR_FILES + [ 'webapps/test.war' ] }

task :mvn_clean do
  rm_f( JAR_FILES + [ 'webapps/test.war' ] )
  sh( 'mvn clean' )
task :clean => :mvn_clean

task :tag do
  tag = "jetty-#{Jetty::VERSION}"
  svn_base = 'svn://localhost/subversion.repo/src/gems'
  tag_url = "#{svn_base}/tags/#{tag}"

  dname = File.dirname( __FILE__ )
  dname = '.' if Dir.getwd == dname
  stat = `svn status #{dname}`
  stat.strip! if stat
  if ( stat && stat.length > 0 )
    $stderr.puts( "Resolve the following before tagging (svn status):" )
    $stderr.puts( stat )
    sh( "svn cp -m 'tag [#{tag}]' #{dname} #{tag_url}" )

hoe = "jetty", Jetty::VERSION ) do |p|
  p.developer( "David Kellum", "" )
  p.rubyforge_name = "rjack"
  p.rdoc_pattern = /^(lib.*\.(rb|txt))|[^\/]*\.txt$/