Maven Must Die

The RubyGems packaging system has several advantages for java-hosted components. It goes places that Maven won’t go, like all the way into production and managing non-trivial executables from the command line or inittab. In my first post on rjack I make some further case for this approach and describe some considerations in packaging ruby/java components as gems. In this post I’ll describe some of the complexities with building such gems, and workarounds to common problems.

Gems with jars

The rjack project is best understood with an install of jruby, optimally hashdot, and for example, the jetty gem:

% jgem install jetty
% jetty-service

This will afford the opportunity to look at how the gem is built, as the full source including the build mechanism is included. The top two levels of the gem directory structure are shown below:

/opt/jruby/gems/gems/jetty-6.1.14.1/
|-- History.txt
|-- Manifest.txt
|-- README.txt
|-- Rakefile
|-- assembly.xml
|-- bin/
|   `-- jetty-service*
|-- lib/
|   |-- jetty/
|   `-- jetty.rb
|-- pom.xml
|-- src/
|   `-- main/
|-- test/
|   `-- test_jetty.rb
`-- webapps/
    |-- test/
    `-- test.war

Note the presence of both a Rakefile, and the Maven pom.xml and assembly.xml. Other projects that produce gems including jars manage the jar files as if they were source, but this becomes a maintenance liability in any of the following scenarios:

Included jar(s) are undergoing frequent updates and the gem should be kept current.
Included jar(s) have additional jar dependencies themselves undergoing updates, and these jars must also be included.
The gem includes original java source code that must be built, packaged as a jar and included in with the gem.
External components, perhaps themselves packaged as gems, have dependencies on the packaged java. Thus the internally built java must also be published for resolving external java dependencies.

Hoe, Rake, Maven: The unholy union

The jetty gem Rakefile (see below) attempts the rather perilous integration of Maven into Rake, with Hoe for gem building and rubyforge publishing support. Here you will find subtle and not-so-subtle workarounds for a variety of issues encountered, including:

In standard gem build tradition, the gem version is actually obtained by loading ruby source. This has the advantage of the version being specified in one (fewer) place. However, there is an unfortunate incompatibility with JRuby and Rake both currently defining Object::import (JRUBY-3171). There is a simple workaround: Include an alternative ruby include that just defines version constants without any java imports. In this case its lib/jetty/base.rb, which defines VERSION and JETTY_VERSION in the Jetty module.
Hoe requires a Manifest.txt file on disk a priori, apparently to protect us from making mistakes in what is included in the gem. The problem with this is that jar names to be included may frequently change, and only maven knows the full dependency details. The workaround is to generate the Manifest.txt file with a :manifest task, but this task must be run and completed in its own process before any Hoe tasks utilize it as part of task definition. In practice we need to remember to update the manifest when jar’s change, less than ideal.
In the jetty gem, the full list of JARS (l28) are mapped from the jetty/base.rb included VERSIONs. In other gems the assembly brings in arbitrary dependencies of mixed versions, it becomes necessary to Dir.glob( “..assembly/*.jar” ) for these and the :manifest task acquires a “mvn package” task dependency.
We know to build the assembly via “mvn package” if it doesn’t yet exist, or if the pom.xml or assembly.xml is newer. The assembly is made a dependency of the :gem and :test tasks (l84). However, for gems packing java source, the dependencies are extended to all java source files as well.
Hoe wants to use a graphics tool called “dot” for rdoc generation. I don’t. Disabled via a crufty ENV[‘NOTDOT’] setter.

These might appear to constitute a rant, but I’m well aware of the fact that all of this software is free. I’m just fishing for a better solution in a relatively new problem space.

Gem/jar dependencies

For a more complicated example, consider a dependency graph between two gems both containing java source:

gem jar dependencies

There is no dependency loops and everything works surprisingly well once gem and maven releases have been made to the respective repositories. However its complicated, manual, and error prone when implementing new versions of both gems in parallel, and needing to share changes with other developers. We must abandon Maven’s mvn release:prepare functionality for tagging a release version and incrementing to the next snapshot. Instead the following steps are typical:

For gem-a, jrake clean to remove older jar versions
Update versions numbers in the pom.xml, lib/gem-a/base.rb, and a new version to History.txt
jrake manifest to update Manifest.text with new jars.
jrake test gem to build, test and package the gem.
mvn install to make a.jar available for gem-b java build.
jgem install pkg/gem-a-VERSION.gem to make gem-a available for gem-b
For gem-b, repeat steps 1-4, etc.

Also consider that much of the above will need to be repeated if gem-a undergoes changes that gem-b is dependent on. One saving feature is that from a Maven perspective, gem-a and gem-b still look like “normal” java jar projects, if gem-a and gem-b java changes can be made in parallel with typical maven means, including use of parent pom declaring gem-a and gem-b as modules. But in general, its clear that a better integrated tool, offering full build automation for gems with jars is needed. An evaluation of alternatives will be saved for a subsequent post.

Rakefile for jetty gem

require 'rubygems'

ENV['NODOT'] = "no thank you"
require 'hoe'

$LOAD_PATH << './lib'
require 'jetty/base'

JARS = %w{ jetty jetty-util jetty-rewrite-handler }.map do |n|
  "#{n}-#{ Jetty::JETTY_VERSION }.jar"
end
JARS << "servlet-api-#{ Jetty::SERVLET_API_VERSION }-#{ Jetty::JETTY_VERSION }.jar"
JARS << 'gravitext-testservlets-1.0.jar'
JAR_FILES = JARS.map { |jar| "lib/jetty/#{jar}" }

desc "Update the Manifest with actual jars"
task :manifest do
  out = File.new( 'Manifest.txt', 'w' )
  begin
    out.write <<END
History.txt
Manifest.txt
README.txt
Rakefile
pom.xml
assembly.xml
bin/jetty-service
lib/jetty.rb
lib/jetty/base.rb
lib/jetty/rewrite.rb
lib/jetty/test-servlets.rb
src/main/java/com/gravitext/testservlets/PerfTestServlet.java
src/main/java/com/gravitext/testservlets/SnoopServlet.java
test/test_jetty.rb
test/test.txt
webapps/test.war
webapps/test/WEB-INF/web.xml
webapps/test/index.html
END
    out.puts JAR_FILES
  ensure
    out.close
  end
end

ASSEMBLY = "target/gravitext-testservlets-1.0-bin.dir"

file 'webapps/test.war' => [ 'webapps/test/index.html',
                             'webapps/test/WEB-INF/web.xml' ] do
  sh( 'jar cvf webapps/test.war ' +
      '-C webapps/test index.html -C webapps/test WEB-INF/web.xml' )
end

file ASSEMBLY => [ 'pom.xml', 'assembly.xml' ] do
  sh( 'mvn package' )
end

JARS.each do |jar|
  file "lib/jetty/#{jar}" => [ ASSEMBLY ] do
    cp_r( File.join( ASSEMBLY, jar ), 'lib/jetty' )
  end
end

[ :gem, :test ].each { |t| task t => JAR_FILES + [ 'webapps/test.war' ] }

task :mvn_clean do
  rm_f( JAR_FILES + [ 'webapps/test.war' ] )
  sh( 'mvn clean' )
end
task :clean => :mvn_clean

task :tag do
  tag = "jetty-#{Jetty::VERSION}"
  svn_base = 'svn://localhost/subversion.repo/src/gems'
  tag_url = "#{svn_base}/tags/#{tag}"

  dname = File.dirname( __FILE__ )
  dname = '.' if Dir.getwd == dname
  stat = `svn status #{dname}`
  stat.strip! if stat
  if ( stat && stat.length > 0 )
    $stderr.puts( "Resolve the following before tagging (svn status):" )
    $stderr.puts( stat )
  else
    sh( "svn cp -m 'tag [#{tag}]' #{dname} #{tag_url}" )
  end
end

hoe = Hoe.new( "jetty", Jetty::VERSION ) do |p|
  p.developer( "David Kellum", "dek-ruby@gravitext.com" )
  p.rubyforge_name = "rjack"
  p.rdoc_pattern = /^(lib.*\.(rb|txt))|[^\/]*\.txt$/
end