Toward the goal of fully self-contained java/ruby service gems and ruby scripted setup, some issues have lingered. With the recent releases of Hashdot 1.4 and the new Iyyov monitor, a complete solution is now available. Here is what it looks like:
Package and deploy your service as a versioned rubygem.
Include a version-specific service init script written in ruby and annotated will all necessary JVM/jruby settings (i.e -server -Xmx2g, etc.)
Hashdot will reliably bootstrap your service as a fully UNIX compliant fork’d, setsid, and IO redirected daemon with an instance exclusive PID file and recognizable process name.
Iyyov will start your service on system boot, keep it running, and automatically restart on config changes or gem upgrades.
No external bash scripts, user environment settings, or other cruft required.
MRI rubygem authors would expect to place a daemon startup script in the gem’s bin directory. On install, rubygems will create a wrapper script for the original that arranges for gem initializing, makes the script available to the user PATH, and provides a convenient version selection mechanism. This is a great mechanism for short lived command-line interaction with gems, but has one very troubling shortcoming for a JRuby service writer:
There is no obvious mechanism to reliably control JVM or jruby settings on a daemon or version specific basis and package these settings within a gem.
Jruby defaults to the -client JVM, 500Mb of heap, and no specific GC tunings. This is reasonable default behavior, but most any production-grade service will want and need a mechanism to override these defaults.
To achieve this we need to package a non-wrapped executable script. Here is a sample from hashdot-test-daemon:
#!/usr/bin/env jruby
#-*- ruby -*-
#. hashdot.profile += daemon
#. hashdot.pid_file = ./hashdot-test-daemon.pid
#. hashdot.io_redirect.file = ./hashdot-test-daemon.log
#. hashdot.vm.options += -Xmx64m
#. hashdot.vm.options += -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
require 'rubygems'
gem( "hashdot-test-daemon", "= 1.1.0" )
require 'rjack-logback'
RJack::Logback.config_console( :full => true, :thread => true )
require 'hashdot-test-daemon'
Hashdot::Daemon::Runner.new.run
Note that if a system level Hashdot dependency gives you concern, you could achieve an approximately similar and perhaps sufficient result with a more elaborate shell script and/or other daemonizing options less complete for JRuby.
In any case, we want to be able to launch the correct service version by referencing the full path to the script in the local gem installation. For example if the gems are installed to /opt/jruby/gems, then the following should launch the 1.1.0 version of service in the background with any 1.1.0 specific settings:
/opt/jruby/gems/gems/hashdot-test-daemon-1.1.0-java/init/hashdot-test-daemon
Given java’s native threads its unlikely that it will be necessary to run a large number of separate daemon instances. However there is still plenty of reasons to want to run different services on the same host or different configurations or versions of the same service. Toward this end I use a separate directory for each service, and relative path references to instance specific configuration, log, and PID files, i.e:
/opt/var/hashdot-test-daemon-1 ├── hashdot-test-daemon.log ├── hashdot-test-daemon.log.1.gz ├── hashdot-test-daemon.pid └── config.rb /opt/var/service-test-daemon-2 └── ...
A fully expanded shell based launch of the service might look like:
cd /opt/var/hashdot-test-daemon-1 && \
exec /opt/jruby/gems/gems/hashdot-test-daemon-1.1.0-java/init/hashdot-test-daemon -c config.rb
And in fact, one approach to boot-time startup or restart would be to encode something like this in an init.d script, inittab, upstart, or crontab. The last three can achieve automatic restart, provided that the service supports reliable instance-specific mutual exclusion (via PID file, etc.) Hashdot supports this reliably.
Once I arrived at the above solution for system layout and packaging service startup details in the gem, I went looking for a process monitor solution that would handle restarts and I could customize to keep the setup simple. I first looked at Monit, and while I was able to get it to work with these daemons in basic form, configuration was “involved”, and customization impractical.
Next I tried God. While God’s integration with kernel events makes it all powerful, and its ruby based configuration is awesome, it is also a bit of a liability to install. Furthermore, God doesn’t run on JRuby, due to a combination of heavy native non-FFI extension, and reliance on low level ruby API’s not yet completely supported. In short, I found God to be a bit more than I needed. So instead I began work on and created Iyyov’s configuration in God’s image. See the Iyyov README for a more detailed feature set and examples. The hashdot-test-daemon in the above example can be configured and launched like so:
Iyyov.context do |c|
c.define_daemon do |d|
d.name = "hashdot-test-daemon"
d.version = "~> 1.0"
d.log_rotate
end
end
Which highlights several Iyyov features:
By default, Iyyov uses the rubygems API to find the highest matching gem version and init script with the same name. Thus in the common case, configuration is absolutely DRY.
Built in log rotation/compression that meshes well with Hashdot’s IO redirection and SIGHUP log reopening support. The end result is a complete logging solution that also redirects the often neglected JVM level output in the event of a JVM crash, uncaught exception, or ‘kill -QUIT’ arranged stack dump. The external Linux logrotate utility could also be used here.
Iyyov is also a practical cron replacement supporting fixed or periodic time scheduling of supporting tasks. For example:
Iyyov.context do |c|
# Backup service state at 1am local time each weekday
# Run :async, in a "background" thread, since this might take a while.
c.schedule_at( :name => "backup", :mode => :async,
:fixed_times => [ '1:00' ], :fixed_days => (1..5) ) do
system( "rsync ..." )
end
end
As a final example, since the configuration is fully scriptable ruby, we might select which service to run by hostname in a single maintained configuration file:
Iyyov.context do |c|
case `hostname -s`.strip
when /^server-[012]$/
c.define_daemon { |d| d.name = "widget-factory" }
when 'server-3'
c.define_daemon { |d| d.name = "assembly-line" }
end
end
Because the above described per-gem init script isn’t wrapped by rubygems, the corresponding gem version should be hard coded within to guarantee the resolution of this and other gem dependencies. This presents a maintenance problem which can be solved through various means. My solution is the Tarpit 1.2 test_line_match feature that may be installed in the gem project’s Rakefile, i.e:
task :check_init_version do
t.test_line_match( 'init/hashdot-test-daemon',
/^gem.+hashdot-test-daemon/, /= #{t.version}/ )
end
A simple error message is generated when attempting to build the gem without this version being set properly. My recent gems use this feature extensively and I think its less net maintenance then the other option I considered: generating such files with ERB templates.