void * blog

Important Crates Graph

2021-01-25T00:00:00-08:00

Vertically rendered so it fits the blog format and is mobile friendly, below. Guide level key features of the graph:

All (rounded box) nodes are (rust lang., crates.io) crates.
Colors of nodes represent rough ownership groups and common release practices;
mine, hyperium, tokio-rs, libs-teamish, etc.
Dashed lines are dev-dependencies (only).
The f label means the dependency is feature dependent, and with (f) its a non-default feature.
A red edge is one that is under review—we may be able to remove it in the future, which is almost always a net benefit.

Pro-Tip: We can produce an over-sized PNG to approximate the above vector graphics (SVG) quality, if hosting this directly in the github host repository proves necessary, though it really seems like github should have implemented an SVG security scanner by now.

In the graph, so far, dot has managed to keep the colored groups mostly together simply by dependency association (weight of intra-group edges.) In more complex cases, we might need to use sub-graphs to achieve the same effect.

There are many more crates not shown. Just the body-image project lock file has 109 crates total (some platform specific, non-default, etc.), so the above is less than ⅕ of the total. Most of them don’t cause enough trouble to be shown here, possibly just because they are infrequently updated, or they are leaf nodes of crates we don’t control. By selecting the important dependencies, we can better customize and read the graph. Compare to an automated tool showing all dependencies. Perhaps a more general interpretation would be the that the whole graph is too big for my best 4k monitor, but there are potentially many sub-graphs which can be extracted and custom formatted to make them both readable and useful.

I’ve been refining this almost daily for maybe two weeks now as I work through new releases of most all of these crates, and some other private and unrelated ones.

Or, on a large monitor, open the horizontally rendered version (frame) below in a separate window, or maximize the page and zoom in:

Pro-Tip: Node label line breaks (with <br/> xhtml) are the key for orienting the graph horizontally in particular, making the nodes more square, reducing width and giving more direct paths for the edge splines.

Use cases

Be honest, you want this—in order to get organized and more easily:

Update crate versions and dependency version ranges based on a consistent, interior compatible strategy, by visually walking the graph in reverse arrow direction, from dependencies to dependees.
Identify subsets and make new releases in reverse dependency order (by visually walking the graph).
Intuit the best strategy for upgrading to originally incompatible new releases (or releases falsely claiming Semver compatibility, e.g. MSRV increases in patch releases of transitive deps, an all too common blight) from dependencies outside your control.
Find the root cause of duplicate dependencies in the overall graph, and infer the best strategy for removing them where possible, since they are either breaking (at worst) or a cosmetic and build time/size disadvantage (at best).

Potential Future Work

I’d like to see if an existing automated tool (ala cargo depgraph) maintainer would be interested in adding enough cli flags, optional features, or configuration to allow me to better automate the above without loss of fidelity, even if subsequent customization to the output dot files remains required. This is the default output of current cargo depgraph --all-deps on body-image as a point comparison:

(Don’t squint—I can’t see anything either.)

Even with my current manually produced version, it might be revealing if it made the distinction between private (hidden) and public (visible in the API) dependency edges. (Yet another 3 year old, unstable, incomplete cargo RFC.)

Blocking Permit

2020-01-13T00:00:00-08:00

No one ever gets in trouble for posting micro benchmarks and making broad assumptions about the cause of observed results! This post will focus on a couple of such benchmarks pertaining to blocking operations on otherwise asynchronous runtimes. Along the way I’ll give only sparse background on these projects I’ve been working on, but plenty of links if you are interested in reading further. This blog post is sort of a followup to an URLO post: Futures 0.3, async♯await experience snapshot, and I’ll cross-post this one to URLO as well.

I don’t care much for laptop benchmark results for software not intended to run on laptops. So before I forget, all tests below were conducted on:

ec2 m5dn.large instances
2x CPU: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
ubuntu 18.04 amd64 hvm-ssd 20191113
kernel 4.15.0-1054-aws
rustc 1.42.0-nightly (da3629b05 2019-12-29)
tokio 0.2.8
hyper 0.3.1
blocking-permit@923bddda
body-image-futio@3bc3760e

I started the blocking-permit crate as a prototype while tokio’s much anticipated 0.2 didn’t have any comparable facility (and I wasn’t sure how to constructively contribute). I continued and released the crate for a few features I want, and which the latest tokio version doesn’t yet have. That gap could certainly be closed, but perhaps others will find these same features useful:

for a DispatchPool and BlockingPermit that are portable across tokio and async-std runtimes; or
simply to add additional pool(s), specific to particular operations, within one of these runtimes; or
for its current performance advantages.

I provide some comparative micro-benchmarks via cargo bench with this crate, below. In these benchmarks I compare different strategies for running blocking operations. These include:

direct: Operations are run directly on a reactor thread without any coordination.
permit: A BlockingPermit is requested via blocking_permit_future (this crate) and a fixed size Semaphore, to limit concurrency of running the operation on a reactor thread, in the tokio case via tokio::task::block_in_place, or otherwise by running directly.
dispatch_rx: Operations are dispatched to a DispatchPool via dispatch_rx (this crate) and results are awaited.
spawn_blocking: Operations are dispatched via tokio::task::spawn_blocking to a similarly dedicated set of tokio managed threads.

On a second axis is the type of the blocking workload under test:

noop: The operation does nothing. This attempts to measure the overhead of any coordination.
expensive: A CPU intensive operation. An array of 300 usize integers is randomly shuffled then sorted in place.
sleep: Sleeps for an exponentially distributed semi-random sleep time of 1 to 49 microseconds. This best simulates a blocking remote operation.

The total thread count is kept consistent across all strategies. The dispatching strategies (dispatch_rx and spawn_blocking) employ 4 core threads and 4 “extra” dispatch-dedicated threads. The direct and permit strategies employ 8 core threads, and 4 Semaphore permits if applicable. For the sleep operations, this is extended to 40 “extra” threads or permits. Note these tests were run on a virtual host with two virtual CPUs.

To simulate concurrency, batches of 100 operations (200 for sleep) are spawned and then awaited concurrently via futures::stream::FuturesUnordered, for each iteration of the benchmark. Some overhead could likely be attributed to the latter’s bookkeeping.

Results

cargo bench --features=tokio-omnibus

test noop_threaded_direct                ... bench:      52,220 ns/iter (+/- 10,434)
test noop_threaded_dispatch_rx           ... bench:     159,234 ns/iter (+/- 57,529)
test noop_threaded_permit                ... bench:      72,940 ns/iter (+/- 3,534)
test noop_threaded_spawn_blocking        ... bench:     147,804 ns/iter (+/- 8,325)
test r_expensive_threaded_direct         ... bench:   1,253,295 ns/iter (+/- 189,310)
test r_expensive_threaded_dispatch_rx    ... bench:   1,544,520 ns/iter (+/- 164,754)
test r_expensive_threaded_permit         ... bench:   1,272,118 ns/iter (+/- 89,466)
test r_expensive_threaded_spawn_blocking ... bench:   1,574,170 ns/iter (+/- 162,377)
test sleep_threaded_direct               ... bench:     649,913 ns/iter (+/- 8,448)
test sleep_threaded_dispatch_rx          ... bench:     992,531 ns/iter (+/- 61,243)
test sleep_threaded_permit               ... bench:     800,128 ns/iter (+/- 41,226)
test sleep_threaded_spawn_blocking       ... bench:   1,624,739 ns/iter (+/- 377,409)

When comparing benchmarks for our dispatch_rx vs. spawn_blocking, noop and expensive are fairly close, but sleep (at 40 threads) using tokio’s spawn_blocking is considerably slower. As the higher thread counts are likely representative of real world use cases with blocking APIs, I find this noteworthy.

The previous DispatchPool of blocking-permit release 0.1.0 used crossbeam’s MPMC channel, which was much slower then the above (released as 1.0.0) results and tokio’s spawn_blocking pool, particularly at higher thread counts, as with sleep.

Comparing permit to direct strategies gives some sense of Semaphore overhead. Comparing permit to either dispatching strategy is likely to be of limited value due to limitations of these benchmarks—most importantly, there is no actual I/O for which reactor threads need to be kept available.

What’s the motivation behind the permit strategy anyway?

Note that tokio::task::block_in_place not only informs tokio of an imminent blocking operation but actively starts the process to enlist a new reactor thread. Without some fixed number of Semaphore permits, the number of threads is effectively unbounded. While tokio 0.1 had a configurable cap on the total number of threads, this was found to be a liability as a dead- or live-lock potential, and was removed in tokio 0.2. If you want your process to have a known maximum (and small) number of threads like I do, then you need to either not use block_in_place or constrain the blocking concurrency by some other means. If the sum of such semaphore(s) permits is less than the configured total number of reactor (core) threads, then we are guaranteed to always have a reactor thread, with disused blocking threads will be kept alive and rotated in as subsequent reactors. This turns out to be reasonably (but not perfectly) efficient.

I’m surprised that something like the permit strategy wasn’t mentioned in Stop worrying about blocking: the new async-std runtime, inspired by Go (async-std blog, by Stejepan Glavina), or in async-std#631 as it would seem to be directly applicable as a strategy there as well. (Much more on “worrying about blocking” in the conclusion.)

Picking an async Semaphore implementation

The above permit benchmarks use tokio’s async Semaphore, which was originally available in 0.2 alphas, made private in the 0.2.0 release, and finally re-exported with interface changes in an 0.2.5 PATCH release. Our BlockingPermitFuture wraps the underlying permit future to provide a consistent, extended interface. In Tokio’s case, that future is an unnamed type returned from an an async fn, so I need to move it to the heap with Pin<Box<dyn Future<_>>.

While the fate of tokio’s Semaphore was uncertain I also integrated the futures-intrusive Semaphore and this is retained for compatibility with other runtimes. A different complication with this Semaphore is that its future type isn’t Sync, but in my usage with hyper, Sync becomes an unexpected requirement. The permit strategy benchmarks are re-run with this Semaphore-type, below:

cargo bench --features=futures-intrusive,tokio-threaded permit

test noop_threaded_permit                ... bench:      74,282 ns/iter (+/- 3,821)
test r_expensive_threaded_permit         ... bench:   1,304,930 ns/iter (+/- 80,482)
test sleep_threaded_permit               ... bench:   6,811,144 ns/iter (+/- 1,446,636)

Note that the sleep workload is significantly deteriorated with the futures-intrusive Semaphore. All permit using benchmarks are currently slower with this Semaphore than with tokio’s. Removing the Sync wrapper in these benchmarks does not measurably improve them.

body-image

Since the prior URLO post, the massive async♯await upgrade has been completed, including a reasonably pleasant experience of re-writing all its tests and benchmarks in generic, async♯await style (here with a total of 34 uses of await and test LoC changes: 1636 insertions, 622 deletions). I’ve released (besides blocking-permit) a new set of body-image-* 2.0.0 crates using the latest tokio, hyper, and futures.

Firstly, its worth summarizing that an original set of stream-to-sink forwarding benchmarks, available in the body-image-futio 1.3.0 release (using tokio 0.1), have all improved at least somewhat in the 2.0.0 release (using tokio 0.2). But I want to focus on a new set of benchmarks, added in 2.0.0, which better simulate my use case for the project.

A new set of client benchmarks compares the performance of making HTTP requests to a server which returns 8MiB chunked responses and caching the body in the various transparent ways that the body-image crate allows: scattered allocations in ram, or written to the file system as received, and optionally memory mapped. The client then summarizes a stream of the response body (either in memory of read back from the filesystem) by finding the highest byte value (always 0xFF, as proof of reading) and verifying the total length.

Like in the prior blocking-permit benchmarks, concurrency is simulating by batching, in this case 16 request-response-summarize operations per benchmark iteration. The total thread count is kept to 4 threads (2 core, 2 “extra” dispatch-dedicated threads) and 2 (tokio) Semaphore permits. The tests were run on the same ec2 host configuration described above.

In results below, a separate server is run via cargo run --release --example server on a separate virtual host of identical type, provided to the benchmarks via a BENCH_SERVER_URL environment variable. If the benchmark is run without this variable, a server is run in process but still communicates with the clients via the localhost TCP-stack.

Terms used in the benchmark names:

ram: Response Bytes chunks are left scattered and pushed to a Vec<Bytes> without copying. Once the body is complete it is summarized by reading each chunk in turn, without copying.
ram_gather: As per ram above, but before processing the body, it is first gathered into a single contiguous Bytes buffer in one allocation. Note this resembles many hyper body handling examples and may be an unfortunate requirement of parsing APIs, etc.
fs: All response chunks are written directly to the filesystem, un-buffered, as they are received. When later readying the body (once it is completely written), 64KiB (uninitialized) buffers are used.
mmap: As per fs above, but before processing, the entire body is first memory mapped (into virtual memory). On unix, madvise(2) is then called on the memory region with MADV_SEQUENTIAL to suggest aggressive read-ahead, before it is read as a single contiguous UniBodyBuf item, with no additional copying.
mmap_*_copy: As per above mmap but copies the memory mapped region to a new Bytes item before processing it. The extra copy isn’t recommended but could possibly justify the mmap if there are multiple read passes and one requires Bytes.
direct: Blocking filesystem write/read operations are run directly on a reactor thread without any coordination.
permit: A BlockingPermit is obtained via blocking_permit_future before running the write and read operations on a reactor thread, in this tokio case via tokio::task::block_in_place.
dispatch: Write/read operations are dispatched to a DispatchPool via dispatch_rx.

Results

% BENCH_SERVER_URL=http://10.0.0.115:9087 cargo bench client

test client_01_ram              ... bench:  58,528,073 ns/iter (+/- 3,776,825)
test client_01_ram_gather       ... bench:  81,692,816 ns/iter (+/- 3,984,386)
test client_10_fs_direct        ... bench: 119,846,890 ns/iter (+/- 1,999,313)
test client_10_fs_permit        ... bench: 141,334,426 ns/iter (+/- 3,529,708)
test client_12_fs_dispatch      ... bench: 151,809,679 ns/iter (+/- 6,100,962)
test client_15_mmap_direct_copy ... bench: 137,397,757 ns/iter (+/- 2,643,698)
test client_15_mmap_permit_copy ... bench: 157,148,683 ns/iter (+/- 6,222,127)
test client_16_mmap_direct      ... bench: 110,491,801 ns/iter (+/- 2,796,833)
test client_17_mmap_permit      ... bench: 124,647,965 ns/iter (+/- 4,390,988)

The aggregate mean transfer and processing rate, in terms of the original body size, for the fastest, ram case is 2.14 GiB/sec or 17.1 Gib/sec. Amazon AWS suggests that one can get “up to 25 Gbps” of (raw TCP) bandwidth for these instances, independent of EBS traffic, and made possible because client and server instances are in the same VPC and availability zone. Since the client instance’s 2 CPUs are approximately saturated in these tests (mostly system time actually), I would expect results to be slower if TLS was also involved (its not).

An interesting if entirely anecdotal comparison, is the maximum ~70 MB/s reported (reading off graph y axis) in that same async-std blog article running on much larger “m5a.16xlarge” and “m5a.8xlarge” instances? Perhaps that benchmark wasn’t so optimized with large body payloads or has other constraints?

The fs (and mmap) results are based an ext4 filesystem on EBS “gp2” SSD network attached storage. The “m5dn.large” ec2 instance type also has access to directly attached NVMe SSD (instance store). However, mounting and using that storage for the body-image temp. files was not found to significantly change the results. At the observed rate, gp2 storage for these sequential writes and sequential (and mapped) reads, does not appear to be a significant bottleneck, but see below for additional optimizations.

Using tmpfs

Yes, tmpfs can behave as a RAM disk, so let me explain the use case lest I’m accused of cheating just for the benchmarks. As async♯await can be described as cooperative multi-tasking, this use of tmpfs can be described as cooperative swapping. Linux tmpfs will start using available swap space if the instance experiences memory pressure. By writing body chunks to tmpfs, I’m telling the kernel that these are currently low-priority pages that it can remove them from RAM if needed. Once the body is downloaded, and processing begins by reading or memory mapping, I’m informing the kernel that these pages are again high-priority. This cooperative swapping will behave much better than the uncooperative swapping that could occur if I just keep all bodies in RAM and cause memory pressure that way. Then any operation in the entire process, reactor or other threads, becomes a potential blocking operation as it may need to wait for executable or heap pages to swap back in! The kernel doesn’t really have much to go on for page prioritization purposes.

Another nice feature of tmpfs on Linux is that it supports Transparent Huge Pages (default 2MiB) without any other required configuration. Huge pages drastically reduce kernel virtual memory bookkeeping overhead for memory mapping, at the cost of over-allocating up to 1 page of memory per mapped file. There may be workarounds for space efficiency as well, but I’m currently not worried much about it.

Below shows the setup for tmpfs with huge pages and then the updated benchmark results:

% sudo mount -t tmpfs -o size=500M,huge=always,uid=1001,gid=1001 tmpfs \
    ../target/testmp

% BENCH_SERVER_URL=http://10.0.0.115:9087 cargo bench client

test client_01_ram              ... bench:  58,392,758 ns/iter (+/- 16,045,924)
test client_01_ram_gather       ... bench:  85,527,831 ns/iter (+/- 18,187,177)
test client_10_fs_direct        ... bench:  94,056,654 ns/iter (+/- 2,915,506)
test client_10_fs_permit        ... bench: 120,463,257 ns/iter (+/- 2,742,582)
test client_12_fs_dispatch      ... bench: 130,157,569 ns/iter (+/- 5,993,078)
test client_15_mmap_direct_copy ... bench: 104,351,507 ns/iter (+/- 2,596,867)
test client_15_mmap_permit_copy ... bench: 129,762,411 ns/iter (+/- 6,165,487)
test client_16_mmap_direct      ... bench:  79,705,427 ns/iter (+/- 3,013,775)
test client_17_mmap_permit      ... bench:  99,289,803 ns/iter (+/- 4,002,980)

As expected the fs and mmap results are all improved. Here mmap_direct actually outperforms ram_gather, and is only a 36% latency overhead beyond ram (without gather). Now, what if I only need to use fs (and mmap) on 10% or 1% of responses exceed some configured size in the Tunables, the remainder staying in ram? That net overhead should be more like 3.6% or 0.36%. With tmpfs, this seems like an effective way to have my cake and eat it too—a very limited resource cost for piece of mind that my process isn’t going to exceed RAM or thrash if it requires any swap space at all, and no need to enforce some low maximum http body size or low concurrency.

Conclusions

Dear Reader,

Having made it this far, did you notice that in all cases tested, the direct strategy (blocking a reactor thread) performs best?

Its true. I have almost nothing positive performance-wise to show for a significant amount of effort spent implementing the permit and dispatch strategies employed in these tests. One minor win is that from preliminary benchmarking I made the decision (in the 3rd rewrite, sigh) to compose permit and dispatch strategies as wrappers over direct AsyncBodyImage (our Stream) and AsyncBodySink types, so the latter may be directly selected at compile time, without further overhead or complication.

One other possible advantage of permit, as hinted in these benchmark results, is that the variance (if not the mean) of request/response handling is reduced with permit. That might be important in some use cases, for example for fairness to multiple users.

How could direct be best performing? What have I done wrong, if the demigods of rust async (well, some of them) tell us that blocking (their? our?) reactor threads is a bad idea? I speculate that this is both true and that I’ll be using the direct strategy in production, for all of the following reasons:

Memory cache locality and coherence is rather important in I/O heavy applications, so moving a read or write operation with a pointer to a buffer in some CPU’s cache or NUMA zone to another thread that might get scheduled anywhere is inherently costly. And this is typically exacerbated on virtualized instances. I believe this is one reason why permit tends to outperform the dispatch strategy in the above client benchmarks.
8KiB or 512KiB filesystem sequential writes to SSDs (network attached or local) simply aren’t blocking enough to matter, at least as compared with the overhead of any other strategy.
Tokio’s threaded runtime, with sufficient number of reactor threads and its work stealing abilities, is quiet capable of handling an occasionally blocked reactor thread without replacement or slowdown, at least for this tested work load.

I think there may still be plenty of other use cases for either the BlockingPermit or the DispatchPool, but they will tend to be much more course grain. For example, I envision dispatching a one-step blocking operation to uncompress HTTP body payloads enmasse, in the background of a particular server, once bodies are completely downloaded.

These findings similarly make me suspect that efforts to provide a direct drop-in std replacement in the form of AsyncRead and AsyncWrite and associated types and functions, is probably attacking the problem at too low a level. At the very least, I don’t suspect those traits as currently conceived will be useful for body-image-futio.

Some related questions:

Does hyper-tls or native-tls or tokio-tls offload packet decryption from the reactor thread? If any of these do, I haven’t been able to find where it happens. Please point it out for me.
Does hyper’s HTTP 1 chunked transfer encoding implementation attempt to offload Base64 encode/decode from the reactor thread? Should it?
Does the newer async-compression worry about blocking a reactor thread? Should it?

Or is the general strategy with these just that only a small block, say 8KiB or 64KiB, is decrypted/decoded/decompressed for each Stream::poll_next()? That would sound a lot like the above best performing direct strategy with blocking filesystem I/O.

Why use the new body-image-* 2.0.0 crates?

If you can do all of your dynamic response body production, POST request body consumption, or client request/response body handling in a purely streaming, chunk-at-a-time fashion, then that’s going to be the best thing to do. Fire and forget. The pattern is generally limited to HTTP proxies and load balancers.

If however, you need to do things like:

parse body payloads, producing large parse graphs in RAM, or
mutate the start of a body based on content in the middle or end of the body, or
produce dynamic content bodies much faster then your clients, or server POST endpoints, can consume them over the network

…then you might find the above tested setups and tunable options with body-image actually gain you efficiencies and/or make you sleep better at night not worrying over memory consumption, unbounded thread growth, or uncontrolled swapping. Let me know what you find!

Cargo build.rs recipes

2019-04-25T00:00:00-07:00

Rust is very open, which is great in comparison to some other language communities. I’m reminded in particular of one run by Sun and later Oracle. Ever tried filing an issue on Java’s public bug tracker? But its hard to look at the Rust RFC process, and in particular at the RFCs one feels they need accepted, without finding some delay or failure evident from the sheer human scale and attempting to achieve such broad consensus.

Take for example, RFC 2495: Minimum Supported Rust Version. Unlike many other RFCs, this one is particularly simple: Add a field to Cargo.toml to express the minimum version of the rust compiler that a project needs to successfully build. Stepping back, its really hard to even imagine, given all the great progress and features of cargo, that it doesn’t yet have this basic feature as found in other languages with cargo like tools. Why wouldn’t we want to at least collect that data point when a crate is released? Why wouldn’t we use this data point for warning messages or refined dependency resolution for users that, for one reason or another, are stuck on older rust releases? RFC 2495 remains open after 10 months, apparently stalled with a notable lack of core team engagement.

RFC 2523: #[cfg(accessible(..) / version(..))] is closely related and could eventually offer a superset of features. It has had the engagement, but suffers from being overly broad in scope. The accessible(..) tests have unresolved implementation questions and version(..) tests are (to my surprise) controversial.

Enforcing MSRV with build.rs

Without these RFCs, Cargo’s build.rs feature allows us to take matters into our own hands. Here is a self contained recipe for enforcing a project’s MSRV. You’ll need to change just the PACKAGE and msrv lines for your own projects:

#![cfg_attr(feature = "cargo-clippy", allow(clippy::all))]

use std::env;
use std::process::Command;

fn main() {
    static PACKAGE: &'static str = "tao-log";
    let msrv = vec![1, 31];

    static VERSION: &'static str = env!("CARGO_PKG_VERSION");
    static M_V: &'static str = "minimum supported rust version (MSRV)";

    let rustv = rustc_version();

    if rustv < msrv {
        panic!(
            "{} v{} {} is {} > {} (this rustc)",
            PACKAGE, VERSION, M_V, join(&msrv), join(&rustv));
    }
}

fn join(ver: &Vec<u16>) -> String {
    let mut out = String::new();
    for v in ver {
        if !out.is_empty() { out.push('.'); }
        out.push_str(&v.to_string());
    }
    out
}

// Parse `rustc --version` and return as vector of integers, or panic.
fn rustc_version() -> Vec<u16> {
    let rustc = env::var("RUSTC").unwrap_or("rustc".to_owned());
    let out = Command::new(rustc).arg("--version").output().unwrap();
    let out = String::from_utf8(out.stdout).unwrap();
    for l in out.lines() {
        if l.starts_with("rustc ") {
            let mut v = &l[6..];
            if let Some(e) = v.find(" ") {
                v = &v[..e];
            }
            let mut vp = v.split("-");
            if let Some(v) = vp.next() {
                let vs: Vec<u16> = v.split(".")
                    .filter_map(|vss| vss.parse().ok())
                    .collect();
                if !vs.is_empty() {
                    return vs;
                }
            }
        }
    }
    panic!("rustc version not found")
}

The above and subsequent code snippets are from tao-log, released under dual MIT and Apache licenses.

Update 2019-6-4: Applied a fix to above function rustc_version.

Update 2021-1-19: Contextually silence clippy lints (line 1), assuming newish clippy while avoiding breakage on old rustc.

Without the much more pleasant, declarative, RFC proposal, we are forced to carefully write (and test) forward/backward compatible rust code that will work with versions as old as might be attempted. This build script will work with rust as old as 1.0.0. Note the use of pre-2018-edition syntax, like the generous helping of 'static lifetimes. We must include the project name directly because env!("CARGO_PKG_NAME") isn’t available until cargo 0.10.0 and rust 1.9.0.

The version_check crate, or the more elaborate rustc_version or autocfg crates, could be used to minimize the per-project build.rs, at the expense of 1 or more additional build time dependencies. I had started with version_check but replaced it with the above, adding only 22 code lines, and with simpler error handling for this use case.

Don’t forget to add a build field to the [package] section of your Cargo.toml so that older cargo releases will always run this build script. Prior to rust/cargo 1.17.0, specifying this field was required:

[package]
build         = "build.rs"

With this in place, for tao-log master branch, if you attempt to compile with say 1.16.0, this is the output:

  Compiling tao-log v0.2.0 (file:///home/david/src/tao-log)
error: failed to run custom build command for
`tao-log v0.2.0 (file:///home/david/src/tao-log)`
process didn't exit successfully: `/home/david/src/tao-log/target/debug/build/
tao-log-318f175d5ac13527/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'tao-log v0.2.0 minimum supported rust version (MSRV)
is 1.31.0 > 1.16.0 (this rustc)', build.rs:20
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Without this, you would get the following, as the first of many fatal errors using 1.16.0:

warning: unused manifest key: package.edition
   Compiling tao-log v0.2.0 (file:///home/david/src/tao-log)
error[E0432]: unresolved import
   --> src/lib.rs:171:9
    |
171 | pub use ::log;
    |         ^^^^^

error: aborting due to previous error

error: Could not compile `tao-log`.

In considering the value of this, please put yourself in the shoes of prospective new rust users, and new users of your project. Would seeing the latter error breed confidence in either? I think I would have assumed I needed to add another dependency. But that really won’t help here.

For 1.27.0 through 1.30.0 the build will fail before the build script, as the included cargo is aware of the edition field, but not ready for the 2018 edition. These rust versions never reach the build script stage. With 1.30.0:

error: Edition 2018 is unstable and only available for nightly builds of rustc.

…which is confusing, from our current vantage point, because the 2018 edition was released in stable 1.31.0. Even a cargo/rustc as new as 1.30.0 is not going to offer the more useful and current truth. With 1.27.0 it is slightly worse, with a suggestion that is distracting at best:

Caused by:
  editions are unstable

Caused by:
  feature `edition` is required

this Cargo does not support nightly features, but if you switch to nightly
channel you can add `cargo-features = ["edition"]` to enable this feature

These last issues are unique to the 2018 Edition, however, and only for 1.27.0 through 1.30.0.

Another potential problem is that your dependencies can fail prior to the build script running. Worse, since builds are run in parallel, they may fail inconsistently based on timing (though they do always fail.) For tao-log, the log dep requires 1.16.0, so below that version log will fail in its own perplexing way, for example with 1.15.0:

error: borrowed value does not live long enough
   --> /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/log-0.4.6/src/lib.rs:830:36
    |
830 |                 args: format_args!(""),
    |                                    ^^ does not live long enough
...
837 |     }
    |     - temporary value only lives until here

If dependencies are updated to enforce MSRV themselves, then this could also be avoided.

The following table summaries the compile outcomes for tao-log using various rust versions, with or without the above build.rs. The first row case depends on the MSRV of dependencies.

rust version	with build.rs	without
—1.15.0 (MSRV of deps)	perplexing (deps) or clear MSRV error	perplexing (ours or deps)
1.16.0—1.26.0	clear MSRV error	perplexing
1.27.0—1.29.0	editions unstable	editions unstable
1.30.0	edition 2018 unstable	edition 2018 unstable
1.31.0	success	success
1.32.0—	success (optimized)	success

As rust versions progress and we decide to raise our MSRV or otherwise use new version features when available (see below), this recipe will continue to work correctly, and the various problem cases should become less frequent. Even for some future new rust edition, we might optimistically hope for better considered error messages.

Configuring for versions beyond MSRV

With the above build.rs in place, it becomes trivial to add additional cfg flags based on versions beyond MSRV, on the end of the main function:

    if rustv >= vec![1, 32] {
        println!("cargo:rustc-cfg=tao_log_trailing_comma");
    }

…then in the code:

#[doc(hidden)]
#[macro_export]
#[cfg(tao_log_trailing_comma)]
macro_rules! __tao_logv {
    ($lvl:expr, $exp:expr $(,)?) => (...)
}

The $(,)? syntax is for ignoring a trailing comma in macro rule patterns. It is first available in rust 1.32.0. In our MSRV 1.31.0, we settle for $(,)* which is more lenient then we really want.

Note the “tao_log_” prefix as namespace. There are other cargo/rustc defined flags, and its unclear if these cfg flags are applied globally. Regardless, the prefix makes it clear that this is defined in tao-log itself.

SyncWrap for Engineers

2014-03-20T00:00:00-07:00

Not too long ago, it was common for an internet company’s dedicated operations staff to manually provision non-virtualized hosts, by following formalized instructions maintained in runbooks. Things changed at a glacial pace by today’s standards, and developers frequently encountered resistance to deploying anything with new provisioning requirements. These days, startups and other kinds of small companies are flocking to virtualized environments, and Amazon EC2 in particular. All too commonly, non-technical management actually believes (or prefers to pretend) that there is no longer any need for operations effort and budget in these environments.

A dirty little secret is that the majority of these companies are not currently using any provisioning automation and many are not even using any application deployment automation.² That’s not really surprising given no time or budget for learning projects, classes, paid support, a dedicated “expert” consultant, or hire.

Without automation, hosts are manually maintained. Some manage to adopt a modicum of discipline in maintaining virtual machine images. While images are better than nothing, as an approach this is essentially cut-and-paste reuse, with very similar limitations.³ Ultimately:

Images end up collecting and propagating cruft, abandoned modifications and additions
Hosts are not reproducible, say for staging or testing, at least not without down time for imaging
Hosts can not be reproduced if unexpectedly lost
Fear, loathing, and practical inability to perform general maintenance and updates (i.e. closing security holes, updating to the latest base-OS, experimental disk storage optimization)
Silos of host ownership and access (often driven by the same fears)
Failed and abandoned attempts to document the manual provisioning/deployment process (runbook style); where the document is never quite accurate and up-to-date.

Provisioning and Deployment Automation

Anyone who has experienced well maintained and fully automated provisioning and deployment knows what its worth in avoiding the above described problems and as a general enabler to rapid system and application improvements.

The provisioning system needs to take us from the current Linux distribution base image to hosts ready for deployment as any of the numerous different roles we need: database servers, web application servers, etc. As provisioning requirements change (i.e new developers need access, ruby needs an upgrade) the provisioning system needs to be able to (1) reliably update existings hosts; and/or (2) enable us to quickly build replacement hosts, test them, rotate them in, and eventually shutdown the old. We need to be able to trust running it repeatable on our most critical production hosts.

The deployment system needs to support continuous release while not adversely impacting current users and stability of the applications. It may need to orchestrate incremental or rolling updates to achieve this.

Its questionable if provisioning vs. deployment is more historic artifact than useful distinction. At the very least, its often quite useful to provision and deploy in a single step within a unified system.

The Case for SyncWrap

Current commercially-backed provisioning tools Chef and Puppet are too complex and abstract to enable the majority of small projects/companies to even start automating provisioning, and are frequently found cumbersome or inadequate for deployment.

SyncWrap aims to provide a unified provisioning and deployment tool in plain Ruby, with minimal learning curve for Ruby programmers. See Provision Programming for some examples of style. The following conditions might also help to distinguish SyncWrap:

If you already have Chef or Puppet working, and are happy with it for both provisioning and deployment, then kudos for achieving full automation. You have most certainly made a significant investment and will not likely want to abandon those efforts.
If you are an engineer, have Ruby and a little Bash scripting ability, and want to automate quickly, with minimal learning curve.
If you expect the first provisioning task to go nearly as fast as the manual alternative. Otherwise you won’t be able to justify automation with yourself or management.
If you can’t afford to learn and maintain two different tools for each of provisioning and deployment, and can’t accept the lack of speed or fine grain control with the alternatives for deployment.
If for that one hour you had to try and automate provisioning, you bounced off the abstraction envelope of Chef or Puppet.
If you previously used Vlad or Capistrano, like the direct approach, but want a more powerful and provisioning capable tool.
If provisioning a correct system Ruby is important. The alternatives that run Ruby on the host being provisioned can have a bit of a chicken-and-egg problem. ⁴ ⁵
If you’d prefer programming in Python instead of Ruby, you might try Fabric or Cuisine which have a similar design philosophy to SyncWrap.

Commercial Tools Target Adminstrators

While I don’t wish to single out Puppet, they did recently provide some material which offers some useful points for comparison:

From Puppet is Executable Documentation (January 7, 2014 Puppet Labs Blog):

Don’t Re-Invent bash in Puppet: If you have a series of exec resources chained together, you’re Doing It Wrong. While you may be used to thinking of things as iterative steps, that’s thinking like a programmer instead of like a documenter.

The author has apparently observed a programmer committing certain crimes of programming with Puppet. Perhaps from the programmer’s perspective, a series of Bash commands was more straightforward than using Puppet’s DSL and declarative features? Later in the same post:

Don’t be afraid to write a little Ruby to create a custom function for that logic. Even if you don’t know Ruby, you’ll find that — 99 percent of the time — googling how to do what you want doesn’t require any previous knowledge of the language. Also, check to make sure the function doesn’t already exist in the stdlib module. There are currently 84 functions available in stdlib alone. For example, the pick function in stdlib allows you take a series of variables and select the first one that has a valid value. The strip function in stdlib will take an array of strings and remove any leading and trailing spaces from each string.

Puppet, or at the very least this post, is not targeting a Ruby developer who would naturally use Ruby core map(&:strip) without blinking or looking for the matching Puppet “stdlib” function. There is an implicit tension to the idea of programming here.

Ultimately, Puppet’s modules and Chef’s cookbooks designs show their roots in the runbook style. In an environment where there is a whole other team doing the programming, it may have been a practical or tactical necessity to abstract out and declare as much as possible. Both Puppet and Chef seem more geared for a classic system administrator role, than for engineers who prefer to directly program.

Capistrano

From Capistrano Version 3 Release Announcement:

Applicability. We’ve always maintained that Capistrano is a terrible tool for system provisioning, and that more often than not servers are better being setup with Chef, Puppet or similar, whilst we still agree with that, the new features in Capistrano really lend themselves to integrating with these kinds of tools.

Despite the author’s position, we’ve seen numerous attempts to use Capistrano for provisioning, many of which have been generally successful at providing some reasonable degree of automation. However, Capistrano also seems to suffer from over-abstraction, framework, and barriers to using basic Ruby language features like classes, methods, inheritance and overrides. V3 attempts to hide all command shell invocation behind another new DSL, embedded in the Rake DSL.

In a blog post from the same month, Using Capistrano v3 With Chef the same author demonstrates using Capistrano to deploy (or is it provision?) Chef Solo (the free version of Chef) which in turn provisions the host, before Capistrano deploys to it.

We’ve been working really hard to push Capistrano v3, after years of languishing behind the cutting edge, and in an environment where Chef and Puppet (which should really restrict themselves to working on infrastructure) encroaching on deploying applications, as well; I’ve been really happy to have had the chance to code a really nice way bridge the gaps, and make Capistrano work better with Chef for deployments where repeatable infrastructure is important, but fast deployments of application code are essential…

SyncWrap takes a different approach and enables clean, direct programming of the low-level details in Bash, which effectively is the Domain Specific Language for low-level remote interaction.

From Wikipedia Jakarta_slumlife66.JPG; Jonathan McIntosh, 2004, under the Creative Commons attribution 2.0 generic license ↩
From my own recent experience and an informal survey of colleagues. ↩
This is not to say that there isn’t still an important role and utility for images; more later. ↩
“Most Vagrant baseboxes install Ruby for use by Chef and Puppet, but the latest ones do this into /opt/ruby rather than using the system packages.” Stackoverflow: Installing Ruby 1.9.2 on Ubuntu using Chef and Vagrant ↩
“Changing the system ruby when you depend on Chef is always a bootstrappy web of emotion and anti-gravity, and moving to Ruby 2 was even more scary given how many dependencies are assumed on Chef’s end.” StatusPage.io: Moving to ruby-2.0.0-p0 on Mac+RVM and Ubuntu 12.04 ↩

Provision Programming

2014-03-19T00:00:00-07:00

Enabling direct programming of your provisioning and deployment automation, rather than obscuring it, is a principal goal of SyncWrap.

Getting Started

Lets consider, as an example, provisioning users (aka roles) for use with a PostgreSQL database. The database engine may be installed via the existing SyncWrap::PostgreSQL component. If doing this manually we might start by adding the first database user we need, simply by running (PostgreSQL CLI) createuser bob on each host. Lets start by simply doing the same with SyncWrap. We create a new MyDatabase component (here directly in the sync.rb, but it could also be require’d from lib/my_database.rb, see LAYOUT) and add it to the appropriate hosts (or roles):

class MyDatabase < SyncWrap::Component
  def install
    sh( <<-SH, user: 'postgres' )
      createuser bob
    SH
  end
end

host 'localhost', MyDatabase.new

The <<-SH here-document is arguably overkill for this simple Bash command, but the syntax will prove helpful as the Bash fragments become more complex. The above assumes we are testing this on a localhost (where PostgreSQL already installed), but you might also want to try plugging this into something like examples/ec2.rb from the SyncWrap git repo and test with a temporary EC2 host.

Initial output using syncwrap -v (–verbose) flag:

syncwrap -f db_setup.rb -v
== localhost #<Module:0x0000000321f6a8>::MyDatabase#install: enqueue
<-- sh localhost (-v coalesce user:postgres live)
createuser bob
--> Exit 0 (success)

Repeatability

Now if we run the above a second time, we’ll get an error:

syncwrap -f db_setup.rb -v
== localhost #<Module:0x000000028a7620>::MyDatabase#install: enqueue
<-- sh localhost (-v coalesce user:postgres live)
createuser bob
createuser: creation of new role failed: ERROR:  role "bob" already exists
== localhost error [1]:
CommandFailure: sudo exit code: 1
/home/david/src/syncwrap/lib/syncwrap/context.rb:261:in `capture_stream'
/home/david/src/syncwrap/lib/syncwrap/context.rb:189:in `run_shell'
...

In general, when automating provisioning and deployment, we need to make the process idempotent, meaning that we can run it multiple times and get the same desired result. Here’s the simplest fix:

class MyDatabase < SyncWrap::Component
  def install
    sh( <<-SH, user: 'postgres' )
      createuser bob || true
    SH
  end
end

This continues to show the error on verbose output but returns success and allows the automation to continue. In general this is preferable to no automation. If this is checked into version control, basic repeatable automation and self documentation has been achieved. A remaining problem is that createuser could fail for more reasons than just that we already created the user, but our script fragment ignores all errors. A more robust approach would be to first check if the user already exists:¹

class MyDatabase < SyncWrap::Component
  def install
    sql_test = "SELECT count(*) FROM pg_user WHERE usename = 'bob'"
    sh( <<-SH, user: 'postgres' )
      if [[ $(psql -tA -c "#{sql_test}") == "0" ]]; then
        createuser bob
      fi
    SH
  end
end

De-/Composition in Ruby

If at some point in the future we need to add a second PostgreSQL user, then the script fragment becomes a good candidate for a Ruby method:

class MyDatabase < SyncWrap::Component
  def install
    %w[ bob joanne ].each { |u| pg_create_user( u ) }
  end

  # Create the PostgreSQL user if not already present
  def pg_create_user( user, flags = [] )
    sql_test = "SELECT count(*) FROM pg_user WHERE usename = '#{user}'"
    sh( <<-SH, user: 'postgres' )
      if [[ $(psql -tA -c "#{sql_test}") == "0" ]]; then
        createuser #{flags.join ' '} #{user}
      fi
    SH
  end
end

The SyncWrap command queue will effectively combine any number of pg_create_user calls into a script to be executed in a single shell session, so there is no significant overhead vs. implementing the loop in Bash. Indeed we can see this by running it with verbose output enabled:

syncwrap -f db_setup.rb -v
== localhost #<Module:0x0000000300aed0>::MyDatabase#install: enqueue
<-- sh localhost (-v coalesce user:postgres live)
if [[ $(psql -tA -c "SELECT count(*) FROM pg_user WHERE usename = 'bob'") == "0" ]]; then
  createuser  bob
fi
psql -tA -c "SELECT count(*) FROM pg_user WHERE usename = 'bob'")
psql -tA -c "SELECT count(*) FROM pg_user WHERE usename = 'bob'"
if [[ $(psql -tA -c "SELECT count(*) FROM pg_user WHERE usename = 'joanne'") == "0" ]]; then
  createuser  joanne
fi
psql -tA -c "SELECT count(*) FROM pg_user WHERE usename = 'joanne'")
psql -tA -c "SELECT count(*) FROM pg_user WHERE usename = 'joanne'"
--> Exit 0 (success)

Note that the script echoing is via Bash’s own set -v. Since the psql test command is run in a subshell it is echoed more than once. If we drop users “bob” and “joanne” and run it again with the -x flag, which uses Bash’s set -x:

syncwrap -f db_setup.rb -vx
== localhost #<Module:0x0000000345ac60>::MyDatabase#install: enqueue
<-- sh localhost (-x coalesce user:postgres live)
++ psql -tA -c 'SELECT count(*) FROM pg_user WHERE usename = '\''bob'\'''
+ [[ 0 == \0 ]]
+ createuser bob
++ psql -tA -c 'SELECT count(*) FROM pg_user WHERE usename = '\''joanne'\'''
+ [[ 0 == \0 ]]
+ createuser joanne
--> Exit 0 (success)

This shows that createuser is indeed run based on the matching condition. If run again, we see that the conditional is working as intended and createuser is not run if the users already exist:

syncwrap -f db_setup.rb -vx
== localhost #<Module:0x000000028a2c60>::MyDatabase#install: enqueue
<-- sh localhost (-x coalesce user:postgres live)
++ psql -tA -c 'SELECT count(*) FROM pg_user WHERE usename = '\''bob'\'''
+ [[ 1 == \0 ]]
++ psql -tA -c 'SELECT count(*) FROM pg_user WHERE usename = '\''joanne'\'''
+ [[ 1 == \0 ]]
--> Exit 0 (success)

As a final example of composition, the fragment below taken from SyncWrap::PostgreSQL demonstrates the block form of sh (here the sudo variant) where Ruby function dist_service enqueues commands from within an outer Bash conditional.

sudo( "if [ ! -d '#{pg_data_dir}/base' ]; then", close: "fi" ) do
  unless pg_data_dir == pg_default_data_dir
    sudo <<-SH
      mkdir -p #{pg_data_dir}
      chown postgres:postgres #{pg_data_dir}
      chmod 700 #{pg_data_dir}
    SH
  end
  dist_service( 'postgresql', 'initdb' )
end

Conclusions

The Bash used above can be thought of as low-level anonymous functions², where all higher level logic is implemented in Ruby. The remote pre-provisioning requirements are limited to sudo access and the bash shell, which comes by default on target Linux distributions. Since provisioning tasks are typically composed from system commands, Bash is a reasonable choice at this level. To access remote state, we need to be able to write bash conditionals and some simple pipelines. The (Wooledge) BashGuide and Bash Hackers Wiki may be useful resources. As shown, syncwrap provides convenient diagnostic options in order to debug your bash fragments.

Should every PostgreSQL user need to implement pg_create_user like methods?. The SyncWrap::PostgreSQL component can and should (#1) be extended to provide these kinds of general utility methods, freeing MyDatabase of needing to implement the same. Given dynamic component binding, the pg_create_user method can simply be moved to the SyncWrap::PostgreSQL component class, with the install method remaining unchanged.

The example demonstrates the low barrier from manual provisioning in an interactive shell, to basic automation in SyncWrap using similar bash fragments, unobscured. For flexibility and reuse, functional decomposition and re-composition can be done in Ruby with real Component objects–no need to deal with Bash functions. Add the ability to push templated configuration files, coordinated in Ruby, and we have a complete and direct system for provisioning and deployment.

More intelligent provisioning might also check if the user privileges have changed since initially created. PostgreSQL’s ALTER ROLE could be used to later promote bob to a super-user, for example. ↩
But be careful with leaking Bash variables, as there is no real closure. ↩

Versioning

2012-07-22T00:00:00-07:00

A colleague recently pointed out that version 1.2.b.1 was invalid Semantic Versioning as referenced in the Iūdex developers guide. I’ll show that SemVer has been a moving target, and the tools involved don’t entirely support it. But at minimum, the intent and meaning of 1.2.b.1 may not be as clear as I had hoped. I’ll document the pitfalls found along the way, and suggest a clearer strategy moving forward.

tl/dr: Given the complexities, incompatibilities and pitfalls involved, don’t bother with pre-release version notations. See conclusion.

Ελληνικά

The b in 1.2.b.1 was for beta. The Greek letter names alpha and beta have long been used to designate pre-release software. I would define this further and pragmatically as software made available with either known problems or without sufficient testing for confidence that it will meet the same quality levels as current or future release versions.¹ I had wanted a short form comparable with the decimal versions, but perhaps 1.2.α.0 and 1.2.β.0 would have been more clear? I joke, but it offers a fitting segue to the underlying issue here: tool compatibility and interoperability. I simply don’t have the fortitude to use ~> 1.2.β in a Rubygems dependency or the equivalent Maven version range [1.2.β,1.3). Fortunately in all specifications and tools discussed below beta and b; alpha and a are interchangable.

Semantic Versioning

The version 1.2.b.1 is invalid, with different possible corrections, with three different versions of SemVer. Since commit 05a00df0 last year, a pre-release modifier/component is required to be delimited with a - hyphen instead of the . dot used in prior versions. The following table shows corrected examples based on each:

SemVer Version	Example Corrections
before 05a00df0	1.2.0.b1 1.2.0.b-1
1.0.0	1.2.0-b1 1.2.0-b-1
2.0.0-rc.1	1.2.0-b.1

Note that only SemVer 2.0.0-rc.1 offers numeric sort order for the 1 following the b. All other versions have the potential pitfall of lexicographic sorting, e.g. b10 < b9 and b-10 < b-9.

But returning to the practical: No release version of SemVer is compatible with Rubygems.

Rubygems

The 1.2.b.1 version is compliant with the rubygems version notation which currently supports the following rule set:

Strict dot delimited notation with no additional modifiers.
Dot delimited versions tokens sort in numeric order if numeric.
Tokens containing non-numeric characters sort in lexicographic order² and always before numeric values.
For the purpose of ordering, it is as if each version is postfixed by an infinite series of 0 tokens. Thus 1.2.b is less than 1.2 because it is less than 1.2.0.0…
Any non-numeric characters indicate the version is a pre-release, and thus won’t be installed by default unless gem --prerelease is specified.

My intent with 1.2.b.1 was to release a 1.2 beta series followed by a 1.2.0 release. The third position would sort as the progression a b 0 1 … The problem with this is that the rubygems authors had a different progression in mind, given the behavior of “pessimistic” or “spermy” version specifiers with pre-releases. The table shows the intended progression and what some specifiers match:

Version	~> 1.1.0	~> 1.2.b	~> 1.2.0
1.1.0	•
1.1.1	•
1.2.a.0
1.2.b.0		•
1.2.b.1		•
1.2.0		•	•
1.2.1		•	•
1.3.b.0		¤
1.3.0		¤
1.4.0		¤

In my own usage, I had expected ~> 1.2.b to accept the beta series and subsequent 1.2.x releases but not 1.3.b.0, 1.3.0 and beyond (¤ above).

The actual implemented behavior of pessimistic specifiers with pre-release tokens is obscure and undocumented. Through my own testing, ~> 1.2.b is effectively equivalent to [ '~> 1.2', '>= 1.2.b' ]. The closest alternative specification for my original intent is [ '>= 1.2.b', '< 1.3.a' ] which is awkward at best.

The reason for this seamingly over-complicated behavior is that the rubygems authors opted to support an alternative progression for beta releases, as shown in the following table:

Version	~> 1.1.0	~> 1.2.0.b	~> 1.2.0
1.1.0	•
1.1.1	•
1.2.0.a.0
1.2.0.b.0		•
1.2.0.b.1		•
1.2.0		•	•
1.2.1		•	•
1.3.0.b.0
1.3.0

So for example, if you intend to put out a pre-release for what will become a final release of 1.4.0, then you had better version it like 1.4.0.rc.0 (replace rc as desired) and not 1.4.rc.0. As this is rather obscure and confusing, several prominent projects have done it incorrectly, for example, the 1.1.pre and 1.1.rc series of bundler, and similar versions of i18n, pg, and mongo.

Maven

If you’re not yet acquainted with my use of Java for performance and Ruby for elegant brevity: I’m using Maven only as a javac with dependencies tool and packing jars in gems. But in some cases there are combined java/ruby dependencies which must be expressed in a compatible way between the two systems.

In a similar fashion, 1.2.b.1 is accepted by Maven but non-standard and has its own set of pitfalls. The most significant of these is that Maven, according to documentation, orders such versions as follows:

1.2.0 < 1.2.1 < 1.2.2 < 1.2.a.1 < 1.2.b.1 < 1.2.b.2

Yet in practice Maven appears, at least in certain contexts to order as:

1.2.0 < 1.2.a.1 < 1.2.b.1 < 1.2.b.2 < 1.2.1 < 1.2.2

So don’t make the mistake of using versions like this in Maven. Fortunately, a Maven version range specifier of [1.2,1.2.9999) filters out the pre-releases despite the above ordering. I will be moving on from this particular issue by making 1.2.1 the next (non-pre) release of Iudex (which appears to avoid issues.)

There is a somewhat comparable mapping of pre-release versions available across the SemVer v2, Rubygems and Maven:

SemVer 2	Rubygems	Maven
1.2.0-b.2	1.2.0.b.2	1.2.0-b.2
1.3.0	1.3.0	1.3.0

However, b.2 is still compared lexicographical in Maven, so with this scheme, you would need to avoid b.10. Furthermore, range specifiers aren’t particularly pre-release aware, so the Maven range equivalent to Rubygems specifier ~> 1.2.0.b.2 would presumably need to be [1.2.0-b.2, 1.2.999) though this isn’t documented and I haven’t tested it.

It may also be worth noting that Maven dependency mediation allows users to avoid version ranges while muddling the vary intent of library authors in specifying compatible dependencies.

Did you notice how relatively simple that 1.3.0 version is looking in the above table?

Conclusion

Despite best intentions by spec. and tool authors, pre-release notations and behaviors are obscure, often conflicting, and with many potential pitfalls.

Consider also that many pre-release progressions end with a production release with no functional change from a prior candidate. Extra release work to support an often arbitrary distinction. Release beauty is ultimately in the eye of the beholder. Perhaps in general, and most certainly with this particular set of tools, we would be better served to dispense with the notion that the highest major.minor release number is necessarily the most stable. And as a corollary, continue to release patches to the previous major.minor branch.

The next Iūdex preview/candidate/beta release will be versioned: 1.3.0 and will be experimental by comparision to 1.2.2.

I’m not sure how this notion of pre-release relates to test-driven development concepts, where any new feature would of course be sufficiently unit tested. ↩
What locale you might ask? Greek? ↩

Bundler for gem development

2012-02-19T00:00:00-08:00

Motivation

We build a lot of gems, both public and private. In projects involving 5 or 10 different kinds of service daemons, it is useful for both consistency and efficiency of install and updates to package each daemon in its own gem. Now anything to be reused across multiple services should also be packaged as a gem. Before you know it, you have a lot of gems in play. For example, a sub-graph of Iūdex project dependencies:

Where several of these gems are actually dependencies of other internal service gems. Some might assume this is a recipe for a speedy descent into dependency hell. I would agree that some care in dependency management is definitely required. For example, maintaining these dependency visualizations, and avoiding loops or unnecessary coupling. What is the return on this effort? Some advantages include:

A patch release for a critical fix to a service daemon, or a single shared gem used by many daemons, does not expand into the need to release everything all at once. (The implication here is, yes, we are using semantic versioning and allowing controlled patch updates in production.)
Testing updates is similarly much less invasive to arrange in your development environment.
Efficiency of upgrade over the network: Deploying an update does not involve pushing monolithic 10’s of megabytes of statically linked software from your development hosts to production. Instead rubygems figures our what incremental gem updates are actually required, dynamically.

The above approach is quite different from the prototypical Ruby on Rails application, deployed direct from git, and using Bundler to bring in the gem dependencies, often vendored, or effectively, statically linked. Bundler grew out of the need to overcome dependency issues, which had as root cause the failure of many gem publishers to practice semantic versioning, i.e. interface or behavioral changes in patch releases breaking your application; or lack of awareness by application developers on the need to lock down gem dependencies against potential future breaking changes, ex: the >= 0 dependency specification.

But Bundler is also particularly useful for developing gems, and managing dependencies in the development environment before release and deployment. The TarPit 2 release fully integrates with Bundler.

Multiple Gem Projects

You won’t find much written on this as a project style or Bundler use case. Rails is however one high-profile example, where several inter-dependent gems are contained in a single git-repo. Notice the the top-level Gemfile here which loads all of the internal gemspecs. In such a project, consider the common development task of making changes across multiple gems in parallel, for example: new feature and tests in gem A, with usage and more testing in gem B.

Before Bundler, you would need to gem install A after each change, for integration testing with gem B. This added a bit of tedium as well as potential for confusion on the state of your local gem repo. (Did I install that change yet or not?)

With Bundler, and in particular using a top-level common Gemfile, Bundler now arranges for gem B’s tests to load gem A directly from the git-repo itself. No install step is required, and the current state of what is being tested is as clear as the current state of your git development repo.

Warning: dependencies may not be as close as they appear

Bundler currently has a shortcoming in support for multi-gem projects: gemspec dependency omissions might not be caught when using a top-level Gemfile. See issue 1668 for full details. For this reason, in projects of this form I suggest occasionally testing with per-gem Gemfile’s, for example prior to release or when making gemspec changes. Here is one way to arrange for that without needing to maintain some sort of parallel git branch with a separate set of gemfiles:

desc "Generate per-gem Gemfiles and jbundle install each"
task :generate_gemfile_per_gem do
  gems.each do |gname|
    Dir.chdir( gname ) do
      puts "=== Gemfile: #{gname} ==="

      File.open( 'Gemfile', 'w' ) do |fout|
        fout.write <<RUBY
# -*- ruby -*-
source :rubygems
gemspec :path => '.', :name => '#{gname}'
RUBY
      end

      sh "jbundle install --path #{ENV['HOME']}/.gem --local"
    end
  end
end

Note that you likely will want to jrake install all gems to your local repo before using jrake generate_gemfile_per_gem and then jrake test to validate your gemspecs.

jbexec

Gem project bin (or init) scripts shouldn’t require Bundler, as that would make Bundler a production dependency. So for ad hoc testing in development you either need to jgem or jrake install everything as was done pre-Bundler, or get the bundle activated through some other means. The jbundle exec as provided by Bundler ends up booting the full bundle environment twice, once in an additional process. This is particularly slow on jruby.

As an alternative, the following little jbexec script placed in your bin directory is much faster and appears adequate for this use case:

#!/usr/bin/env jruby

require 'rubygems'
require 'bundler/setup'

load ARGV.shift

Iyyov

2010-03-13T00:00:00-08:00

Toward the goal of fully self-contained java/ruby service gems and ruby scripted setup, some issues have lingered. With the recent releases of Hashdot 1.4 and the new Iyyov monitor, a complete solution is now available. Here is what it looks like:

Package and deploy your service as a versioned rubygem.
Include a version-specific service init script written in ruby and annotated will all necessary JVM/jruby settings (i.e -server -Xmx2g, etc.)
Hashdot will reliably bootstrap your service as a fully UNIX compliant fork’d, setsid, and IO redirected daemon with an instance exclusive PID file and recognizable process name.
Iyyov will start your service on system boot, keep it running, and automatically restart on config changes or gem upgrades.
No external bash scripts, user environment settings, or other cruft required.

Packaging your Service Gem

MRI rubygem authors would expect to place a daemon startup script in the gem’s bin directory. On install, rubygems will create a wrapper script for the original that arranges for gem initializing, makes the script available to the user PATH, and provides a convenient version selection mechanism. This is a great mechanism for short lived command-line interaction with gems, but has one very troubling shortcoming for a JRuby service writer:

There is no obvious mechanism to reliably control JVM or jruby settings on a daemon or version specific basis and package these settings within a gem.

Jruby defaults to the -client JVM, 500Mb of heap, and no specific GC tunings. This is reasonable default behavior, but most any production-grade service will want and need a mechanism to override these defaults.

To achieve this we need to package a non-wrapped executable script. Here is a sample from hashdot-test-daemon:

#!/usr/bin/env jruby
#-*- ruby -*-
#. hashdot.profile         += daemon
#. hashdot.pid_file         = ./hashdot-test-daemon.pid
#. hashdot.io_redirect.file = ./hashdot-test-daemon.log
#. hashdot.vm.options += -Xmx64m
#. hashdot.vm.options += -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled

require 'rubygems'
gem( "hashdot-test-daemon", "= 1.1.0" )

require 'rjack-logback'
RJack::Logback.config_console( :full => true, :thread => true )

require 'hashdot-test-daemon'
Hashdot::Daemon::Runner.new.run

Note that if a system level Hashdot dependency gives you concern, you could achieve an approximately similar and perhaps sufficient result with a more elaborate shell script and/or other daemonizing options less complete for JRuby.

In any case, we want to be able to launch the correct service version by referencing the full path to the script in the local gem installation. For example if the gems are installed to /opt/jruby/gems, then the following should launch the 1.1.0 version of service in the background with any 1.1.0 specific settings:

/opt/jruby/gems/gems/hashdot-test-daemon-1.1.0-java/init/hashdot-test-daemon

Service runtime layout

Given java’s native threads its unlikely that it will be necessary to run a large number of separate daemon instances. However there is still plenty of reasons to want to run different services on the same host or different configurations or versions of the same service. Toward this end I use a separate directory for each service, and relative path references to instance specific configuration, log, and PID files, i.e:

/opt/var/hashdot-test-daemon-1
├── hashdot-test-daemon.log
├── hashdot-test-daemon.log.1.gz
├── hashdot-test-daemon.pid
└── config.rb
/opt/var/service-test-daemon-2
└── ...

A fully expanded shell based launch of the service might look like:

cd /opt/var/hashdot-test-daemon-1 && \
   exec /opt/jruby/gems/gems/hashdot-test-daemon-1.1.0-java/init/hashdot-test-daemon -c config.rb

And in fact, one approach to boot-time startup or restart would be to encode something like this in an init.d script, inittab, upstart, or crontab. The last three can achieve automatic restart, provided that the service supports reliable instance-specific mutual exclusion (via PID file, etc.) Hashdot supports this reliably.

Iyyov

Once I arrived at the above solution for system layout and packaging service startup details in the gem, I went looking for a process monitor solution that would handle restarts and I could customize to keep the setup simple. I first looked at Monit, and while I was able to get it to work with these daemons in basic form, configuration was “involved”, and customization impractical.

Next I tried God. While God’s integration with kernel events makes it all powerful, and its ruby based configuration is awesome, it is also a bit of a liability to install. Furthermore, God doesn’t run on JRuby, due to a combination of heavy native non-FFI extension, and reliance on low level ruby API’s not yet completely supported. In short, I found God to be a bit more than I needed. So instead I began work on and created Iyyov’s configuration in God’s image. See the Iyyov README for a more detailed feature set and examples. The hashdot-test-daemon in the above example can be configured and launched like so:

Iyyov.context do |c|
  c.define_daemon do |d|
    d.name     = "hashdot-test-daemon"
    d.version  = "~> 1.0"
    d.log_rotate
  end
end

Which highlights several Iyyov features:

By default, Iyyov uses the rubygems API to find the highest matching gem version and init script with the same name. Thus in the common case, configuration is absolutely DRY.
Built in log rotation/compression that meshes well with Hashdot’s IO redirection and SIGHUP log reopening support. The end result is a complete logging solution that also redirects the often neglected JVM level output in the event of a JVM crash, uncaught exception, or ‘kill -QUIT’ arranged stack dump. The external Linux logrotate utility could also be used here.

Iyyov is also a practical cron replacement supporting fixed or periodic time scheduling of supporting tasks. For example:

Iyyov.context do |c|

  # Backup service state at 1am local time each weekday
  # Run :async, in a "background" thread, since this might take a while.
  c.schedule_at( :name => "backup", :mode => :async,
                 :fixed_times => [ '1:00' ], :fixed_days => (1..5) ) do
    system( "rsync ..." )
  end

end

As a final example, since the configuration is fully scriptable ruby, we might select which service to run by hostname in a single maintained configuration file:

Iyyov.context do |c|

  case `hostname -s`.strip

  when /^server-[012]$/
    c.define_daemon { |d| d.name = "widget-factory" }

  when 'server-3'
    c.define_daemon { |d| d.name = "assembly-line" }

  end
end

Gem maintenance

Because the above described per-gem init script isn’t wrapped by rubygems, the corresponding gem version should be hard coded within to guarantee the resolution of this and other gem dependencies. This presents a maintenance problem which can be solved through various means. My solution is the Tarpit 1.2 test_line_match feature that may be installed in the gem project’s Rakefile, i.e:

task :check_init_version do
  t.test_line_match( 'init/hashdot-test-daemon',
                     /^gem.+hashdot-test-daemon/, /= #{t.version}/ )
end

A simple error message is generated when attempting to build the gem without this version being set properly. My recent gems use this feature extensively and I think its less net maintenance then the other option I considered: generating such files with ERB templates.

RJack Updates via Gemcutter

2009-12-19T00:00:00-08:00

Another batch of RJack updates has been accumulating, but was blocked behind figuring out what has changed with Gemcutter and gem publishing to the canonical repo. Gemcutter appears to be good forward progress in modernizing gem distribution. While the blog posts and FAQ provide clues, it would help more if there was some canonical status page (not a twitter timeline) that provides a clear summary of the current state of accounts, publishing routes, and gem repo behavior. Somewhere I thought I read that mirroring of the old rubyforge project gem file hosting has stopped, and a look at rubyforge.org shows the last published gem recorded there was on 2009-12-10. So without risking an old style rubyforge release (Hoe’s :release task) with unknown results, I decided it was time to start using “gem push”.

The previous introduction of rjack-tarpit made the changes for rjack much simpler. I first released a new rjack-tarpit-1.1.0 which provides new “push” and “install” tasks as shown below:

# Define gem push and install tasks
def define_gem_tasks
  desc "gem push (gemcutter)"
  task :push => [ :gem ] do
    require 'rubygems'
    cm = Gem::CommandManager::instance
    cm.run( [ 'push', '-V', "pkg/#{name}-#{version}.gem" ] )
  end

  desc "gem install (default install dir)"
  task :install => [ :gem ] do
    require 'rubygems'
    cm = Gem::CommandManager::instance
    cm.run( [ 'install', '--no-ri', '-V', "pkg/#{name}-#{version}.gem" ] )
  end
end

(I hope CommandManager::run is a “supported” approach to running gem commands in process. This has a speed advantage under jruby by avoiding a separate interpreter start-up.)

Using the new gemcutter based :push task I was able to release all of these accumulated updates:

This eliminates one Hoe dependency, and with tarpit encapsulating these concerns, this change feels like a step in the direction of removing the Hoe dependency (and the quirky Manifest handling) all together. I like my implementation better than the recently released Hoe 2.4 Gemcutter plugin.

Jekyll

2009-11-04T00:00:00-08:00

A new post for a newly revamped void * blog and a new personal home page constructed with Jekyll, a ruby based static site generator. In relation to my needs, Jekyll offers:

A back-to-basics approach of static site generation from “source” as HTML templates, CSS, and markdown, all easily maintained by me in git and emacs.
Supports blog-style dated posts with excerpts and template driven atom feed generation.
Usable as a simple but complete and powerful CMS, not just blog posts. I want this for both maintaining a home page for myself, but also for developing technical site docs (usage, design, reference, etc.) for various projects.
Supports inline code sample syntax highlighting, something that was lacking in the previous blog.

For now, the generated site remains on my home server and is served via Nginx. But I could easily push it elsewhere, for example, S3; as it requires no dynamic server component.

A missing feature vs. my previous blog is comment support, which has traditionally required a dynamic server, means of storage, spam detection, and many other complications. But java script embeddable comment services like DISQUS are now winning conversions even from mainstream blog platforms based on superior features and identity integration. Its on my list to integrate one of these. Its both amusing and convenient to have Web 2.0 handling the last mile, and allowing me to get back to basics with a Web 0.1 static site generator.

I’ll end the post with an example, the liquid template used to produce this blog’s atom feed:

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

 <title>void * blog</title>
 <link href="http://gravitext.com/atom.xml" rel="self"/>
 <link href="http://gravitext.com/"/>
 <updated>{​{ site.time | date_to_xmlschema }}</updated>
 <id>http://gravitext.com/</id>
 <author>
   <name>David Kellum</name>
   <email>dek-blog@gravitext.com</email>
 </author>

 {​% for post in site.posts %}
 <entry>
   <title>{​{ post.title }}</title>
   <link href="http://gravitext.com{​{ post.url }}"/>
   <updated>{​{ post.date | date_to_xmlschema }}</updated>
   <id>id:{​{ post.id }}</id>
   <content type="html">{​{ post.content | xml_escape }}</content>
 </entry>
 {​% endfor %}

</feed>

RJack Updates

2009-10-24T00:00:00-07:00

This week’s addition to the growing set of rjack gems is rjack-rome-1.0.0 based on the ROME RSS/Atom feed reader/generator.

A new SVG intra-gem dependency diagram, built using the Graphviz Dot tool, was also added to the rjack project home page. The gems are linked to the RDoc:

RJack Updates

2009-10-18T00:00:00-07:00

The RJack project home page has been further improved. New and updated gems are detailed below.

New

The following new gems have been released after consultations with the upstream project owners:

rjack-jdom-1.1.0.0 (JDOM XML processor)
rjack-jets3t-0.7.1.0 (Amazon S3/CloudFront client)

The rjack-jets3t gem in particular is demonstrating returns on the gem based dependency management approach of rjack. This new gems depends on rjack-httpclient-3, rjack-commons-codec, and rjack-slf4j.

With these additions, rjack is up to 31 third-party jars packaged in 13 gems. With the recently released rjack-tarpit making it much easier to setup and maintain these gems, I am much more inclined to push all Java third-party jars into rjack gems rather than packing them directly in the consuming projects. This makes the consuming projects easier to maintain and distribute, and avoids version conflicts.

Next up for inclusion is an rjack-rome packaging of the ROME feed parser (which will also dependent on rjack-jdom.)

Updated

Additional gem updates since the last RJack post include:

TarPit

2009-10-11T00:00:00-07:00

Admission: I was a bit rash to have declared Jihad on Maven back in December. (Is it ever wise to write “Jihad” on your blog?) Now 10 months later I’m still using Maven to build java and manage cross-java dependencies at build time. In the interest of peace and reconciliation let me say that Maven has really been performing amiably in its now reduced responsibility of invoking javac with a proper classpath, and packaging jars from the results. While this stalemate still seems less than ideal, I think I might continue to be able to get along with Maven, provided I don’t need to write any “plugins” for it, or wrestle with it for any less well trodden tasks.

Besides the war mongering, the original post does manage to outline the goals and associated set of issues around building gems for JRuby with java jars included. I’ve had good success with this strategy for releasing componentized systems to both dedicated colo and Amazon EC2 production environments in the last year. Enough so that investing some more time in build automation has become warranted. This comes in the form of a new Rake helper gem called TarPit, released as part of the RJack project. What was a very wet and copied around jars-in-gem rakefile recipe, can now be expressed in a short and declarative style. For example, the rjack-logback Rakefile:

require 'rjack-tarpit'

t = RJack::TarPit.new( 'rjack-logback', RJack::Logback::VERSION )

t.specify do |h|
  h.developer( "David Kellum", "dek-oss@gravitext.com" )
  h.extra_deps << [ 'rjack-slf4j', '~> 1.5.8' ]
  h.rubyforge_name = "rjack"
  h.remote_rdoc_dir = "logback"
end

t.jars = %w{ core classic access }.map do |n|
  "logback-#{n}-#{ RJack::Logback::LOGBACK_VERSION }.jar"
end

file 'Manifest.txt' => [ "lib/#{t.name}/base.rb" ]

t.assembly_version = 1.0

t.define_tasks

How it all works

The t.specify() call wraps Hoe.spec() (Hoe 2.x) where h is the hoe object. All of the normal Hoe.spec options are still available, but TarPit importantly gains control of when in the Rake setup process Hoe.spec() is actually called (hint: last in define_tasks, once any needed Manifest.txt update has already been performed.)
There are some strategy options on the initial TarPit.new call, including: :jars_from_assembly (where only Maven knows the versioned jars to be included) and :no_assembly (when you are only including an internal jar built from source). In the :jars_from_assembly case, given that Hoe still insists on a Manifest.txt complete on disk at start of Rake setup, you still need to manually run jrake manifest as needed. If instead you can specify the jars, for example, as above from a ruby version number, then Manifest.txt will be built as needed and before any dependent hoe tasks.
In either case TarPit will setup Rake to run “mvn package” as a dependency whenever any java source files (src/**), pom.xml, or assembly.xml have been updated. The Maven produced jars, including “assembled” external jars are symlinked from the normal Maven target location to your lib/gemname/ directory.

RJack Expanded

2009-10-10T00:00:00-07:00

The RJack project has expanded its mission in providing common open-source java components as gems for use with JRuby.

Name Changes

When I started RJack with the slf4j and logback gems, I figured I was pretty safe in using these names verbatim, since the gems were a very minimal and direct packaging of the same named upstream projects. Same with the jetty inclusion. Over time however I found potential inclusions colliding with existing ruby gems, for example: hc-httpclient was so named because httpclient, not surprisingly, was not available. With the later additions a certain rjack style also started to jell:

SLF4J is the logging interface: every included rjack gem will use it to resolve java logging dependencies where found, including via legacy interfaces like commons logging, etc.
RJack will strive to expose every aspect of component setup, which might be referred to as “configuration” in the upstream java project, as easy to use and extend ruby classes, instead of less flexible XML, YAML, or the like.
When appropriate and useful, RJack gems will include ruby adapters to make these components more amendable to direct use from ruby code.

Given these stylistic choices, and in order to better consolidate the effort and prepare for further expansion, I’ve decided at this point to release new versions of the existing gems under a “rjack-“ gem prefix name and ::RJack ruby module. The following table shows old and new names and the current release version. New rjack-gems are also shown.

Old Name	New Name	Updated Version
slf4j	rjack-slf4j	1.5.8.1
logback	rjack-logback	0.9.17.1
hc-httpclient	rjack-httpclient-3	3.1.3
-	rjack-httpclient-4	4.0.0
jetty	rjack-jetty	6.1.20.1
jetty-jsp	rjack-jetty-jsp	6.1.20.2.1.1
-	rjack-commons-codec	1.4.0
-	rjack-commons-dbcp	1.2.2.0
-	rjack-commons-dbutils	1.2.0
-	rjack-commons-pool	1.5.3.0

The rjack-httpclient-[3,4] gems were a special consideration. I maintain continuity between the two versions, and the ruby setup interfaces are compatible, but require’ing both versions in the same process is permissible (like the associated Java code). This supports incremental transition to 4.x, a reality given lack of backward compatibility in the java interface and lack of significant pressure to migrate from 3.x.

In my other projects that use these gems, I’ve found converting to the new names a pretty easy grep and replace operation. Since they are also some cross dependencies between these gems, I recommend transition en masse. The gems with the old names will remain in the RubyForge gem repo, and significant bug fixes can be back ported if needed.

New Features

Finally, there are some other new features included:

The SLF4J gem now has better support for both ruby and java exceptions logged from ruby, and in mapping Ruby Module::Class names into java/SLF4J Logger dot notation.
Mapped Diagnostic Content support in SLF4J and Logback
A new reusable jar-in-gem build tool, rjack-tarpit the subject of a future post.

Insidious Whitespace

2009-06-28T00:00:00-07:00

Over the past few months I’ve converted just about every active project of mine from Subversion to Git. This now includes the Evri development projects, as well as my open source projects, and even home linux server code (i.e. the ruby script that runs this blog.)

Background

I was originally drawn to git based on its merits in supporting free forking of open source projects. Publishing several of my personal open source projects to github has cemented my appreciation. That said, as has been discussed broadly, the learning curve of git is quite steep. With this kind of power must come some complexity. Its relatively simple to push your sole project contributions to a github repo, but the first time I offered commits and a pull request off a fork of activerecord-jdbc-adapter I was a bit embarrassed to find that I really didn’t yet know what I was doing. That was a good lesson for me, perhaps learned the only practical way, and since then I’ve put a little extra git education time into just about every project. Milestones crossed:

I’m no longer terrified by the unknown that was “git rebase”
I’m only mildly leery of “git svn.”

Yes I will make my history look the way it should: nice independent linear patches, forward-only merges when possible, etc. And I think this is slowly starting to pay dividends in my development process. I’ve re-enabled some development tactics that I had previously decided, over the years of using SVN, CVS, and those before it, just weren’t worth the trouble. And I’ve learned some new tactics.

My mode of operation with Subversion had been to avoid branches and merging at all costs:

Don’t check that in until I tag the next release!
Lets split this code base up so we can work better in parallel!

While there remains merit in the last approach (appropriate modular decomposition), it shouldn’t be driven by limitations of a tool, and besides I never figured out exactly where to put those “trunk” directories.

Making the tools work

What parts of my development tool set need to change with git?

Maven might have been an issue (isn’t it always), but my recent conversion to java/ruby gems packaging has the convenient side effect of liberating me from the maven release plugin (though I’m told there is workable maven/git integration). I use rake for that now, and a simple “rake tag” task covers my git interaction.

I’m still inclined to use Eclipse, while its not ideal, as a java code editor, primarily for the refactoring support that java seems to need, as a junit test launcher, and occasional debugger. The egit plugin for eclipse works well for basic operations, but the command line is still my main interface to git.

But with merging in play, I really want my commits to be recognizable as clean atomic patch sets. An important aspect of this is not cluttering up those patches with arbitrary white noise in the form of randomly changing whitespace. With emacs, there is whitespace-mode and surrounding commands. Problem solved for all files accept java, but Eclipse is just not that sophisticated as a text editor.

cleanws script

So I worked up a new tool. Its a simple text file whitespace filter, but I’ve also added some nice git integration and color coding to make it reasonably convenient. Ruby BTW is quite nice for this kind of task. Before doing a “git add”, I run “cleanws -g” to ensure good canonical whitespace in new/updated files. See the following example:

The second call fixes the file and (-w)rites it to disk. Any detected evil tab characters are left in place to be manually resolved.

Here is the full script. Perhaps it will find its way into a gem and github it there is interest or if I continue to expand on it. Enjoy, and please keep it clean out there.

Hashdot 1.3 Released

2009-03-09T00:00:00-07:00

Hashdot 1.3 has been released on SourceForge.

1.3 (2009-3-9)

Added hashdot.user.home property to current real user home directory.
Added hashdot.header.comment property in support of languages not using ‘#’ as a line comment. The groovy.hdp and rhino.hdp profiles now set this property to “//”. Associated examples demonstrate alternative header formats.
Added hashdot.script.dir. This and hashdot.script properties are now always absolute paths. Clarified use of java.class.path.
Added clojure (clj) profile and example. [Phil Hagelberg]
Refined examples, jruby is only default INSTALL_SYMLINK.

Hashdot 1.2 Released

2008-12-22T00:00:00-08:00

Hashdot 1.2 has been released on SourceForge. Special thanks to Matt Sanford for contributing the initial Mac OS X port and helping me through some additional OS X idiosyncrasies. Details in the change log below:

1.2 (2008-12-22)

Added HUP signal handler to reopen STDOUT/STDERR when both daemonize and io_redirect are used
Added support for hashdot.chdir
Ported to Mac OS X. See default.hdp for differences in profile configuration. Note on Mac, the final test/test_cmdline.rb currently fails due to lack of a suitable equivalent to the Linux prctl() for process rename. However, this is not strictly required for useful operation.
Simplified default.hdp and shortlived.hdp using new hashdot.vm.lib delayed expansion property.
Added explicit int casts to avoid x64 warnings
Improved INSTALL notes.

Maven Must Die

2008-12-07T00:00:00-08:00

The RubyGems packaging system has several advantages for java-hosted components. It goes places that Maven won’t go, like all the way into production and managing non-trivial executables from the command line or inittab. In my first post on rjack I make some further case for this approach and describe some considerations in packaging ruby/java components as gems. In this post I’ll describe some of the complexities with building such gems, and workarounds to common problems.

Gems with jars

The rjack project is best understood with an install of jruby, optimally hashdot, and for example, the jetty gem:

% jgem install jetty
% jetty-service

This will afford the opportunity to look at how the gem is built, as the full source including the build mechanism is included. The top two levels of the gem directory structure are shown below:

/opt/jruby/gems/gems/jetty-6.1.14.1/
|-- History.txt
|-- Manifest.txt
|-- README.txt
|-- Rakefile
|-- assembly.xml
|-- bin/
|   `-- jetty-service*
|-- lib/
|   |-- jetty/
|   `-- jetty.rb
|-- pom.xml
|-- src/
|   `-- main/
|-- test/
|   `-- test_jetty.rb
`-- webapps/
    |-- test/
    `-- test.war

Note the presence of both a Rakefile, and the Maven pom.xml and assembly.xml. Other projects that produce gems including jars manage the jar files as if they were source, but this becomes a maintenance liability in any of the following scenarios:

Included jar(s) are undergoing frequent updates and the gem should be kept current.
Included jar(s) have additional jar dependencies themselves undergoing updates, and these jars must also be included.
The gem includes original java source code that must be built, packaged as a jar and included in with the gem.
External components, perhaps themselves packaged as gems, have dependencies on the packaged java. Thus the internally built java must also be published for resolving external java dependencies.

Hoe, Rake, Maven: The unholy union

The jetty gem Rakefile (see below) attempts the rather perilous integration of Maven into Rake, with Hoe for gem building and rubyforge publishing support. Here you will find subtle and not-so-subtle workarounds for a variety of issues encountered, including:

In standard gem build tradition, the gem version is actually obtained by loading ruby source. This has the advantage of the version being specified in one (fewer) place. However, there is an unfortunate incompatibility with JRuby and Rake both currently defining Object::import (JRUBY-3171). There is a simple workaround: Include an alternative ruby include that just defines version constants without any java imports. In this case its lib/jetty/base.rb, which defines VERSION and JETTY_VERSION in the Jetty module.
Hoe requires a Manifest.txt file on disk a priori, apparently to protect us from making mistakes in what is included in the gem. The problem with this is that jar names to be included may frequently change, and only maven knows the full dependency details. The workaround is to generate the Manifest.txt file with a :manifest task, but this task must be run and completed in its own process before any Hoe tasks utilize it as part of task definition. In practice we need to remember to update the manifest when jar’s change, less than ideal.
In the jetty gem, the full list of JARS (l28) are mapped from the jetty/base.rb included VERSIONs. In other gems the assembly brings in arbitrary dependencies of mixed versions, it becomes necessary to Dir.glob( “..assembly/*.jar” ) for these and the :manifest task acquires a “mvn package” task dependency.
We know to build the assembly via “mvn package” if it doesn’t yet exist, or if the pom.xml or assembly.xml is newer. The assembly is made a dependency of the :gem and :test tasks (l84). However, for gems packing java source, the dependencies are extended to all java source files as well.
Hoe wants to use a graphics tool called “dot” for rdoc generation. I don’t. Disabled via a crufty ENV[‘NOTDOT’] setter.

These might appear to constitute a rant, but I’m well aware of the fact that all of this software is free. I’m just fishing for a better solution in a relatively new problem space.

Gem/jar dependencies

For a more complicated example, consider a dependency graph between two gems both containing java source:

There is no dependency loops and everything works surprisingly well once gem and maven releases have been made to the respective repositories. However its complicated, manual, and error prone when implementing new versions of both gems in parallel, and needing to share changes with other developers. We must abandon Maven’s mvn release:prepare functionality for tagging a release version and incrementing to the next snapshot. Instead the following steps are typical:

For gem-a, jrake clean to remove older jar versions
Update versions numbers in the pom.xml, lib/gem-a/base.rb, and a new version to History.txt
jrake manifest to update Manifest.text with new jars.
jrake test gem to build, test and package the gem.
mvn install to make a.jar available for gem-b java build.
jgem install pkg/gem-a-VERSION.gem to make gem-a available for gem-b
For gem-b, repeat steps 1-4, etc.

Also consider that much of the above will need to be repeated if gem-a undergoes changes that gem-b is dependent on. One saving feature is that from a Maven perspective, gem-a and gem-b still look like “normal” java jar projects, if gem-a and gem-b java changes can be made in parallel with typical maven means, including use of parent pom declaring gem-a and gem-b as modules. But in general, its clear that a better integrated tool, offering full build automation for gems with jars is needed. An evaluation of alternatives will be saved for a subsequent post.

Rakefile for jetty gem

require 'rubygems'

ENV['NODOT'] = "no thank you"
require 'hoe'

$LOAD_PATH << './lib'
require 'jetty/base'

JARS = %w{ jetty jetty-util jetty-rewrite-handler }.map do |n|
  "#{n}-#{ Jetty::JETTY_VERSION }.jar"
end
JARS << "servlet-api-#{ Jetty::SERVLET_API_VERSION }-#{ Jetty::JETTY_VERSION }.jar"
JARS << 'gravitext-testservlets-1.0.jar'
JAR_FILES = JARS.map { |jar| "lib/jetty/#{jar}" }

desc "Update the Manifest with actual jars"
task :manifest do
  out = File.new( 'Manifest.txt', 'w' )
  begin
    out.write <<END
History.txt
Manifest.txt
README.txt
Rakefile
pom.xml
assembly.xml
bin/jetty-service
lib/jetty.rb
lib/jetty/base.rb
lib/jetty/rewrite.rb
lib/jetty/test-servlets.rb
src/main/java/com/gravitext/testservlets/PerfTestServlet.java
src/main/java/com/gravitext/testservlets/SnoopServlet.java
test/test_jetty.rb
test/test.txt
webapps/test.war
webapps/test/WEB-INF/web.xml
webapps/test/index.html
END
    out.puts JAR_FILES
  ensure
    out.close
  end
end

ASSEMBLY = "target/gravitext-testservlets-1.0-bin.dir"

file 'webapps/test.war' => [ 'webapps/test/index.html',
                             'webapps/test/WEB-INF/web.xml' ] do
  sh( 'jar cvf webapps/test.war ' +
      '-C webapps/test index.html -C webapps/test WEB-INF/web.xml' )
end

file ASSEMBLY => [ 'pom.xml', 'assembly.xml' ] do
  sh( 'mvn package' )
end

JARS.each do |jar|
  file "lib/jetty/#{jar}" => [ ASSEMBLY ] do
    cp_r( File.join( ASSEMBLY, jar ), 'lib/jetty' )
  end
end

[ :gem, :test ].each { |t| task t => JAR_FILES + [ 'webapps/test.war' ] }

task :mvn_clean do
  rm_f( JAR_FILES + [ 'webapps/test.war' ] )
  sh( 'mvn clean' )
end
task :clean => :mvn_clean

task :tag do
  tag = "jetty-#{Jetty::VERSION}"
  svn_base = 'svn://localhost/subversion.repo/src/gems'
  tag_url = "#{svn_base}/tags/#{tag}"

  dname = File.dirname( __FILE__ )
  dname = '.' if Dir.getwd == dname
  stat = `svn status #{dname}`
  stat.strip! if stat
  if ( stat && stat.length > 0 )
    $stderr.puts( "Resolve the following before tagging (svn status):" )
    $stderr.puts( stat )
  else
    sh( "svn cp -m 'tag [#{tag}]' #{dname} #{tag_url}" )
  end
end

hoe = Hoe.new( "jetty", Jetty::VERSION ) do |p|
  p.developer( "David Kellum", "dek-ruby@gravitext.com" )
  p.rubyforge_name = "rjack"
  p.rdoc_pattern = /^(lib.*\.(rb|txt))|[^\/]*\.txt$/
end

RJack Updates

2008-12-06T00:00:00-08:00

The RJack project home page has been beautified.

The gem updates include:

jetty and jetty-jsp 6.1.14
slf4j 1.5.6
logback 0.9.13

Also, as part of this update I tested compatibility with newly released JRuby 1.1.6-RC1. Assuming the RC status is meaningfully applied and temporary, all should be in good for the 1.1.6 general release.

The void * blog is now running these gems.

Scripted Server Setup

2008-11-09T00:00:00-08:00

I’ve made several posts on Hashdot and the RJack gems, but have yet to post on the why? or what? of it. This post should fill that gap by way of a non-trivial example.

At Evri I’m responsible for content acquisition and text extraction. We’re parsing volumes of news and general web content, and separating the kernels of semantically rich text from the chaff i.e. site navigational boxes, advertisements, etc. Normally this content is processed “offline”, but I had reason recently to package it as a HTTP-XML service for transforming web content in real time. As the content processing system already existed as a mixture of java (for the heavy lifting) and ruby, I went looking for a solution for service construction that would be simple, flexible enough to reuse the existing processing code and setup, and well performing.

As the XML production (output) code was already in java, and because I wanted the performance advantages of the JVM’s native threads, I decided to write two java servlets: one to simply test obtaining the web content by URL as a proxy (using thread-safe and connection pooling commons-httpclient), and the second extending the first to actually transform the content HTML into an internal XML representation. So now I have two servlets that need to be setup in an HTTP server, I want them to share a common HTTPClient setup, and the transforming servlet also needs a reference to a custom setup content processor. How was I going to wire all this together?

Standards, Frameworks, and XML configuration

Classical java would have me create a Web Application Archive (WAR) which contains the compiled Servlets plus a web.xml Deployment Descripter providing the mapping of servlets to URIs. Since the container will be constructing the servlets, access to the HTTPClient and processor must be arranged by non-direct means, presumably through a ServletContext which allows Object-type attributes. Back in the day. with earlier Servlet specs, this would have been done with some form of initialization servlet. Now its possible to register a context listener which does the same. I could write all the java classes and XML myself, or depend on something like the Spring Framework’s ContextLoaderListener. In either case it will be a challenge to keep the ruby based content processor setup without resorting to an additionally complex integration, with java setting up a jruby interpreter (via BSF or otherwise). Note also that none of this complexity helps to address such matters as how to get this all installed in a production environment, how the service will be started, monitored, etc. The two servlets are themselves pretty simple. How might we shed some of the surrounding baggage?

Un-Inversion of Control

Enter the beautifully designed Jetty Web Server which offers full compliance with all of the committee Servlet/JSP standards, but also offers direct programmatic setup under the guise of embedding. It was quick work to prototype the same embedded HTTP server setup in jruby. I later took up authoring a generic jetty gem packaging and setup façade as a side project.

As I’ve setup a lot of classic java web applications in the past, reviewing the script for this post gave me an eery feeling that somehow there must be a few more XML configuration files or bash scripts somewhere that were needed to make this all work. That’s not the case here. Lets quickly review the prerequisites:

Java JDK or JRE
Hashdot installed with JRuby in /opt.
RJack logback and jetty gems:
```
  % jgem install logback jetty
```
A commons-httpclient gem which has not yet been published.
The Evri proprietary content_processor and proxy_transformer ruby Modules which, sorry to say, I won’t be sharing.

(Note: I’ve also changed a few of the setup details of these to avoid giving too many details.)

Dependency Injection

Fancy terminology aye? The HTTP Client and content processor are simply created as local variables http (l28) and processor (l33) in the script, and passed directly on construction of the ProxyServlet and TransformServlet (l46-7). Isn’t that refreshingly direct? The Jetty::ServerFactory provided in the jetty gem is used to create a the complete HTTP server, with a healthy dose of sensible defaults. The set_context_servlets call (l35) uses a ruby hash for mapping paths of the root '/' context to these servlets. There is no Web Application, because there is no need for one. (However the jetty gem supports full webapps as well when needed.)

Lifecycle Events

More fancy terminology. We have some shutdown work we’d like to do after the server port is closed. Once Jetty is started via server.start (l51), the server.join call (l52) will block until the server exits, for example via SIGTERM. We then have an opportunity to cleanup, for example, gracefully closing the HTTP client and any kept-alive connections (l55). Other example might include writing out some final reporting information.

Code or Configuration?

The original Jetty embedding examples are written in Java and thus need to be compiled. Any settings such as the HTTP port number are thus hard coded in the jetty examples. The distinction is a good bit less clear in the case of this ruby script. The script itself is almost entirely in a declarative style, save perhaps for the final server start, join and cleanup (l51-55). It also uses some local variables, but this isn’t unlike named object references in Jetty or Spring XML configuration. It would certainly be easy enough to break a final set of configuration elements out into a separate XML configuration file (or YAML, or even java proprieties), but what is the gain? Is it not easy enough to change any necessary settings in this script directly? Is there not a clear advantage to having all aspects of the service setup in a single unified file, rather than spread out into multiple XML configuration files and the typically required bash wrapper script?

Where is the Ruby at Runtime?

While ruby is used for setup, wiring, dependency injection, or whatever you’d like to call it, ruby is interestingly absent as an explicit reference in any of the Java code, where there is not a a single JRuby or BSF dependency reference. Furthermore, the ProxyServlet never utilizes a single line of ruby code in its runtime execution. We’ve effectively made good use of ruby to setup this servlet but take on absolutely zero ruby performance penalties during runtime.

The content processor by comparison, as used by the TransformServlet uses a pipeline or chain-of-responsibility pattern with some links in the chain implemented in ruby. The ruby links are non-performance intensive processing logic steps, and don’t measurably detract from the performance. Note that if they did become a performance bottleneck, they could be rewritten in pure-java without needing to modify or recompile the TransformServlet. Thus by putting ruby in control of service boot-strapping, ruby can easily be injected into the service and mixed with java components where advantageous and without any complexities incurred on the java side. Some additional example code for this kind of ruby injection can be found in the Jetty::ServerFactory rdoc, where for example, I demonstrate implementing a Servlet in ruby and passing it to Jetty.

Logging, Daemonizing, and other oft ignored Subtleties

The Hashdot launcher and hashdot properties are used to arrange for the service to daemonize, redirect STDOUT/STDERR to a log file, and set a Java heap size. This will make it simple to add the service to inittab or other UNIX process monitor. The logback gem is loaded (l10,14) which loads the slf4j gem as a dependency. Jetty in turn detects the presence of SLF4J and will use it for logging. SLF4J via Logback output to STDOUT at INFO level is arranged pragmatically (l22-5), thus avoiding a separate logback.xml. Below is an example of log output using jetty-service script provided with the jetty gem itself (and with the logback gem available).

1    INFO  org.mortbay.log - Logging to Logger[org.mortbay.log] via org.mortbay.log.Slf4jLog
58   INFO  org.mortbay.log - jetty-6.1.12
280  INFO  org.mortbay.log - NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
388  INFO  org.mortbay.log - Started SelectChannelConnector@0.0.0.0:38225
388  INFO  jetty-service - Listening on port: 38225

Its important to redirect STDOUT and STDERR because any fatal errors like uncaught runtime exceptions or java crash dumps will end up here. Its also convenient when debugging a problem to be able to pkill -HUP proxy-transform-service and get all of the thread stack dumps into the log. Finally, its quite common for an application which brings in many open-source components to have cases where these components write to STDOUT or STDERR under unusual circumstances. An approach that redirects STDOUT/STDERR to /dev/null or an obscure alternative log file will commonly result in this information being lost. Thus I find a certain old school elegance in coalescing all the logging output to STDOUT, using Hashdot to redirect this to a file at the system (not just java) level, and using an external tool like logrotate to provide log rotation.

proxy-transform-service script

#!/opt/bin/jruby
#. hashdot.profile += daemon
#. hashdot.vm.options += -Xmx256m
#. hashdot.io_redirect.file = /var/log/proxy-transformer.log

require 'rubygems'

gem 'logback', '~> 0.9'
gem 'jetty', '~> 6.1'
gem 'commons-httpclient', '~> 3.1'

require 'logback'
require 'jetty'
require 'commons-httpclient'

require 'content_processor'
require 'proxy_transformer'

# Logback
Logback.configure do
  Logback.root.add_appender( Logback::ConsoleAppender.new )
  Logback.root.level = Logback::INFO
end

# HTTP Client
http = CommonsHttpClient::Facade.new
http.max_total_connections = 100
http.connection_timeout = 1500 #ms

# Content Processor
processor = ContentProcessor.new do |p|
  p.centroid_weight = 0.3
  p.filter_threads = 1
end

# HTTP Server
factory = Jetty::ServerFactory.new
factory.max_threads = 50
factory.port = 8080
factory.max_idle_time_ms = 10000

include ProxyTransformer
factory.set_context_servlets( '/',
  { '/proxy'     => ProxyServlet.new( http.client ),
    '/transform' => TransformServlet.new( http.client, processor ) }

server = factory.create

server.start
server.join

# Shutdown, cleanup
http.shutdown

RJack Updates

2008-11-09T00:00:00-08:00

Two gem updates released to the RJack project on RubyForge:

based on the latest Jetty Web Server 6.1.12.

Hashdot 1.2 Dev

2008-11-07T00:00:00-08:00

A quick status update on Hashdot:

To better facilitate merging changes with a colleague who is investigating the Mac OS X port, I’ve enabled the SVN repository for Hashdot on SourceForge, imported the 1.1 source, and then committed a set of changes for Hashdot 1.2 I had previously made locally. The changes are listed below as a preview for interested parties:

* Added change_log starting with version 1.2

svn merge -r 464:HEAD

* Added HUP signal handler to reopen STDOUT/STDERR for logrotate
  compatibility.
* Fixed hashbang.
* Added explicit int casts to avoid x64 warnings
* VERSION -> 1.2
* Added support for hashdot.chdir, incl. test and doc.

RJack Updates

2008-11-06T00:00:00-08:00

On Monday, I managed to complete the release of two new gems and two gem updates, all part of the RJack project hosted on RubyForge. As this is my first post on RJack, here is a an introduction without quoting the linked project summary, and some details on jar in gem packaging.

RJack provides gem packaging of various broadly used Java components for use with JRuby. Included in this release set are upstream updates to the slf4j and logback logging system, and the new gems jetty and jetty-jsp, which package the Jetty Web Server. A guiding design principal is that each of these gems may be used simply and solely to deliver java package dependencies to a consuming ruby/java application. However, three of the four gems in the current set also provide either façades for setup or adapters for direct use from ruby code. For these inclusions, care is given to avoid incurring any additional external dependencies.

Why gems?

The RubyGems package management system offers several advantages for working with server application components in both development and production environments, including a straight forward command line interface, multiple remote and local gem repositories, versioned dependency management, support for multiple versions of the same component, and support for installing command-line accessable executables. While in common use for pure and C-extended ruby components, JRuby provides an opportunity to similarly utilize gems for java-extended ruby or even near-pure java components. The nearest alternative from the java comunity would be Maven and its derivatives, but these are far more appropriate for handling java dependencies at compile time than for production deployment.

Once the logical decision to deploy gems is made, the first order of business is determining how to divide the various components and their java jar dependencies into gems for maximum benefit including reuse, reliability, and ease/safety of upgrade with independent release cycles. Java components are ultimately loaded by searching through an order list of jars or directories, the “classpath”. If multiple versions of the same java package are found on the classpath, the first instance wins. Problems arise if two utilized components include incompatible versions of the same package. It is therefore advantageous to break common java dependencies out into individual gems and utilize the version management offered by the rubygems system.

As a prime example, the SLF4J system is predicated on the application developer selecting a final logging destination by installing the appropriate logging adapter in the classpath. If however various gems with some java dependencies each in turn include their own versions of these or other logging system jars, then this logging output flexibility is lost. As SLF4J is now the best available logging interface for Java, I’ve released an independent slf4j gem, for reuse and in the hope of encouraging other java gem packagers to use the slf4j gem to resolve any java logging dependencies, in lieu of embedding alternative logging components in their gems.

Gems package projects

Since Maven manages dependencies at the jar level, a one-to-one mapping would suggest a gem per jar. While some discussion has taken place on automating direct jar to gem packing, a significant offering has yet to be published. I’m also not convinced this is the right mapping. Consider that in ruby, the require method is included in normal program flow, and its common to use multiple require calls to selectively bring in different parts of a single gem. JRuby extends require to support adding a named jar file to the classpath of its custom class loader. Its therefore quite reasonable for gems to map directly to a java project which provides more than one jar. A clear example of this is the RJack slf4j gem. See the SLF4J module documentation for require mappings to eight additional input/output adapter jars beyond the slf4j-api-version.jar loaded with the base require 'slf4j'. There will be other cases where a single jar is best packaged as a gem, but its clearly not necessary to do so when a project releases multiple jar’s in version lock-step.

A counter argument for finer grain gems is the case where, as with the jetty-jsp gem, large jars are included and some of these are commonly not needed. The jetty gem contains the core server components and servlet-api (4 gems) at 0.7M where as the jetty-jsp gem adds JSP support (4 more gems) for an additional 5.2M. There are also cases where external project jar dependencies are added to a gem. The argument for this would be that it is unlikely that a particular dependency will be used in another context. Of course this could be a mistake, but its one that can be fixed later with a patch release, breaking the dependency out into its own gem.

Other considerations

Finally its worth noting that these gems defer to the upstream java project version numbers, adding a single gem package release number to the end of the upstream version. This puts emphasis on version compatibility in upstream project upgrades and will encourage timing any significant ruby layer changes with upstream major releases.

Along similar lines, each of these gems is released under identical license terms to the upstream project, thereby simplifying license considerations for consuming applications.

JRuby setup with Hashdot

2008-10-27T00:00:00-07:00

In Introducing Hashdot I laid out various problems with using the standard ‘java’ launcher and the ‘jruby’ wrapper bash script provided with the JRuby distribution. In this post, I’m going to describe in detail an alternative installation layout and setup of JRuby using Hashdot to best advantage. Some goals for this setup:

Any and all invocations of ‘jruby’ should be launched via hashdot to take advantage of central configuration, better process report, and hashdot script headers where appropriate.
No user environment dependencies. We want JRuby to function properly for any user on the system, without requiring each user to maintain environment variables, etc. in their own profile.
A separate gem repository for JRuby, independent of the versioned JRuby distribution. This will enable JRuby upgrades without needing to rebuild or copy to a new gem repository.
MRI-ruby and JRuby need to coexist. This means ‘ruby’ launches MRI and ‘jruby’ launches JRuby. Such utilities as ‘ri’, ‘gem’, and ‘rake’ should invoke MRI ruby, while we also want convenient access to the same via JRuby as ‘jri’, ‘jgem’, ‘jrake’, etc.
The current latest ‘jruby’ interpreter, ‘jgem’, ‘jri’, ‘jrake’ and any bin scripts installed in the jruby gem repository should be available on a system standard PATH.

Layout in /opt

The ‘/opt’ directory appears most appropriate for this kind of system wide but custom setup. Alternatively ‘/usr/local’ could be used. The following structure is recommended and will be described in detail below.

/opt
|-- bin
|   |-- hashdot
|   |-- jgem
|   |-- jirb -> ../dist/jruby/bin/jirb
|   |-- jrake -> ../jruby/gems/bin/rake
|   |-- jri -> ../dist/jruby/bin/ri
|   |-- jruby -> hashdot
|   `-- jruby-shortlived -> hashdot
|-- dist
|   |-- jdk_sun_1.6.0_05_x32
|   |-- jdk_sun_1.6.0_07_x32
|   |-- jruby -> jruby-1.1.4
|   |-- jruby-1.1.3
|   `-- jruby-1.1.4
|-- hashdot
|   `-- profiles
`-- jruby
    `-- gems

Create the /opt skeleton

As root:

% mkdir /opt /opt/bin /opt/dist /opt/jruby

Configure, make, install Hashdot

Configuring hashdot currently involves editing the Makefile including with the source distribution. Set the following Makefile variables:

INSTALL_BIN=/opt/bin
PROFILE_DIR=/opt/hashdot/profiles

Hashdot may then be built or rebuilt via ‘make clean all’. You might want to then customize the various profiles before running ‘make install’ as root. You can also customize them in /opt/hashdot/profiles after install.

Install distributions

As root, unpack the Java JDK (or JRE) and JRuby under /opt/dist. Note I prefer to use a more descriptive name including the full java version number. Since we also reference some jruby distro provided bin scripts, symlink the latest jruby-1.1.4 to jruby. As shown, its possible for multiple versions of java/jruby to coexist for testing purposes.

Create /opt/jruby/gems repository

To create a starting gems repository, as root:

% cp -a /opt/dist/jruby-1.1.4/lib/ruby/gems /opt/dist/jruby/gems

To make sure this repository is used for all jruby invocations, GEM_HOME and GEM_PATH environment variables are set via Hashdot in the jruby.hdp profile (see below.) This has an important advantage over setting in a user profile: The MRI ruby gem home/path will not be effected and can be kept separate.

Create /opt/bin symlinks

The above list directory tree for /opt/bin should be self explanatory. We create ‘jirb’ and ‘jri’ symlinks to dist/jruby/bin in order to make these easy to use. Note however that ‘jruby -S ri’ will also still work for these. Similarly we create jrake as a symlink to gems/bin/rake, since in a development environment we will likely want MRI and jruby versions both available. Finally, we choose to customize ‘jgem’ by copying and changing the hashbang to:

#!/opt/bin/jruby-shortlived

Which gives a bit of a performance advantage in common usage. Note that with the given hashdot profiles, the server vm is default, which is almost always the best choice for any long lived application.

System PATH

Adjusting the standard system PATH is Linux (or UNIX) distribution dependent and might involve modifying /etc/profile or an equivalent. In any case, /opt/bin should be found early in the path (since we only include ‘j*’ commands here), and /opt/jruby/gems/bin should be added late in the path to get additional jruby gem bin scripts available, but without overriding any MRI ruby gem equivalents. For example:

/opt/bin:/usr/local/bin:/bin:/usr/sbin:/usr/bin:/opt/jruby/gems/bin

In the case where you need both the MRI and jruby gem bin script versions to coexist in the PATH, create a ‘j*’ symlink in /opt/bin for the jruby version.

Hashdot profiles

The customized profiles I use on my 32bit development environment are given below.

default.hdp:

# HashDot default launch profile
# This profile is always read before any other profiles or hashdot
# properties

## Setup the default JVM

# JVM install directory
hashdot.vm.home = /opt/dist/jdk_sun_1.6.0_07_x32

# Use "amd64" instead of i386 for linux 64bit platforms
hashdot.vm.arch     = i386

# Use server by default (see alternative in shortlived.hdp)
hashdot.vm.mode     = server

hashdot.vm.libbase  = ${hashdot.vm.home}/jre/lib/${hashdot.vm.arch}

# Setup libpath to find libjvm.so and dependencies.
hashdot.vm.libpath  = ${hashdot.vm.libbase}
hashdot.vm.libpath += ${hashdot.vm.libbase}/${hashdot.vm.mode}

# TZ often needed for Java
hashdot.env.TZ = America/Los_Angeles

jruby.hdp:

# HashDot launch profile for JRuby (http://jruby.codehaus.org/)

jruby.home    = /opt/dist/jruby-1.1.4
jruby.lib     = ${jruby.home}/lib
jruby.script  = jruby
jruby.shell   = /bin/sh

# Identical defaults to jruby launcher
# These can still be overridden by the individual scripts
hashdot.vm.options += -Xmx500m -Xss1024k

# Only jruby.jar is required for typical usage (scripts can require
# bsf.jar or JIP profiler if desired).
hashdot.vm.options += -Xbootclasspath/a:${jruby.lib}/jruby.jar

hashdot.main = org.jruby.Main

# Arguments following these flags are _not_ a script to scan for
# hashdot headers.
hashdot.parse_flags.value_args = -F -I -r

# Give up looking for a script header with any of these
hashdot.parse_flags.terminal = -C -e -S

# GEM PATH/HOME
hashdot.env.GEM_HOME = /opt/jruby/gems
hashdot.env.GEM_PATH = ${hashdot.env.GEM_HOME}

shortlived.hdp:

# HashDot profile for short lived processes.
# Overrides default.hdp to use the client VM.

# Note: for current amd64 java distros, this entire profile should be
# commented out to disable it, since these only include a server VM

hashdot.vm.mode     = client

hashdot.vm.libpath  = ${hashdot.vm.libbase}
hashdot.vm.libpath += ${hashdot.vm.libbase}/${hashdot.vm.mode}

jruby-shortlived.hdp:

# HashDot profile for shortlived jruby scripts.

# Extends shortlived (client VM) and (jruby) profiles with further
# startup time tweaks.
hashdot.profile = shortlived jruby

# JMX setup slows startup and wont be used
jruby.management.enabled=false

Introducing Hashdot

2008-10-22T00:00:00-07:00

From the summary of Hashdot as published on SourceForge:

Hashdot elevates Java-platform script interpreters to first class status on Unix-like operating systems. It provides a script aware replacement to the stock ‘java’ launcher, and thus avoids numerous issues in using the ‘java’ launcher to bootstrap a script interpreter. All relevant interpreter and JVM options (i.e: Java heap size) may be specified directly in a script header and/or via system profiles, without resorting to environment variables, command line arguments, and the additional wrapper shell scripts needed to maintain them.

There are some interesting aspects to this work. Firstly, Hashdot really needed to be written in C for tight UNIX integration (fork(), dlopen(), prctl(), setsid(), etc.) Though I had developed in C and C++ for many years, it’s been a while, and I was surprised at how cumbersome I now found it.

Its a bit of of an odd predicament to be forced to write C, essentially as glue code between UNIX, java, and a Java-platform script interpreter. When I took on my first large scale java project in the late 90’s (Java 1.1, Netscape Search federation and UI) I reserved the right to do any heavy lifting that was required in C++. Surprisingly, it didn’t come to that then, and only rarely has in the last 10 years. I have spent some time optimizing inner loops in a style of java that looks all too close to C. So here I am writing actual C again, but as glue code rather than for performance reasons, and simply to gain some modest comforts while writing ruby and java for the JVM!

Its doubly odd to be writing C glue code in order to make decidedly incremental and cosmetic improvements to a UNIX, Java, and JRuby stack. What if however, the aggregate total of current stack integration warts, annoyances, and complexities might otherwise retard broad acceptance of that stack by any of the UNIX, Java, or Ruby camps? Maybe then it would be worth diving in.

The Stock ‘java’ Launcher

Understanding the why of Hashdot must begin with a problem statement, and the problems begin with the stock ‘java’ launcher command line:

% java -h
Usage: java [-options] class [args...]
           (to execute a class)
   or  java [-options] -jar jarfile [args...]
           (to execute a jar file)

where options include:
    -client       to select the "client" VM
    -server       to select the "server" VM
    -cp <class search path of directories and zip/jar files>
    -classpath <class search path of directories and zip/jar files>
                  A : separated list of directories, JAR archives,
                  and ZIP archives to search for class files.
    -D<name>=<value>
                  set a system property
    ...

Like many before and after, the Java programming language started life as a toy. It must have seemed reasonable at that point, for a user starting a java program via a class’s Main method to effectively link its dependencies by passing -cp on the command line. Anyone using Java in any sort of performance intensive application would know that also setting an appropriate maximum heap size (-Xmx) and, for long lived services, using the -server VM is essentual. All told the following example “command line” is typical for a full production java service:

java -server -Xss512k -Xms32m -Xmx256m -XX:NewSize=16m -Xnoclassgc
-Dsun.net.inetaddr.ttl=30 -Dsun.net.inetaddr.negative.ttl=0
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=/home/david/run/\
  apache-tomcat-5.5.20/conf/logging.properties
-Djava.endorsed.dirs=/home/david/run/apache-tomcat-5.5.20/common/endorsed
-classpath :/home/david/run/apache-tomcat-5.5.20/bin/bootstrap.jar\
  :/home/david/run/apache-tomcat-5.5.20/bin/commons-logging-api.jar
-Dcatalina.base=/home/david/run/apache-tomcat-5.5.20
-Dcatalina.home=/home/david/run/apache-tomcat-5.5.20
-Djava.io.tmpdir=/home/david/run/apache-tomcat-5.5.20/temp
org.apache.catalina.startup.Bootstrap start

Since its rather impractical to type this on the command line, the obvious solution is to write a shell script wrapper that exec’s it. Ironically, by not offering a better facility for process boot strapping, and in the name of portability, Sun pushed developers into writing highly non-portable shell scripts. For a developer starting his career in Java, this was an awkward introduction to (ba)sh scripts. For a complete server such as Tomcat used in the above example, I count 1276 lines of shell script (sh + bat) and a maintenance job in its own right.

Enter JRuby

Scripting languages like ruby, perl, or python have all done a much better job than java in supporting cross-platform portability while also not hamstringing OS, file system, or console operablility. This was but one aspect that drew me to jruby. Would it not be possible to deploy entire, potentially complex, java-based services as easily installed gems and directly executable ruby scripts, rather than the typical amalgam of tarballs, XML config files, and wrapper bash scripts?

I believe the answer is a resounding “yes!” and that it will be well worth the effort, but there is some work to do to really get there. Lets consider the following very simple theoretical service implemented in java with a jruby based launch script:

#!/opt/dist/jruby-1.1.4/bin/jruby

require 'myservice.jar'
import 'com.foobar.MyService'

service = MyService.new
service.run

Note the use of a UNIX-standard ‘#!’ hashbang (or shebang) to reference the interpreter for the script. The UNIX shell or system exec calls will detect the hashbang and invoke the named interpreter executable, passing the script file as an argument. While this works fine with any native script interpreter (ruby, perl, python, bash, etc.) it won’t work out of the box with jruby or any other java-platform script interpreters not resorting to native code. The reason is that the referenced ‘jruby’ launcher is itself a bash script which needs to exec a java command line with the required details. On Linux at least, multiple scripts/intpreters can not be chained in this way. The work around presented for jruby is:

#!/usr/bin/env jruby

…which then requires something like:

PATH=/opt/dist/jruby-1.1.4/bin:$PATH

…to be set in a user or system global profile. Besides raising some traditional security concerns, the use of the env workaround has notable side effects. First, it introduces a PATH dependency that must be managed external to the script. Next consider wanting to set various jruby intpreter and java options made available as command line flags to jruby, for example enabling full compilation and the java max heap size:

#!/usr/bin/env jruby -X+C -J-Xmx256m

But this also doesn’t work, at least on current versions of Linux. When used as a hashbang, the jruby -X+C -J-Xmx256m is interpreted as the single parameter, and (fortunately) isn’t found in the path. Similarly the jruby script launcher supports JRUBY_OPTS as an alternative means to set these options:

#!/usr/bin/env JRUBY_OPTS='-X+C -J-Xmx256m' jruby

But again, on current versions of Linux, this approach fails even more spectacularly: it causes an infinite loop! Brilliant!

You could tune all of the java and jruby option flags by directly modifying the jruby launcher script in the jruby distribution, but then you are stuck with one set of settings per host and jruby install. Poor at best. The only practical option (short of native code) is to write a wrapper script to launch any ruby script we want to set options for:

#!/bin/bash

# Hard code myservice.rb location or have additional fun determining
# relative location via $0
exec /opt/dist/jruby-1.1.4/bin/jruby -X+C -J-Xmx256m -J-server \
     /home/david/run/myservice.rb

Isn’t this fun? We are now back where we started with the java launcher: needing to write wrapper scripts around any significant production application. Besides being a nuisance and hiding important details in a seperate wrapper script file, this approach is particularly at odds with the bin script support offered by RubyGems. Without a native launcher like Hashdot, does this lack of exec integration not deminish jruby by comparison, not with java, but with MRI (native) ruby?

In my next post on Hashdot, I’ll demonstrate how Hashdot solves these and other historic blemishes and limitations of java and java-platform script interpreters like JRuby.