Gems in a Jar with RedBridge

Using JRuby embed API (RedBridge), it is easy to use Ruby gems from Java. However, packaging might not be simple, so people occasionally struggle to create a "portable" package. Being portable is important for a Java app. All stuffs of the Java app should be packaged in a jar archive, which should work on another PC, or even different OS. This is the good side of Java. This blog illustrates one solution for packaging.

I intentionally didn't use jruby command since the command hides what are going on behind. You might be exhausted by lines of "java -jar ....," sorry. But, it helps to understand a packaging process.

Directories

Since I used bundler to install gems, I created directories below to fit them to bundler.

Linden -+- lib -+- jruby -+- 1.8
        +- src -+- linden
        +- build

"Linden" is a top directory and the name doesn't have any special meaning. You can name it whatever you like. "lib/jruby/1.8" is the directory gems will be installed. "src" is for Java code, and "build" is for compiled *.class files.

Bundler Installation

I intentionally used jruby-complete.jar to assure gems won't be installed in the default location. The path to jruby-complete.jar can be anything since typing a full path to jruby-complete.jar works. But, I like a short path to type, so I copied jruby-complete.jar under "Linden/lib." Now, the directories/files were as in below:

Linden -+- lib -+- jruby -+- 1.8
                +- jruby-complete.jar
        +- src -+- linden
        +- build

Then, the bundler installation went:

cd lib
java -jar jruby-complete.jar -S gem install bundler jruby-openssl --no-ri --no-rdoc -i jruby/1.8

Bundler and jruby-openssl gems were installed and the directories became as in below:

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
                |                 +- cache -+- ...
                |                 +- doc
                |                 +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
                |                 |        +- bundler-1.0.3 -+- ...
                |                 |        +- jruby-openssl-0.7.1 -+- ...
                |                 +- specifications -+- ...
                +- jruby-complete.jar
        +- src -+- linden
        +- build

Let's see whether gems are really installed.

export GEM_PATH=`pwd`/jruby/1.8
java -jar jruby-complete.jar -S gem list

This command should print out the installed three gems and pre-installed gems.

*** LOCAL GEMS ***

bouncy-castle-java (1.5.0145.2)
bundler (1.0.3)
columnize (0.3.1)
jruby-openssl (0.7.1)
rake (0.8.7)
rspec (1.3.0)
ruby-debug (0.10.3)
ruby-debug-base (0.10.3.2)
sources (0.0.1)

OK. Bundler was installed successfully.

Installation of other gems

Next step is to install other gems using bundler, so I used bundle command. Theoretically, PATH environment variable should work to find bundle command by ruby. Unfortunately, this didn't work in my "java -jar ..." usage. Instead, I typed a path to bundle command after -S option.

java -jar jruby-complete.jar -S jruby/1.8/bin/bundle init

Then, I added "twitter" gem to Gemfile.

# A sample Gemfile
source "http://rubygems.org"

# gem "rails"

gem "twitter"

The installation went on:

java -Xmx500m -jar jruby-complete.jar -S jruby/1.8/bin/bundle install --path=.

The java command option "-Xmx500m" is for performance and to avoid memory outage. Now, 6 gems were installed and the directories/files became:

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
                |                 |       +- httparty
                |                 |       +- oauth
                |                 +- cache -+- ...
                |                 +- doc
                |                 +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
                |                 |        +- bundler-1.0.3 -+- ...
                |                 |        +- crack-0.1.8 -+- ...
                |                 |        +- hashie-0.4.0 -+- ...
                |                 |        +- httparty-0.6.1 -+- ...
                |                 |        +- jruby-openssl-0.7.1 -+- ...
                |                 |        +- multi_json-0.0.4 -+- ...
                |                 |        +- oauth-0.4.3 -+- ...
                |                 |        +- twitter-0.9.12 -+- ...
                |                 +- specifications -+- ...
                +- jruby-complete.jar
                +- Gemfile
                +- Gemfile.lock
        +- src -+- linden
        +- build

Java code to use twitter gem

Since all gems were ready, I wrote Java code to see gems worked. This time, I relied on GEM_PATH environment variable to find gems. The fist code had Java package, linden, and class name, SearchSample. So, I created SearchSample.java file under "src/linden" directory.

package linden;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.ScriptingContainer;

public class SearchSample {
    private String jarname = "sample.jar";
    
    private SearchSample() {
        String basepath = System.getProperty("user.dir");
        System.out.println("basepath: " + basepath);
        ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
        System.out.println("jrubyhome: " + container.getHomeDirectory());
        container.runScriptlet("ENV['GEM_PATH']='" + basepath + "/lib/jruby/1.8'");
 
        String script = 
            "require 'rubygems'\n" +
            "require 'twitter'\n" +
            "require 'pp'\n" +
            "pp Twitter::Search.new('#jruby').fetch.results.first";
        container.runScriptlet(script);
    }

    public static void main(String[] args) {
        new SearchSample();
    }
}

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
                |                 |       +- httparty
                |                 |       +- oauth
                |                 +- cache -+- ...
                |                 +- doc
                |                 +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
                |                 |        +- bundler-1.0.3 -+- ...
                |                 |        +- crack-0.1.8 -+- ...
                |                 |        +- hashie-0.4.0 -+- ...
                |                 |        +- httparty-0.6.1 -+- ...
                |                 |        +- jruby-openssl-0.7.1 -+- ...
                |                 |        +- multi_json-0.0.4 -+- ...
                |                 |        +- oauth-0.4.3 -+- ...
                |                 |        +- twitter-0.9.12 -+- ...
                |                 +- specifications -+- ...
                +- jruby-complete.jar
                +- Gemfile
                +- Gemfile.lock
        +- src -+- linden -+- SearchSample.java
        +- build

Compile and run using rake-ant integration

Everything was ready. OK, how do I compile and run it? Of course, classic "javac" and "java" command were the options. But, these commands are not very convenient to repeat compile/run. So, I used JRuby's rake-ant integration. Yes, I wrote "Rakefile" to compile and run Java code. Isn't it nice? Here's Rakefile:

require 'ant'

namespace :ant do
  task :compile => :clean do
    ant.javac :srcdir => "src", :destdir => "build"
  end
end

namespace :ant do
  task :java => :compile do
    ant.java :classname => "linden.SearchSample" do
      classpath do
        pathelement :location => "lib/jruby-complete.jar"
        pathelement :path => "build"
      end
    end
  end
end

require 'rake/clean'

CLEAN.include '*.class', '*.jar'

I created Rakefile under the Linden directory, so now the directories/files were:

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
                |                 |       +- httparty
                |                 |       +- oauth
                |                 +- cache -+- ...
                |                 +- doc
                |                 +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
                |                 |        +- bundler-1.0.3 -+- ...
                |                 |        +- crack-0.1.8 -+- ...
                |                 |        +- hashie-0.4.0 -+- ...
                |                 |        +- httparty-0.6.1 -+- ...
                |                 |        +- jruby-openssl-0.7.1 -+- ...
                |                 |        +- multi_json-0.0.4 -+- ...
                |                 |        +- oauth-0.4.3 -+- ...
                |                 |        +- twitter-0.9.12 -+- ...
                |                 +- specifications -+- ...
                +- jruby-complete.jar
                +- Gemfile
                +- Gemfile.lock
        +- src -+- linden -+- SearchSample.java
        +- build
        +- Rakefile

When I typed "rake ant:java", the Java code above worked and printed out one tweet.

java -jar lib/jruby-complete.jar -S rake ant:java
(in /Users/yoko/Works/tmp/Linden)
basepath: /Users/yoko/Works/tmp/Linden
jrubyhome: file:/Users/yoko/Works/tmp/Linden/lib/jruby-complete.jar!/META-INF/jruby.home
{"profile_image_url"=>
  "http://a3.twimg.com/profile_images/76084835/web-profile_normal.jpg",
 "created_at"=>"Fri, 22 Oct 2010 15:38:09 +0000",
 "from_user"=>"mccrory",
 "metadata"=>{"result_type"=>"recent"},
 "to_user_id"=>nil,
 "text"=>
  "RT @carlosqt: #IronRuby 1.1.1 a released! download from: http://ironruby.codeplex.com/ #programming #JRuby #Ruby #Rails #dotnet #code",
 "id"=>28415816774,
 "from_user_id"=>1757577,
 "geo"=>nil,
 "iso_language_code"=>"en",
 "source"=>"<a href="http://twitter.com/">web</a>"}

Packaging

This time I can't rely on GEM_PATH environment variable. If GEM_PATH had worked also for the path in a jar, unfortunately, it didn't. Instead of GEM_PATH, I set all path to gems to ScriptingContainer. This was not complicated since all gems were installed in the same directory. I fixed the jar name, "sample.jar," so this name appeared in the Java code. This name could have been given via a command line argument. Edited SearchSample.java is in below:

package linden;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
import java.util.zip.ZipEntry;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.ScriptingContainer;

public class SearchSample {
    private String jarname = "sample.jar";
    
    private SearchSample() throws IOException {
        String basepath = System.getProperty("user.dir");
        System.out.println("basepath: " + basepath);
        ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
        System.out.println("jrubyhome: " + container.getHomeDirectory());
        container.setLoadPaths(getGemPaths(jarname, basepath));
 
        String script = 
            "require 'rubygems'\n" +
            "require 'twitter'\n" +
            "require 'pp'\n" +
            "pp Twitter::Search.new('#jruby').fetch.results.first";
        container.runScriptlet(script);
    }
    
    private List<String> getGemPaths(String jarname, String basepath) throws IOException {
        JarFile jarFile = new JarFile(basepath + "/" + jarname);
        Enumeration<JarEntry> entries = jarFile.entries();
        String gempath = "lib/jruby/1.8/gems/";
        Set<String> gemnames = new HashSet<String>();
        while (entries.hasMoreElements()) {
            ZipEntry entry = (ZipEntry) entries.nextElement();
            String entryName = entry.getName();
            if (entryName.startsWith(gempath) && entryName.length() > gempath.length()) {
                String n  = entryName.substring(gempath.length());
                String m = n.substring(0, n.indexOf("/"));
                gemnames.add(m);
            }
        }
        List<String> gemPaths = new ArrayList<String>();
        for (String gem : gemnames) {
            gemPaths.add("file:" + basepath + "/" + jarname + "!/lib/jruby/1.8/gems/" + gem + "/lib");
        }
        return gemPaths;
    }

    public static void main(String[] args) throws IOException {
        new SearchSample();
    }
}

To make jar archive, I added jar task to my Rakefile.

require 'ant'

namespace :ant do
  task :compile => :clean do
    ant.javac :srcdir => "src", :destdir => "build"
  end
end

namespace :ant do
  task :java => :compile do
    ant.java :classname => "linden.SearchSample" do
      classpath do
        pathelement :location => "lib/jruby-complete.jar"
        pathelement :path => "build"
      end
    end
  end
end

namespace :ant do
  task :jar => :compile do
    ant.jar :basedir => ".", :destfile => "sample.jar" do
      fileset :dir => "build" do
          include :name => "**/*.class"
      end
      include :name => "lib/jruby/1.8/gems/**/*"
      manifest do
        attribute :name => "Main-Class", :value => "linden.SearchSample"
      end
    end
  end
end

require 'rake/clean'

CLEAN.include '*.class', '*.jar'

All right, let's create jar archive.

java -jar lib/jruby-complete.jar -S rake ant:jar

The ant:jar task created the sample.jar archive in Linden directory. To test this archive was really gems in a jar, I unset GEM_PATH environment variable first. Then, I typed java command and got one tweet, Yay!

unset GEM_PATH; echo $GEM_PATH
java -cp lib/jruby-complete.jar:sample.jar linden.SearchSample

To ensure that the jar archive had gems in the jar, I moved sample.jar to a different directory and typed java command with the full path to jruby-complete.jar. It worked.

cp sample.jar ../.
cd ..
java -cp Linden/lib/jruby-complete.jar:sample.jar linden.SearchSample

This might be one answer of packaging gems in a jar and using gems from RedBridge.