The second step to Rails on RedBridge

This is the second attempt to make Rails work on RedBrdige (JRuby embedding API). I believe Rails on RedBridge got closer to a real application. The first attempt is also in this blog, A small step to Rails on RedBridge. The blog post had some impact on a few people who want to control Rails (or Sinatra) from Java. The example there successfully showed how we could wake Rails up from Java Servlet, but was almost a hello world example. Meanwhile, I demonstrated simple Rails app from Groovy (Groovy Servlet, precisely) in my session at RubyConf 2010, "RubyGems to All JVM Languages." Rails on RedBridge way is an approach from Java API on Servlet, so all JVM languages are possibly available to use Rails *gem* from them over JRuby's Java API. That demonstration at RubyConf 2010 impressed the audiences there enough to have a clap, but I knew it was still a hello world example. Recently, I tried to develop Rails on RedBridge more so that Rails app got work like other rackup style frameworks. RailsCrossing is the outcome. It is the Java API to use Rails.


OK. I'm going to write how to setup and write code using RailsCrossing.


1. Creating Java Web Application Project

The first job is to create a Java web application project. I used NetBeans for that. You can use whatever IDE you like that has the feature to create a Java Servlet based web application project. My web app project has a name "Spruce" and the server to run it is Tomcat 7.0.4.

Initially, a directory tree of the Spruce project looks like below (I didn't write NetBeans specific files/directories):

Spruce -+- src
        +- web -+- META-INF -+- context.xml
                +- WEB-INF
                +- index.jsp

The directory tree is slightly different when other IDEs are used. That's not the matter. Just look at WEB-INF directory. The WEB-INF directory is special for Java Servlet web app. It is the place to put libraries and is protected from accesses originated from web clients.


2. Prepare for Rails APP

RailsCrossing assumes Rails app is located under WEB-INF directory. It could not be necessarily there. However, I made that constraint since Java Servlet web application should be portable. Everything needed to the web app work should be in the web app directory tree. It's the policy.

Let's prepare the app.

First, get jruby-complete.jar and put it in WEB-INF/lib directory. (Get the latest (master) branch of JRuby. Or, you can use jruby-complete.jar of RailsCrossing project)

Spruce -+- src
        +- web -+- META-INF -+- context.xml
                +- WEB-INF -+- lib -+- jruby-complete.jar
                +- index.jsp

Next, install bundler gem. Make sure bundler will be installed under WEB-INF directory. So, let's go to lib directory and type as in below (you can use jruby command instead):

$ java -jar jruby-complete.jar -S gem install bundler --no-ri --no-rdoc -i jruby/1.8
Fetching: bundler-1.0.10.gem (100%)
Successfully installed bundler-1.0.10
1 gem installed

Don't forget -i option to install bundler gem under lib directory. The command above installs bundler gem in WEB-INF/lib/jruby/1.8. So, you can find bundle command in the path, WEB-INF/lib/jruby/1.8/bin/bundle.

Spruce -+- src
        +- web -+- META-INF -+- context.xml
                +- WEB-INF -+- lib -+- jruby-complete.jar
                |                   +- jruby -+- 1.8 -+- bin -+- bundle
                |                                     +- cache -+- ...
                |                                     +- doc
                |                                     +- gem -+- bundler-1.0.10 -+- ..
                |                                     +- specifications -+- ...
                | 
                +- index.jsp

Let's set GEM_PATH to use bundler to install other gems.

$ export GEM_PATH=[path to lib directory]/jruby/1.8

Write or create Gemfile. If you want to use bundle command rather than open a new file by editor, type as in below:

java -jar jruby-complete.jar -S jruby/1.8/bin/bundle init

Gemfile should be something like this:

# A sample Gemfile
source "http://rubygems.org"

gem "rails"

I just deleted the comment sign (#) on the rails line after bundle init command. The last preparation is to install rails gem.

java -Xmx500m -jar jruby-complete.jar -S jruby/1.8/bin/bundle install --path .

Don't forget "--path ." to install gems under the same path as bundler has been installed in. When bundle install command successfully finished, bunch of gems are in jruby/1.8.

Spruce -+- src
        +- web -+- META-INF -+- context.xml
                +- WEB-INF -+- lib -+- Gemfile
                |                   +- Gemfile.lock
                |                   +- jruby-complete.jar
                |                   +- jruby -+- 1.8 -+- bin -+- bundle
                |                                     |       +- erubis
                |                                     |       +- ....
                |                                     +- cache -+- ...
                |                                     +- doc
                |                                     +- gem -+- abstract-1.0.0 -+- ..
                |                                     +- gem -+- .... -+- ..
                |                                     +- gem -+- bundler-1.0.10 -+- ..
                |                                     +- gem -+- .... -+- ..
                |                                     +- specifications -+- ...
                | 
                +- index.jsp

3. Create Rails App

Probably, I don't need to say anything. I typed below:

java -jar jruby-complete.jar -S jruby/1.8/bin/rails new blog --template http://jruby.org

Now, blog Rails app is created under WEB-INF/lib/blog.

Spruce -+- src
        +- web -+- META-INF -+- context.xml
                +- WEB-INF -+- lib -+- Gemfile
                |                   +- Gemfile.lock
                |                   +- blog -+- Gemfile
                |                   |        +- README
                |                   |        +- Rakefile
                |                   |        +- ...
                |                   +- jruby-complete.jar
                |                   +- jruby -+- 1.8 -+- bin -+- bundle
                |                                     |       +- erubis
                |                                     |       +- ....
                |                                     +- cache -+- ...
                |                                     +- doc
                |                                     +- gem -+- abstract-1.0.0 -+- ..
                |                                     +- gem -+- .... -+- ..
                |                                     +- gem -+- bundler-1.0.10 -+- ..
                |                                     +- gem -+- .... -+- ..
                |                                     +- specifications -+- ...
                | 
                +- index.jsp

In general, database setup follows. But, to make it understandable to know how it works, let's create a controller.

$ cd blog (Now, I'm in Spruce/web/WEB-INF/lib/blog directory)
$ java -Xmx500m -jar ../jruby-complete.jar ../jruby/1.8/bin/bundle install --path ../.  (Don't forget --path ../. option!)
$ java -jar ../jruby-complete.jar script/rails g controller home index

So far so good. These steps to creating Rails app should be familiar with.


4. Simple Sample Servlet to run Rails' controller

Here comes RailsCrossing API. You can write Servlet using methods in CrossingHelpers or subclassing CrossingServlet. The CrossingServlet does everything to run Rails' controller. For example, SimpleSample below would be the simplest example.

package com.servletgarden.spruce;

import com.servletgarden.railsxing.CrossingServlet;
import java.io.IOException;
import javax.servlet.Servlet;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.annotation.WebInitParam;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**
 *
 * @author Yoko Harada 
 */
@WebServlet(name="SimpleSample",
        urlPatterns={"/*"},
        initParams={@WebInitParam(name="rails_path", value="/lib/blog"), @WebInitParam(name="gem_path", value="/lib/jruby/1.8")})
public class SimpleSample extends CrossingServlet {
    
    /**
     * @see Servlet#init(ServletConfig)
     */
    @Override
    public void init(ServletConfig config) throws ServletException {
        super.init(config);
    }
   
    /** 
     * Processes requests for both HTTP GET and POST methods.
     * @param request servlet request
     * @param response servlet response
     * @throws ServletException if a servlet-specific error occurs
     * @throws IOException if an I/O error occurs
     */
    protected void processRequest(HttpServletRequest request, HttpServletResponse response)
    throws ServletException, IOException {
        
        dispatch(request, response);
    } 

    /** 
     * Handles the HTTP GET method.
     * @param request servlet request
     * @param response servlet response
     * @throws ServletException if a servlet-specific error occurs
     * @throws IOException if an I/O error occurs
     */
    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response)
    throws ServletException, IOException {
        processRequest(request, response);
    } 

    /** 
     * Handles the HTTP POST method.
     * @param request servlet request
     * @param response servlet response
     * @throws ServletException if a servlet-specific error occurs
     * @throws IOException if an I/O error occurs
     */
    @Override
    protected void doPost(HttpServletRequest request, HttpServletResponse response)
    throws ServletException, IOException {
        processRequest(request, response);
    }

    /** 
     * Returns a short description of the servlet.
     * @return a String containing servlet description
     */
    @Override
    public String getServletInfo() {
        return "the simplest example used RailsCrossing";
    }

}

Let's look at what's going on in detail.

In the SimpleSample init method, it calls init method of a super class, CrossingServlet. In CrossingServlet init method, initialization of ScriptingContainer (JRuby embedding API a.k.a. RedBridge) and Rails are done. Then, Rails routes mappings are parsed.

    public void init(ServletConfig config) throws ServletException {
        container = CrossingHelpers.initialize(config);
        routes = CrossingHelpers.parseRoutes(container);
    }

While initialization is going on CrossingServlet uses "rails_path" and "gem_path" initParams relative to WEB-INF directory. These are necessary to fire up Rails.


Well, how Rails routes mapping can be parsed? Probably, many of you have used rails console. RailsCrossing does the same thing on a Servlet as people do on rails console. For example,

$ java -jar ../jruby-complete.jar script/rails c
irb(main):007:0> route_ary = ActionDispatch::Routing::Routes.routes
=> [#<ActionDispatch::Routing::Route:0x555669ae @name="home_index"...(snip)
irb(main):004:0> route_ary[0].to_a[1]
=> {:path_info=>/\A\/home\/index(?:\.([^\/.?]+))?\Z/, :request_method=>/^GET$/}
irb(main):005:0> route_ary[0].to_a[2]
=> {:controller=>"home", :action=>"index"}
irb(main):006:0> route_ary[0].to_a[3]
=> "home_index"

CrossingHelpers.parseRoutes method exactly evaluates these commands to draw mapping info out from Rails.


Let get back to SimpleSample Servlet. Look at the processRequest method. In this method, CrossingServlet's dispatch method is called. Just that. What's this dispatch?
Here it is:

    protected void dispatch(HttpServletRequest request, HttpServletResponse response) throws IOException {
        CrossingRoute route = findMatchedRoute(request.getContextPath(), request.getPathInfo(), request.getMethod());
        if (route == null) return;
        Map env = getEnvMap(request);
        CrossingResponse crossingResponse =  CrossingHelpers.dispatch(container, route, env);
        response.setStatus(crossingResponse.getStatus());
        Set keys = crossingResponse.getResponseHeader().keySet();
        for (String key : keys) {
            String value = crossingResponse.getResponseHeader().get(key);
            response.setHeader(key, value);
        }
        PrintWriter writer = response.getWriter();
        writer.write(crossingResponse.getBody());
    }

Based on the parsed routes, CrossingServlet finds what controller should be used from path_info in HttpServletRequest. If matched controller is found, it creates a map of each HTTP request params including HTTP request header and query string. Then, dispatching. Here, RailsCrossing does rails console commands on Servlet again.

irb(main):006:0> env = {"rack.input" => "", "REQUEST_METHOD" => "GET"}
=> {"rack.input"=>"", "REQUEST_METHOD"=>"GET"}
irb(main):008:0> HomeController.action('index').call(env)[0]     
=> 200
irb(main):009:0> HomeController.action('index').call(env)[1]
=> {"Content-Type"=>"text/html; charset=utf-8", "ETag"=>"\"a5e60d2fa2208dc316b0f09cef107bba\"", "Cache-Control"=>"max-age=0, private, must-revalidate"}
irb(main):010:0> HomeController.action('index').call(env)[2].body
=> "<!DOCTYPE html>\n<html>\n<head>\n  <title>Blog</title>\n...

Once CrossingServlet gets the response, it sets HTTP response status code and header params, then writes html part out to Servlet's writer.

These are what RaisCrossing does.


5. One More Setup

The difference of a context path lies between Java web app and Rails app. In general, Java web application is referenced by http://servername:port/context_name/servlet_name/path_info, while rails is done by http://servername:port/path_info. We need to fix these to make them coincide.

Let's change Rails first editing config/routes.rb:

Blog::Application.routes.draw do
scope "/blog" do
  get "home/index"
  ...
end
end

I added scope in config/routes.rb. Now, blog app is referenced by http://servername:port/blog/path_info. Next is a tomcat setting. I edited META_INF/context.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Context antiJARLocking="true" path="/blog"/>

Originally, it was path="/Spruce" since I named the web app Spruce. After Spruce has been changed to "blog," every HTTP request to http://servername:port/blog/* has been directed to SimpleSample Servlet.


6. Run Servlet

Since I created web app on NetBeans, I can get SimpleSample Servlet started from NetBeans menu with the path, "/home/index" If not, start tomcat up and request "http://localhost:8080/blog/home/index" You'll see familiar controller default output on a browser.


7. Thoughts

RailsCrossing uses Rails metal introduced in version 3. This rack style invocation was easy to get it run. However, it is mostly undocumented area. I attempted many commands on rails console and tried to find the best way; however, there might be still better way.

A good side of RailsCrossing is ... it is a Java API. This means, JVM languages such as Scala, Groovy, or other can use RailsCrossing over java integration feature. I'll try Rails from Scala next.

Embedding API refinements for JRuby 1.6.0RC1

TThe biggest release ever --- JRuby 1.6.0RC1 has been released on Jan. 10, 2011. Headius wrote a blog about this biggest-ever release, JRuby 1.6.0 RC1 Released. Ruby 1.9 compatibility, performance improvement, built-in graph profiler, and more. How about embedding API (RedBridge)? In the release note (http://jruby.org/2011/01/10/jruby-1-6-0-rc1.html), you'll find a line mentioned about RedBridge, "Embedding API refinements." I'm going to write what refinements RedBridge had in JRuby 1.6.0RC1 in this blog post.


1. New Context Instance Type, concurrent

So far, RedBridge has had three types of context models, singleton, singlethread, and threadsafe. JRuby 1.6.0 added one more type, concurrent. The concurrent type is somehow mixture of singleton and threadsafe. Ruby runtime is a singleton, variables (except global variables) are thread local. This model would be good to use RedBridge on servlet based web applications. For example, we can instantiate Ruby runtime and evaluate all Ruby codes during servlet’s initialization, then use them in HTTP request processing methods, such as doGet() or doPost(). Threadsafe context model is similar, but Ruby runtimes are on each thread (each HTTP request on a web app). On the other hand, the runtime in the same classloader will be used on the concurrent context model on all threads, if Ruby runtime has been already initialized by another app.

The concurrent model is a bit complicated and assuming users have enough knowledge about Java web application and Ruby's variable/constant scope. Global variables are totally unprotected from concurrent processing since Ruby runtime is a singleton. As you may know the scope of a global variable is global on Ruby runtime, so is the concurrent context. Global variables of concurrent model are looked up from all threads (all HTTP requests on a web app) simultaneously. The scope of other variable types and a constant depends on how those are used. If those are top-level variables/constants, the scope will be global since Runtime is a singleton. If not, those are thread local and protected from concurrent processing.

When you use the concurrent context model, specify as in below:

Embed core
ScriptingContainer container = new ScriptingContainer(LocalContextScope.CONCURRENT);

JSR223
System.property(“org.jruby.embed.localcontext.scope”, “concurrent”);

2. Receiver aware sharing variables

I’ve written about this new feature in the blog post, http://yokolet.blogspot.com/2010/09/new-featues-of-embedding-api-for-jruby.html ScriptingContainer.get()/put()/remove() methods had a receiver object in those method arguments. If the receiver object is not specified, runtime top-self is used as a receiver. This means those operations are not thread safe on concurrent context type. Be careful.
Also, see the lazy variable retrieval feature of the same blog post. This is for performance improvement.


3. Classloader option for JSR223

The new option to set classloader into ScriptingContainer via ScriptEngineFactory has been added from JRuby 1.6.0RC1. This is for users who want to use JSR223 ScriptEngine for JRuby on some custom classloader. Setting a classloader of ScriptingContainer itself to Ruby runtime is for now well-known technique on custom classloader. The problem was JSR223 doesn’t have a method for that. So, JRuby 1.6.0 sets the classloader that is used to instantiate ScriptingContainer to Ruby runtime by default. This is done in ScriptEngineFactory#getScriptEngine() method internally. This behavior can be disabled by a system property, org.jruby.embed.classloader.

System.setProperty(“org.jruby.embed.classloader”, “current”); // default, sets classloader
System.setProperty(“org.jruby.embed.classloader”, “none”); // doesn't set classloader

The option is JSR223 only. The org.jruby.embed.classloader system property has no effect when you directly instantiate ScriptingContainer. Set whatever appropriate classloader using ScriptingContainer.setClassLoader() method.


4. New Map proxy

This is not a embedding API but JRuby core. Also the feature is not only for RedBridge users but also for all JRuby users. However, the new proxy would make RedBridge users happy especially, so I’ll add this feature here.

Since JRuby 1.6.0RC1, a java.util.Map type object has all Ruby Hash methods by default. Let’s see example code.

ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
ConcurrentHashMap map1 = new ConcurrentHashMap();
map1.put("a", 100);
map1.put("b", 200);
Map map2 = new HashMap();
map2.put("b", 254);
map2.put("c", 300);
container.put("map1", map1);
container.put("map2", map2);
String script =
    "map1.merge!(map2) {|k,o,n| o+n }\n" +
    "puts \"#{map1.inspect}, length = #{map1.length}\"\n" +
    "puts \"class: #{map1.class}\"";
container.runScriptlet(script);

Set entries = map1.entrySet();
for (Map.Entry entry : entries) {
    System.out.print(entry.getKey() + ": " + entry.getValue() + ", ");
}

Above code prints:

{"a"=>100, "b"=>454, "c"=>300}, length = 3
class: Java::JavaUtilConcurrent::ConcurrentHashMap
a: 100, b: 454, c: 300,

Two Map type objects were merged as instructed, map1.merge!(map2) {|k,o,n| o+n }. Also in Java world, map1 was changed exactly the same way as was in Ruby world.


These are the brief summary of embedding API refinement done in JRuby 1.6.0RC1. Have fun and please report if you find a bug or encounter a weird behavior.

Gems in a Jar with RedBridge

Using JRuby embed API (RedBridge), it is easy to use Ruby gems from Java. However, packaging might not be simple, so people occasionally struggle to create a "portable" package. Being portable is important for a Java app. All stuffs of the Java app should be packaged in a jar archive, which should work on another PC, or even different OS. This is the good side of Java. This blog illustrates one solution for packaging.

I intentionally didn't use jruby command since the command hides what are going on behind. You might be exhausted by lines of "java -jar ....," sorry. But, it helps to understand a packaging process.

Directories

Since I used bundler to install gems, I created directories below to fit them to bundler.

Linden -+- lib -+- jruby -+- 1.8
        +- src -+- linden
        +- build

"Linden" is a top directory and the name doesn't have any special meaning. You can name it whatever you like. "lib/jruby/1.8" is the directory gems will be installed. "src" is for Java code, and "build" is for compiled *.class files.

Bundler Installation

I intentionally used jruby-complete.jar to assure gems won't be installed in the default location. The path to jruby-complete.jar can be anything since typing a full path to jruby-complete.jar works. But, I like a short path to type, so I copied jruby-complete.jar under "Linden/lib." Now, the directories/files were as in below:

Linden -+- lib -+- jruby -+- 1.8
                +- jruby-complete.jar
        +- src -+- linden
        +- build

Then, the bundler installation went:

cd lib
java -jar jruby-complete.jar -S gem install bundler jruby-openssl --no-ri --no-rdoc -i jruby/1.8

Bundler and jruby-openssl gems were installed and the directories became as in below:

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
                |                 +- cache -+- ...
                |                 +- doc
                |                 +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
                |                 |        +- bundler-1.0.3 -+- ...
                |                 |        +- jruby-openssl-0.7.1 -+- ...
                |                 +- specifications -+- ...
                +- jruby-complete.jar
        +- src -+- linden
        +- build

Let's see whether gems are really installed.

export GEM_PATH=`pwd`/jruby/1.8
java -jar jruby-complete.jar -S gem list

This command should print out the installed three gems and pre-installed gems.

*** LOCAL GEMS ***

bouncy-castle-java (1.5.0145.2)
bundler (1.0.3)
columnize (0.3.1)
jruby-openssl (0.7.1)
rake (0.8.7)
rspec (1.3.0)
ruby-debug (0.10.3)
ruby-debug-base (0.10.3.2)
sources (0.0.1)

OK. Bundler was installed successfully.

Installation of other gems

Next step is to install other gems using bundler, so I used bundle command. Theoretically, PATH environment variable should work to find bundle command by ruby. Unfortunately, this didn't work in my "java -jar ..." usage. Instead, I typed a path to bundle command after -S option.

java -jar jruby-complete.jar -S jruby/1.8/bin/bundle init

Then, I added "twitter" gem to Gemfile.

# A sample Gemfile
source "http://rubygems.org"

# gem "rails"

gem "twitter"

The installation went on:

java -Xmx500m -jar jruby-complete.jar -S jruby/1.8/bin/bundle install --path=.

The java command option "-Xmx500m" is for performance and to avoid memory outage. Now, 6 gems were installed and the directories/files became:

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
                |                 |       +- httparty
                |                 |       +- oauth
                |                 +- cache -+- ...
                |                 +- doc
                |                 +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
                |                 |        +- bundler-1.0.3 -+- ...
                |                 |        +- crack-0.1.8 -+- ...
                |                 |        +- hashie-0.4.0 -+- ...
                |                 |        +- httparty-0.6.1 -+- ...
                |                 |        +- jruby-openssl-0.7.1 -+- ...
                |                 |        +- multi_json-0.0.4 -+- ...
                |                 |        +- oauth-0.4.3 -+- ...
                |                 |        +- twitter-0.9.12 -+- ...
                |                 +- specifications -+- ...
                +- jruby-complete.jar
                +- Gemfile
                +- Gemfile.lock
        +- src -+- linden
        +- build

Java code to use twitter gem

Since all gems were ready, I wrote Java code to see gems worked. This time, I relied on GEM_PATH environment variable to find gems. The fist code had Java package, linden, and class name, SearchSample. So, I created SearchSample.java file under "src/linden" directory.

package linden;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.ScriptingContainer;

public class SearchSample {
    private String jarname = "sample.jar";
    
    private SearchSample() {
        String basepath = System.getProperty("user.dir");
        System.out.println("basepath: " + basepath);
        ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
        System.out.println("jrubyhome: " + container.getHomeDirectory());
        container.runScriptlet("ENV['GEM_PATH']='" + basepath + "/lib/jruby/1.8'");
 
        String script = 
            "require 'rubygems'\n" +
            "require 'twitter'\n" +
            "require 'pp'\n" +
            "pp Twitter::Search.new('#jruby').fetch.results.first";
        container.runScriptlet(script);
    }

    public static void main(String[] args) {
        new SearchSample();
    }
}

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
                |                 |       +- httparty
                |                 |       +- oauth
                |                 +- cache -+- ...
                |                 +- doc
                |                 +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
                |                 |        +- bundler-1.0.3 -+- ...
                |                 |        +- crack-0.1.8 -+- ...
                |                 |        +- hashie-0.4.0 -+- ...
                |                 |        +- httparty-0.6.1 -+- ...
                |                 |        +- jruby-openssl-0.7.1 -+- ...
                |                 |        +- multi_json-0.0.4 -+- ...
                |                 |        +- oauth-0.4.3 -+- ...
                |                 |        +- twitter-0.9.12 -+- ...
                |                 +- specifications -+- ...
                +- jruby-complete.jar
                +- Gemfile
                +- Gemfile.lock
        +- src -+- linden -+- SearchSample.java
        +- build

Compile and run using rake-ant integration

Everything was ready. OK, how do I compile and run it? Of course, classic "javac" and "java" command were the options. But, these commands are not very convenient to repeat compile/run. So, I used JRuby's rake-ant integration. Yes, I wrote "Rakefile" to compile and run Java code. Isn't it nice? Here's Rakefile:

require 'ant'

namespace :ant do
  task :compile => :clean do
    ant.javac :srcdir => "src", :destdir => "build"
  end
end

namespace :ant do
  task :java => :compile do
    ant.java :classname => "linden.SearchSample" do
      classpath do
        pathelement :location => "lib/jruby-complete.jar"
        pathelement :path => "build"
      end
    end
  end
end

require 'rake/clean'

CLEAN.include '*.class', '*.jar'

I created Rakefile under the Linden directory, so now the directories/files were:

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
                |                 |       +- httparty
                |                 |       +- oauth
                |                 +- cache -+- ...
                |                 +- doc
                |                 +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
                |                 |        +- bundler-1.0.3 -+- ...
                |                 |        +- crack-0.1.8 -+- ...
                |                 |        +- hashie-0.4.0 -+- ...
                |                 |        +- httparty-0.6.1 -+- ...
                |                 |        +- jruby-openssl-0.7.1 -+- ...
                |                 |        +- multi_json-0.0.4 -+- ...
                |                 |        +- oauth-0.4.3 -+- ...
                |                 |        +- twitter-0.9.12 -+- ...
                |                 +- specifications -+- ...
                +- jruby-complete.jar
                +- Gemfile
                +- Gemfile.lock
        +- src -+- linden -+- SearchSample.java
        +- build
        +- Rakefile

When I typed "rake ant:java", the Java code above worked and printed out one tweet.

java -jar lib/jruby-complete.jar -S rake ant:java
(in /Users/yoko/Works/tmp/Linden)
basepath: /Users/yoko/Works/tmp/Linden
jrubyhome: file:/Users/yoko/Works/tmp/Linden/lib/jruby-complete.jar!/META-INF/jruby.home
{"profile_image_url"=>
  "http://a3.twimg.com/profile_images/76084835/web-profile_normal.jpg",
 "created_at"=>"Fri, 22 Oct 2010 15:38:09 +0000",
 "from_user"=>"mccrory",
 "metadata"=>{"result_type"=>"recent"},
 "to_user_id"=>nil,
 "text"=>
  "RT @carlosqt: #IronRuby 1.1.1 a released! download from: http://ironruby.codeplex.com/ #programming #JRuby #Ruby #Rails #dotnet #code",
 "id"=>28415816774,
 "from_user_id"=>1757577,
 "geo"=>nil,
 "iso_language_code"=>"en",
 "source"=>"<a href="http://twitter.com/">web</a>"}

Packaging

This time I can't rely on GEM_PATH environment variable. If GEM_PATH had worked also for the path in a jar, unfortunately, it didn't. Instead of GEM_PATH, I set all path to gems to ScriptingContainer. This was not complicated since all gems were installed in the same directory. I fixed the jar name, "sample.jar," so this name appeared in the Java code. This name could have been given via a command line argument. Edited SearchSample.java is in below:

package linden;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
import java.util.zip.ZipEntry;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.ScriptingContainer;

public class SearchSample {
    private String jarname = "sample.jar";
    
    private SearchSample() throws IOException {
        String basepath = System.getProperty("user.dir");
        System.out.println("basepath: " + basepath);
        ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
        System.out.println("jrubyhome: " + container.getHomeDirectory());
        container.setLoadPaths(getGemPaths(jarname, basepath));
 
        String script = 
            "require 'rubygems'\n" +
            "require 'twitter'\n" +
            "require 'pp'\n" +
            "pp Twitter::Search.new('#jruby').fetch.results.first";
        container.runScriptlet(script);
    }
    
    private List<String> getGemPaths(String jarname, String basepath) throws IOException {
        JarFile jarFile = new JarFile(basepath + "/" + jarname);
        Enumeration<JarEntry> entries = jarFile.entries();
        String gempath = "lib/jruby/1.8/gems/";
        Set<String> gemnames = new HashSet<String>();
        while (entries.hasMoreElements()) {
            ZipEntry entry = (ZipEntry) entries.nextElement();
            String entryName = entry.getName();
            if (entryName.startsWith(gempath) && entryName.length() > gempath.length()) {
                String n  = entryName.substring(gempath.length());
                String m = n.substring(0, n.indexOf("/"));
                gemnames.add(m);
            }
        }
        List<String> gemPaths = new ArrayList<String>();
        for (String gem : gemnames) {
            gemPaths.add("file:" + basepath + "/" + jarname + "!/lib/jruby/1.8/gems/" + gem + "/lib");
        }
        return gemPaths;
    }

    public static void main(String[] args) throws IOException {
        new SearchSample();
    }
}

To make jar archive, I added jar task to my Rakefile.

require 'ant'

namespace :ant do
  task :compile => :clean do
    ant.javac :srcdir => "src", :destdir => "build"
  end
end

namespace :ant do
  task :java => :compile do
    ant.java :classname => "linden.SearchSample" do
      classpath do
        pathelement :location => "lib/jruby-complete.jar"
        pathelement :path => "build"
      end
    end
  end
end

namespace :ant do
  task :jar => :compile do
    ant.jar :basedir => ".", :destfile => "sample.jar" do
      fileset :dir => "build" do
          include :name => "**/*.class"
      end
      include :name => "lib/jruby/1.8/gems/**/*"
      manifest do
        attribute :name => "Main-Class", :value => "linden.SearchSample"
      end
    end
  end
end

require 'rake/clean'

CLEAN.include '*.class', '*.jar'

All right, let's create jar archive.

java -jar lib/jruby-complete.jar -S rake ant:jar

The ant:jar task created the sample.jar archive in Linden directory. To test this archive was really gems in a jar, I unset GEM_PATH environment variable first. Then, I typed java command and got one tweet, Yay!

unset GEM_PATH; echo $GEM_PATH
java -cp lib/jruby-complete.jar:sample.jar linden.SearchSample

To ensure that the jar archive had gems in the jar, I moved sample.jar to a different directory and typed java command with the full path to jruby-complete.jar. It worked.

cp sample.jar ../.
cd ..
java -cp Linden/lib/jruby-complete.jar:sample.jar linden.SearchSample

This might be one answer of packaging gems in a jar and using gems from RedBridge.

pure Java Nokogiri - XSLT extension function -

Here's a memo of pure Java Nokogiri about an implementation of XSLT extension function. In short, I concluded pure Java version is unable to support Nokogiri style XSLT extension function. I've tried possible ways to make it happen, but for an inevitable reason, I settled to this conclusion. However, in future, this part might be reconsidered when XML libraries and APIs are replaced to others. For the future version of pure Java Nokogiri, I'm going to write down what I did and what was the problem. Hopefully, this memo will help to retry the implementation later.


1. What is XSLT extension function?

XSLT extension is defined in "14 Extensions" of XSL Transformations (XSLT)
Version 1.0 (http://www.w3.org/TR/xslt), which allows users to delegate an XSLT processing to a specified function/method written in a programming languages such as Ruby, Java, JavaScript. As in Nokogiri's test case,

<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:f="http://e.org/functions"
  extension-element-prefixes="f">

  <xsl:template match="text()">
    <xsl:copy-of select="f:capitalize(.)"/>
  </xsl:template>
....

the element, "extension-element-prefixes," indicates the function is tied to this namespace. This XSL file should be common to all languages used to write the function.


2. Nokogiri style function mapping

Nokogiri maps the namesapce to the function as in below:

foo = Class.new do
  def capitalize nodes
    nodes.first.content.upcase
  end
end

XSLT.register "http://e.org/functions", foo

Thus, a receiver object is registered to XSLT processor with the URL tag. It is nice Ruby friendly design. <xsl:copy-of select="f:capitalize(.)"/> above executes a "capitalize" method of the "foo" object.


3. How Java handles this?

As far as I googled about a Java way, not many documents, blogs, articles were out there. Probably, Extending XSLT with Java - Chapter 17. XSLT would be the best described one. This explains how to delegate the process to a Java method tied to the namespace. Xalan has the document, Xalan-Java Extensions; however, this uses BSF (Bean Scripting Framework: http://jakarta.apache.org/bsf/) to execute a function/method written in an XSL file. So, the first one has a possibility to realize Nokogiri style.


While I tried a couple of patterns, the "extension-element-prefixes" element seemed not to have a much meaning. Instead, "xmlns:java="http://xml.apache.org/xslt/java" and "xmlns:foo="xalan://[fully qualified class name]" worked. OK, so pure Java version of Nokogiri needs a specific rule to use XSLT extension function. This might be better than unsupported. Then, I wrote a Java class below to see whether it worked or not:

package Canna;

public class ExtensionFoo {
    public static Object exec(String method, Object value) {
        .....
    }
}

The method should be static to be called from XSLT processor. The first argument is a method name to make the style resemble to the Nokogiri way. Since I thought

foo = Class.new do
  def capitalize nodes
    nodes.first.content.upcase
  end
end

XSLT.register "http://e.org/functions", foo

xsl = Nokogiri.XSLT(<<-EOXSL)
<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:foo="xalan://nokogiri.internals.XsltExtensionFunction"
  extension-element-prefixes="foo">

  <xsl:template match="text()">
    <xsl:copy-of select="foo:exec('capitalize' .)"/>
  </xsl:template>
...

would not be a bad substitution. Users need to have just a small rule only in the xsl file.


Here are entire files that I tried XSLT extension function in action by Java.

[extension.xsl]
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:foo="xalan://Canna.ExtensionFoo"
                extension-element-prefixes="foo"
                version="1.0">
  <xsl:template match="text()">
    <xsl:copy-of select="foo:exec('capitalize', .)"/>
  </xsl:template>
</xsl:stylesheet>

[extension.xml]
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1 plus MathML 2.0//EN"
 "http://www.w3.org/TR/MathML2/dtd/xhtml-math11-f.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
  <head>
    <meta http-equiv="Content-type" content="application/xhtml+xml"/>
    <title>Foo</title>
  </head>
  <body>
    <h1>Foo</h1>
    <p>Lorem ipsum.</p>
  </body>
</html>

[ExtensionFoo.java]
package Canna;

public class ExtensionFoo {
    public static Object exec(String method, Object value) {
        if (value != null && (value instanceof String)) {
            return ((String)value).toUpperCase();
        } else {
            return "hello?";
        }
    }
}

[TransformSample.java]
package Canna;

import java.io.File;
import javax.xml.transform.Result;
import javax.xml.transform.Source;
import javax.xml.transform.Templates;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerConfigurationException;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.stream.StreamResult;
import javax.xml.transform.stream.StreamSource;

public class TransformSample {
    private static String userdir = System.getProperty("user.dir");
    private static String templateName = "extension.xsl";
    private static String documentName = "extension.xml";

    private TransformSample() throws TransformerConfigurationException, TransformerException {
        Source templateSource = new StreamSource(new File(userdir + "/ext/java/Canna/" + templateName));
        Source documentSource = new StreamSource(new File(userdir + "/ext/java/Canna/" + documentName));
        Result result = new StreamResult(System.out);
        TransformerFactory factory = TransformerFactory.newInstance();
        Templates templates = factory.newTemplates(templateSource);
        Transformer transformer = templates.newTransformer();
        transformer.transform(documentSource, result);
    }

    public static void main(String[] args) throws TransformerConfigurationException, TransformerException {
        new TransformSample();
    }
}

What was the result? The program output a bunch of "hello?"s. Why? The given object of a method argument wasn't the String type but DTMNodeProxy. What's DTMNodeProxy? This is com.sun.org.apache.xml.internal.dtm.ref.DTMNodeProxy, org.apache.xml.internal.dtm.ref.DTMNodeProxy, or another XSLT processor's internal type. The most affordable choice would be org.apache.xml.internal.dtm.ref.DTMNodeProxy, but Nokogiri needs to add xalan.jar to its jar list. Definitely, Nokogiri will be fat. Otherwise, pure Java Nokogiri will lose portability. Thankfully, there is an option. Users can convert values from XSL to Java and hand a desired type in to the method. So, I changed one line in XSL file:

<xsl:copy-of select="foo:exec('capitalize', string(.))"/>

OK, this worked. Every text became an upper case.


4. Inevitable API conflict

Although there were pure Java specific rules, Nokogiri style XSLT extension function seemed to work. However, the result was NOT. Puzzled. I moved the sample code above under Nokogir source tree then figured out the culprit.

When xercesImpl.jar or jing.jar is on a classpath, the sample code failed to parse the XSL file.

Sigh... I haven't found what's wrong with that yet, but the conflict lies there. Pure Java Nokogiri uses an internal API of Xerces for SAX and Jing for Relaxng processing. Both xercesImpl.jar and jing.jar are necessary APIs.

Probably, the best choice would be not to support XSLT extension of Nokogiri style right now. In future, pure Java Nokogiri might choose other XML APIs. Or, someone might give me a good advice to avoid the conflict. So, still, there is a possibility to make it happen later. At the time, this memo hopefully helps to restart implementing the XSLT extension feature.

New featues of embedding API for JRuby 1.6

This month, a lot of work for JRuby's embedding API (RedBridge) has been done. Mainly, bug fixing. While I was fixing bugs, I eventually landed to add a new feature and change the area of sharing variables. These will be in JRuby 1.6. Currently, snapshot is available at http://ci.jruby.org/snapshots/ if you want to test it. I believe the changes are good to use Ruby more naturally, but those might affect the code already written a little bit. That why I'm writing this. If you are the user of JRuby's embedding API or JSR223, be aware of upcoming changes.


1. Sharing variables becomes a receiver sensitive

Before the changes, embedding API didn't mind the difference of receivers. A receiver means Ruby's receiver, which is returned as a result of evaluation. Variables and constants to be shared are injected to a top level, in other words, runtime's top self. Also, those should be retrieved from the top level variables and constants. However, this logic didn't work perfectly.

Firstly, a trouble happened in sharing instance variables. The reason is that embedding API didn't use consistent receivers to inject and retrieve instance variables. When multiple receivers were involved, multiple values were assigned to the same key. This ended up unwanted results occasionally.

In light of this, I added three methods to ScriptingContainer.

get(receiver, key)
put(receiver, key, value)
remove(receiver, key, value)

These methods explicitly interact with a given receiver. Existing get/put/remove methods will have top self receivers for the argument.

Let me show you example. I'm going to use the Ruby code, tree_sample.rb below:

class Tree
  attr_accessor :name, :shape, :type
  
  def initialize name, shape, type
    @name = name
    @shape = shape
    @type = type
  end

  def name= name
    @name = name
  end

  def to_s
    "#{name.capitalize} is a(n) #{shape} shaped, #{type} tree."
  end
end

When the code is evaluated by:

ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
container.runScriptlet(PathType.CLASSPATH, "ruby/tree_sample.rb");

The runtime caches the Tree class and returns nil, which is converted to null for Java code. Then, suppose two objects are instantiated:

Object tree1 = container.runScriptlet("Tree.new('any', 'pyramidal', 'evergreen')");
Object tree2 = container.runScriptlet("Tree.new('any', 'oval', 'deciduous')");

Tree1 and tree2 above are receivers. Before the changes, container retrieved instance variables from the receivers and saved in an internal variable table tied to the instance name at the end of runScriptlet method. As you know, there're two objects. Two values of each instance variable were assigned to the single key without any receiver info. The retrieval should be receiver sensitive as well as injecting.

After the change, the instance variable values are tied to both the key and receiver. In JRuby 1.6, you'll get expected results even though multiple receivers of the same class are there. For example, suppose callMethod methods are run for each receiver object:

container.callMethod(tree1, "name=", "pine");
container.callMethod(tree2, "name=", "poplar");
System.out.println(container.callMethod(tree1, "to_s", String.class));
System.out.println(container.callMethod(tree2, "to_s", String.class));

The result is the one expected:

Pine is a(n) pyramidal shaped, evergreen tree.
Poplar is a(n) oval shaped, deciduous tree.

For clarity, let me add more lines of sharing variable related method usages:

container.put(tree1, "@name", "camellia");
container.put(tree2, "@name", "cherry");
container.put(tree1, "@shape", "oval");
container.put(tree2, "@shape", "round");

System.out.println(container.callMethod(tree1, "to_s", String.class));
System.out.println(container.callMethod(tree2, "to_s", String.class));
        
System.out.println("@type of tree1: " + container.get(tree1, "@type"));
System.out.println("@type of tree2: " + container.get(tree2, "@type"));

Above prints:

Camellia is a(n) oval shaped, evergreen tree.
Cherry is a(n) round shaped, deciduous tree.
@type of tree1: evergreen
@type of tree2: deciduous

So far, I talked about instance variables. Constants are also receiver sensitive in JRuby 1.6. Look at the code below:

ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
String script = 
    "COLOR = 'pink'\n" +
    "class ColorSample\n" +
    "  COLOR = 'orange'\n" +
    "end\n" +
    "ColorSample.new";
Object receiver = container.runScriptlet(script);
System.out.println("top level: " + container.get("COLOR"));
System.out.println("class: " + container.get(receiver, "COLOR"));

There're two constants which have the same name, COLOR. While container.get("COLOR")) gets a top level contant, container.get(receiver, "COLOR")) gets a constant from a given receiver. So, the output is:

top level: pink
class: orange

How about global and local variables? Global variables are receiver insensitive because those should be referenced globally. Local variables are always injected to/retrieved from the top level. It might have been possible to tie to the receiver, but that is not a good idea because Ruby code might use gems or third party libraries. The injected local variables from Java might be extraterrestrials for those libraries and cause unexpected results.

For JSR223 user, I added an "org.jruby.embed.receiver" attribute, but this needs more work to make it happen.


2. Variable/constant value retrieval becomes lazy by default


Before the change ScriptingContainer tried to retrieve variable/constant values as much as possible at the end of runScriptlet/callMethod and EvalUnit.run methods. This was convenient to get variables/constants defined in Ruby ready for Java. However, too many values were saved in the internal variable table. Those variables were injected to Ruby code in succeeding evaluations implicitly. Obviously, this behavior made performance and memory usage worse. When multiple gems were used, it would be serious.

In JRuby 1.6, all variables/constants are retrieved lazily except persistent local variables. This means the internal variable table will have minimum key-value pairs. When a get method of ScriptingContainer is called, the requested key-value pair is saved in the variable table. When key-value pairs have been put to the internal variable table before the evaluation, those will be updated right after the evaluation. Others are not. However, persistent local variables are exception. The values are retrieved eagerly as it was done before. This is because of the internal of JRuby and is helped by the policy that local variables to be shared are only top level ones. Probably, I can consider top level local variables are few.

The change will bring a good result; however, JSR223 users might be affected. The existence of JSR223's SimpleBindings and SimpleScriptContext are always headache to me. It's very hard to add a trick to those two. Workaround is to set false in lazy option and to use the method, ScriptEngine#getContext().getAttrubute() method. Otherwise, set a key and dummy value pair to the bindings. Then, JRubyEngine updates the value right after the evaluation. By the release of 1.6, I'll improve JSR223 support.


By this change, users program will be more natural in terms of Ruby coding even on JRuby's embedding API. My work on these has not yet finished. I'll add more tests and some measures to make these on JSR223.

My Presentation at JRubyKaigi 2010

I spoke at JRubyKaigi 2010 about JRuby embed API (RedBridge). Since attendees were almost all Japanese, I spoke in Japanese. However, my slide was written using simple English words, so that even non-Japanese speanker could understand what's going on. Don't worry about the Language of my slide.

Now, you can see my slide at http://servletgarden-point.appspot.com/slideshow. As the url shows, my slide itself was a demo built on Google App Engine using JRuby on Rails. This app uses jQuery UI and ajax. Click on buttons on the left side, then right side contents will change, and you'll see accordion. Also, you can try examples I showed in my presentation. Cloning the git repo,

    git://github.com/yokolet/jrubykaigi_examples.git

and look at README to run examples.

Cucumber on RedBridge

With JRuby's RedBridge, Ruby applications get started from Java. For example Cucumber, does. Let's see how Cucumber runs in Java code.

Firstly, people use Cucumber like this:

jruby -S cucumber addition.feature

"cucumber" is installed in $JRUBY_HOME/bin directory and looks like a command. But "cucumber" is a Ruby script, so "cucumber" can be evaluated using JRuby Embed, RedBridge, API. How about "addition.feature"? Cucumber receives the feature name via ARGV constant. This means adding a file name to ARGV will work. From these analysis, I wrote CucumberRunner.

package evergreen;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.PathType;
import org.jruby.embed.ScriptingContainer;

public class CucumberRunner {
    private String jrubyhome =  "/Users/yoko/Projects/jruby";
    private String cucumber = jrubyhome + "/bin/cucumber";
    private String feature = jrubyhome + "/lib/ruby/gems/1.8/gems/cucumber-0.8.5/examples/i18n/en/features/addition.feature";
    
    private CucumberRunner() {
        ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
        container.setHomeDirectory(jrubyhome);
        container.runScriptlet("ARGV << '" + feature + "'");
        container.runScriptlet(PathType.ABSOLUTE, cucumber);
    }

    public static void main(String[] args) {
        new CucumberRunner();
    }
}

This printed out:

# language: en
Feature: Addition
  In order to avoid silly mistakes
  As a math idiot
  I want to be told the sum of two numbers

  Scenario Outline: Add two numbers                    # /Users/yoko/Projects/jruby/lib/ruby/gems/1.8/gems/cucumber-0.8.5/examples/i18n/en/features/addition.feature:7
    Given I have entered <input_1> into the calculator # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:14
    And I have entered <input_2> into the calculator   # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:14
    When I press <button>                              # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:18
    Then the result should be <output> on the screen   # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:22

    Examples: 
      | input_1 | input_2 | button | output |
      | 20      | 30      | add    | 50     |
      | 2       | 5       | add    | 7      |
      | 0       | 40      | add    | 40     |

3 scenarios (3 passed)
12 steps (12 passed)
0m0.179s
/Users/yoko/Projects/jruby/lib/ruby/gems/1.8/gems/cucumber-0.8.5/bin/cucumber:19:in `load': exit (SystemExit)
 from /Users/yoko/Projects/jruby/bin/cucumber:19
Exception in thread "main" org.jruby.embed.EvalFailedException: exit
 at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:127)
 at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1149)
 at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1194)
 at evergreen.CucumberRunner.(CucumberRunner.java:16)
 at evergreen.CucumberRunner.main(CucumberRunner.java:20)
Caused by: org.jruby.exceptions.RaiseException: exit
 at (unknown).(unknown)(/Users/yoko/Projects/jruby/lib/ruby/gems/1.8/gems/cucumber-0.8.5/bin/cucumber:19)
 at Kernel.load(/Users/yoko/Projects/jruby/bin/cucumber:19)
 at (unknown).(unknown)(:1)

OK. Cucumber seems to execute SystemExit in the end. Java code isn't happy with this sort of exception since evaluation itself has been succeeded. So, I'm going to catch org.jruby.embed.EmbedEvalFailedException and error message raised in Ruby.

package evergreen;

import java.io.StringWriter;

import org.jruby.embed.EvalFailedException;
import org.jruby.embed.LocalContextScope;
import org.jruby.embed.PathType;
import org.jruby.embed.ScriptingContainer;

public class CucumberRunner1 {
    private String jrubyhome =  "/Users/yoko/Projects/jruby";
    private String cucumber = jrubyhome + "/bin/cucumber";
    private String feature = jrubyhome + "/lib/ruby/gems/1.8/gems/cucumber-0.8.5/examples/i18n/en/features/addition.feature";
    
    private CucumberRunner1() {
        StringWriter errorWriter = new StringWriter();
        try {
            ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
            container.setError(errorWriter);
            container.setHomeDirectory(jrubyhome);
            container.runScriptlet("ARGV << '" + feature + "'");
            container.runScriptlet(PathType.ABSOLUTE, cucumber);
        } catch (EvalFailedException e) {
            System.out.println("Cuke says: " + e.getMessage());
        } finally {
            System.out.println("Cuke also says: " + errorWriter.toString());
        }
    }

    public static void main(String[] args) {
        new CucumberRunner1();
    }
}

All right, CuumberRunner1 printed out:

# language: en
Feature: Addition
  In order to avoid silly mistakes
  As a math idiot
  I want to be told the sum of two numbers

  Scenario Outline: Add two numbers                    # /Users/yoko/Projects/jruby/lib/ruby/gems/1.8/gems/cucumber-0.8.5/examples/i18n/en/features/addition.feature:7
    Given I have entered <input_1> into the calculator # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:14
    And I have entered <input_2> into the calculator   # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:14
    When I press <button>                              # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:18
    Then the result should be <output> on the screen   # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:22

    Examples: 
      | input_1 | input_2 | button | output |
      | 20      | 30      | add    | 50     |
      | 2       | 5       | add    | 7      |
      | 0       | 40      | add    | 40     |

3 scenarios (3 passed)
12 steps (12 passed)
0m0.238s
Cuke says: exit
Cuke also says: /Users/yoko/Projects/jruby/lib/ruby/gems/1.8/gems/cucumber-0.8.5/bin/cucumber:19:in `load': exit (SystemExit)
 from /Users/yoko/Projects/jruby/bin/cucumber:19

Now, everything is controllable from Java.

Then, what if Cucumber result is showed up in Swing window? Isn't it interesting? So, I set StringWriter to ScriptingContainer so that all standard output from Ruby will be caught in StringWriter. Then, I added small Swing code.

package evergreen;

import java.io.StringWriter;

import javax.swing.JFrame;
import javax.swing.JTextArea;

import org.jruby.embed.EvalFailedException;
import org.jruby.embed.LocalContextScope;
import org.jruby.embed.PathType;
import org.jruby.embed.ScriptingContainer;

public class CucumberRunner2 {
    private String jrubyhome =  "/Users/yoko/Projects/jruby";
    private String cucumber = jrubyhome + "/bin/cucumber";
    private String feature = jrubyhome + "/lib/ruby/gems/1.8/gems/cucumber-0.8.5/examples/i18n/en/features/addition.feature";
    
    private CucumberRunner2() {
        StringWriter writer = new StringWriter();
        StringWriter errorWriter = new StringWriter();
        try {
            ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
            container.setError(errorWriter);
            container.setHomeDirectory(jrubyhome);
            container.setWriter(writer);
            container.runScriptlet("ARGV << '" + feature + "'");
            container.runScriptlet(PathType.ABSOLUTE, cucumber);
        } catch (EvalFailedException e) {
            System.out.println("Cuke says: " + e.getMessage());
        } finally {
            System.out.println("Cuke also says: " + errorWriter.toString());
            writeOnWindow(writer.toString());
        }
    }
    
    private void writeOnWindow(String message) {
        JFrame frame = new JFrame();
        JTextArea text = new JTextArea(message);
        frame.add(text);
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.pack();
        frame.setVisible(true);
    }

    public static void main(String[] args) {
        new CucumberRunner2();
    }
}

When I ran above code, a simple Swing window showed up with the result of Cucumber test.

Cucumber is Swing friendly with RedBridge. NetBeans plugin for Cucumber might be possible. Or, other Java based plugins for Cucumber might be easily implemented with RedBrdige.

Clojure uses DataMapper

Clojure is one of the JVM languages. People know this fact well. Also, people know well JRuby is among JVM languages. On the JVM languages, many people have used Java APIs from Clojure or Ruby code. But, we can do more since the JVM languages are able to communicate each other. The communication is done over the API exposed to Java such as JSR223. This means libraries and tools for a particular language are available to use in other JVM languages. For example, Clojure can choose DataMapper gem to interact databases. Usage is not so complicated. How could I make it happen? Here's a small example.


1. Installation

Clojure and JRuby themselves don't need to be installed. Grab the archives and unzip them. JRuby has installers, so you can use those if you like. After setting the path to jruby command, install 3 DataMapper gems. In this example, I used Sqlite3 for my DBMS. This is why I installed the DataMapper adapter for Sqlite3.

jruby -S gem install --no-ri --no-rdoc dm-core dm-sqlite-adapter dm-migrations

2. Run Clojure with the classpath to jruby.jar

My JRuby resides in /Users/yoko/Tools/jruby-1.5.1, so the path to jruby.jar is /Users/yoko/Tools/jruby-1.5.1/lib/jruby.jar. I added one more path. It is a directory for Ruby code for DataMapper, and the path name is /Users/yoko/Works/Samples/datamapper.
Move to the directory where Clojure was expanded, and run Clojure as in below:

java -cp clojure.jar:/Users/yoko/Tools/jruby-1.5.1/lib/jruby.jar:/Users/yoko/Works/Samples/datamapper clojure.main

3. Setup ScriptingContainer

This example uses JRuby's embed core, RedBridge, since it is easier to use compared to JSR223. The first step is to instantiate ScriptingContainer and set it up so that gems can be loaded.

user=> (import '(org.jruby.embed ScriptingContainer PathType))
org.jruby.embed.PathType
user=> (def c (ScriptingContainer.))
#'user/c
user=> (. c setHomeDirectory "/Users/yoko/Tools/jruby-1.5.1")
nil

To verify the setting was correct, I printed out Ruby's $LOAD_PATH.

user=> (. c runScriptlet "p $LOAD_PATH")
["clojure.jar", "/Users/yoko/Tools/jruby-1.5.1/lib/jruby.jar", "/Users/yoko/Works/Samples/datamapper", 
"/Users/yoko/Tools/jruby-1.5.1/lib/ruby/site_ruby/1.8",
"/Users/yoko/Tools/jruby-1.5.1/lib/ruby/site_ruby/shared", "/Users/yoko/Tools/jruby-1.5.1/lib/ruby/1.8", "."]
nil

The path seems OK. Let's go forward.


4. Load gems and connect to the database

The second step is to load gems.

user=> (. c runScriptlet "require 'rubygems'; require 'dm-core'; require 'dm-migrations'")
true

Then, setup DataMapper.

user=> (. c runScriptlet "DataMapper.setup(:default, 'sqlite::memory:')")
#<RubyObject #<DataMapper::Adapters::SqliteAdapter:0xa75865>>

Writing Ruby code in method argument is not very nice. Instead, I wrote Ruby code in the *.rb files and evaluated each file loaded from classpath, /Users/yoko/Works/Samples/datamapper.

user=> (. c runScriptlet PathType/CLASSPATH "category_def.rb")
#<RubyObject #<DataMapper::Model::DescendantSet:0x3b9617>>

See category_def.rb below:

# Definition of the Category model
class Category
  include DataMapper::Resource

  property :id, Serial
  property :category, String
  property :created_at, DateTime
end

# Migration
DataMapper.auto_migrate!

Since the Category table has been created, I added three new records:

user=> (. c runScriptlet PathType/CLASSPATH "categories.rb")
true

See categories.rb below:

# Create new records
c1 = Category.new
c1.category = 'Kitchen & Food'
c1.save

c2 = Category.new
c2.category = 'Bed & Bath'
c2.save

c3 = Category.new
c3.category = 'Dining'
c3.save

Let's see what were input to Sqlite3.

user=> (. c runScriptlet "p Category.all")
[#<Category @id=1 @category="Kitchen & Food" @created_at=nil>, #<Category
 @id=2 @category="Bed & Bath" @created_at=nil>, #<Category @id=3 @category="Dining" @created_at=nil>]
nil

As in this example, DataMapper worked with Clojure! MySQL and PostgreSQL are available to use from Clojure via DataMapper. Not just DataMapper. Other Ruby gems are also friends of Clojure.

A Small Step to Rails on RedBridge

When RedBridge (JRuby Embed) was released included in JRuby 1.4.0RC1 for the first time ever, I wrote“'Rails on Red Bridge' will be my exciting challenge" in my blog, What's the embedding API of JRuby 1.4.0RC1?. Since then, RedBridge had many improvements and bug fixes, so I tackled the issue this week. As a result, it went good. I could successfully made a small step to Rials on RedBridge. However, the process was a bit tricky, so I'm going to write down how I could make it, step by step.

The most difficult part was gem bundling. Since Rails on RedBridge is built on Java web application, all gems should be in a Java based web application structure. Besides, the gems should be looked up by a Rails app on a servlet. After I tried some, I concluded Rails 3 was the best choice because of its built-in feature of gem bundler. So, I created a Java web application project with Apache Tomcat on Eclipse and Rails 3 app within.

First, I built jruby-complete.jar from JRuby 1.5.0 source archive. At this moment, JRuby 1.5.0.RC3 is the latest.

$ tar zxfv /Users/yoko/Downloads/jruby-src-1.5.0.RC3.tar.gz
$ cd jruby-1.5.0.RC3
$ ant jar-complete
$ export JRUBY_HOME=`pwd`
$ PATH=$JRUBY_HOME/bin:$PATH

Then, I installed rails 3 and jdbc adapter gems. In my case, I used sqlite3, so I typed a gem name for jdbc sqlite3 adapter.

$ jruby -S gem install rails --pre --no-rdoc --no-ri
$ jruby -S gem install activerecord-jdbcsqlite3-adapter --no-rdoc --no-ri

Next, I created a Java web application project, Hemlock, on Eclipse. Once the project was built, I had the tree structure below:

Hemlock -+- WebContent -+- META-INF --- MANIFEST.MF
         |              +- WEB-INF -+- lib
         |                          +- web.xml
         +- build -+- classes
         +- src

Then, I created a Rails app under WebContent/WEB-INF/lib. This is because a web application adds the path, WEB-INF/lib, into the classpath automatically. WebContent/WEB-INF/classes would have been among the choices for the same reason.

My Rails app is based on a classic tutorial, "Four Days on Rails." This original document got outdated, but Japanese version was updated up to Rails 2.3.2. You can download the PDF file from Ita-san's blog: Four Days on Rails 2.0. Online translation service might help you to read the PDF.

$ cd [path-to-workspace]/Hemlock/WebContent/WEB-INF/lib
$ jruby -S rails todo
$ cd todo

Then, I edited Gemfile and config/database.yml to replace adapter gem and setting from sqlite3 to jdbcsqlite3.

Gemfile

#gem 'sqlite3-ruby', :require => 'sqlite3'
gem 'activerecord-jdbcsqlite3-adapter'

config/database.yml

#adapter: sqlite3
adapter: jdbcsqlite3

Before proceeding further, I did scaffolding to see database connection was set up right.

$ jruby -S rails g scaffold Category category:string
$ jruby -S rake db:migrate
$ jruby -S rails server

Requesting http://localhost:3000/categories on my browser, I added three categories.


I did one more setup on Rails --- Rails metal. Rails metal was originally invented to improve perfomance. But, metal exposes Rack's bare interface, so metal makes it easy to handle Rails app in the servlet.

$ jruby -S rails g metal poller

OK, let's setup gems for a servlet.

$ jruby -S bundle install vendor --disable-shared-gems

This command installed all necessary gems in WebContent/WEB-INF/lib/todo/vendor/gems directory.

Then, I moved on to the web application setting and the servlet. I copied jruby-compelete.jar to WEB-INF/lib, first. Next, after refreshing the Hemlock Eclipse project, I added jruby-complete.jar by Properties -> Java Build Path -> Add External JARS -> [on a JAR Selection window, I chose Hemlock/WebContent/WEB-INF/lib/jruby-complete.jar] -> OK. My servlet became as in below:

package com.servletgarden.hemlock;

import java.io.IOException;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.ScriptingContainer;

public class SimpleMetalServlet extends HttpServlet {
 private static final long serialVersionUID = 1L;
 private String basepath;
    private ScriptingContainer container;
    private List loadPaths;

    public SimpleMetalServlet() {
        super();
    }
    
    @Override
    public void init() {
        basepath = getServletContext().getRealPath("/WEB-INF");
        String[] paths = {
                basepath + "/lib/todo/vendor/gems/bundler-0.9.25/lib/",
                basepath + "/lib/todo"
                };
        loadPaths = Arrays.asList(paths);
        container = new ScriptingContainer(LocalContextScope.THREADSAFE);
    }
    
    @Override
    protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        container.setLoadPaths(loadPaths);
        container.runScriptlet("ENV['BUNDLE_GEMFILE'] = \"" + basepath + "/lib/todo/Gemfile\"");
        container.runScriptlet("require 'config/environment'");
        Map env = getEnvMap(request);
        container.put("env", env);
        List result = (List) container.runScriptlet("Poller.call env");
        returnResult(response, result);
    }
    
    private Map getEnvMap(HttpServletRequest request) {
        Map map = new HashMap();
        map.put("PATH_INFO", "/poller");
        return map;
    }
    
    private void returnResult(HttpServletResponse response, List result) throws IOException {
        response.setStatus(((Long)result.get(0)).intValue());
        Map headers = (Map)result.get(1);
        Set keys = headers.keySet();
        for (String key : keys) {
            if ("Content-Type".equals(key)) {
                response.setContentType((String)headers.get(key));
            }
        }
        List contents = (List)result.get(2);
        for (String content : contents) {
            response.getWriter().write(content);
        }
    }
}

When I right-clicked on the servlet source and chose Run As -> Run on Server, my Eclipse showed that simple "Hello, World!"


OK, let's look at what I did in SimpleMetalServlet. The first one is a load path setting.

Hemlock tree

Hemlock -+- WebContent -+- META-INF --- MANIFEST.MF
         |              +- WEB-INF -+- lib -+- todo -+- Gemfile
         |                          |       |        +- app -+- controllers --- ...
         |                          |       |        |       +- metal --- poller.rb
         |                          |       |        |       +- ...
         |                          |       |        +- config -+- environment.rb
         |                          |       |        +- vendor -+- gems -+- bundler-0.9.25 -+- lib
         |                          |       |                           -+- ...
         |                          |       |        +- ...
         |                          |       +- jruby-complete.jar
         |                          +- web.xml
         +- build -+- classes
         +- src -+- com -+- servletgarden -+- hemlock -+- SimpleMetalServlet.java

in init() method:

basepath = getServletContext().getRealPath("/WEB-INF");
String[] paths = {
        basepath + "/lib/todo/vendor/gems/bundler-0.9.25/lib/",
        basepath + "/lib/todo"
};
loadPaths = Arrays.asList(paths);

in service() method:

container.setLoadPaths(loadPaths);

The path to bundler is to use bundler gem to load bundled gems. Bundler gem is installed in the todo/vendor/gems directory, but before loading bundled gem, SimpleMetalServlet can't load bundler itself. So, the servlet is telling JRuby runtime where the bundler gem is. The path to todo, Rails app top directory, is to tell JRuby where Rails app top directory is located.

The second one is the path to Gemfile.

container.runScriptlet("ENV['BUNDLE_GEMFILE'] = \"" + basepath + "/lib/todo/Gemfile\"");

Gemfile matters to Rails. To make Rails up and running, SimpleMetalServlet needs to give info about Gemfile. In this case, SimpleMetalServlet set the real path to Gemfile tied to the key "BUNDLE_GEMFILE" in ENV hash.

Then,

container.runScriptlet("require 'config/environment'");

Todo Rails app should get started by this.

Everything should be setup now, so SimpleMetalServlet is able to kick the metal method. Before talking about how the servlet kicked it, let's see what code Rails tailored.

poller.rb

# Allow the metal piece to run in isolation
require File.expand_path('../../../config/environment',  __FILE__) unless defined?(Rails)

class Poller
  def self.call(env)
    if env["PATH_INFO"] =~ /^\/poller/
      [200, {"Content-Type" => "text/html"}, ["Hello, World!"]]
    else
      [404, {"Content-Type" => "text/html", "X-Cascade" => "pass"}, ["Not Found"]]
    end
  end
end

Poller class has a class method, call, which needs an argument, env. The value, env, is a hash, in which PATH_INFO key and the value tied to it is expected. So, SimpleMetalServlet creates java.util.Map type object, puts to ScriptingContainer and evaluates Poller.call method with the argument, env.

Map env = getEnvMap(request);
container.put("env", env);
List result = (List) container.runScriptlet("Poller.call env");

private Map getEnvMap(HttpServletRequest request) {
    Map map = new HashMap();
    map.put("PATH_INFO", "/poller");
    return map;
}

At last, response handling comes in.

returnResult(response, result);

private void returnResult(HttpServletResponse response, List result) throws IOException {
    response.setStatus(((Long)result.get(0)).intValue());
    Map headers = (Map)result.get(1);
    Set keys = headers.keySet();
    for (String key : keys) {
        if ("Content-Type".equals(key)) {
            response.setContentType((String)headers.get(key));
        }
    }
    List contents = (List)result.get(2);
    for (String content : contents) {
        response.getWriter().write(content);
    }
}

Rack based application returns the response as an array of a status code, hash of http response headers, and array of body contents. SimpleMetalServlet parses the response and sends back to a browser. Since Poller class returns so simple reponse, SimpleMetalServlet doesn't do much in this case.

I tweaked poller.rb a bit so that registered categories are showed up.

poller.rb

#[200, {"Content-Type" => "text/html"}, ["Hello, World!"]]
[200, {"Content-Type" => "application/xhtml+xml"}, [Category.all.to_xml]]

Refreshing Hemlock project on Eclipse, restarting Tomcat also on Eclipse, then requesting http://localhost:8080/Hemlock/SimpleMetalServlet gave me the XML below.


It seems Rails on RedBridge worked.

Finally, pure Java Nokgiri worked on Google App Engine

Pure Java version of Nokogiri is on the way to its very first version. It is "pure Java," not backed by libxml2, so people will expect that Nokogiri works on Google App Engine. In fact, Nokogiri on GAE is one of the purposes of Java port. Then... does it really work? Yes, it does! I could manage to get it work on GAE. However, it took pretty longer time than I thought, so I'm going to write down how I made it. Hoping this will help *pure Java* Nokogiri users.

1. Create pure Java Nokogiri gem

See Nokogiri Java Port: Help Us Finish It! for details.

$ git clone git://github.com/tenderlove/nokogiri.git
$ cd nokogiri
$ git checkout -b java origin/java
(use C Ruby for following two commands)
$ sudo gem install racc rexical rake-compiler hoe
$ rake gem:dev:spec
$ jruby -S rake java:gem

You'll have pure Java Nokogiri gem in "pkg" directory. The name is something like nokogiri-1.4.0.20100424131857-java.gem.

Then, install this gem to your JRuby.
$ jruby -S gem install [path to nokogiri]/pkg/nokogiri-1.4.0.20100424131857-java.gem


2. Create a web app project for GAE

I used Google Plugin for Ecplise to create a web application project. This is Java's web application project. I used Java project because I wrote a servlet using JRuby Embed (RedBridge). Testing Nokogiri with JRuby Embed (RedBridge) is good to know how it works.

3. Create one jar archive including Nokogiri gem

I could have put Ruby library and jar archives of Nokogiri under WEB-INF, but instead, I created one jar archive. This is because GAE limits a number of files as well as each file size. Before creating jar archive, you need to do "gem bundle" so that JRuby can load ruby files. Gem bundler is good tool to put gems together to create a jar archive. See Using the New Gem Bundler Today to learn how you can bundle gems.

First, install gem bundler.

$ jruby -S gem install bundler08

Go to the top directory of your web application. If you created a web application project using Eclipse plugin, go to [project's top directory]/war. Then, create "Gemfile." For example,

$ cd /Users/yoko/workspace/Dahlia/war
$ vi Gemfile

Here's an example of Gemfile.

# Critical default settings:
disable_system_gems
disable_rubygems
bundle_path ".gems/bundler_gems"

# List gems to bundle here:
gem "nokogiri", "1.4.0.20100424131857"

Make sure nokogiri gem is installed in your JRuby by "jruby -S gem list."

$ jruby -S gem bundle

This command creates .gems directory and puts gems listed in Gemfile under .gems/bundler_gems directory.

Next, edit .gems/bundler_gems/environment.rb as in below:

require 'rbconfig'
engine  = defined?(RUBY_ENGINE) ? RUBY_ENGINE : 'ruby'
version = Config::CONFIG['ruby_version']
#require File.expand_path("../#{engine}/#{version}/environment", __FILE__)
require File.dirname(__FILE__) + "/#{engine}/#{version}/environment"

File.expand_path tries to expand the path to a full path starting from "/" (root). This full path doesn't work as a load path on Java based web application. This environment.rb file will be in a jar archive, so the path should be something like "file:/base/data/home/apps/servletgarden-dahlia/17.341465334854239323/WEB-INF/lib/gems.jar!/bundler_gems/jruby/1.8/environment." The path starting from "/" never works.

Now, you have everything under .gems/bundler_gems directory, next thing you do is to create a jar archive. For example, I did as in below:

$ cd .gems
$ jar -J-Duser.language=en cfv ../WEB-INF/lib/gems.jar bundler_gems/

In this case, gems.jar will be created in WEB_INF/lib directory. If you change the top directory in the jar archive, you will have a different load path setting I did in my servlet.

You need to one more job here. Pure Java Nokogiri gem has six jar archives in it. However, you need to move or copy Java archives in WEB-INF/lib directory to be loaded to your web application The jar archives are in .gems/bundler_gems/jruby/1.8/gems/nokogiri-1.4.0.20100424131857-java/lib/ directory. So, assuming you are in .gems direotry,

$ cp undler_gems/jruby/1.8/gems/nokogiri-1.4.0.20100424131857-java/lib/*.jar ../WEB-INF/lib/.
$ cp undler_gems/jruby/1.8/gems/nokogiri-1.4.0.20100424131857-java/lib/nokogiri/nokogiri.jar ../WEB-INF/lib/.

4. Have patched JRuby archive

JRuby 1.5.0RC2 released in April 28 2010 fixed all problems described here. Have JRuby 1.5.0RC2! Then, what you do is to create two jar archives by rake task.

This part was the most painful to make Nokogiri work on GAE. Unfortunately, you can't do this on JRuby 1.5.0RC1. You need the latest JRuby in git repo because of the problems.
The first problem is that JRuby 1.5.0RC1's source archive doesn't have gem directory. In the gem directory, the tool to split jruby-complete.jar up into two jar archives is there. Because jruby-complete.jar is too big to upload GAE, this jar file need to be split into smaller jars. GAE has --enable_jar_splitting option, but jruby-complete.jar is not just a bunch of .class files. It includes .rb files, which should be found under jruby.home. So, I don't think --enable_jar_splitting option will work. This problem was fixed in master already. If you have the latest JRuby,

$ ant dist
$ cd gem
$ jruby -S rake

will do the job.

The second problem is JRuby Embed raises NullPointerException form SystemPropertyCathcer. JRuby Embed haven't suspected "java.class.path" system property might be null, but it is on GAE. This bug is also fixed in master.

However, the third problem is now under the way. The third one, maybe serious one, is that "require 'rbconfig'" failes on GAE. When RbConfigLibrary is loaded on JRuby it raises NullPointerException because Platform.ARCH is null on GAE. I filed this in JIRA, http://jira.codehaus.org/browse/JRUBY-4749 with a patch. After that, Charles attached a new patch, which is supposed to be applied to jruby-1_5 branch. Probably, it won't take long to solve this issue. If you want to give Nokogiri a try on GAE now, apply the patch and build JRuby, and split jruby-complete.jar up into two jars.

5. Write a Servlet

Here's a very simple Servlet that uses Nokogiri.

package com.servletgarden.dahlia;

import java.io.IOException;
import java.util.Arrays;
import java.util.List;

import javax.servlet.http.*;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.ScriptingContainer;

@SuppressWarnings("serial")
public class DahliaServlet extends HttpServlet {
    private ScriptingContainer container;
    private String script = 
        "doc = Nokogiri::XML \"\"\n" +
        "puts doc.to_xml";
    
    public void init() {
        String basepath = getServletContext().getRealPath("/WEB-INF");
        String[] paths = {"file:"+ basepath + "/lib/gems.jar!/bundler_gems"};
        List loadPaths = Arrays.asList(paths);
        container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
        container.getProvider().getRubyInstanceConfig().setDebug(true);
        container.setLoadPaths(loadPaths);
    }
    public void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws IOException {
        resp.setContentType("text/plain; charset=UTF-8");
        synchronized (container) {
            container.setWriter(resp.getWriter());
            container.runScriptlet("require 'environment'");
            container.runScriptlet("require 'nokogiri'");
            container.runScriptlet(script);
            container.terminate();
        }
    }
}

The load path setting in this servlet depends on what top directory you chose for the gem.jar. Don't forget "require 'environment'" since this app uses gem bundler instead of rubygems.

If this servlet successfully works on GAE, you'll get this simple response on your browser.

<?xml version="1.0"?>
<root>
  <foo/><quux/><bar/>
</root>

You can see this at http://servletgarden-dahlia.appspot.com/dahlia

RedBridge, what are new and improved in JRuby 1.5.0RC1

As you may know, Tom Enebo announced the release of JRuby 1.5.0RC1 on Apr. 15 saying "aged like a fine wine." @headius tweated "Over 1250 commits for JRuby 1.5, our largest amount of work ever for any individual release." Also, RedBridge is. RedBridge has been improved since last release based on user inputs. It's API had many changes to become more useful and organized API. Although I've already written about all changes in this blog, I'm going to put them together here for convenience.

New and Deprected Configuration API
RedBridge in JRuby 1.5.0 has a lot of Ruby runtime configuration methods. Before, those were available through getProvider().getRubyInstanceConfig() method, however, this was not a good idea. Since the method exposes JRuby's internal API, users' code might be affected by internal API changes. This fact is against to the purpose of RedBridge. RedBrdige should cover JRuby's internal API and absorb internal changes so that users don't need to fix their code by themselves. Avoid using getProvider().getRubyInstanceConfig() method as much as possible. If you want more runtime configuration methods, please request us.

New runtime configuration methods of ScriptingContainer:

• get/setInput
  
• get/setOutput
  
• get/setError
  
• get/setCompileMode
  
• get/setRunRubyInProcess
  
• get/setCompatVersion
  
• get/setObjectSpaceEnabled
  
• get/setEnvironment
  
• get/setCurrentDirectory
  
• get/setHomeDirectory
  
• get/setClassCache
  
• get/setClassLoader
  
• get/setProfile
  
• get/setLoadServiceCreator
  
• get/setArgv
  
• get/setScriptFileName
  
• get/setRecordSeparator
  
• get/setKCode
  
• get/setJITLogEvery
  
• get/setJITThreshold
  
• get/setJITMax
  
• get/setJITMaxSize
  

Deprecated configuration methods:

• getRuntime()
  
• getProvider().setLoadPaths()
  
• getProvider().setClassCache()
  

Usage example:

[JRuby 1.4.0]
ScriptingContainer container = new ScriptingContainer();
container.getProvider().getRubyInstanceConfig().setJRubyHome(jrubyhome);

[JRuby 1.5.0]
ScriptingContainer container = new ScriptingContainer();
container.setHomeDirectory(jrubyhome);

New Options
Also, RedBridge got two new options: SHARING_VARIABLES and TERMINATION. The first, SHARING_VARIABLES, option turns on/off a sharing variables feature.This is an essential feature for some users while useless for other users. For those people, sharing variables is just a source of performance degradation. When the feature is turned off, the performance will be a bit better.
Usage example:

[Embed Core]
container.setAttribute(AttributeName.SHARING_VARIABLES, false);

[JSR223]
engine.getContext().setAttribute("org.jruby.embed.sharing.variables", false, ScriptContext.ENGINE_SCOPE);

The second, TERMINATION, option is for JSR223 users to call terminate. This option was added in light of RedBridge's behavior change. This is a big change, so I'll discuss more about this.

Changed Behaviors

* No termination in each evaluation and method call

RedBridge in JRuby 1.4.0 always called Ruby runtime's terminate method at the end of each evaluation and method call. This was to execute at_exit blocks and release resources automatically. The idea came from JSR223 API, which doesn't have a terminate method defined. However, I eliminated this behavior in JRuby 1.5.0 for three reasons. The first one is for performance. The terminate() method is so slow and was the biggest culprit of bad performance. The more Ruby code uses ruby files, instance variables, etc, the more it takes time. The second one is to make RedBridge's behavior more natural. Since the terminate() method fires at_exit blocks automatically, users might have unexpected results when they use third party libraries. Users should have a chance to fire at_exit blocks by themselves. The third one is that JRuby's memory leak was fixed. Thus, RedBridge doesn't need to invoke the terminate method just for releasing resources. Make sure, you have the terminate method in the right place.

Usage examples:

[Embed Core]

ScriptingContainer container = null;
try {
    container = new ScriptingContainer();
    container.runScriptlet(PathType.CLASSPATH, testname);
} catch (Throwable t) {
    t.printStackTrace();
} finally {
    container.terminate();
}

[JSR223]

ScriptEngineManager manager = new ScriptEngineManager();
ScriptEngine engine = manager.getEngineByName("jruby");
engine.eval("$x='GVar'");
engine.eval("at_exit { puts \"#{$x} in an at_exit block\" }");  // nothing is printed here
engine.getContext().setAttribute(AttributeName.TERMINATION.toString(), true, ScriptContext.ENGINE_SCOPE);
engine.eval("");  // prints "GVar in an at_exit block"

* Global runtime

When a global ruby runtime exists on a single JVM, a singleton model of RedBridge uses the global runtime in JRuby 1.5.0. This works behind the scene and seems not so attractive, but is interesting. Here's a bit tricky usage:

$ pwd
/Users/yoko/Tools/jruby-1.5.0.RC1
$ jruby --1.9 -Ctest -S irb
irb(main):001:0> require 'java'
=> true
irb(main):002:0> container=org.jruby.embed.ScriptingContainer.new
=> org.jruby.embed.ScriptingContainer@771eb1
irb(main):003:0> container.compat_version                <---- --1.9 option
=> RUBY1_9
irb(main):004:0> container.current_directory             <---- -Ctest option
=> "/Users/yoko/Tools/jruby-1.5.0.RC1/test"
irb(main):009:0> container.home_directory                <---- jruby home
=> "/Users/yoko/Tools/jruby-1.5.0.RC1"
irb(main):010:0> container.run_scriptlet "at_exit { puts \"see you, later\" }" 
=> #<Proc:0x88a1b@<script>:1>
irb(main):011:0> container.terminate
see you, later
=> nil

This new behavior might be useful in a complicated application.
However, you should be aware that setting a runtime configuration doesn't work if the global runtime is there already. This is because the runtime configuration is read only when the runtime is instantiated. You should be careful not miss the timing to set configuration.

* Lazy Runtime Initialization

RedBridge (in this case, I mean Embed Core) delays ruby runtime initialization as much as possible. This is to improve ScriptingContainer's start up time. You may know loading Ruby runtime is a huge job and takes pretty much time. This might cause frustration if it happens right after the ScriptinContainer gets started. The question is when runtime is up and running. Some of ScriptingContainer's methods will kick ruby runtime to wake up. Here's a list:

• put()
  
• runScriptlet()
  
• setWriter()
  
• resetWriter()
  
• setErrorStream()
  
• resetErrorStream()
  
• setReader()
  

Thus, when you want configuration settings to work, you need to set them before these methods.
Meanwhile, JSR223 implementation doesn't delay ruby runtime initialization. It was not easy without breaking JSR223's requirement.

* Lazy Java Library Loading

Red Bridge doesn't load a java library while ruby runtime is initialized in JRuby 1.5.0. This is also for performance improvement. Loading libraries on to ruby runtime is quite a cumbersome job. Checking loaded library tables up to see whether a specified library has not yet loaded, judging how to load the library, then loading, caching... Even though Java library is not loaded while initialization, it will be loaded internally if necessary. Or you can load Java library explicitly:

container.runScriptlet("require 'java'");

Performance Tuning Tips
RedBridge's performance has been improved compared to older version, but you can tweak a bit more. For example, you can remove variables for sharing or clear sharing variable table at some point:

org.jruby.embed.ScriptingContainer#remove(String key)
org.jruby.embed.ScriptingContainer#clear()

RedBridge retrieves instance variables and constants as much as possible at the end of each evaluation and method call. All retrieved values are injected to runtime when the next script or method is evaluated. You can cut down the time for injection by removing unnecessary values.

Remaining jobs
RedBridge couldn't resolve all issues and has remaining jobs. Among them, OSGi and configuration on JSR223 impl would be two big issues. By the final release of JRuby 1.5.0, I want to improve these.


Finally, your input will help us to make RedBridge more perfect API. Give it a try and report us!