Friday, October 22, 2010

Gems in a Jar with RedBridge

Using JRuby embed API (RedBridge), it is easy to use Ruby gems from Java. However, packaging might not be simple, so people occasionally struggle to create a "portable" package. Being portable is important for a Java app. All stuffs of the Java app should be packaged in a jar archive, which should work on another PC, or even different OS. This is the good side of Java. This blog illustrates one solution for packaging.

I intentionally didn't use jruby command since the command hides what are going on behind. You might be exhausted by lines of "java -jar ....," sorry. But, it helps to understand a packaging process.

Directories


Since I used bundler to install gems, I created directories below to fit them to bundler.

Linden -+- lib -+- jruby -+- 1.8
+- src -+- linden
+- build

"Linden" is a top directory and the name doesn't have any special meaning. You can name it whatever you like. "lib/jruby/1.8" is the directory gems will be installed. "src" is for Java code, and "build" is for compiled *.class files.

Bundler Installation


I intentionally used jruby-complete.jar to assure gems won't be installed in the default location. The path to jruby-complete.jar can be anything since typing a full path to jruby-complete.jar works. But, I like a short path to type, so I copied jruby-complete.jar under "Linden/lib." Now, the directories/files were as in below:

Linden -+- lib -+- jruby -+- 1.8
+- jruby-complete.jar
+- src -+- linden
+- build


Then, the bundler installation went:

cd lib
java -jar jruby-complete.jar -S gem install bundler jruby-openssl --no-ri --no-rdoc -i jruby/1.8

Bundler and jruby-openssl gems were installed and the directories became as in below:

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
| +- cache -+- ...
| +- doc
| +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
| | +- bundler-1.0.3 -+- ...
| | +- jruby-openssl-0.7.1 -+- ...

| +- specifications -+- ...
+- jruby-complete.jar
+- src -+- linden
+- build

Let's see whether gems are really installed.

export GEM_PATH=`pwd`/jruby/1.8
java -jar jruby-complete.jar -S gem list

This command should print out the installed three gems and pre-installed gems.

*** LOCAL GEMS ***

bouncy-castle-java (1.5.0145.2)
bundler (1.0.3)
columnize (0.3.1)
jruby-openssl (0.7.1)
rake (0.8.7)
rspec (1.3.0)
ruby-debug (0.10.3)
ruby-debug-base (0.10.3.2)
sources (0.0.1)

OK. Bundler was installed successfully.

Installation of other gems


Next step is to install other gems using bundler, so I used bundle command. Theoretically, PATH environment variable should work to find bundle command by ruby. Unfortunately, this didn't work in my "java -jar ..." usage. Instead, I typed a path to bundle command after -S option.

java -jar jruby-complete.jar -S jruby/1.8/bin/bundle init

Then, I added "twitter" gem to Gemfile.

# A sample Gemfile
source "http://rubygems.org"

# gem "rails"

gem "twitter"

The installation went on:

java -Xmx500m -jar jruby-complete.jar -S jruby/1.8/bin/bundle install --path=.

The java command option "-Xmx500m" is for performance and to avoid memory outage. Now, 6 gems were installed and the directories/files became:

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
| | +- httparty
| | +- oauth

| +- cache -+- ...
| +- doc
| +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
| | +- bundler-1.0.3 -+- ...
| | +- crack-0.1.8 -+- ...
| | +- hashie-0.4.0 -+- ...
| | +- httparty-0.6.1 -+- ...

| | +- jruby-openssl-0.7.1 -+- ...
| | +- multi_json-0.0.4 -+- ...
| | +- oauth-0.4.3 -+- ...
| | +- twitter-0.9.12 -+- ...

| +- specifications -+- ...
+- jruby-complete.jar
+- Gemfile
+- Gemfile.lock

+- src -+- linden
+- build


Java code to use twitter gem


Since all gems were ready, I wrote Java code to see gems worked. This time, I relied on GEM_PATH environment variable to find gems. The fist code had Java package, linden, and class name, SearchSample. So, I created SearchSample.java file under "src/linden" directory.

package linden;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.ScriptingContainer;

public class SearchSample {
private String jarname = "sample.jar";

private SearchSample() {
String basepath = System.getProperty("user.dir");
System.out.println("basepath: " + basepath);
ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
System.out.println("jrubyhome: " + container.getHomeDirectory());
container.runScriptlet("ENV['GEM_PATH']='" + basepath + "/lib/jruby/1.8'");

String script =
"require 'rubygems'\n" +
"require 'twitter'\n" +
"require 'pp'\n" +
"pp Twitter::Search.new('#jruby').fetch.results.first";
container.runScriptlet(script);
}

public static void main(String[] args) {
new SearchSample();
}
}

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
| | +- httparty
| | +- oauth
| +- cache -+- ...
| +- doc
| +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
| | +- bundler-1.0.3 -+- ...
| | +- crack-0.1.8 -+- ...
| | +- hashie-0.4.0 -+- ...
| | +- httparty-0.6.1 -+- ...
| | +- jruby-openssl-0.7.1 -+- ...
| | +- multi_json-0.0.4 -+- ...
| | +- oauth-0.4.3 -+- ...
| | +- twitter-0.9.12 -+- ...
| +- specifications -+- ...
+- jruby-complete.jar
+- Gemfile
+- Gemfile.lock
+- src -+- linden -+- SearchSample.java
+- build



Compile and run using rake-ant integration



Everything was ready. OK, how do I compile and run it? Of course, classic "javac" and "java" command were the options. But, these commands are not very convenient to repeat compile/run. So, I used JRuby's rake-ant integration. Yes, I wrote "Rakefile" to compile and run Java code. Isn't it nice? Here's Rakefile:

require 'ant'

namespace :ant do
task :compile => :clean do
ant.javac :srcdir => "src", :destdir => "build"
end
end

namespace :ant do
task :java => :compile do
ant.java :classname => "linden.SearchSample" do
classpath do
pathelement :location => "lib/jruby-complete.jar"
pathelement :path => "build"
end
end
end
end

require 'rake/clean'

CLEAN.include '*.class', '*.jar'

I created Rakefile under the Linden directory, so now the directories/files were:

Linden -+- lib -+- jruby -+- 1.8 -+- bin -+- bundle
| | +- httparty
| | +- oauth
| +- cache -+- ...
| +- doc
| +- gems -+- bouncy-castle-java-1.5.0145.2 -+- ...
| | +- bundler-1.0.3 -+- ...
| | +- crack-0.1.8 -+- ...
| | +- hashie-0.4.0 -+- ...
| | +- httparty-0.6.1 -+- ...
| | +- jruby-openssl-0.7.1 -+- ...
| | +- multi_json-0.0.4 -+- ...
| | +- oauth-0.4.3 -+- ...
| | +- twitter-0.9.12 -+- ...
| +- specifications -+- ...
+- jruby-complete.jar
+- Gemfile
+- Gemfile.lock
+- src -+- linden -+- SearchSample.java
+- build
+- Rakefile


When I typed "rake ant:java", the Java code above worked and printed out one tweet.

java -jar lib/jruby-complete.jar -S rake ant:java
(in /Users/yoko/Works/tmp/Linden)
basepath: /Users/yoko/Works/tmp/Linden
jrubyhome: file:/Users/yoko/Works/tmp/Linden/lib/jruby-complete.jar!/META-INF/jruby.home
{"profile_image_url"=>
"http://a3.twimg.com/profile_images/76084835/web-profile_normal.jpg",
"created_at"=>"Fri, 22 Oct 2010 15:38:09 +0000",
"from_user"=>"mccrory",
"metadata"=>{"result_type"=>"recent"},
"to_user_id"=>nil,
"text"=>
"RT @carlosqt: #IronRuby 1.1.1 a released! download from: http://ironruby.codeplex.com/ #programming #JRuby #Ruby #Rails #dotnet #code",
"id"=>28415816774,
"from_user_id"=>1757577,
"geo"=>nil,
"iso_language_code"=>"en",
"source"=>"<a href="http://twitter.com/">web</a>"}


Packaging


This time I can't rely on GEM_PATH environment variable. If GEM_PATH had worked also for the path in a jar, unfortunately, it didn't. Instead of GEM_PATH, I set all path to gems to ScriptingContainer. This was not complicated since all gems were installed in the same directory. I fixed the jar name, "sample.jar," so this name appeared in the Java code. This name could have been given via a command line argument. Edited SearchSample.java is in below:

package linden;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
import java.util.zip.ZipEntry;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.ScriptingContainer;

public class SearchSample {
private String jarname = "sample.jar";

private SearchSample() throws IOException {
String basepath = System.getProperty("user.dir");
System.out.println("basepath: " + basepath);
ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
System.out.println("jrubyhome: " + container.getHomeDirectory());
container.setLoadPaths(getGemPaths(jarname, basepath));

String script =
"require 'rubygems'\n" +
"require 'twitter'\n" +
"require 'pp'\n" +
"pp Twitter::Search.new('#jruby').fetch.results.first";
container.runScriptlet(script);
}

private List<String> getGemPaths(String jarname, String basepath) throws IOException {
JarFile jarFile = new JarFile(basepath + "/" + jarname);
Enumeration<JarEntry> entries = jarFile.entries();
String gempath = "lib/jruby/1.8/gems/";
Set<String> gemnames = new HashSet<String>();
while (entries.hasMoreElements()) {
ZipEntry entry = (ZipEntry) entries.nextElement();
String entryName = entry.getName();
if (entryName.startsWith(gempath) && entryName.length() > gempath.length()) {
String n = entryName.substring(gempath.length());
String m = n.substring(0, n.indexOf("/"));
gemnames.add(m);
}
}
List<String> gemPaths = new ArrayList<String>();
for (String gem : gemnames) {
gemPaths.add("file:" + basepath + "/" + jarname + "!/lib/jruby/1.8/gems/" + gem + "/lib");
}
return gemPaths;
}


public static void main(String[] args) throws IOException {
new SearchSample();
}
}

To make jar archive, I added jar task to my Rakefile.

require 'ant'

namespace :ant do
task :compile => :clean do
ant.javac :srcdir => "src", :destdir => "build"
end
end

namespace :ant do
task :java => :compile do
ant.java :classname => "linden.SearchSample" do
classpath do
pathelement :location => "lib/jruby-complete.jar"
pathelement :path => "build"
end
end
end
end

namespace :ant do
task :jar => :compile do
ant.jar :basedir => ".", :destfile => "sample.jar" do
fileset :dir => "build" do
include :name => "**/*.class"
end
include :name => "lib/jruby/1.8/gems/**/*"
manifest do
attribute :name => "Main-Class", :value => "linden.SearchSample"
end
end
end
end


require 'rake/clean'

CLEAN.include '*.class', '*.jar'

All right, let's create jar archive.

java -jar lib/jruby-complete.jar -S rake ant:jar

The ant:jar task created the sample.jar archive in Linden directory. To test this archive was really gems in a jar, I unset GEM_PATH environment variable first. Then, I typed java command and got one tweet, Yay!

unset GEM_PATH; echo $GEM_PATH
java -cp lib/jruby-complete.jar:sample.jar linden.SearchSample

To ensure that the jar archive had gems in the jar, I moved sample.jar to a different directory and typed java command with the full path to jruby-complete.jar. It worked.

cp sample.jar ../.
cd ..
java -cp Linden/lib/jruby-complete.jar:sample.jar linden.SearchSample



This might be one answer of packaging gems in a jar and using gems from RedBridge.

Monday, October 11, 2010

pure Java Nokogiri - XSLT extension function -

Here's a memo of pure Java Nokogiri about an implementation of XSLT extension function. In short, I concluded pure Java version is unable to support Nokogiri style XSLT extension function. I've tried possible ways to make it happen, but for an inevitable reason, I settled to this conclusion. However, in future, this part might be reconsidered when XML libraries and APIs are replaced to others. For the future version of pure Java Nokogiri, I'm going to write down what I did and what was the problem. Hopefully, this memo will help to retry the implementation later.


1. What is XSLT extension function?

XSLT extension is defined in "14 Extensions" of XSL Transformations (XSLT)
Version 1.0 (http://www.w3.org/TR/xslt), which allows users to delegate an XSLT processing to a specified function/method written in a programming languages such as Ruby, Java, JavaScript. As in Nokogiri's test case,

<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:f="http://e.org/functions"
extension-element-prefixes="f">

<xsl:template match="text()">
<xsl:copy-of select="f:capitalize(.)"/>
</xsl:template>
....

the element, "extension-element-prefixes," indicates the function is tied to this namespace. This XSL file should be common to all languages used to write the function.


2. Nokogiri style function mapping

Nokogiri maps the namesapce to the function as in below:

foo = Class.new do
def capitalize nodes
nodes.first.content.upcase
end
end

XSLT.register "http://e.org/functions", foo

Thus, a receiver object is registered to XSLT processor with the URL tag. It is nice Ruby friendly design. <xsl:copy-of select="f:capitalize(.)"/> above executes a "capitalize" method of the "foo" object.


3. How Java handles this?

As far as I googled about a Java way, not many documents, blogs, articles were out there. Probably, Extending XSLT with Java - Chapter 17. XSLT would be the best described one. This explains how to delegate the process to a Java method tied to the namespace. Xalan has the document, Xalan-Java Extensions; however, this uses BSF (Bean Scripting Framework: http://jakarta.apache.org/bsf/) to execute a function/method written in an XSL file. So, the first one has a possibility to realize Nokogiri style.


While I tried a couple of patterns, the "extension-element-prefixes" element seemed not to have a much meaning. Instead, "xmlns:java="http://xml.apache.org/xslt/java" and "xmlns:foo="xalan://[fully qualified class name]" worked. OK, so pure Java version of Nokogiri needs a specific rule to use XSLT extension function. This might be better than unsupported. Then, I wrote a Java class below to see whether it worked or not:

package Canna;

public class ExtensionFoo {
public static Object exec(String method, Object value) {
.....
}
}

The method should be static to be called from XSLT processor. The first argument is a method name to make the style resemble to the Nokogiri way. Since I thought

foo = Class.new do
def capitalize nodes
nodes.first.content.upcase
end
end

XSLT.register "http://e.org/functions", foo

xsl = Nokogiri.XSLT(<<-EOXSL)
<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:foo="xalan://nokogiri.internals.XsltExtensionFunction"
extension-element-prefixes="foo">

<xsl:template match="text()">
<xsl:copy-of select="foo:exec('capitalize' .)"/>
</xsl:template>
...

would not be a bad substitution. Users need to have just a small rule only in the xsl file.


Here are entire files that I tried XSLT extension function in action by Java.

[extension.xsl]
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:foo="xalan://Canna.ExtensionFoo"
extension-element-prefixes="foo"
version="1.0">
<xsl:template match="text()">
<xsl:copy-of select="foo:exec('capitalize', .)"/>
</xsl:template>
</xsl:stylesheet>

[extension.xml]
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1 plus MathML 2.0//EN"
"http://www.w3.org/TR/MathML2/dtd/xhtml-math11-f.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta http-equiv="Content-type" content="application/xhtml+xml"/>
<title>Foo</title>
</head>
<body>
<h1>Foo</h1>
<p>Lorem ipsum.</p>
</body>
</html>

[ExtensionFoo.java]
package Canna;

public class ExtensionFoo {
public static Object exec(String method, Object value) {
if (value != null && (value instanceof String)) {
return ((String)value).toUpperCase();
} else {
return "hello?";
}
}
}

[TransformSample.java]
package Canna;

import java.io.File;
import javax.xml.transform.Result;
import javax.xml.transform.Source;
import javax.xml.transform.Templates;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerConfigurationException;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.stream.StreamResult;
import javax.xml.transform.stream.StreamSource;

public class TransformSample {
private static String userdir = System.getProperty("user.dir");
private static String templateName = "extension.xsl";
private static String documentName = "extension.xml";

private TransformSample() throws TransformerConfigurationException, TransformerException {
Source templateSource = new StreamSource(new File(userdir + "/ext/java/Canna/" + templateName));
Source documentSource = new StreamSource(new File(userdir + "/ext/java/Canna/" + documentName));
Result result = new StreamResult(System.out);
TransformerFactory factory = TransformerFactory.newInstance();
Templates templates = factory.newTemplates(templateSource);
Transformer transformer = templates.newTransformer();
transformer.transform(documentSource, result);
}

public static void main(String[] args) throws TransformerConfigurationException, TransformerException {
new TransformSample();
}
}

What was the result? The program output a bunch of "hello?"s. Why? The given object of a method argument wasn't the String type but DTMNodeProxy. What's DTMNodeProxy? This is com.sun.org.apache.xml.internal.dtm.ref.DTMNodeProxy, org.apache.xml.internal.dtm.ref.DTMNodeProxy, or another XSLT processor's internal type. The most affordable choice would be org.apache.xml.internal.dtm.ref.DTMNodeProxy, but Nokogiri needs to add xalan.jar to its jar list. Definitely, Nokogiri will be fat. Otherwise, pure Java Nokogiri will lose portability. Thankfully, there is an option. Users can convert values from XSL to Java and hand a desired type in to the method. So, I changed one line in XSL file:

<xsl:copy-of select="foo:exec('capitalize', string(.))"/>

OK, this worked. Every text became an upper case.


4. Inevitable API conflict

Although there were pure Java specific rules, Nokogiri style XSLT extension function seemed to work. However, the result was NOT. Puzzled. I moved the sample code above under Nokogir source tree then figured out the culprit.

When xercesImpl.jar or jing.jar is on a classpath, the sample code failed to parse the XSL file.

Sigh... I haven't found what's wrong with that yet, but the conflict lies there. Pure Java Nokogiri uses an internal API of Xerces for SAX and Jing for Relaxng processing. Both xercesImpl.jar and jing.jar are necessary APIs.

Probably, the best choice would be not to support XSLT extension of Nokogiri style right now. In future, pure Java Nokogiri might choose other XML APIs. Or, someone might give me a good advice to avoid the conflict. So, still, there is a possibility to make it happen later. At the time, this memo hopefully helps to restart implementing the XSLT extension feature.

Monday, September 20, 2010

New featues of embedding API for JRuby 1.6

This month, a lot of work for JRuby's embedding API (RedBridge) has been done. Mainly, bug fixing. While I was fixing bugs, I eventually landed to add a new feature and change the area of sharing variables. These will be in JRuby 1.6. Currently, snapshot is available at http://ci.jruby.org/snapshots/ if you want to test it. I believe the changes are good to use Ruby more naturally, but those might affect the code already written a little bit. That why I'm writing this. If you are the user of JRuby's embedding API or JSR223, be aware of upcoming changes.


1. Sharing variables becomes a receiver sensitive

Before the changes, embedding API didn't mind the difference of receivers. A receiver means Ruby's receiver, which is returned as a result of evaluation. Variables and constants to be shared are injected to a top level, in other words, runtime's top self. Also, those should be retrieved from the top level variables and constants. However, this logic didn't work perfectly.

Firstly, a trouble happened in sharing instance variables. The reason is that embedding API didn't use consistent receivers to inject and retrieve instance variables. When multiple receivers were involved, multiple values were assigned to the same key. This ended up unwanted results occasionally.

In light of this, I added three methods to ScriptingContainer.

get(receiver, key)
put(receiver, key, value)
remove(receiver, key, value)

These methods explicitly interact with a given receiver. Existing get/put/remove methods will have top self receivers for the argument.

Let me show you example. I'm going to use the Ruby code, tree_sample.rb below:

class Tree
attr_accessor :name, :shape, :type

def initialize name, shape, type
@name = name
@shape = shape
@type = type
end

def name= name
@name = name
end

def to_s
"#{name.capitalize} is a(n) #{shape} shaped, #{type} tree."
end
end

When the code is evaluated by:

ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
container.runScriptlet(PathType.CLASSPATH, "ruby/tree_sample.rb");

The runtime caches the Tree class and returns nil, which is converted to null for Java code. Then, suppose two objects are instantiated:

Object tree1 = container.runScriptlet("Tree.new('any', 'pyramidal', 'evergreen')");
Object tree2 = container.runScriptlet("Tree.new('any', 'oval', 'deciduous')");

Tree1 and tree2 above are receivers. Before the changes, container retrieved instance variables from the receivers and saved in an internal variable table tied to the instance name at the end of runScriptlet method. As you know, there're two objects. Two values of each instance variable were assigned to the single key without any receiver info. The retrieval should be receiver sensitive as well as injecting.

After the change, the instance variable values are tied to both the key and receiver. In JRuby 1.6, you'll get expected results even though multiple receivers of the same class are there. For example, suppose callMethod methods are run for each receiver object:

container.callMethod(tree1, "name=", "pine");
container.callMethod(tree2, "name=", "poplar");
System.out.println(container.callMethod(tree1, "to_s", String.class));
System.out.println(container.callMethod(tree2, "to_s", String.class));

The result is the one expected:

Pine is a(n) pyramidal shaped, evergreen tree.
Poplar is a(n) oval shaped, deciduous tree.

For clarity, let me add more lines of sharing variable related method usages:

container.put(tree1, "@name", "camellia");
container.put(tree2, "@name", "cherry");
container.put(tree1, "@shape", "oval");
container.put(tree2, "@shape", "round");

System.out.println(container.callMethod(tree1, "to_s", String.class));
System.out.println(container.callMethod(tree2, "to_s", String.class));

System.out.println("@type of tree1: " + container.get(tree1, "@type"));
System.out.println("@type of tree2: " + container.get(tree2, "@type"));

Above prints:

Camellia is a(n) oval shaped, evergreen tree.
Cherry is a(n) round shaped, deciduous tree.
@type of tree1: evergreen
@type of tree2: deciduous


So far, I talked about instance variables. Constants are also receiver sensitive in JRuby 1.6. Look at the code below:

ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
String script =
"COLOR = 'pink'\n" +
"class ColorSample\n" +
" COLOR = 'orange'\n" +
"end\n" +
"ColorSample.new";
Object receiver = container.runScriptlet(script);
System.out.println("top level: " + container.get("COLOR"));
System.out.println("class: " + container.get(receiver, "COLOR"));

There're two constants which have the same name, COLOR. While container.get("COLOR")) gets a top level contant, container.get(receiver, "COLOR")) gets a constant from a given receiver. So, the output is:

top level: pink
class: orange


How about global and local variables? Global variables are receiver insensitive because those should be referenced globally. Local variables are always injected to/retrieved from the top level. It might have been possible to tie to the receiver, but that is not a good idea because Ruby code might use gems or third party libraries. The injected local variables from Java might be extraterrestrials for those libraries and cause unexpected results.

For JSR223 user, I added an "org.jruby.embed.receiver" attribute, but this needs more work to make it happen.


2. Variable/constant value retrieval becomes lazy by default


Before the change ScriptingContainer tried to retrieve variable/constant values as much as possible at the end of runScriptlet/callMethod and EvalUnit.run methods. This was convenient to get variables/constants defined in Ruby ready for Java. However, too many values were saved in the internal variable table. Those variables were injected to Ruby code in succeeding evaluations implicitly. Obviously, this behavior made performance and memory usage worse. When multiple gems were used, it would be serious.

In JRuby 1.6, all variables/constants are retrieved lazily except persistent local variables. This means the internal variable table will have minimum key-value pairs. When a get method of ScriptingContainer is called, the requested key-value pair is saved in the variable table. When key-value pairs have been put to the internal variable table before the evaluation, those will be updated right after the evaluation. Others are not. However, persistent local variables are exception. The values are retrieved eagerly as it was done before. This is because of the internal of JRuby and is helped by the policy that local variables to be shared are only top level ones. Probably, I can consider top level local variables are few.

The change will bring a good result; however, JSR223 users might be affected. The existence of JSR223's SimpleBindings and SimpleScriptContext are always headache to me. It's very hard to add a trick to those two. Workaround is to set false in lazy option and to use the method, ScriptEngine#getContext().getAttrubute() method. Otherwise, set a key and dummy value pair to the bindings. Then, JRubyEngine updates the value right after the evaluation. By the release of 1.6, I'll improve JSR223 support.


By this change, users program will be more natural in terms of Ruby coding even on JRuby's embedding API. My work on these has not yet finished. I'll add more tests and some measures to make these on JSR223.

Saturday, September 04, 2010

My Presentation at JRubyKaigi 2010

I spoke at JRubyKaigi 2010 about JRuby embed API (RedBridge). Since attendees were almost all Japanese, I spoke in Japanese. However, my slide was written using simple English words, so that even non-Japanese speanker could understand what's going on. Don't worry about the Language of my slide.

Now, you can see my slide at http://servletgarden-point.appspot.com/slideshow. As the url shows, my slide itself was a demo built on Google App Engine using JRuby on Rails. This app uses jQuery UI and ajax. Click on buttons on the left side, then right side contents will change, and you'll see accordion. Also, you can try examples I showed in my presentation. Cloning the git repo,

git://github.com/yokolet/jrubykaigi_examples.git

and look at README to run examples.

Thursday, July 15, 2010

Cucumber on RedBridge

With JRuby's RedBridge, Ruby applications get started from Java. For example Cucumber, does. Let's see how Cucumber runs in Java code.

Firstly, people use Cucumber like this:

jruby -S cucumber addition.feature

"cucumber" is installed in $JRUBY_HOME/bin directory and looks like a command. But "cucumber" is a Ruby script, so "cucumber" can be evaluated using JRuby Embed, RedBridge, API. How about "addition.feature"? Cucumber receives the feature name via ARGV constant. This means adding a file name to ARGV will work. From these analysis, I wrote CucumberRunner.

package evergreen;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.PathType;
import org.jruby.embed.ScriptingContainer;

public class CucumberRunner {
private String jrubyhome = "/Users/yoko/Projects/jruby";
private String cucumber = jrubyhome + "/bin/cucumber";
private String feature = jrubyhome + "/lib/ruby/gems/1.8/gems/cucumber-0.8.5/examples/i18n/en/features/addition.feature";

private CucumberRunner() {
ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
container.setHomeDirectory(jrubyhome);
container.runScriptlet("ARGV << '" + feature + "'");
container.runScriptlet(PathType.ABSOLUTE, cucumber);
}

public static void main(String[] args) {
new CucumberRunner();
}
}

This printed out:

# language: en
Feature: Addition
In order to avoid silly mistakes
As a math idiot
I want to be told the sum of two numbers

Scenario Outline: Add two numbers # /Users/yoko/Projects/jruby/lib/ruby/gems/1.8/gems/cucumber-0.8.5/examples/i18n/en/features/addition.feature:7
Given I have entered <input_1> into the calculator # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:14
And I have entered <input_2> into the calculator # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:14
When I press <button> # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:18
Then the result should be <output> on the screen # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:22

Examples:
| input_1 | input_2 | button | output |
| 20 | 30 | add | 50 |
| 2 | 5 | add | 7 |
| 0 | 40 | add | 40 |

3 scenarios (3 passed)
12 steps (12 passed)
0m0.179s
/Users/yoko/Projects/jruby/lib/ruby/gems/1.8/gems/cucumber-0.8.5/bin/cucumber:19:in `load': exit (SystemExit)
from /Users/yoko/Projects/jruby/bin/cucumber:19
Exception in thread "main" org.jruby.embed.EvalFailedException: exit
at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:127)
at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1149)
at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1194)
at evergreen.CucumberRunner.(CucumberRunner.java:16)
at evergreen.CucumberRunner.main(CucumberRunner.java:20)
Caused by: org.jruby.exceptions.RaiseException: exit
at (unknown).(unknown)(/Users/yoko/Projects/jruby/lib/ruby/gems/1.8/gems/cucumber-0.8.5/bin/cucumber:19)
at Kernel.load(/Users/yoko/Projects/jruby/bin/cucumber:19)
at (unknown).(unknown)(:1)

OK. Cucumber seems to execute SystemExit in the end. Java code isn't happy with this sort of exception since evaluation itself has been succeeded. So, I'm going to catch org.jruby.embed.EmbedEvalFailedException and error message raised in Ruby.

package evergreen;

import java.io.StringWriter;

import org.jruby.embed.EvalFailedException;
import org.jruby.embed.LocalContextScope;
import org.jruby.embed.PathType;
import org.jruby.embed.ScriptingContainer;

public class CucumberRunner1 {
private String jrubyhome = "/Users/yoko/Projects/jruby";
private String cucumber = jrubyhome + "/bin/cucumber";
private String feature = jrubyhome + "/lib/ruby/gems/1.8/gems/cucumber-0.8.5/examples/i18n/en/features/addition.feature";

private CucumberRunner1() {
StringWriter errorWriter = new StringWriter();
try {
ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
container.setError(errorWriter);
container.setHomeDirectory(jrubyhome);
container.runScriptlet("ARGV << '" + feature + "'");
container.runScriptlet(PathType.ABSOLUTE, cucumber);
} catch (EvalFailedException e) {
System.out.println("Cuke says: " + e.getMessage());
} finally {
System.out.println("Cuke also says: " + errorWriter.toString());
}
}

public static void main(String[] args) {
new CucumberRunner1();
}
}

All right, CuumberRunner1 printed out:

# language: en
Feature: Addition
In order to avoid silly mistakes
As a math idiot
I want to be told the sum of two numbers

Scenario Outline: Add two numbers # /Users/yoko/Projects/jruby/lib/ruby/gems/1.8/gems/cucumber-0.8.5/examples/i18n/en/features/addition.feature:7
Given I have entered <input_1> into the calculator # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:14
And I have entered <input_2> into the calculator # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:14
When I press <button> # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:18
Then the result should be <output> on the screen # cucumber-0.8.5/examples/i18n/en/features/step_definitons/calculator_steps.rb:22

Examples:
| input_1 | input_2 | button | output |
| 20 | 30 | add | 50 |
| 2 | 5 | add | 7 |
| 0 | 40 | add | 40 |

3 scenarios (3 passed)
12 steps (12 passed)
0m0.238s
Cuke says: exit
Cuke also says: /Users/yoko/Projects/jruby/lib/ruby/gems/1.8/gems/cucumber-0.8.5/bin/cucumber:19:in `load': exit (SystemExit)
from /Users/yoko/Projects/jruby/bin/cucumber:19

Now, everything is controllable from Java.

Then, what if Cucumber result is showed up in Swing window? Isn't it interesting? So, I set StringWriter to ScriptingContainer so that all standard output from Ruby will be caught in StringWriter. Then, I added small Swing code.

package evergreen;

import java.io.StringWriter;

import javax.swing.JFrame;
import javax.swing.JTextArea;

import org.jruby.embed.EvalFailedException;
import org.jruby.embed.LocalContextScope;
import org.jruby.embed.PathType;
import org.jruby.embed.ScriptingContainer;

public class CucumberRunner2 {
private String jrubyhome = "/Users/yoko/Projects/jruby";
private String cucumber = jrubyhome + "/bin/cucumber";
private String feature = jrubyhome + "/lib/ruby/gems/1.8/gems/cucumber-0.8.5/examples/i18n/en/features/addition.feature";

private CucumberRunner2() {
StringWriter writer = new StringWriter();
StringWriter errorWriter = new StringWriter();
try {
ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
container.setError(errorWriter);
container.setHomeDirectory(jrubyhome);
container.setWriter(writer);
container.runScriptlet("ARGV << '" + feature + "'");
container.runScriptlet(PathType.ABSOLUTE, cucumber);
} catch (EvalFailedException e) {
System.out.println("Cuke says: " + e.getMessage());
} finally {
System.out.println("Cuke also says: " + errorWriter.toString());
writeOnWindow(writer.toString());
}
}

private void writeOnWindow(String message) {
JFrame frame = new JFrame();
JTextArea text = new JTextArea(message);
frame.add(text);
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.pack();
frame.setVisible(true);
}

public static void main(String[] args) {
new CucumberRunner2();
}
}

When I ran above code, a simple Swing window showed up with the result of Cucumber test.

Cucumber is Swing friendly with RedBridge. NetBeans plugin for Cucumber might be possible. Or, other Java based plugins for Cucumber might be easily implemented with RedBrdige.

Monday, July 12, 2010

Clojure uses DataMapper

Clojure is one of the JVM languages. People know this fact well. Also, people know well JRuby is among JVM languages. On the JVM languages, many people have used Java APIs from Clojure or Ruby code. But, we can do more since the JVM languages are able to communicate each other. The communication is done over the API exposed to Java such as JSR223. This means libraries and tools for a particular language are available to use in other JVM languages. For example, Clojure can choose DataMapper gem to interact databases. Usage is not so complicated. How could I make it happen? Here's a small example.


1. Installation

Clojure and JRuby themselves don't need to be installed. Grab the archives and unzip them. JRuby has installers, so you can use those if you like. After setting the path to jruby command, install 3 DataMapper gems. In this example, I used Sqlite3 for my DBMS. This is why I installed the DataMapper adapter for Sqlite3.

jruby -S gem install --no-ri --no-rdoc dm-core dm-sqlite-adapter dm-migrations



2. Run Clojure with the classpath to jruby.jar

My JRuby resides in /Users/yoko/Tools/jruby-1.5.1, so the path to jruby.jar is /Users/yoko/Tools/jruby-1.5.1/lib/jruby.jar. I added one more path. It is a directory for Ruby code for DataMapper, and the path name is /Users/yoko/Works/Samples/datamapper.
Move to the directory where Clojure was expanded, and run Clojure as in below:

java -cp clojure.jar:/Users/yoko/Tools/jruby-1.5.1/lib/jruby.jar:/Users/yoko/Works/Samples/datamapper clojure.main



3. Setup ScriptingContainer

This example uses JRuby's embed core, RedBridge, since it is easier to use compared to JSR223. The first step is to instantiate ScriptingContainer and set it up so that gems can be loaded.

user=> (import '(org.jruby.embed ScriptingContainer PathType))
org.jruby.embed.PathType
user=> (def c (ScriptingContainer.))
#'user/c
user=> (. c setHomeDirectory "/Users/yoko/Tools/jruby-1.5.1")
nil

To verify the setting was correct, I printed out Ruby's $LOAD_PATH.

user=> (. c runScriptlet "p $LOAD_PATH")
["clojure.jar", "/Users/yoko/Tools/jruby-1.5.1/lib/jruby.jar", "/Users/yoko/Works/Samples/datamapper",
"/Users/yoko/Tools/jruby-1.5.1/lib/ruby/site_ruby/1.8",
"/Users/yoko/Tools/jruby-1.5.1/lib/ruby/site_ruby/shared", "/Users/yoko/Tools/jruby-1.5.1/lib/ruby/1.8", "."]
nil

The path seems OK. Let's go forward.


4. Load gems and connect to the database

The second step is to load gems.

user=> (. c runScriptlet "require 'rubygems'; require 'dm-core'; require 'dm-migrations'")
true


Then, setup DataMapper.

user=> (. c runScriptlet "DataMapper.setup(:default, 'sqlite::memory:')")
#<RubyObject #<DataMapper::Adapters::SqliteAdapter:0xa75865>>


Writing Ruby code in method argument is not very nice. Instead, I wrote Ruby code in the *.rb files and evaluated each file loaded from classpath, /Users/yoko/Works/Samples/datamapper.

user=> (. c runScriptlet PathType/CLASSPATH "category_def.rb")
#<RubyObject #<DataMapper::Model::DescendantSet:0x3b9617>>

See category_def.rb below:

# Definition of the Category model
class Category
include DataMapper::Resource

property :id, Serial
property :category, String
property :created_at, DateTime
end

# Migration
DataMapper.auto_migrate!

Since the Category table has been created, I added three new records:

user=> (. c runScriptlet PathType/CLASSPATH "categories.rb")
true

See categories.rb below:

# Create new records
c1 = Category.new
c1.category = 'Kitchen & Food'
c1.save

c2 = Category.new
c2.category = 'Bed & Bath'
c2.save

c3 = Category.new
c3.category = 'Dining'
c3.save

Let's see what were input to Sqlite3.

user=> (. c runScriptlet "p Category.all")
[#<Category @id=1 @category="Kitchen & Food" @created_at=nil>, #<Category
@id=2 @category="Bed & Bath" @created_at=nil>, #<Category @id=3 @category="Dining" @created_at=nil>]
nil



As in this example, DataMapper worked with Clojure! MySQL and PostgreSQL are available to use from Clojure via DataMapper. Not just DataMapper. Other Ruby gems are also friends of Clojure.

Tuesday, May 04, 2010

A Small Step to Rails on RedBridge

When RedBridge (JRuby Embed) was released included in JRuby 1.4.0RC1 for the first time ever, I wrote“'Rails on Red Bridge' will be my exciting challenge" in my blog, What's the embedding API of JRuby 1.4.0RC1?. Since then, RedBridge had many improvements and bug fixes, so I tackled the issue this week. As a result, it went good. I could successfully made a small step to Rials on RedBridge. However, the process was a bit tricky, so I'm going to write down how I could make it, step by step.

The most difficult part was gem bundling. Since Rails on RedBridge is built on Java web application, all gems should be in a Java based web application structure. Besides, the gems should be looked up by a Rails app on a servlet. After I tried some, I concluded Rails 3 was the best choice because of its built-in feature of gem bundler. So, I created a Java web application project with Apache Tomcat on Eclipse and Rails 3 app within.

First, I built jruby-complete.jar from JRuby 1.5.0 source archive. At this moment, JRuby 1.5.0.RC3 is the latest.

$ tar zxfv /Users/yoko/Downloads/jruby-src-1.5.0.RC3.tar.gz
$ cd jruby-1.5.0.RC3
$ ant jar-complete
$ export JRUBY_HOME=`pwd`
$ PATH=$JRUBY_HOME/bin:$PATH

Then, I installed rails 3 and jdbc adapter gems. In my case, I used sqlite3, so I typed a gem name for jdbc sqlite3 adapter.

$ jruby -S gem install rails --pre --no-rdoc --no-ri
$ jruby -S gem install activerecord-jdbcsqlite3-adapter --no-rdoc --no-ri


Next, I created a Java web application project, Hemlock, on Eclipse. Once the project was built, I had the tree structure below:

Hemlock -+- WebContent -+- META-INF --- MANIFEST.MF
| +- WEB-INF -+- lib
| +- web.xml
+- build -+- classes
+- src

Then, I created a Rails app under WebContent/WEB-INF/lib. This is because a web application adds the path, WEB-INF/lib, into the classpath automatically. WebContent/WEB-INF/classes would have been among the choices for the same reason.

My Rails app is based on a classic tutorial, "Four Days on Rails." This original document got outdated, but Japanese version was updated up to Rails 2.3.2. You can download the PDF file from Ita-san's blog: Four Days on Rails 2.0. Online translation service might help you to read the PDF.

$ cd [path-to-workspace]/Hemlock/WebContent/WEB-INF/lib
$ jruby -S rails todo
$ cd todo

Then, I edited Gemfile and config/database.yml to replace adapter gem and setting from sqlite3 to jdbcsqlite3.

Gemfile

#gem 'sqlite3-ruby', :require => 'sqlite3'
gem 'activerecord-jdbcsqlite3-adapter'

config/database.yml

#adapter: sqlite3
adapter: jdbcsqlite3

Before proceeding further, I did scaffolding to see database connection was set up right.

$ jruby -S rails g scaffold Category category:string
$ jruby -S rake db:migrate
$ jruby -S rails server

Requesting http://localhost:3000/categories on my browser, I added three categories.


I did one more setup on Rails --- Rails metal. Rails metal was originally invented to improve perfomance. But, metal exposes Rack's bare interface, so metal makes it easy to handle Rails app in the servlet.

$ jruby -S rails g metal poller


OK, let's setup gems for a servlet.

$ jruby -S bundle install vendor --disable-shared-gems

This command installed all necessary gems in WebContent/WEB-INF/lib/todo/vendor/gems directory.

Then, I moved on to the web application setting and the servlet. I copied jruby-compelete.jar to WEB-INF/lib, first. Next, after refreshing the Hemlock Eclipse project, I added jruby-complete.jar by Properties -> Java Build Path -> Add External JARS -> [on a JAR Selection window, I chose Hemlock/WebContent/WEB-INF/lib/jruby-complete.jar] -> OK. My servlet became as in below:

package com.servletgarden.hemlock;

import java.io.IOException;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.ScriptingContainer;

public class SimpleMetalServlet extends HttpServlet {
private static final long serialVersionUID = 1L;
private String basepath;
private ScriptingContainer container;
private List loadPaths;

public SimpleMetalServlet() {
super();
}

@Override
public void init() {
basepath = getServletContext().getRealPath("/WEB-INF");
String[] paths = {
basepath + "/lib/todo/vendor/gems/bundler-0.9.25/lib/",
basepath + "/lib/todo"
};
loadPaths = Arrays.asList(paths);
container = new ScriptingContainer(LocalContextScope.THREADSAFE);
}

@Override
protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
container.setLoadPaths(loadPaths);
container.runScriptlet("ENV['BUNDLE_GEMFILE'] = \"" + basepath + "/lib/todo/Gemfile\"");
container.runScriptlet("require 'config/environment'");
Map env = getEnvMap(request);
container.put("env", env);
List result = (List) container.runScriptlet("Poller.call env");
returnResult(response, result);
}

private Map getEnvMap(HttpServletRequest request) {
Map map = new HashMap();
map.put("PATH_INFO", "/poller");
return map;
}

private void returnResult(HttpServletResponse response, List result) throws IOException {
response.setStatus(((Long)result.get(0)).intValue());
Map headers = (Map)result.get(1);
Set keys = headers.keySet();
for (String key : keys) {
if ("Content-Type".equals(key)) {
response.setContentType((String)headers.get(key));
}
}
List contents = (List)result.get(2);
for (String content : contents) {
response.getWriter().write(content);
}
}
}

When I right-clicked on the servlet source and chose Run As -> Run on Server, my Eclipse showed that simple "Hello, World!"


OK, let's look at what I did in SimpleMetalServlet. The first one is a load path setting.

Hemlock tree

Hemlock -+- WebContent -+- META-INF --- MANIFEST.MF
| +- WEB-INF -+- lib -+- todo -+- Gemfile
| | | +- app -+- controllers --- ...
| | | | +- metal --- poller.rb
| | | | +- ...
| | | +- config -+- environment.rb
| | | +- vendor -+- gems -+- bundler-0.9.25 -+- lib
| | | -+- ...
| | | +- ...
| | +- jruby-complete.jar
| +- web.xml
+- build -+- classes
+- src -+- com -+- servletgarden -+- hemlock -+- SimpleMetalServlet.java

in init() method:

basepath = getServletContext().getRealPath("/WEB-INF");
String[] paths = {
basepath + "/lib/todo/vendor/gems/bundler-0.9.25/lib/",
basepath + "/lib/todo"
};
loadPaths = Arrays.asList(paths);

in service() method:

container.setLoadPaths(loadPaths);

The path to bundler is to use bundler gem to load bundled gems. Bundler gem is installed in the todo/vendor/gems directory, but before loading bundled gem, SimpleMetalServlet can't load bundler itself. So, the servlet is telling JRuby runtime where the bundler gem is. The path to todo, Rails app top directory, is to tell JRuby where Rails app top directory is located.

The second one is the path to Gemfile.

container.runScriptlet("ENV['BUNDLE_GEMFILE'] = \"" + basepath + "/lib/todo/Gemfile\"");

Gemfile matters to Rails. To make Rails up and running, SimpleMetalServlet needs to give info about Gemfile. In this case, SimpleMetalServlet set the real path to Gemfile tied to the key "BUNDLE_GEMFILE" in ENV hash.

Then,

container.runScriptlet("require 'config/environment'");

Todo Rails app should get started by this.

Everything should be setup now, so SimpleMetalServlet is able to kick the metal method. Before talking about how the servlet kicked it, let's see what code Rails tailored.

poller.rb

# Allow the metal piece to run in isolation
require File.expand_path('../../../config/environment', __FILE__) unless defined?(Rails)

class Poller
def self.call(env)
if env["PATH_INFO"] =~ /^\/poller/
[200, {"Content-Type" => "text/html"}, ["Hello, World!"]]
else
[404, {"Content-Type" => "text/html", "X-Cascade" => "pass"}, ["Not Found"]]
end
end
end

Poller class has a class method, call, which needs an argument, env. The value, env, is a hash, in which PATH_INFO key and the value tied to it is expected. So, SimpleMetalServlet creates java.util.Map type object, puts to ScriptingContainer and evaluates Poller.call method with the argument, env.

Map env = getEnvMap(request);
container.put("env", env);
List result = (List) container.runScriptlet("Poller.call env");

private Map getEnvMap(HttpServletRequest request) {
Map map = new HashMap();
map.put("PATH_INFO", "/poller");
return map;
}


At last, response handling comes in.

returnResult(response, result);

private void returnResult(HttpServletResponse response, List result) throws IOException {
response.setStatus(((Long)result.get(0)).intValue());
Map headers = (Map)result.get(1);
Set keys = headers.keySet();
for (String key : keys) {
if ("Content-Type".equals(key)) {
response.setContentType((String)headers.get(key));
}
}
List contents = (List)result.get(2);
for (String content : contents) {
response.getWriter().write(content);
}
}

Rack based application returns the response as an array of a status code, hash of http response headers, and array of body contents. SimpleMetalServlet parses the response and sends back to a browser. Since Poller class returns so simple reponse, SimpleMetalServlet doesn't do much in this case.

I tweaked poller.rb a bit so that registered categories are showed up.

poller.rb

#[200, {"Content-Type" => "text/html"}, ["Hello, World!"]]
[200, {"Content-Type" => "application/xhtml+xml"}, [Category.all.to_xml]]

Refreshing Hemlock project on Eclipse, restarting Tomcat also on Eclipse, then requesting http://localhost:8080/Hemlock/SimpleMetalServlet gave me the XML below.


It seems Rails on RedBridge worked.

Friday, April 23, 2010

Finally, pure Java Nokgiri worked on Google App Engine

Pure Java version of Nokogiri is on the way to its very first version. It is "pure Java," not backed by libxml2, so people will expect that Nokogiri works on Google App Engine. In fact, Nokogiri on GAE is one of the purposes of Java port. Then... does it really work? Yes, it does! I could manage to get it work on GAE. However, it took pretty longer time than I thought, so I'm going to write down how I made it. Hoping this will help *pure Java* Nokogiri users.

1. Create pure Java Nokogiri gem

See Nokogiri Java Port: Help Us Finish It! for details.
$ git clone git://github.com/tenderlove/nokogiri.git
$ cd nokogiri
$ git checkout -b java origin/java
(use C Ruby for following two commands)
$ sudo gem install racc rexical rake-compiler hoe
$ rake gem:dev:spec
$ jruby -S rake java:gem

You'll have pure Java Nokogiri gem in "pkg" directory. The name is something like nokogiri-1.4.0.20100424131857-java.gem.

Then, install this gem to your JRuby.
$ jruby -S gem install [path to nokogiri]/pkg/nokogiri-1.4.0.20100424131857-java.gem


2. Create a web app project for GAE

I used Google Plugin for Ecplise to create a web application project. This is Java's web application project. I used Java project because I wrote a servlet using JRuby Embed (RedBridge). Testing Nokogiri with JRuby Embed (RedBridge) is good to know how it works.

3. Create one jar archive including Nokogiri gem

I could have put Ruby library and jar archives of Nokogiri under WEB-INF, but instead, I created one jar archive. This is because GAE limits a number of files as well as each file size. Before creating jar archive, you need to do "gem bundle" so that JRuby can load ruby files. Gem bundler is good tool to put gems together to create a jar archive. See Using the New Gem Bundler Today to learn how you can bundle gems.

First, install gem bundler.

$ jruby -S gem install bundler08

Go to the top directory of your web application. If you created a web application project using Eclipse plugin, go to [project's top directory]/war. Then, create "Gemfile." For example,

$ cd /Users/yoko/workspace/Dahlia/war
$ vi Gemfile

Here's an example of Gemfile.

# Critical default settings:
disable_system_gems
disable_rubygems
bundle_path ".gems/bundler_gems"

# List gems to bundle here:
gem "nokogiri", "1.4.0.20100424131857"

Make sure nokogiri gem is installed in your JRuby by "jruby -S gem list."

$ jruby -S gem bundle

This command creates .gems directory and puts gems listed in Gemfile under .gems/bundler_gems directory.

Next, edit .gems/bundler_gems/environment.rb as in below:
require 'rbconfig'
engine = defined?(RUBY_ENGINE) ? RUBY_ENGINE : 'ruby'
version = Config::CONFIG['ruby_version']
#require File.expand_path("../#{engine}/#{version}/environment", __FILE__)
require File.dirname(__FILE__) + "/#{engine}/#{version}/environment"

File.expand_path tries to expand the path to a full path starting from "/" (root). This full path doesn't work as a load path on Java based web application. This environment.rb file will be in a jar archive, so the path should be something like "file:/base/data/home/apps/servletgarden-dahlia/17.341465334854239323/WEB-INF/lib/gems.jar!/bundler_gems/jruby/1.8/environment." The path starting from "/" never works.

Now, you have everything under .gems/bundler_gems directory, next thing you do is to create a jar archive. For example, I did as in below:

$ cd .gems
$ jar -J-Duser.language=en cfv ../WEB-INF/lib/gems.jar bundler_gems/

In this case, gems.jar will be created in WEB_INF/lib directory. If you change the top directory in the jar archive, you will have a different load path setting I did in my servlet.

You need to one more job here. Pure Java Nokogiri gem has six jar archives in it. However, you need to move or copy Java archives in WEB-INF/lib directory to be loaded to your web application The jar archives are in .gems/bundler_gems/jruby/1.8/gems/nokogiri-1.4.0.20100424131857-java/lib/ directory. So, assuming you are in .gems direotry,

$ cp undler_gems/jruby/1.8/gems/nokogiri-1.4.0.20100424131857-java/lib/*.jar ../WEB-INF/lib/.
$ cp undler_gems/jruby/1.8/gems/nokogiri-1.4.0.20100424131857-java/lib/nokogiri/nokogiri.jar ../WEB-INF/lib/.

4. Have patched JRuby archive

JRuby 1.5.0RC2 released in April 28 2010 fixed all problems described here. Have JRuby 1.5.0RC2! Then, what you do is to create two jar archives by rake task.


This part was the most painful to make Nokogiri work on GAE. Unfortunately, you can't do this on JRuby 1.5.0RC1. You need the latest JRuby in git repo because of the problems.
The first problem is that JRuby 1.5.0RC1's source archive doesn't have gem directory. In the gem directory, the tool to split jruby-complete.jar up into two jar archives is there. Because jruby-complete.jar is too big to upload GAE, this jar file need to be split into smaller jars. GAE has --enable_jar_splitting option, but jruby-complete.jar is not just a bunch of .class files. It includes .rb files, which should be found under jruby.home. So, I don't think --enable_jar_splitting option will work. This problem was fixed in master already. If you have the latest JRuby,

$ ant dist
$ cd gem
$ jruby -S rake

will do the job.

The second problem is JRuby Embed raises NullPointerException form SystemPropertyCathcer. JRuby Embed haven't suspected "java.class.path" system property might be null, but it is on GAE. This bug is also fixed in master.

However, the third problem is now under the way. The third one, maybe serious one, is that "require 'rbconfig'" failes on GAE. When RbConfigLibrary is loaded on JRuby it raises NullPointerException because Platform.ARCH is null on GAE. I filed this in JIRA, http://jira.codehaus.org/browse/JRUBY-4749 with a patch. After that, Charles attached a new patch, which is supposed to be applied to jruby-1_5 branch. Probably, it won't take long to solve this issue. If you want to give Nokogiri a try on GAE now, apply the patch and build JRuby, and split jruby-complete.jar up into two jars.

5. Write a Servlet

Here's a very simple Servlet that uses Nokogiri.
package com.servletgarden.dahlia;

import java.io.IOException;
import java.util.Arrays;
import java.util.List;

import javax.servlet.http.*;

import org.jruby.embed.LocalContextScope;
import org.jruby.embed.ScriptingContainer;

@SuppressWarnings("serial")
public class DahliaServlet extends HttpServlet {
private ScriptingContainer container;
private String script =
"doc = Nokogiri::XML \"\"\n" +
"puts doc.to_xml";

public void init() {
String basepath = getServletContext().getRealPath("/WEB-INF");
String[] paths = {"file:"+ basepath + "/lib/gems.jar!/bundler_gems"};
List loadPaths = Arrays.asList(paths);
container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
container.getProvider().getRubyInstanceConfig().setDebug(true);
container.setLoadPaths(loadPaths);
}
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
resp.setContentType("text/plain; charset=UTF-8");
synchronized (container) {
container.setWriter(resp.getWriter());
container.runScriptlet("require 'environment'");
container.runScriptlet("require 'nokogiri'");
container.runScriptlet(script);
container.terminate();
}
}
}

The load path setting in this servlet depends on what top directory you chose for the gem.jar. Don't forget "require 'environment'" since this app uses gem bundler instead of rubygems.

If this servlet successfully works on GAE, you'll get this simple response on your browser.
<?xml version="1.0"?>
<root>
<foo/><quux/><bar/>
</root>

You can see this at http://servletgarden-dahlia.appspot.com/dahlia

Friday, April 16, 2010

RedBridge, what are new and improved in JRuby 1.5.0RC1

As you may know, Tom Enebo announced the release of JRuby 1.5.0RC1 on Apr. 15 saying "aged like a fine wine." @headius tweated "Over 1250 commits for JRuby 1.5, our largest amount of work ever for any individual release." Also, RedBridge is. RedBridge has been improved since last release based on user inputs. It's API had many changes to become more useful and organized API. Although I've already written about all changes in this blog, I'm going to put them together here for convenience.

New and Deprected Configuration API
RedBridge in JRuby 1.5.0 has a lot of Ruby runtime configuration methods. Before, those were available through getProvider().getRubyInstanceConfig() method, however, this was not a good idea. Since the method exposes JRuby's internal API, users' code might be affected by internal API changes. This fact is against to the purpose of RedBridge. RedBrdige should cover JRuby's internal API and absorb internal changes so that users don't need to fix their code by themselves. Avoid using getProvider().getRubyInstanceConfig() method as much as possible. If you want more runtime configuration methods, please request us.

New runtime configuration methods of ScriptingContainer:

  • get/setInput
  • get/setOutput
  • get/setError
  • get/setCompileMode
  • get/setRunRubyInProcess
  • get/setCompatVersion
  • get/setObjectSpaceEnabled
  • get/setEnvironment
  • get/setCurrentDirectory
  • get/setHomeDirectory
  • get/setClassCache
  • get/setClassLoader
  • get/setProfile
  • get/setLoadServiceCreator
  • get/setArgv
  • get/setScriptFileName
  • get/setRecordSeparator
  • get/setKCode
  • get/setJITLogEvery
  • get/setJITThreshold
  • get/setJITMax
  • get/setJITMaxSize

Deprecated configuration methods:

  • getRuntime()
  • getProvider().setLoadPaths()
  • getProvider().setClassCache()

Usage example:

[JRuby 1.4.0]
ScriptingContainer container = new ScriptingContainer();
container.getProvider().getRubyInstanceConfig().setJRubyHome(jrubyhome);

[JRuby 1.5.0]
ScriptingContainer container = new ScriptingContainer();
container.setHomeDirectory(jrubyhome);


New Options
Also, RedBridge got two new options: SHARING_VARIABLES and TERMINATION. The first, SHARING_VARIABLES, option turns on/off a sharing variables feature.This is an essential feature for some users while useless for other users. For those people, sharing variables is just a source of performance degradation. When the feature is turned off, the performance will be a bit better.
Usage example:

[Embed Core]
container.setAttribute(AttributeName.SHARING_VARIABLES, false);

[JSR223]
engine.getContext().setAttribute("org.jruby.embed.sharing.variables", false, ScriptContext.ENGINE_SCOPE);

The second, TERMINATION, option is for JSR223 users to call terminate. This option was added in light of RedBridge's behavior change. This is a big change, so I'll discuss more about this.

Changed Behaviors

* No termination in each evaluation and method call

RedBridge in JRuby 1.4.0 always called Ruby runtime's terminate method at the end of each evaluation and method call. This was to execute at_exit blocks and release resources automatically. The idea came from JSR223 API, which doesn't have a terminate method defined. However, I eliminated this behavior in JRuby 1.5.0 for three reasons. The first one is for performance. The terminate() method is so slow and was the biggest culprit of bad performance. The more Ruby code uses ruby files, instance variables, etc, the more it takes time. The second one is to make RedBridge's behavior more natural. Since the terminate() method fires at_exit blocks automatically, users might have unexpected results when they use third party libraries. Users should have a chance to fire at_exit blocks by themselves. The third one is that JRuby's memory leak was fixed. Thus, RedBridge doesn't need to invoke the terminate method just for releasing resources. Make sure, you have the terminate method in the right place.

Usage examples:

[Embed Core]

ScriptingContainer container = null;
try {
container = new ScriptingContainer();
container.runScriptlet(PathType.CLASSPATH, testname);
} catch (Throwable t) {
t.printStackTrace();
} finally {
container.terminate();
}

[JSR223]

ScriptEngineManager manager = new ScriptEngineManager();
ScriptEngine engine = manager.getEngineByName("jruby");
engine.eval("$x='GVar'");
engine.eval("at_exit { puts \"#{$x} in an at_exit block\" }"); // nothing is printed here
engine.getContext().setAttribute(AttributeName.TERMINATION.toString(), true, ScriptContext.ENGINE_SCOPE);
engine.eval(""); // prints "GVar in an at_exit block"


* Global runtime

When a global ruby runtime exists on a single JVM, a singleton model of RedBridge uses the global runtime in JRuby 1.5.0. This works behind the scene and seems not so attractive, but is interesting. Here's a bit tricky usage:

$ pwd
/Users/yoko/Tools/jruby-1.5.0.RC1
$ jruby --1.9 -Ctest -S irb
irb(main):001:0> require 'java'
=> true
irb(main):002:0> container=org.jruby.embed.ScriptingContainer.new
=> org.jruby.embed.ScriptingContainer@771eb1
irb(main):003:0> container.compat_version <---- --1.9 option
=> RUBY1_9
irb(main):004:0> container.current_directory <---- -Ctest option
=> "/Users/yoko/Tools/jruby-1.5.0.RC1/test"
irb(main):009:0> container.home_directory <---- jruby home
=> "/Users/yoko/Tools/jruby-1.5.0.RC1"
irb(main):010:0> container.run_scriptlet "at_exit { puts \"see you, later\" }"
=> #<Proc:0x88a1b@<script>:1>
irb(main):011:0> container.terminate
see you, later
=> nil

This new behavior might be useful in a complicated application.
However, you should be aware that setting a runtime configuration doesn't work if the global runtime is there already. This is because the runtime configuration is read only when the runtime is instantiated. You should be careful not miss the timing to set configuration.

* Lazy Runtime Initialization

RedBridge (in this case, I mean Embed Core) delays ruby runtime initialization as much as possible. This is to improve ScriptingContainer's start up time. You may know loading Ruby runtime is a huge job and takes pretty much time. This might cause frustration if it happens right after the ScriptinContainer gets started. The question is when runtime is up and running. Some of ScriptingContainer's methods will kick ruby runtime to wake up. Here's a list:

  • put()
  • runScriptlet()
  • setWriter()
  • resetWriter()
  • setErrorStream()
  • resetErrorStream()
  • setReader()

Thus, when you want configuration settings to work, you need to set them before these methods.
Meanwhile, JSR223 implementation doesn't delay ruby runtime initialization. It was not easy without breaking JSR223's requirement.

* Lazy Java Library Loading

Red Bridge doesn't load a java library while ruby runtime is initialized in JRuby 1.5.0. This is also for performance improvement. Loading libraries on to ruby runtime is quite a cumbersome job. Checking loaded library tables up to see whether a specified library has not yet loaded, judging how to load the library, then loading, caching... Even though Java library is not loaded while initialization, it will be loaded internally if necessary. Or you can load Java library explicitly:
container.runScriptlet("require 'java'");


Performance Tuning Tips
RedBridge's performance has been improved compared to older version, but you can tweak a bit more. For example, you can remove variables for sharing or clear sharing variable table at some point:
org.jruby.embed.ScriptingContainer#remove(String key)
org.jruby.embed.ScriptingContainer#clear()

RedBridge retrieves instance variables and constants as much as possible at the end of each evaluation and method call. All retrieved values are injected to runtime when the next script or method is evaluated. You can cut down the time for injection by removing unnecessary values.

Remaining jobs
RedBridge couldn't resolve all issues and has remaining jobs. Among them, OSGi and configuration on JSR223 impl would be two big issues. By the final release of JRuby 1.5.0, I want to improve these.


Finally, your input will help us to make RedBridge more perfect API. Give it a try and report us!

Thursday, February 25, 2010

JRuby Embed (Red Bridge) Update: termination and skipping sharing variables

Hers' recent update of RedBridge. Performance got much better, but the change on termination might affect your code. If you expect at_exit blocks to be executed automatically, you need to add termination.

By the recent change, Embed Core, JSR223, and BSF, all three implementations had changes in their behaviors of evaluation and method invocation. Termination is no longer executed automatically. This means, at_exit blocks are not executed at the end of every runScriplet, run and callMethod (eval, invokeMethod and invoekeFunction in JSR223). It is effective since commit 673df9f.

This rather big changes was made for two reasons. One is to avoid possible unexpected behavior caused by at_exit blocks to be executed too early. For example, a gem might have a class that has an at_exit block, which should run after other code have finished. The second reason is a performance improvement. Terminate method takes much time to complete. Because of this, evaluation and method invocation of embedding API were very slow. Now, embedding API got much better performance than before.

Then, how to do that? To terminate explicitly, call terminate method on Embed Core and BSF. It's simple. However, JSR223 doesn't have terminate method defined by the specification. So, use newly added attribute, AttributeName.TERMINATION or org.jruby.embed.termination to trun termination on. Next, evaluate blank code.

Usage examples:

[Embed Core]

ScriptingContainer container = null;
try {
container = new ScriptingContainer();
conatiner.runScriptlet(PathType.CLASSPATH, testname);
} catch (Throwable t) {
t.printStackTrace();
} finally {
container.terminate();
}

[JSR223]

ScriptEngineManager manager = new ScriptEngineManager();
ScriptEngine engine = manager.getEngineByName("jruby");
engine.eval("$x='GVar'");
engine.eval("at_exit { puts \"#{$x} in an at_exit block\" }"); // nothing is printed here
engine.getContext().setAttribute(AttributeName.TERMINATION.toString(), true, ScriptContext.ENGINE_SCOPE);
engine.eval(""); // prints "GVar in an at_exit block"


Let's move on to the second change. A new option has been added to skip sharing variables. Sharing variables is a useful or required feature for users from JSR223 and BSF background. But, it is not a necessary for others especially who have directly used JRuby's internal API. Sharing variables just slowed down the evaluations and method invocations. When sharing variables is skipped, the performance will be a bit better.

Usage example:

[Embed Core]
container.setAttribute(AttributeName.SHARING_VARIABLES, false);

[JSR223]
engine.getContext().setAttribute(AttributeName.SHARING_VARIABLES.toString(), false, ScriptContext.ENGINE_SCOPE);


Have fun with RedBridge!

Thursday, February 11, 2010

Rava - pure Ruby JavaVM

Since JRuby 1.4.0, become_java! method has been available to use to create a real Java class from Ruby class. This new feature always reminds me "Rava," which was written by Koichi Sasada(Ko1) back in 2002. Ko1 is, of course, famous Ruby committer and the author of YARV. When the days that Ruby was infamous while Java was thriving, Ko1 wrote Rava. Although I could not make it work, I think the code itself is still worth glancing at.

- Rava / JavaVM on Ruby (2) (Rava version 0.0.2)
- Rava / JavaVM on Ruby (Rava version 0.0.1)

According to Ko1, Rava is pure Ruby JavaVM and joke software. He said in the article for a Japanese Magazine that Rava could load and interpret a Java class. Rava was not perfect but had basic features, for example:

  • interprets most of bytecode

  • invokes static/non-static methods

  • reads/writes static/non-static fields

  • handles exceptions

  • runs threads


Also, Ko1 created a prototype of a JIT compiler. He made all of those in a week or so. Ko1 explained that's Ruby.

Here're excerpts from the article about Rava in depth.

  • Operand Stack : Rava used Ruby Array to handle Java's operand stack since Ruby Array has enough feature to manage the stack.

  • Types and data: Rava mapped Java's primitive types to Ruby's Number or its descendant. Java's reference type was converted into a field and Ruby object that has a reference to the original object. Java's field was mapped to Hash with keys of field names.

  • Method invocation : Rava had its method frame as in below. JVM stack was in a single array, which includes operand stack.

  • [Rava Method Frame]
    +----------+ --
    operand stack --->| | |
    stack pointer --->+----------+ |
    invoker frame info --->| | | method fame
    +----------+ |
    local variable are --->| | |
    frame pointer ---> +----------+ --
    invoker method frame --> | |
    +----------+
    JVM stack

  • JIT compiler : Rava converted bytecode into an equvalent Ruby script. To choose what bytecode should be compiled, Rava had a profiler to count a number of method invocation.



JRuby interprets Ruby on Java, while Rava interprets Java on Ruby. JRuby's become_java! converts Ruby class into Java bytecode, while Rava's JIT compiler converts Java bytecode into Ruby class. Unlike JRuby, Rava was outdated, which is a big difference; however, exploring Rava code might be fun.

Friday, February 05, 2010

Hacking JRuby - add all Hash methods to Map

I recently filed JRUBY-4528, whose patch adds all Ruby's Hash methods to a java.util.Map type object. Applying the patch, I confirmed that I could use "add_ruby_methods" method on Map type object, then, Hash methods for Map object. This would be useful especially for embedding API users since they often want to share Map object between Java and Ruby, back and forth.

What's the problem of current JRuby? When an instance of java.util.HashMap is sent into Ruby code, the object is converted into a "usable" Java object in Ruby world. This is what org.jruby.javasupport.JavaEmbedUtils.javaToRuby() method does, and we can't get Java Map converted into Ruby Hash automatically. Why? People might want to use that object as it is, HashMap type object itself, for other Java APIs used in Ruby. However, no built-in method converts Map to Hash so far although some of methods are added to.

My patch is attempt to add all Hash methods to Map type object by "add_ruby_methods" method.
For example:

irb(main):004:0> require 'java'
=> true
irb(main):005:0> jhash = java.util.HashMap.new
=> {}
irb(main):006:0> jhash.put("1", 100)
=> nil
irb(main):007:0> jhash.put("2", 200)
=> nil
irb(main):008:0> jhash.inspect
=> "{2=200, 1=100}"
irb(main):009:0> rhash = jhash.add_ruby_methods
=> {"2"=>200, "1"=>100}
irb(main):010:0> rhash.inspect
=> "{\"2\"=>200, \"1\"=>100}"
irb(main):011:0> p rhash.values
[200, 100]
=> nil
irb(main):012:0> rhash.merge!({"2"=>222, "3"=>333})
=> {"3"=>333, "2"=>222, "1"=>100}
irb(main):013:0> jhash.inspect
=> "{3=333, 2=222, 1=100}"

On jirb, I created java.util.HashMap object and put two key-value pairs using Java API, which was inspected by automatically added "inspect" method while converting. Then, I used add_ruby_methods method. After that, I could use Hash's inspect, values and merge! methods. Operations for "rhash" object above is also operations to "jhash," so when I inspected jhash, key-value pairs were also updated.

What if I create a Map object in Java code and give it to Ruby? Key-value pairs in a Java Map object was completely manipulated by Ruby. For example, see code below:

ScriptingContainer container = new ScriptingContainer(LocalContextScope.SINGLETHREAD);
ConcurrentHashMap map1 = new ConcurrentHashMap();
map1.put("a", 100);
map1.put("b", 200);
Map map2 = new HashMap();
map2.put("b", 254);
map2.put("c", 300);
container.put("h1", map1);
container.put("h2", map2);
container.put("num", 0);
String script =
"rh = h1.add_ruby_methods\n" +
"puts \"num: #{num}\"\n" +
"rh.merge!(h2.add_ruby_methods) {|k,o,n| num += 1; o+n }";
container.runScriptlet(script);
Set entries = map1.entrySet();
for (Map.Entry entry : entries) {
System.out.print(entry.getKey() + ": " + entry.getValue() + ", ");
}

outputs:

b: 454, a: 100, c: 300,

As you see, merge! method worked. All java.util.Map type such as ConcurrentHashMap or TreeMap are available to apply the method.

Possible problem of this attempt is that contents of an object after add_ruby_methods applied are Java objects. For this reason, a direct comparison to a Ruby Hash object fails. In this case, to_hash method would work since to_hash method returns real Ruby Hash of converted Java Map.

This is just an attempt, and I'm not sure this patch will be applied or not. If you think this is useful, leave a comment on JIRA.

Tuesday, February 02, 2010

JRuby Embed (Red Bridge) Gotchas: on jirb

I haven't used like that before, but there are people who want to use JRuby Embed API on jirb. I fixed a bug, http://jira.codehaus.org/browse/JRUBY-4521, and tried what I could do on jirb.

At first, I instantiated ScriptingContainer and checked what initial parameters were set.

irb(main):001:0> require 'java'
=> true
irb(main):002:0> container = org.jruby.embed.ScriptingContainer.new
=> org.jruby.embed.ScriptingContainer@ea7549
irb(main):003:0> p container.get_home_directory
"/Users/yoko/DevSpace/jruby~main"
=> nil
irb(main):004:0> p container.load_paths
[/Users/yoko/DevSpace/jruby~main/lib/profile.jar, /Users/yoko/NetBeansProjects/cirrus/build/classes]
=> nil
irb(main):005:0> p container.class_loader
org.jruby.util.JRubyClassLoader@5e2075
=> nil
irb(main):006:0> p container.current_directory
"/Users/yoko/NetBeansProjects/cirrus"
=> nil
irb(main):007:0> p container.compat_version
RUBY1_8
=> nil
irb(main):008:0> p container.supported_ruby_version
"jruby 1.5.0.dev (ruby 1.8.7 patchlevel 174) (2010-02-02 0505fb1) (Java HotSpot(TM) Client VM 1.5.0_22) [i386-java]"
=> nil

Hmmm.... interesting. Of course, no compilation at all. Perhaps, ScriptingContainer's API is useful to see jirb internal settings.

Then, how evaluations go?

irb(main):009:0> script = "puts \"Hello World\""
=> "puts \"Hello World\""
irb(main):010:0> container.run_scriptlet(script)
Hello World
=> nil
irb(main):011:0> message = "Hi, there!"
=> "Hi, there!"
irb(main):012:0> container.put("message", message)
=> nil
irb(main):013:0> container.run_scriptlet("puts \"message: #{message}\"")
message: Hi, there!
=> nil

OK, evaluations as well as sharing variables between Java(?) (or jirb?) and Ruby seem to work.
How about method call?

irb(main):014:0> script = "def say\nputs \"oh!\"\nend"
=> "def say\nputs \"oh!\"\nend"
irb(main):015:0> recv = container.run_scriptlet(script)
=> nil
irb(main):016:0> container.call_method(recv, "say", java.lang.Object.class)
:1:in `say': wrong # of arguments(1 for 0) (ArgumentError)
from :1
NativeException: org.jruby.embed.InvokeFailedException: wrong # of arguments(1 for 0)
from org/jruby/embed/internal/EmbedRubyObjectAdapterImpl.java:387:in `call'
from org/jruby/embed/internal/EmbedRubyObjectAdapterImpl.java:326:in `callMethod'
from org/jruby/embed/ScriptingContainer.java:1268:in `callMethod'
from :1

No, it failed. This is because jirb chose "public Object callMethod(Object receiver, String methodName, Object... args)" for callMethod. Unfortunately, in this case, Ruby doesn't know the difference of several callMethod methods.
Ok, then, no argument for "say" method. Will it work?

irb(main):017:0> container.call_method(recv, "say")
CallableSelector.java:196:in `assignableOrDuckable': java.lang.ArrayIndexOutOfBoundsException: 2
from CallableSelector.java:22:in `access$200'
from CallableSelector.java:163:in `accept'
from CallableSelector.java:101:in `findCallable'
from CallableSelector.java:86:in `findMatchingCallableForArgs'
from CallableSelector.java:39:in `matchingCallableArityN'
from RubyToJavaInvoker.java:170:in `findCallable'
from InstanceMethodInvoker.java:29:in `call'
from InstanceMethodInvoker.java:67:in `call'
from AliasMethod.java:66:in `call'
from CachingCallSite.java:329:in `cacheAndCall'
...
...

Oh dear, jirb was blown up.

So far, ScriptingContainer on jirb doesn't work enough but might be fun for quick hack.

........

Wrap this up.

We have java_send method and can specify exact Java method by its argument, but this also didn't work well. When I tried to run "volume" method of this Ruby code:

def volume(r)
4.0 / 3.0 * Math::PI * r ** 3.0
end

irb(main):008:0> ret = container.run_scriptlet(org.jruby.embed.PathType::CLASSPATH, "ruby/sphere.rb")
=> nil
irb(main):009:0> container.java_send :callMethod, [java.lang.Object, java.lang.String, java.lang.Class], self, "volume", java.lang.Integer.new(3)
TypeError: for method ScriptingContainer.callMethod expected [class java.lang.Object, class java.lang.String, class java.lang.Class]; got: [org.jruby.RubyObject,java.lang.String,java.lang.Integer]; error: argument type mismatch
from :1

like in the above, I got TypeError. A receiver object was the problem. Java program could cast java.lang.RubyObject to java.lang.Object, but jirb could not.

........

Wrap up, part 2.

While testing ScriptingContainer on jirb, I found a bug. Singleton model didn't see the same RubyInstanceConfig when Ruby runtime had already instantiated preceding ScriptingContainer. After the fix, some of runtime configurations can be changed through ScriptingContainer's methods. For example, I could change Ruby version to be used. The Ruby code below uses a block local variable introduced in Ruby 1.9.

# This snippet is borrowed from http://gihyo.jp/dev/serial/01/ruby/0003
# defines local variable x
x = "bear"

# A block local variable x is used in this block. (Two "x"s work together)
["dog", "cat", "panda"].each do |x|
# This x is a block local variable.
p x
break if x == "cat"
end

# This x is a local variable since it is used outside of the block.
p x

When I tried this code by setting both Ruby 1.8 and 1.9, I got appropriate outputs for both mode on jirb.

irb(main):001:0> require 'java'
=> true
irb(main):002:0> container = org.jruby.embed.ScriptingContainer.new
=> org.jruby.embed.ScriptingContainer@ea7549
irb(main):003:0> container.compat_version
=> RUBY1_8
irb(main):004:0> container.run_scriptlet(org.jruby.embed.PathType::CLASSPATH, "ruby/block-param-scope.rb")
"dog"
"cat"
"cat"
=> nil
irb(main):005:0> container.set_compat_version(org.jruby.CompatVersion::RUBY1_9)
=> nil
irb(main):006:0> container.compat_version
=> RUBY1_9
irb(main):007:0> container.run_scriptlet(org.jruby.embed.PathType::CLASSPATH, "ruby/block-param-scope.rb")
"dog"
"cat"
"bear"
=> nil

So, again, SciprtingContainer on jirb is interesting. :)

Tuesday, January 19, 2010

JRuby Embed (Red Bridge) Update: configuration and global runtime

Here're recent updates of JRuby Embed (Red Bridge). Since I wrote about updates last time, Red Bridge's API has been changed a lot around configuration. Although this would directly affect to users' code, new API is easier to use and more stable. For example, setting jruby home directory has been changed from the fist to second one below:

[JRuby 1.4.0]
ScriptingContainer container = new ScriptingContainer();
container.getProvider().getRubyInstanceConfig().setJRubyHome(jrubyhome);

[JRuby 1.5.0.dev]
ScriptingContainer container = new ScriptingContainer();
container.setHomeDirectory(jrubyhome);

As you see clearly, getProvider().getRubyInstanceConfig() was gone. This is to hide JRuby's internal API so that internal API changes won't force users to adjust thier code. This also fixes the problem that current Red Bridge API exposes internal API too much and doesn't absorb internal changes well enough. To absorb internal changes is a Red Bridge's job, and by this update, more jobs have been covered. Other than get/setHomeDirectory(), we had following configuration methods in ScriptingContainer.

  • get/setInput
  • get/setOutput
  • get/setError
  • get/setCompileMode
  • get/setRunRubyInProcess
  • get/setCompatVersion
  • get/setObjectSpaceEnabled
  • get/setEnvironment
  • get/setCurrentDirectory
  • get/setHomeDirectory
  • get/setClassCache
  • get/setClassLoader
  • get/setProfile
  • get/setLoadServiceCreator
  • get/setArgv
  • get/setScriptFileName
  • get/setRecordSeparator
  • get/setKCode
  • get/setJITLogEvery
  • get/setJITThreshold
  • get/setJITMax
  • get/setJITMaxSize

Please go to API documantation and JRuby Options to know what are those.

The second update info is about global runtime. The "global runtime" means a static instance of Ruby runtime (org.jruby.Ruby) and, now, is used in a singleton model. The singleton model is a default local context type in JRuby 1.5.0, so unless users choose threadsafe or singlethread models explicitly, they will use static, JVM global Ruby runtime. Changing it a JVM global instance, we are supposed to get exactly the same Ruby runtime instance everywhere on a single JVM. This would be useful for some cases such as using Red Brdige for DSL. Meanwhile, we should be careful when Ruby code gets run on multi-threaded environment. Since Ruby runtime holds many states, most of them will be shared globally.

Red Bridge is still under the way to the stable API and may have more changes. However, it is surely on the way to more feasible API. Please feel free to suggest new methods or features over Red Bridge.