Saturday, August 29, 2009

Yaml doesn't work on Google App Engine

While I was testing JRuby Embed API on Google App Engine, I encountered this awkward problem. Yaml never worked on GAE. Exactly the same Servlet successfully worked on GlassFish. Servlet and Ruby codes were:

# yaml_snippet.rb

require 'yaml'

content = YAML::load @text

def format element
case element
when String: print "<p>#{element}</p>"
when Array:
print "<ul>"
element.each do |child|
print "<li>"
format child
print "</li>"
end
puts "</ul>"
when Hash:
element.each do |key, value|
print "<ul><li>#{key}"
format value
print "</li></ul>"
end
end
end

content.each do |heading, paragraph|
puts "<h4>#{heading}</h4>"
paragraph.each do |element|
format element
end
end

package olive.jruby.example;

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;
import java.util.Arrays;
import java.util.List;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.jruby.embed.ScriptingContainer;
import org.jruby.javasupport.JavaEmbedUtils.EvalUnit;

public class YamlSampleServlet extends HttpServlet {
private ScriptingContainer container;
private EvalUnit yaml_unit;
private String text =
"Trees:\n" +
"- This is a small example to general HTML.\n" +
"- - Quince\n" +
" - flower: Red\n" +
"- - Apple\n" +
" - fruit: Red\n" +
"- - Maple\n" +
" - leaf: Red";

@Override
public void init() {
String classpath = getServletContext().getRealPath("/WEB-INF/classes");
List<String> loadPaths = Arrays.asList(classpath.split(File.pathSeparator));
container = new ScriptingContainer();
container.getProvider().setLoadPaths(loadPaths);
String filename = "ruby/yaml_snippet.rb";
InputStream istream = container.getRuntime().getJRubyClassLoader().getResourceAsStream(filename);
yaml_unit = container.parse(istream, filename);
}

protected void processRequest(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
container.setWriter(out);
try {
out.println("<html>");
out.println("<head>");
out.println("<title>Servlet YamlSampleServlet</title>");
out.println("</head>");
out.println("<body>");
out.println("<h3>Servlet YamlSampleServlet at " + request.getContextPath() + "</h3>");
container.put("@text", text);
yaml_unit.run();
out.println("</pre></body>");
out.println("</html>");
} finally {
out.close();
}
}

@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
processRequest(request, response);
}

@Override
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
processRequest(request, response);
}

@Override
public String getServletInfo() {
return "Yaml Sample";
}
}

And the programs produced:

<html>
<head>
<title>Servlet YamlSampleServlet</title>
</head>
<body>
<h3>Servlet YamlSampleServlet at /Olive</h3>
<h4>Trees</h4>
<p>This is a small example to general HTML.</p><ul><li><p>Quince</p></li><li><ul><li>flower<p>Red</p></li></ul></li></ul>
<ul><li><p>Apple</p></li><li><ul><li>fruit<p>Red</p></li></ul></li></ul>
<ul><li><p>Maple</p></li><li><ul><li>leaf<p>Red</p></li></ul></li></ul>
</pre></body>
</html>

However, this test servlet never worked on GAE because of the exception:

[java] yaml/constants:15:in `const_missing': uninitialized constant YAML::Yecht::Resolver (NameError)
[java] from yaml:88
[java] from yaml:2:in `require'
[java] from ruby/yaml_snippet.rb:2
[java] ...internal jruby stack elided...
[java] from Module.const_missing(yaml:88)
[java] from (unknown).(unknown)(yaml:2)
[java] from (unknown).(unknown)(yaml:2)
[java] from Kernel.require(ruby/yaml_snippet.rb:2)
[java] from (unknown).(unknown)(:1)

JRuby has yaml library under the "builtin" directory in its source tree, so all of necessary scripts here are in jruby.jar as well as jruby-complete.jar. But, yaml library needs jruby.home system property to be set correctly, so jruby-complete.jar is the only choice on a web application. In case of an application, I could set jruby.home explicitly as in below and got the result as I expected:

package brick;

import java.io.InputStream;
import org.jruby.embed.ScriptingContainer;
import org.jruby.javasupport.JavaEmbedUtils.EvalUnit;

public class YamlSample {

private String filename = "ruby/yaml_snippet.rb";
private String text =
"Trees:\n" +
"- This is a small example to general HTML.\n" +
"- - Quince\n" +
" - flower: Red\n" +
"- - Apple\n" +
" - fruit: Red\n" +
"- - Maple\n" +
" - leaf: Red";

private YamlSample() {
System.setProperty("jruby.home", "/Users/yoko/Works/080909-jruby/jruby~main");
ScriptingContainer container = new ScriptingContainer();
InputStream istream = container.getRuntime().getJRubyClassLoader().getResourceAsStream(filename);
EvalUnit yaml_unit = container.parse(istream, filename);
container.put("@text", text);
yaml_unit.run();
}

public static void main(String[] args) {
new YamlSample();
}
}

I tried to copy all yaml library under the source tree of my web application, and clean & build, redeploy, and request again... with no luck. Sigh... I tried to set the path to the yaml library in jar archive,

String classpath = getServletContext().getRealPath("/WEB-INF/lib/jruby-complete.jar!/builtin");
List loadPaths = Arrays.asList(classpath.split(File.pathSeparator));
container = new ScriptingContainer();
container.getProvider().setLoadPaths(loadPaths);

... no luck. The same exception, as ever.

Since the program worked on GlassFish, I guess the difference in class loading mechanism between GlassFish and GAE might have caused the exception on GAE. But, I haven't figured the culprit out so far. Any idea?

Wednesday, August 26, 2009

NekoBean Fall Version


NetBeans' mascot, NekoBean, is enjoying cool air in fall surrounded by colored foliage.

More at:

http://nekobean.net/2009/08/post-18.html.

Tuesday, August 25, 2009

JRuby Embed API Update: Servlet Examples

I added Servlet Examples section in JRuby Embed API Wiki. Right now, just three examples are in that section. (I'll add more examples later.) Those are:

  • HelloWorldServlet
    Simple "Hello World" example, but helpful to get started.

  • GreetingServlet
    Two methods written in Ruby are called from Servlet.

  • SortableServlet
    Java interface is implemented in two ways in Ruby.


I tested these Servlets on Google App Engine and felt relieved since all three Servlets worked well. I've wanted to verify that Embed API works on GAE, which has some restrictions in programming on it. Embed API doesn't use any unsupported API, so there should not be any problem. However, I realized that I had to specify a classpath explicitly, and the classpath to be specified should not include appengine-tools-api.jar. If no classpath is given, Embed API sees java.class.path system property that has a path to appengine-tools-api.jar. This means, JRuby tries to load appengine-tools-api.jar onto Ruby runtime. The result is ... simply getting an exception. Thus, when Embed API is used with Servlet, especially, with Google App Engine, setting classpath is really important.

Embed API has two ways of setting a classpath. One is to use org.jruby.embed.class.path system property. This is easy, but not a preferred way in a web application. Since system property is common on Java VM, so the value is shared by every Servlet in more than one war archives and mutliple web applications on a single web application server. Some servlet might set classpath "A" using org.jruby.embed.class.path. At the same time another servlet might try to set classpath "B" using org.jruby.embed.class.path. We don't know what classpath is actually used.

Another way of setting classpath is to use setLoadPaths method of API. For example,

public class HelloWorldServlet extends HttpServlet {
private ScriptingContainer container;

@Override
public void init() {
String classpath = getServletContext().getRealPath("/WEB-INF/classes");
List loadPaths = Arrays.asList(classpath.split(File.pathSeparator));
container =
new ScriptingContainer(LocalContextScope.SINGLETHREAD);
container.getProvider().setLoadPaths(loadPaths);
}
...

Technically, we don't need to set the classpath to /WEB-INF/classes, since it has been already set by a server. But, some path with no further trouble is needed, so I chose that.

JRuby Embed API has a public method to set classpath, but JSR 223 implementation is unable to have such method. The specification doesn't define such method. The only way to set classpath for JSR 223 implementation is to use system property. It is true also in RedBridge. So, be careful to choose a harmless classpath to all Servlets on a web application server.

Saturday, August 15, 2009

RedBridge and JRuby Embed API update

I updated both JRuby Embed API and RedBridge and the latest version is, now, 0.0.1.1. By this update, three types of local variable behaviors were added in light of the discussion, http://www.nabble.com/Call-for-discussion-about-embed-API-tc24528478.html. Before the update of Embed API and RedBridge, Ruby's local variables always survived over the multiple evaluations. Thus, local variables used in the first script evaluation were always reused in the second, third, or fourth evaluation even though scripts has no relation to each other. Of course, users could delete unwanted local variables explicitly before the following evaluations went on, but this didn't happen in default. This feature was useful especially for ex-BSF users; however, it was not semantically correct. So, Tom Enebo concerned about it. During the discussion was going on, a nice idea of a toggle-able local variable was suggested (Thank you, Adam ;) ), and seemed to satisfy conflicting needs. The latest version supported the toggle-able local variable.

New local variable behavior has three options, transient, persistent and global. The first default behavior, transient, is a faithful behavior to Ruby semantics. So, local variables vanish after each evaluation. Java programs can't get local variables used in Ruby scripts. If you want to use the same value or object as a local variable in more than one script, you need to reset it again and again. However, instance and global variables survive over the evaluations as those were in the previous version.

The second variable behavior, persistent, is the behavior that the previous version did. Thus, the same local variables can be used in multiple script evaluations. Also, those can be retrieved from Ruby and used in Java. As well as a local variable, an instance and global variables, and a constant are persistent over multiple evaluations.

Example for JRuby Embed API:

package brick;

import java.util.Map;
import java.util.Set;
import org.jruby.embed.ScriptingContainer;
import org.jruby.embed.LocalVariableBehavior;

public class Sample1 {

private Sample1() {

ScriptingContainer container = new ScriptingContainer(LocalVariableBehavior.PERSISTENT);
container.runScriptlet("p=9.0");
container.runScriptlet("q = Math.sqrt p");
container.runScriptlet("puts \"square root of #{p} is #{q}\"");
Map m = container.getVarMap();
Set<String> keys = container.getVarMap().keySet();
for (String key : keys) {
System.out.println(key + ", " + m.get(key));
}
System.out.println("Ruby used: p = " + container.get("p") +
", q = " + container.get("q"));
}

public static void main(String[] args) {
new Sample1();
}
}

Example for RedBridge (JSR223):

package redbridge;

import java.util.Set;
import javax.script.Bindings;
import javax.script.ScriptContext;
import javax.script.ScriptEngine;
import javax.script.ScriptException;
import org.jruby.embed.jsr223.JRubyScriptEngineManager;

public class EvalStringSample {

private EvalStringSample() throws ScriptException {
System.out.println("[" + getClass().getName() + "]");
System.setProperty("org.jruby.embed.localvariable.behavior", "persistent");
JRubyScriptEngineManager manager = new JRubyScriptEngineManager(Thread.currentThread().getContextClassLoader());
ScriptEngine engine = manager.getEngineByName("jruby");
engine.eval("p=9.0");
engine.eval("q = Math.sqrt p");
engine.eval("puts \"square root of #{p} is #{q}\"");

Bindings bindings = engine.getBindings(ScriptContext.ENGINE_SCOPE);
Set<String> keys = bindings.keySet();
for (String key : keys) {
System.out.println(key + ", " + bindings.get(key));
}
System.out.println("Ruby used: p = " + engine.get("p") +
", q = " + engine.get("q"));
}

public static void main(String[] args) throws ScriptException {
new EvalStringSample();
}
}

Output:

square root of 9.0 is 3.0
MANT_DIG, 53
MAX_10_EXP, 308
DIG, 15
MIN_EXP, -1021
ROUNDS, 1
MAX, 1.7976931348623157E308
RADIX, 2
EPSILON, 2.220446049250313E-16
MIN, 4.9E-324
q, 3.0
p, 9.0
MIN_10_EXP, -307
MAX_EXP, 1024
Ruby used: p = 9.0, q = 3.0


The third variable behavior, global, is a backwards compatibility option for users who have used JSR223 reference implementation released form scripting.dev.java.net. The reference implementation (RI) uses Ruby's global variable to share variables between Java and Ruby. And the name of variables used in Java has the same form as the one of a local variable in Ruby. I mean, Java sees "message" while Ruby sees "$message." However, Embed API and RedBridge enable not only the global variable but also the instance and local variable and constant sharing. On Redbridge, when people use the name "message" in Java, they also use "message" in Ruby. When it is "$message" in Java, also, "$message" in Ruby. So that RI users can move on to RedBridge easily, I added this local variable behavior.

Example for RedBridge (JSR223):

# greetings_globalvars.rb

def greet
message = "How are you? #{$who}."
end

def sayhi
$, = ","
$\ = "\n"
print "Hi", $people
$, = ""
$\ = nil
end

def count
$people.size + 1
end

// OldVariableBehaviorSample.java
package redbridge;

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.Reader;
import java.util.ArrayList;
import java.util.List;
import javax.script.Invocable;
import javax.script.ScriptEngine;
import javax.script.ScriptException;
import org.jruby.embed.jsr223.JRubyScriptEngineManager;

public class OldVariableBehaviorSample {
private final static String basedir = "/Users/yoko/NetBeansProjects/Birch";

private OldVariableBehaviorSample()
throws ScriptException, FileNotFoundException, NoSuchMethodException {
System.setProperty("org.jruby.embed.localvariable.behavior", "old");
JRubyScriptEngineManager manager = new JRubyScriptEngineManager();
ScriptEngine engine = manager.getEngineByName("jruby");
String filename = basedir + "/src/ruby/greetings_globalvars.rb";
Reader reader = new FileReader(filename);
engine.put("who", "Anakin");
List people = new ArrayList();
people.add("Obi-Wan");
people.add("C-3PO");
people.add("R2-D2");
engine.put("people", people);
engine.eval(reader);
Object[] args = null;
Object result = ((Invocable)engine).invokeFunction("greet", args);
System.out.println(result.toString());
((Invocable)engine).invokeFunction("sayhi", args);
result = ((Invocable)engine).invokeFunction("count", args);
System.out.println("counted: " + result.toString());
}

public static void main(String[] args)
throws ScriptException, FileNotFoundException, NoSuchMethodException {
new OldVariableBehaviorSample();
}
}

Output:

How are you? Anakin.
Hi,[Obi-Wan, C-3PO, R2-D2]
counted: 4


See wiki pages for details.
JRuby Embed API: http://kenai.com/projects/jruby-embed/pages/Home
RedBridge: http://kenai.com/projects/redbridge/pages/Home

Friday, August 07, 2009

RedBridge Update: JRubyScriptEngineManager

Today, I added two classes, JRubyScriptEngineManager and ServiceFinder, to RedBridge (JSR 223 JRuby engine). ServiceFinder is used from JRubyScriptEngineManager, and not for users. This update will be helpful especially for OS X users. Now, RedBridge works on both JDK 1.5 and 1.6 on OS X Java Update 4.

Since its first release, RedBridge hasn't had JRubyScriptEngineManager mainly because of copyright. JSR 223 JRuby Engine released from Scripting Project at dev.java.net has the same name and behavior class. Although I wrote that class, I couldn't simply include it in RedBridge since Sun has copyright. Thus, I've tested RedBridge on JDK 1.6 though RedBridge itself was compiled on JDK 1.5. However, after OS X's Java has been updated in last June, JDK 1.6's service discovery failed to locate RedBridge. So, I decided to write it. Like other classes of RedBridge, I totally rewrote JRubyScriptEgineManager, too, so that RedBridge won't suffer from unexpectd legal issues. The new JRubyScriptEngineManager isn't just an modified version of the old one. I wrote it as simple as possible because, I think, JSR 223 is, in many cases, used with frameworks. Keeping it vanilla would be better for users. Less headache.

Now, the snippet will be:

package redbridge;

import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
import org.jruby.embed.jsr223.JRubyScriptEngineManager;

public class EvalStringSample {

private EvalStringSample() throws ScriptException {
System.out.println("[" + getClass().getName() + "]");
System.setProperty("org.jruby.embed.localcontext.scope", "singlethread");
//ScriptEngineManager manager = new ScriptEngineManager();
JRubyScriptEngineManager manager = new JRubyScriptEngineManager();
ScriptEngine engine = manager.getEngineByName("jruby");
engine.eval("p=9.0");
engine.eval("q = Math.sqrt p");
engine.eval("puts \"square root of #{p} is #{q}\"");
System.out.println("q = " + engine.get("q"));
}

public static void main(String[] args) throws ScriptException {
new EvalStringSample();
}
}

Result:

[redbridge.EvalStringSample]
square root of 9.0 is 3.0
q = 3.0


JRubyScriptEngineManager can have a classloader in its constructor argument. When no classloader is given, JRubyScriptEngineManager uses System classloader. For example, to use a current context classloader:

JRubyScriptEngineManager manager =
new JRubyScriptEngineManager(Thread.currentThread().getContextClassLoader());

See, Wiki at the RedBridge project for other usages.