Extending JRuby, Compile and Jar Java Extension Code

I wrote about how to extend JRuby by Java in my blog post, Extending JRuby. At that time, I didn't compile Java code since Eclipse performed that automatically. To jar Java code, I used jar command and made an archive manually. However, this is definitely not nice. In general, Java people use maven or ant to package Java code into a single jar. Although maven and ant are among choices to package JRuby extension code, there is a more Ruby-like way of packaging. It is rake-compiler. Nokogiri uses rake-compiler for both CRuby and pure Java versions to compile and package.


Rake-compiler might be known as a cross-compiling tool, but it is also a compiling tool for Java code. Not like maven and ant, we don't write XML files. Instead, we write Rakefile. This means we compile and package Java code by a rake task on JRuby.


Let's get started. First, you need to install rake-compiler gem to your *JRuby* since it is Java code compilation. Make sure you are using JRuby.

jruby -S rake gem install rake-compiler

Next, make sure your Java code is under an *ext* directory. Rake-compiler assumes extension codes are under the *ext* directory in terms of Convention over Configuration. My Eclipse project created a *src* directory for Java sources, so I refactored the name from src to ext on Eclipse. Then, my Rakefile became as in below:

# -*- ruby -*-

require 'rubygems'
require 'rake/javaextensiontask'

Rake::JavaExtensionTask.new('commons/math/fraction') do |ext|
 jruby_home = ENV['MY_RUBY_HOME'] # this is available of rvm
 jars = ["#{jruby_home}/lib/jruby.jar"] + FileList['lib/*.jar']
 ext.classpath = jars.map {|x| File.expand_path x}.join ':'
 ext.name = 'commons/math/poplar'
end

To compile Java code for JRuby extension, requiring 'rake/javaextensiontask' and writing Rake::JavaExtensionTask are all you need. The question would be what should be in JavaExtensionTask. The constructor argument specifies the directory Java code resides. The jruby_home is to know where jruby.jar is. jars is just an array to set ext.classpath effectively. You may assign whole classpath directly to ext.classpath variable. If you are on Windows, you need to change classpath delimiter from ':' to ';' I also specified ext.name parameter. Without this, JavaExtensionTask creates an jar archive, "lib/commons/math/fraction.jar" from the constructor arguments. In my case, this name is confusing since I have lib/commons/math/fraction.rb, too. Once JRuby finds fraction.rb, JRuby happily quits searching loop. So, lib/commons/math/fraction.jar won't be loaded. You can also specify other parameters like Ruby exntesion, for example,

• ext.gem_spec
  
• ext.tmp_dir
  
• ext.lib_dir
  
• ext.platform
  
• ext.config_options
  
• ext.source_pattern
  

For Java extension, we can set

• source_version
  
• target_version

The default values of these parameters are 1.5.

Everything is ready. Let's compile and jar Java extension code. On your terminal, type

rake compile

Just this compiles and jars Java extension code. The created jar archive is lib/commons/math/poplar.jar. This directory is natural for JRuby extension gems. You can see other rake tasks from rake -T. On your terminal, rake clean, rake clobber and two compile tasks will show up. Next time, you might type

rake clean
rake compile

Like above, rake-compiler works to compile and jar JRuby Java extension code. This might be more familiar and better to J/Rubyists.


All of my code are on GitHub, https://github.com/yokolet/Poplar

Rubinius on JRuby … ?

Of course, compiled Rubinius binary, *.rbc, file doesn’t work on JRuby. This post is about JRuby’s rubinius branch. I’m not sure how may people are aware that, but JRuby does have a rubinius branch. As far as I looked at that branch, it is not to be merged into master, at least, in near future. Maybe it is headius’ pet project at this moment, and is implemented as a JRuby extension. Yes, it is a JRuby extension. The branch attempts “extending JRuby,” like I wrote about in my blog post (http://yokolet.blogspot.com/2011/05/extending-jruby.html). Since this extension is still small, it would be a good practice to figure out how extension works. So, I’m going to write how you can decipher existing JRuby extension. Hopefully, this will help you to write your own extension.


To try the rubinius branch, you have two preparations to be done. The first one is to build JRuby of the rubinius branch. It is easy. Just clone out JRuby, checkout rubinius branch, and run ant command as in below:

git clone git://github.com/jruby/jruby.git
cd jruby
git checkout rubinius

ant clean-all
ant

Next, you need Rubinius source. You don’t need to build rubinius just to see how it works. But, the rubinius extension uses Rubinius’ kernal sources, which is included only in the source archive. That’s why you need the source. Rubinius source archive is available to download from http://rubini.us/. Get the archive and unzip it.


Up until now, you had rubinius branch JRuby and Rubinius source. Set the environment variables. Suppose the JRuby’s home directory is /Users/yoko/Projects/jruby, then, set JRUBY_HOME and PATH like in below. Adjust them to fit in to your system:

export JRUBY_HOME=/Users/yoko/Projects/jruby
PATH=$JRUBY_HOME/bin:$JRUBY

Suppose, Rubinius source are in /Users/yoko/Projects/rubinius-1.2.2, then set RBX_KERNEL as in below:

export RBX_KERNEL=/Users/yoko/Projects/rubinius-1.2.2/kernel

Everything should be ready. Let’s try this out.

bash-3.2 rubinius$ jruby -S irb
irb(main):001:0> require 'rubinius'
=> true

Yay! Rubinus was successfully loaded on JRuby. What’s next? Look at the RubiniusLibrary.java (https://github.com/jruby/jruby/blob/rubinius/src/org/jruby/ext/rubinius/RubiniusLibrary.java). At the line 51, “Rubinius” module is defined.

RubyModule rubinius = runtime.getOrCreateModule("Rubinius");

So, there should be a constant, Rubinius.

irb(main):002:0> Rubinius
=> Rubinius
irb(main):003:0> Rubinius.class
=> Module

So far, so good. Then, at the line 56, you can see

RubyTuple.createTupleClass(runtime);

This means you should go to RubyTuple.java (https://github.com/jruby/jruby/blob/rubinius/src/org/jruby/ext/rubinius/RubyTuple.java). On the line 55-62 of RubyTuple.java, createTupleClass method is defined. In this method, "Tuple" class is defined under the "Rubinius" module. Then, annotated methods, which have Java annotation @JRubyMethod, are defined. Looking at the rest of the code in RubyTuple.java, you can see three annotated methods (new, [], and []=), and one override method (dup) are there. Let’s try these.

irb(main):013:0> Rubinius::Tuple
=> Rubinius::Tuple
irb(main):014:0> tuple = Rubinius::Tuple.new 3
=> #<Rubinius::Tuple:0x3df89785>

This constructor needs one argument because rbNew method is defined as:

public static IRubyObject rbNew(ThreadContext context, IRubyObject tupleCls, IRubyObject cnt) {

Here's the rule. First two arguments are given internally, and the rest of the arguments are given from users. So, I typed “3” as an argument. Let’s keep going on.

irb(main):018:0> tuple[0]
=> nil
irb(main):019:0> tuple[0]=123
=> 123
irb(main):020:0> tuple[0]
=> 123
irb(main):021:0> tuple_dup = tuple.dup
=> #
irb(main):022:0> tuple_dup[0]
=> 123

All right, methods worked.


Next, get back to RubiniusLibrary.java and let’s look at the lines 84-88.

runtime.getObject().deleteConstant("Hash");
runtime.getLoadService().lockAndRequire(rbxHome + "/common/hash.rb");
RubyClass hash = (RubyClass)runtime.getClass("Hash");
hash.defineAnnotatedMethods(RubiniusHash.class);
runtime.setHash(hash);

Soooo interesting! Could you figure out what’s going on here? Hash is redefined using Rubinius code! JRuby’s Hash is entirely written in Java (https://github.com/jruby/jruby/blob/rubinius/src/org/jruby/RubyHash.java). But, once “require rubinius” is done, the Hash is totally replaced by Rubinius’ Hash, which IS written in Ruby. See? JRuby can be extended also in Ruby, not just Java. To do double check this, let’s add one line in initialize method of kernel/common/hash.rb:

def initialize(key, key_hash, value)
  @key      = key
  @key_hash = key_hash
  @value    = value
  @next     = nil
  puts "Rubinius Hash!!"   # this line is added
end

Then, restart irb and re-request rubinius.

bash-3.2 rubinius$ jruby -S irb
irb(main):001:0> require 'rubinius'
=> true
irb(main):002:0> h = Hash.new
Rubinius Hash!!
=> {}

Yay, Hash is really Rubinius’ Hash. What an idea!


As I wrote, you have a lot of options for “extending JRuby.” You can extend using mature Java APIs and, also, cutting edge Ruby code. Why don’t you try this fantastic extension? It should be fun.

Extending JRuby

As pure Java Nokogiri does, we can extend JRuby writing a library backed by Java API. Other than Nokogiri, Weakling (https://github.com/headius/weakling), Warbler(https://github.com/nicksieger/warbler), JSON(https://github.com/flori/json) and more are examples of JRuby extension by Java. If you use google code search with a keyword, "BasicLibraryService," you'll find some more gems. This BasicLibraryService is a sign that the gem is implemented by Java. BasicLibraryService is an interface and has just one method, basicLoad(Ruby runtime). Simple. However, questions might come up in people's mind. What should I write in basicLoad method? How is it called? Not many answers are out there. The helpful answer I could find was a comment on LoadService.java (or LoadService19.java for 1.9 mode). But, it would be still short to write a JRuby extension for JRuby users who want to write their own. So, I wrote a sample code to see how JRuby can be extended. This sample is quite a simple one and far from real JRuby extensions such as pure Java Nokogiri or others. But, the first thing is to understand how it works. This sample will help to get started.


Before going deeper, let's look at a usage of Java API directly from JRuby. I chose Apache Commons Math API (http://commons.apache.org/math/). This API is interesting. It makes many mathematical calculations easy and natural. Among them, I picked up a fraction package. Everybody knows. In Japan, elementary school kids study how to add, subtract or common denominator, etc. It should be easy, but neither Java or Ruby doesn't have such API in a standard library.

Below is a JRuby code that uses fraction Java API directly. This code adds up reciprocals of 1 to 4, Harmonic series of n = 4. The answer is obvious, 1/1 + 1/2 + 1/3 + 1/4 = 25/12.

require 'java'
$: << '/Users/yoko/Tools/commons-math-2.2'
require 'commons-math-2.2'

java_import org.apache.commons.math.fraction.Fraction

f = Fraction.new(1, 1)
(2..4).each do |i|
  f = f.add(Fraction.new(1, i))
end
puts f

My commons-math-2.2.jar is in /Users/yoko/Tools/commons-math-2.2 directory, so I added that path to $LOAD_PATH, then, required that jar archive. The .jar suffix is optional when requiring something on JRuby. JRuby searches from every possible paths adding .class, .rb, .jar or .bundle suffixes. Next, I imported Fraction class and calculated in a straightforward way.

Let's think how this code can be improved to more Ruby like one. Ruby programmer might like f.add!(something) rather than f = f.add(something). So, in this JRuby extension sample, I implemented "add!" method.

Firstly, I wrote FractionService class, which implements BasicLibraryService interface. But, wait. API design should come in before starting it because XXXService class works based on convention over configuration. Java's package name and Ruby's module structure must coincide. In my design, the Fraction class is Commons::Math::Fraction::Fraction in Ruby. This means FractionService class should be in a commons.math.fraction package, and require statement in Ruby should be "require 'commons/math/fraction/fraction.' This is how basicLoad() method is called.

Next would what we should write in basicLoad() method. In general, defining module structures/classes and object allocators are done in this method. Ola Bini's blog, "The JRuby Tutorial #4: Writing Java extensions for JRuby" (http://ola-bini.blogspot.com/2006/10/jruby-tutorial-4-writing-java.html) would be worth to read how to write the method. My simple FractionService became as in below:

package commons.math.fraction;

import java.io.IOException;

import org.jruby.Ruby;
import org.jruby.RubyClass;
import org.jruby.RubyModule;
import org.jruby.runtime.ObjectAllocator;
import org.jruby.runtime.builtin.IRubyObject;
import org.jruby.runtime.load.BasicLibraryService;

public class FractionService implements BasicLibraryService {

   @Override
   public boolean basicLoad(Ruby runtime) throws IOException {
       RubyModule commons = runtime.defineModule("Commons");
       RubyModule math = commons.defineModuleUnder("Math");
       RubyModule fractionModule = math.defineModuleUnder("Fraction");
       RubyClass fraction = fractionModule.defineClassUnder("Fraction", runtime.getObject(), FRACTION_ALLOCATOR);
       fraction.defineAnnotatedMethods(Fraction.class);
       return true;
   }
  
   private static ObjectAllocator FRACTION_ALLOCATOR = new ObjectAllocator() {
       public IRubyObject allocate(Ruby runtime, RubyClass klazz) {
           return new Fraction(runtime, klazz);
       }
   };
}

Then, I wrote commons.math.fraction.Fraction class. In this class, I defined "add!" and "to_s" methods. We can't use "!" in a method name in Java, so the Java method name is add_bang instead. Ruby method name is define in @JRubyMethod annotation. Also, I wrote Ruby's constructor method "new," which is "rbNew" method in Java. The "new" method should be a class method, so it is a static method in Java. Annotations of methods are important. Three annotated *JRubyMethods* in Fraction class get fired up by fraction.defineAnnotatedMethods(Fraction.class); in FractionService class. Because of this, we can use Java methods in Ruby. See my Fraction class below:

package commons.math.fraction;

import org.jruby.Ruby;
import org.jruby.RubyClass;
import org.jruby.RubyObject;
import org.jruby.anno.JRubyClass;
import org.jruby.anno.JRubyMethod;
import org.jruby.runtime.Arity;
import org.jruby.runtime.ThreadContext;
import org.jruby.runtime.builtin.IRubyObject;

@JRubyClass(name="Commons::Math::Fraction")
public class Fraction extends RubyObject {
   private org.apache.commons.math.fraction.Fraction j_fraction = null;
  
   @JRubyMethod(name="new", meta = true, rest = true)
   public static IRubyObject rbNew(ThreadContext context, IRubyObject klazz, IRubyObject[] args) {
       Fraction fraction = (Fraction) ((RubyClass)klazz).allocate();
       fraction.init(context, args);
       return fraction;
   }

   public Fraction(Ruby runtime, RubyClass klass) {
       super(runtime, klass);
   }
  
   void init(ThreadContext context, IRubyObject[] args) {
       Arity.checkArgumentCount(context.getRuntime(), args, 2, 2);
       int numerator = (Integer) args[0].toJava(Integer.class);
       int denominator = (Integer) args[1].toJava(Integer.class);
       j_fraction = new org.apache.commons.math.fraction.Fraction(numerator, denominator);
   }
  
   org.apache.commons.math.fraction.Fraction getJFraction() {
       return j_fraction;
   }
  
   @JRubyMethod(name = "add!")
   public IRubyObject add_bang(ThreadContext context, IRubyObject other) {
       if (other instanceof Fraction) {
           org.apache.commons.math.fraction.Fraction other_fraction = ((Fraction)other).getJFraction();
           j_fraction = j_fraction.add(other_fraction);
           return this;
       } else {
           throw context.getRuntime().newArgumentError("argument should be Commons::Math::Fraction type");
       }
   }
  
   @JRubyMethod
   public IRubyObject to_s(ThreadContext context) {
       return context.getRuntime().newString(j_fraction.toString());
   }
}

I haven't written Rakefile for packaging at this moment, so I manually create jar archive of two Java classes.

jar -J-Duser.language=en -cvf ../lib/commons/math/poplar.jar commons

OK, all Java classes are ready for my simple JRuby extension, so let's work on Ruby code. We might have an option to require Java classes directory, but that is not nice. Since users themselves must require FractionService or other, internal change will affect users code. Besides, it doesn't look like Rubygems. So, I wrote commons_math_fraction.rb to hook up FractionService.

require 'commons-math-2.2'
require 'commons/math/fraction'

Surely, this code needs to be brush up, for example, paths. But, I kept simple since this sample is to understand the idea of extending JRuby.
Then, one more Ruby code, commons/math/fraction.rb:

require 'commons/math/poplar'
require 'commons/math/fraction/fraction'

module Commons
 module Math
   module Fraction
   end
 end
end

The first line requires poplar.jar archive, and the second does FractionService class.


Everything is ready, so let's write Ruby code using this tiny, shiny JRuby extension. The file name is fraction_sample.rb:

require 'java'

$: << '/Users/yoko/Documents/workspace/Poplar/lib'
require 'commons_math_fraction'
f = Commons::Math::Fraction::Fraction.new(1, 1)
(2..4).each do |i|
  f.add!(Commons::Math::Fraction::Fraction.new(1, i))
end
puts f

Since my JRuby extension is not yet gem, GEM_PATH or other gem loading ways don't work. I set a load path to my *Poplar* project. Under the lib directory, I have jars and Ruby files.

jruby fraction_sample.rb

gave me the answer 25 / 12.


Like this, we can extend JRuby using Java API. Using Java API under the hood, we can create Rubygems. Some are re-implementation by Java like pure Java Nokgoiri. Others are Java originated Ruby API. JRuby extension is an example that Java effectively complements Ruby. So, add Ruby API to your favorite Java tools.


All codes of this sample are https://github.com/yokolet/Poplar.

Attempt to get Nokogiri work on Android

Conclusion

As a result, Nokogiri was loaded on Android successfully but didn't work on it. When I tried to parse XML document, I got tons of errors something like:

W/dalvikvm(  374): Unable to resolve superclass of Lorg/apache/xerces/dom/DeferredDocumentImpl; (2008)
W/dalvikvm(  374): Link of class 'Lorg/apache/xerces/dom/DeferredDocumentImpl;' failed

I'm pretty sure this sort of error messages complain there aren't enough interfaces of org.w3c packages defined in Android SDK. Actually, Android SDK's org.w3c API is a subset of JDK's. This is the problem. Xerces needs a full-set of org.w3c packages to work. Pure Java Nokogiri heavily relies on Xerces and nekoHTML/nekoDTD, which are built on top of Xerces. So, pure Java Nokogiri also needs the fullset of org.w3c packages to keep compatibility with libxml2 backed, CRuby version. This is why Nokogiri ended up in raising an exception as in below:

W/dalvikvm(  374): threadid=10: thread exiting with uncaught exception (group=0x40014760)
E/AndroidRuntime(  374): FATAL EXCEPTION: runWithLargeStack
E/AndroidRuntime(  374): java.lang.NoClassDefFoundError: org.apache.xerces.dom.DeferredDocumentImpl
E/AndroidRuntime(  374):  at org.apache.xerces.parsers.AbstractDOMParser.startDocument(Unknown Source)
E/AndroidRuntime(  374):  at org.apache.xerces.impl.dtd.XMLDTDValidator.startDocument(Unknown Source)
(snip)

Is this avoidable? Might be. Googling led me some discussions about replacing org.w3c and related other packages. If I can include Xerces' xml-apis.jar (this defines org.w3c/org.w3c.xxx, javax.xml.xxx, org.xml.xxx) in my Android app and override some of core packages, Nokogiri will start working exactly the same as a web app on Rails. But, it should not be a good workaround. Surgery on SDK might incur other applications that use replaced packages.


Probably, the best answer will create a subset of Nokogiri for Android. I'm not sure such limited version of Nokogiri still attracts users. But, I think it's better than nothing.



Thoughts on Ruboto and Android

Although my small Nokogiri app didn't work, I'm going to write about what I learned and did. This might help some poeple who want to make Ruby gems to work.

• JDK should be 1.6.0_24 on OS X

Ruboto people might not develop JRuby on Rails on Google App Engine, but I do. Just before I tried Ruboto, I had to downgrade JDK version for Google App Engine gem. So, when I started, my JDK was 1.6.0_22. I spent pretty much time to figure out why ruboto didn't work on my PC at all. Once the JDK got back to the latest, ruboto worked like a magic. Make sure what version of JDK you are using.

• Android API level should be 11

Not all Ruboto samples needs level 11 API. For example, samples of https://www.ibm.com/developerworks/web/library/wa-ruby/ worked on level 8. But, Nokogiri needs level 11. I'm not sure the reason, but, the activerecord (and jdbc) sample, https://github.com/ruboto/ruboto-core/wiki/Tutorial%3A-Using-an-SQLite-database-with-ActiveRecord-and-RubyGems, was also tested on level 11, which is Java backed rubygems like Nokogiri.

• Jar archives should be moved to project's libs directory

This happens on an environment that uses custom classloader, for example, Google App Engine. So, I have all jars in my project's libs directory, https://github.com/yokolet/cranberry/tree/master/libs, so that custom classloader can load all jars. If those jars failed to be loaded, Nokogiri raises a mysterious, "undefined method `next_sibling' for class `Nokogiri::XML::Node'," error. I didn't get that error, so jars should be loaded.

Also, I commented line 18-24 out from nokogiri.rb (https://github.com/yokolet/cranberry/blob/master/assets/vendor/gems/1.8/gems/nokogiri-1.5.0.beta.4-java/lib/nokogiri.rb) so that Nokogiri doesn't try to load those jars again.

• Configuration and setup are key to load gems

Loading gems on Ruboto was tricky. In the article, https://www.ibm.com/developerworks/web/library/wa-ruby/, the author rearranged all ruby files into single directory. This might work for small rubygems but never does for Nokogiri. For example, Nokogiri has nokogiri/html/document.rb and nokogiri/xml/document.rb. Instead, the way described in https://github.com/ruboto/ruboto-core/wiki/Tutorial%3A-Using-an-SQLite-database-with-ActiveRecord-and-RubyGems worked well. It looks complicated, but I realized that the thread based gem loading way was really necessary while I was trying other stuff. My config.rb is https://github.com/yokolet/cranberry/blob/master/assets/scripts/config.rb if you want look at it. Also, I edited src/irg/ruboto/Script.java (https://github.com/yokolet/cranberry/blob/master/src/org/ruboto/Script.java) and added "vendor" directory.

When I clicked on "Cranberry" Ruboto icon right after "rake install" said "Success," all Nokogiri files were copied to /data/data/.... directory. To cut down the time for copying, I deleted Nokogiri's test and ext directories, which are unnecessary to run Nokogiri.

• Needs threads to become a nifty app

Android expects developers' "responsiveness" (http://developer.android.com/guide/practices/design/responsiveness.html). According the document, database or network access should not be performed on a main thread. In my Nokogiri sample, I tried to get rss feed, on the main thread firstly, so I got the error:

W/System.err(  343): org.jruby.exceptions.RaiseException: Native Exception: 'class android.os.NetworkOnMainThreadException'; Message: null; StackTrace: android.os.NetworkOnMainThreadException
W/System.err(  343):  at android.os.StrictMode$AndroidBlockGuardPolicy.onNetwork(StrictMode.java:1077)
W/System.err(  343):  at java.net.InetAddress.lookupHostByName(InetAddress.java:481)
(snip)

This is why config.rb uses threads to require rubygems.

• No need to reinstall app when scripts are updated

"rake update_scripts" updates Ruby scripts of installed app. So, you don't need reinstall the app. This was a great help for me since an installing process took many many minutes.

• ... but, it doesn't work. What's going on ???

As an Android newbie, I very often fell into troubles to get Android SDK and the app to work. Sometimes, app icons didn't show up, or rake install failed. The troubleshooting, https://github.com/ruboto/ruboto-core/wiki/Troubleshooting, was so helpful. Especially, "adb kill-server; adb start-server" commands were the best. Also, I made a rule to type "ruby -v" before I started something. As you know, rake tasks start working on CRuby. But, those won't complete tasks as you expect.

I'd like to add "uninstall" the app to the troubleshooting. You can uninstall the app on emulator as well as adb uninstall command. On the emulator, do the long-click on the icon you want to uninstall. Then, trash bin and the word "uninstall" appears. Dragging the icon on trash bin will delete the app. Or adb uninstall [package name of app] will delete the app. For example, my app's package name is com.servletgarden.ruboto.cranberry, so "adb uninstall com.servletgarden.ruboto.cranberry" deleted my app from emulator. In case you forget the package name, look at the path to XXXActivity.java file. That path corresponds to package layer.



How I made this app

In the end, I'm going to add how I made this app and how to start it. This app won't work, but for myself, to try this app in future again, I'll leave this memo.

1. install ruboto-core gem

  $ ruby -v    (double check I'm on JRuby)
$ gem install ruboto-core

2. set path to android tools

  $ cd path/to/android-sdk-mac_x86
$ PATH=`pwd`/tools:`pwd`/platform-tools:$PATH

3. create emulator image

  $ android -s create avd -f -n cranberry-11 -t android-11

This possibly creates it. Actually, I created my virtual image using Eclipse's ADT. It's way easy. Prior to using android command, I installed platforms. I also used Eclipse's ADT for that.


4. create ruboto app

  ruboto gen app --package com.servletgarden.ruboto.cranberry --target android-11

This generated cranberry directory and whole stuff under that.


5. add emulator task to Rakefile

  $ cd cranberry
$ [edit Rakefile]
(line 42-45 of https://github.com/yokolet/cranberry/blob/master/Rakefile)

Since my virutal image name is cranberry-11 (step 3) "-avd cranberry-11" is there. If the app is small, you don't need -partition-size option.


6. install nokogiri gem

  $ mkdir -p assets/vendor/gems/1.8
$ gem install --install-dir assets/vendor/gems/1.8 nokogiri -v 1.5.0.beta.4
$ rm -rf assets/vendor/gems/1.8/cache
$ rm -rf assets/vendor/gems/1.8/doc
$ pushd assets/vendor/gems/1.8/gems/nokogiri-1.5.0.beta.4-java
$ rm -rf ext test
$ popd

7. add config.rb file, one line in assets/scripts/cranberry_activity.rb and edit Script.java

https://github.com/yokolet/cranberry/blob/master/assets/scripts/config.rb

line 2 of https://github.com/yokolet/cranberry/blob/master/assets/scripts/cranberry_activity.rb
line 186-188 of https://github.com/yokolet/cranberry/blob/master/src/org/ruboto/Script.java


8. start emulator

  $ rake emulator

The emulator took many minutes to boot up on my MacBook. Occasionally, it showed up without dark blue hexagons. In such case, emulator didn't work correctly. I tried a couple of times "adb kill-server" and "adb start-server." When that attempt didn't work, I shut the emulator down and did "adb kill-server," then restarted the emulator.


9. start log monitor

  $ adb logcat

This prints out verbose infos, errors, and others. It is a bit noisy, but a great help to figure out what's going on.


10. install app

  $ rake install

Be patient.


10. click Cranberry ruboto icon
Be patient again. JRuby needs long time to activate.
Ruboto default app, Figure 4 of https://www.ibm.com/developerworks/web/library/wa-ruby/ will show up.



11. edit ruby files and do "rake update_scripts"
Then, back to Apps view and clock ruboto icon. Updated version should work, or troubleshooting time starts.


Whew...!

Nokogiri on Google App Engine

Nokogiri 1.5.0 is on its way right now. Sure, it should be soonish. This version is also the first release of pure Java Nokogiri. We call it *pure Java*, but the name might not express itself precisely. Since it is written half Ruby and half Java, so *pure JRuby* (pragdave called so) would be the best name. This pure JRuby version implements methods, which are implemented in C, using xerces, nekoHTML, jing and a couple more Java Tools, while CRuby version uses libxml and libxslt. When people use Nokogiri 1.5.0 on JRuby, they use pure Java version.
What's the beauty of pure Java Nokogiri? It works smoothly on various platforms if Java runs on them. On OS X, Linux, Windows, and even Google App Engine, Nokogiri starts working painlessly. Really frequently asked questions for Nokogiri are "I can't install Nokogiri," or "Nokogiri doesn't work." Definitely, pure Java Nokogiri doesn't have these problems.


To see pure Java Nokogiri works fine, I gave it a try on Google App Engine (GAE). As you know, GAE supports python or Java only. Using libxml is out of scope. In short, pure Java Nokogiri just worked. Easy. (Unexpectedly, I struggled to get GAE work, so I'll write how I made it.) Although I don't have many to write about, I'm going to note what I did for people who don't know they can use Nokogiri on GAE.


First, I installed gems following the instruction, https://gist.github.com/825451. The instruction says, "Do not use rvm," but, I used rvm. Using rvm is not the matter. Rubygems' version is the matter. After I installed Ruby 1.8.7 using rvm, I downgraded rubygems to 1.3.7. Don't forget, google-appengine gem needs version 1.3.7 (or before) of rubygems. Otherwise, bundler08 will fail to install gem command *bundle*. This will end up in raising an error when appengine gem tries to install gems in .gems/bundler_gems/jruby/1.8/gems directory. Make sure *bundle* is listed in there when you type "gem help commands." See http://groups.google.com/group/appengine-jruby/browse_thread/thread/2db62b1a51896098 for a detail.

You do need to have CRuby but don't need to install JRuby. Appengine gem will install jruby-jar gem when it is needed. The gem, jruby-jar, has JRuby's stdlib in a jar archive. JRuby gets stared using this jar archive. So, google-appengine gem mostly works on CRuby and uses jruby-jar gem when JRuby is needed. Therefore, all gems should be installed on CRuby. Below is what I did.

rvm 1.8.7
sudo gem install google-appengine (Since I installed rvm to /usr/local, I need *sudo*)
sudo gem install rails -v 2.3.11
sudo gem install rails_dm_datastore
sudo gem install activerecord-nulldb-adapter
mkdir rails_app; cd rails_app
curl -O http://appengine-jruby.googlecode.com/hg/demos/rails2/rails2311_appengine.rb
ruby rails2311_appengine.rb

Then, rails app is ready to run. To start app on a development server,

./script/server.sh

This should start Jetty and rails app on that.

However, I was among unlucky people. I got Segmentation fault because my Java was Java SE 6 Update 4 for Mac OS X. Googling, I followed "Comment 39" of http://code.google.com/p/googleappengine/issues/detail?id=4712. I didn't want to downgrade JDK, but there seemed no better choice. Anyways, rails app successfully worked on update 3.


Next, I added Nokogiri in Gemfile. Currently 1.5.0.beta.4 is the latest.

gem 'nokogiri', '1.5.0.beta.4'

One more. The latest version of jruby-jar gem is 1.6.1, but, sadly, the jar archive in the gem is too big to upload. JRuby 1.6.1 grew bigger. As far as I remember, 1.6.0 is also too big to upload. Again, downgrade came in. I used version 1.5.6, and my Gemfile became as in below:

# Critical default settings:
disable_system_gems
disable_rubygems
bundle_path '.gems/bundler_gems'

# List gems to bundle here:
gem 'rails_dm_datastore'
gem 'jruby-jars', '1.5.6'
gem 'jruby-openssl'
gem 'jruby-rack', '1.0.5'
gem 'rails', '2.3.11'
gem 'nokogiri', '1.5.0.beta.4'

OK, my platform has been ready. Let's create a simple Nokogiri sample. In this sample, I got the rss feed from cnn.com (http://rss.cnn.com/rss/cnn_topstories.rss), parsed it using Nokogiri, and displayed news list. Since this is just a simple sample of Nokogiri, I generated a controller only.

./script/generate controller newsfeeds index

The rss I used was like https://gist.github.com/921058. From this XML document, I collected item elements using xpath. Then, I extracted pubDate, title, link, and description children elements of item also using xpath.

# newsfeeds_controller.rb
require 'nokogiri'
require 'open-uri'

class Entry
  attr_reader :title, :url, :description, :pubdate
  def initialize(title, url, description, pubdate)
    @title = title
    @url = url
    @description = description
    @pubdate = pubdate
  end
end

class NewsfeedsController < ApplicationController
  def index
    doc = Nokogiri::XML(open("http://rss.cnn.com/rss/cnn_topstories.rss"))
    items = doc.xpath("//item")
    @entries = []
    items.each do |item|
      title = item.xpath("title").text
      url = item.xpath("link").text
      description = item.xpath("description").text
      pubdate = item.xpath("pubDate").text
      @entries << Entry.new(title, url, description, pubdate)
    end
  end
end

# newsfeeds/index.html.erb
<h1>Newsfeeds#index</h1>
<% @entries.each do |entry| %>
  <dl>
    <dt><%= entry.pubdate %></dt>
    <dt><b><%= entry.title %></b>  [<%= link_to("Read", entry.url) %>]</dt>
    <dt><%= entry.description %></dt>
  </dl>
<% end %>

When I restarted the server./script/server.h and requested http://localhost:8080/newsfeeds/, I could see news list something like this.



The last thing I did was uploading. I set my application id on the line "application:" in WEB-INF/app.yaml, then uploaded it by ./script/publish.sh. Now my Nokogiri sample is working at http://4.latest.servletgarden-in-red.appspot.com/newsfeeds/.


In the end, I'm going to add a link to the blog talked about Nokogiri on Google App Engine. This would be helpful, too.

- Google App Engine, JRuby, Sinatra and some fun!


So far, pure Java Nokogiri worked just fine on Google App Engine. Give it a try!