iLike frequently receives large XML files from our partners that we need to analyze.

For ad-hoc analysis of XML from command line, xml_grep is a really valuable tool. Instead of extracting lines that match regular expressions, xml_grep extracts xml nodes that match xpath expressions.

However, it took me a bit of trial and error to figure out how install and use it successfully. Here's a quick cookbook of how I did it; I'll update this post with additional useful examples as I encounter them.

Installation

In most *nix distributions, you can install by simple typing

sudo cpan -i "XML::Twig"

Answer 'y' when Cpan asks you if you want to install other packages that xml_grep depends on.

(xml_grep is included as a part of the XML::Twig package in CPAN. XML::Twig also provides a rich Perl API; however the command line examples here do not require any Perl knowledge.)

Usage examples

Suppose you have the following xml:

xml_grep allows you to specify both --root (the node to match and print out) and --cond (an xpath expression relative to root that filters the results.) If more than one root is provided, the results are combined using OR; likewise with cond.

1. Here's the simplest possible example. --root is an xpath expression specifying which nodes will be printed. -p causes the results to be pretty printed:

Print all events : xml_grep -p --root="Event" foo.xml

2. The next example shows how we can use --cond to filter the results based on an attribute of a subnode of the root.

Print all events for a given artist: xml_grep -p --root="Event" --cond='ArtistID[@ID='806533']' foo.xml

3. Here's how we can match against the text contents of a subelements:

Print all events on a given date: xml_grep -p --root="Event" --cond='EventDate[string()="2009-07-25"]' foo.xml

Multiple cond flags are combined using OR. To print nodes matching one of several artists:

4.Print all events for a given artist: xml_grep -p --root="Event" --cond='ArtistID[@ID='806533']' --cond='ArtistID[@ID='806532']' foo.xml

5. The result of xml_grep is itself an XML document and can be piped into another xml_grep to do additional extraction.

Extract the time and performance name of events on a given date: xml_grep -p --root="Event" --cond='EventDate[string()="2009-07-25"]' foo.xml | xml_grep --root='EventTime' --root='PerformanceName'

5. The result of xml_grep is itself an XML document and can be piped into another xml_grep to do additional extraction.

6. Match a regular expression: xml_grep --root="EventName[string() =~ /Bil.* El.*/]" foo.xml

Early this year, the Orkut team became aware that "a small subset of OpenSocial applications [were] being used to spread phishing attacks to Orkut users."

To solve this, the Orkut team phased out support for Flash and Javascript in profile views and required that app developers use Opensocial data pipelining and templates to specify profile markup.

Data pipelining is declarative way to specify a set of REST calls to fetch the JSON data for a page, these include both OpenSocial and arbitrary REST calls. Templates are an HTML markup language (akin to JSTL or RHTML) that is interpreted on the Opensocial servers to generate the page markup. It includes simple expressions, conditional, and looping constructs, bound against the JSON data from data pipelining.

Orkut is currently the only container that has implemented these features, which are are expected to be part of the Opensocial 0.9 spec.

The specs for these features are still evolving and some of the online tutorials and wiki entries are inconsistent and no longer work. I thought it might be helpful to document an working example (as of August 2009) based on the iLike profile view.

The first step is to require the opensocial-data and opensocial-templates features in the ModulePrefs.

Next we declare the profile content...

...and specify the JSON requests required by the profile view using the os:HttpRequest declaration:

The ilike_async_get_cache_key request returns JSON like this: {"key": 139222}.

The profile_tracks_json request returns JSON like this:
{"tracks":[{,"artist_name":"Bouncing Souls","name":"The Pizza Song"}, {"artist_name":"The Aquabats","name":"Pizza Day"}],"track_count":2}}.

Note how the second request includes query string parameters (${cache.content.key}) derived from the JSON data in the first request. In this case, the profile_tracks_json is set to have a very long cache lifetime; the key is a profile timestamp use to force a fresh version of the profile to be fetched when the profile changes. (The timestamp is maintained in memached and can be returned very quickly by the server.)

It is poorly documented that Orkut's os:HttpRequest wraps this content in a "content" hash wrapper, which is why we have to say "cache.content.key".

Now that we have the data, we declare the template that binds to the json data to generate the markup. Note the use of "if" and "repeat" attributes for conditionals and loops and the "${}" expressions; also note the required use of osx:NavigateToApp to link to canvas pages.

In our case, the template is itself generated using an RHTML template in the Rails framework. We can use Ruby expressions to help simplify generation of the static template, but not, of course, any of the dynamic content or conditionals that depend upon the JSON data.

For example, the grungy escaped NavigateToApp parameters are actually generated by this more readable RHTML code: