5. Active Datasets

Active Datasets are datasets which contain data objects that are "live" - when the objects change they cause messages to be sent back to the server to report the changes. Jitsu uses JSON to support active datasets.

This document is based on the "bookmarks" PHP sample application, which is split across three different directories:

[htdocs]/jitsu/services/bookmarks/
index.php             - defines the bookmarks service
bookmarks.types.xml   - defines types used by the service
[htdocs]/jitsu/services/utils/
jitsu.php             - the JitsuCommandProcessor class
json.php              - PHP utilities to generate JSON messages
[htdocs]/jitsu/quicktours/
active_bookmarks.src.html  - a Jitsu app using the bookmarks service

1. Defining DataTypes

To create an active dataset, first define the data types that the dataset we will contain. The Bookmarks application has a single datatype, defines in bookmarks.data.xml: 

    <type name="Bookmark">
        <property name="title" />
        <property name="url" />
    </type>

Its a simple object with two properties: a title and a url (both strings)..

2. Creating an active dataset

To get active data onto a page, you first load the datatypes for the dataset, and then create an active DataSet instance. An active dataset is a dataset whose src attribute is the URL of the service which supplies the active data. For the Bookmarks sample (see active_bookmarks.src.html in the quicktours directory), we use: 

<html xmlns="http://www.w3.org/1999/xhtml" 
      xmlns:j="http://www.jitsu.org/schemas/2006">

    <j:using src="../services/bookmarks/bookmarks.types.xml"/>

    <body>
      <j:App>
        <j:data>
          <!-- a blank notes dataset-->
          <j:DataSet id="bookmarks" src="/jitsu/services/bookmarks/"/>
        </j:data>

        ...
      </j:App>

The two highlighted statements load the bookmarks.types.xml declarations (i.e.the Bookmark type defined above) and create a blank dataset called "bookmarks", whose data is provided by the "/jitsu/services/bookmarks/" URL.

When Jitsu compiles a <DataSet src="...">, the src is a URL. If the src attribute starts with "file:...", the page compiler loads the dataset as a static resource (i.e. built at compile time). For datasets whose src attribute do not start with "file:", the Jitsu runtime assumes the src attribute is the URL of a JSON data service.

3. Running the Bookmarks application

You run the bookmarks application by loading the active_bookmarks.html quicktour. This loads in the Jitsu app and then requests data from the "/jitsu/services/bookmarks/" URL.

In the /jitsu/services/bookmarks/ directory is an index.php file that handles service request:

class BookmarksDataService { 
    ...
    function run() {
        $cmdProcessor = new JitsuCommandProcessor();
        $cmdProcessor->process($this);        
    }
    ...
}

This file is using JitsuCommandProcessor, a PHP class that is included in the htdocs/jitsu/services/utils/jitsu.php file.

Jitsu apps talk to the server using HTTP PUT. This lets Jitsu send the server a list of commands to process in one go. The server parses all the commands and sends a lsit of responses abck to Jitsu.

JitsuCommandProcessor is a helper class for encoding and decoding Jitsu messages. It uses Michal Migurski's Services_JSON open source package to encode and decode messages in JSON.

JSON

In Jitsu, the client and server use JSON to communicate in both directions - the client uses a HTTP "PUT" to send a list of JSON commands to the server. The server processes the commands and sends a list of JSON responses to the client.

JSON is a lightweight JavaScript object notation. In PHP, PHP associative arrays can easily be converted to JSON - the JitsuCommandProcessor uses the "Service_JSON" class to do this conversion.

BookmarksDataService has the following method for converting a single raw bookmark object into a PHP object ready for JSON encoding:

    // Get a single bookmark object in PHP object format ready for
    // conversion to JSON)
    //
    function marshallRawBookmarkForJson($rawBookmark) {
        return array(
            "jsType"    => "Bookmark", 
            "aref"      => $rawBookmark['id'], // Make an aref for the bookmark.
            "title"     => $rawBookmark['title'],
            "url"       => $rawBookmark['url']);
    }

This method takes a PHP associative array containing "id", "title", and "url" properties. It maps this to a new associative array, containing an "aref", "jsType", "title" and "url" propreties.

Here's what the associative array looks like once its serialized out in JSON:

    {
        "jsType":   "Bookmark",
        "aref":     "b12",
        "title":    "New York Times",
        "url":      "http://www.nytimes.com",
    }

Some things to note:

In the Bookmarks application, we use a letter "b" followed by a simple integer as an aref. Most applications would use a more complex scheme for generating unique ids, but for demo purposes this works well.

The Bookmarks DataSet

We've seen a single Bookmark in JSON. A DataSet is simply a list of objects. Here's the code for obtaining the entire Bookmarks dataset:

    function getBookmarksDataSet() {

        // Get the bookmarks associative 
        // array from the PHP SESSION
        $bookmarks = $this->getAllRawBookmarksFromSession();
        
        // Marshall each raw bookmark object (except for deleted bookmarks)
        // into an object suitable for JSON
        //
        $items = array();
        foreach ($bookmarks as $id => $rawBookmark) {
            if ($rawBookmark['deleted']) {
                continue;
            }
            $items[] = $this->marshallRawBookmarkForJson($rawBookmark);
        }

        // Return JSON object for a DataSet 
        return array(
            'jsType'        => "DataSet",
            'aref'          => 'bookmarks',
            'items'         => $items);
    }

This code constructs an array of bookmarks by iterating over each raw bookmark in the session, converting the raw bookmark into a Json-ready bookmark. It returns an object representing the dataset, with three properties:

        // Return JSON object for a DataSet 
        return array(
            'jsType'        => "DataSet",
            'aref'          => 'bookmarks',
            'items'         => $items);
    }

When this object is written out as JSON, it looks something like:

    {
        "jsType":"DataSet",
        "aref":"bookmarks",
        "items":[
            {
                "jsType":"Bookmark",
                "aref":"b1",
                "title":"My first bookmark",
                "url":"http:\/\/www.attap.com"
            },
            {
                "jsType":"Bookmark",
                "aref":"b2",
                "title":"My second bookmark",
                "url":"http:\/\/www.google.com"
            }
        ]
    }

i.e. it is an object whose "jsType" is "DataSet", and whose "items" is a list of Bookmark objects.

Processing Commands

The Jitsu client and server are very similar in how they process messages. The only major difference is that the server receives commands and generates responses, whereas the client client sends commands and receives responses. In both cases, commands and responses are small JSON objects - here's a "set" command sent from the client to the server: 

    {
        "op":"set",
        "path":["bookmarks","bb7"],
        "key":"title",
        "val":"Hello there"
    }

and a "consoleMessage" response:

    {
        "op": "consoleMessage",
        "val": "Here is a message to write to the console",
    }

Notice that both commands and responses have an "op" ('operation') property that identifies the action to perform. Each operation has its own list of parameters.

Commands and responses are batched: The Jitsu client collects several commands together and sends them as a list to the server. The server processes all the commands in the list and sends a list of responses back. For example, here's a list of two commands:

{
    "dataSet":"bookmarks",
    "commands":[
        {
            "op":"set",
            "path":["bookmarks","bb7"],
            "key":"title",
            "val":"Hello there"
        }
        {
            "op":"load",
            "path":["bookmarks"]
        }
    ]
}

JitsuCommandProcessor is a PHP helper class we include with Jitsu for parsing Jitsu commands:

    $cmdProcessor = new JitsuCommandProcessor();
    $cmdProcessor->process($this);

JitsuCommandProcessor loads the input for an HTTP PUT Request and breaks it up into its constituent commands (a "set" command and a "load" command in the example above). For each command, it invokes an associated method on the command handler object you pass in as the argument to process():

The methods called are:

doLoad Called to load a dataset or a dataobject - the data service should respond with a "data" response that contains the requested object in JSON format.
doAdd Called to add an object to the dataset. The data service responds by sending a "data" response with the full definition of the added object.
doRemove Called to remove an object from the dataset. No response is required for this command.
doSet Called to set the property of a specific object to a new value. No response is required.
doAutoLoad Called when a binding on the page hits a property on a DataObject which is undefined. Servers should respond with a "data" response if there is more data on the object. This allows the server to lazy-loading properties on demand. The presidents sample uses this feature to load bios as the user clicks on presidents in the list view.
doApplicationCommand Called for any command whose "op" property is not recognized. Lets you create application-specific commands.

In some cases due to poor or slow connections, the Jitsu front end may issue the same command several times. The back end data service should behave robustly if the same command is reissued multiple times.

System-Generated Commands and Responses

Here are more details on each of the commands generated by Jitsu.

Load

The Jitsu runtime generates a "load" command to load data objects and datasets, e.g. the first time the user hits the page. Here is a sample "load" command generated by the Jitsu front end, to load the "bookmarks" dataset:

message={
    "dataSet":"bookmarks",
    "commands":[
        {
            "op":"load",
            "path":["bookmarks"]
        }
    ]
}

The data service responds to a "load" message by sending a "data" response message back to the client. Here's an example "data" response sent back:

reply={
    "responses":[
        {
            "op":"data",
            "path":["bookmarks"],
            "val":{
                "jsType":"DataSet",
                "aref":"bookmarks",
                "items":[
                    {
                        "jsType":"Bookmark",
                        "aref":"b1",
                        "title":"My first bookmark",
                        "url":"http:\/\/www.attap.com"
                    },
                    {
                        "jsType":"Bookmark",
                        "aref":"b2",
                        "title":"My second bookmark",
                        "url":"http:\/\/www.google.com"
                    }
                ]
            }
        },
    ]
};

Add

When an item is added to an active DataList or DataSet, Jitsu generates a message like this:

message={
    "dataSet":"bookmarks",
    "commands": [
        {
            "op":"add",
            "path":["bookmarks"],
            "index":2
            "val": {
                "jsType":"Bookmark",
                "aref":"urn:tmp11337614199081",
                "title":"Add Bookmark 1",
                "url":"http://www.attap.com"
            },
        }
    ]
}

The response to an "add" should be a "data" message. e.g. Here is the corresponding reply generated by the Bookmarks PHP server for the add above:

reply={
    "responses": [
        {
            "op":"data",
            "path": ["bookmarks","urn:tmp11337614199081"],
            "val": {
                    "jsType":"Bookmark",
                    "aref":"b7",
                    "title":"Add Bookmark 1",
                    "url":"http:\/\/www.attap.com"
            }
        },
    ]
};

The path in the "add" command is the path to the parent container that is being added to - here we are adding to the "bookmarks" dataset itself.

The "val" in the add command contains the initial properties for the added value. Notice that the "aref" for this value is a unique temporary aref generated by the Jitsu client. This is the one case where Jitsu creates new arefs itself.

In the "data" response, the "path" should be the path to the added object (the final aref in this path must match the temporary aref generated by the client for the new item). The "val" should be the JSON for the data value that was added - notice that in this case the "aref" is the actual ID of the added object ("b7" in this example), not the temporary ID that the client generated.

Remove

The "remove" command is issued when the user removes a DataObject from a dataset. Here is a sample "remove" command generated by the Jitsu front end:

message={
    "dataSet":"bookmarks",
    "commands":[
        {
            "op":"remove",
            "path":["bookmarks","bb7"],
            "index":2
        },
    ]
}

There is no required reply for a remove.

Set

The "set" command is fired when the user modifies a property of a DataObject. Here is a sample "set" command generated by the Jitsu front end:

message={
    "dataSet":"bookmarks",
    "commands": [
        {
            "op":"set",
            "path":["bookmarks","bb7"],
            "key":"title",
            "val":"Hello there"
        }
    ]
}

There is no reply required for a set.

AutoLoad

The "autoload" command has the same semantics as the load command - Jitsu automatically fires "autoload" on an object the first time a page binds to a property on the object that is undefined - this gives the server an opportunity to demand-load data, e.g. to load the body of an email message when the user clicks on the title of the message in a list view.

Console Messages and alerts

The list of responses sent back to the Jitsu client by the server may include console messages for debugging, and alerts to signal errors to the user, e.g.:

reply={
    "responses": [
        {
            "op":"consoleMessage",
            "val":"set on bb7 title to Hello there ok."
        }
        {
            "op":"consoleAlert",
            "val":"Nice to meet you."
        }
    ]
};