JACK2002

Background

From 1996 through 1998 I worked as a junior developer on an application titled "Liquid Motion". The application first saw life as an offering from a small startup named Dimension X and was actually released as "Liquid Motion Pro", but following that company's acquisition by Microsoft, and after substantially more development effort, it was rebranded as Microsoft Liquid Motion.

The application was an animation authoring tool for web developers (loosely comparable with Macromedia Flash). It output animation files in the "JACK" format (for an early name for the product, Java Animation Construction Kit) with the JCK extension (conceptually similar to the SWF format). The runtime was an "applet" - a set of Java classes which would be downloaded to the client Web browser when visiting a page hosting an animation.

The Java-based runtime was the advantage which was promoted over the competitive product, Macromedia Flash. At the time, Java was widely supported by Web browsers (predominantly Netscape Navigator 2.0 and Microsoft Internet Explorer 3.0). When navigating to a page containing a Java applet, the browsers would download the Java class files, cache them locally (for fast access when returning to the page), and run the applet in a sandboxed virtual machine - all without prompting the user or requiring setup steps, and independently of the operating system (Windows, Macintosh, Unix). In contrast, the Flash web runtime (Shockwave for Flash, hence the SWF extension) was a binary plugin component which was not available cross-platform, and had to be downloaded and installed manually before a page containing the animation was correctly visible. Also, the Liquid Motion runtime was extensible - the JCK file could call on a custom Java class type added by a developer which would also be seamlessly downloaded. At the time, this sort of extensibility was not supported by Flash.

The web animation landscape changed rapidly along several axes. On the Windows platform, the addition of ActiveX download support enabled single-click install, letting non-technical users get a seamless experience. As Flash became a de facto standard, Web browser vendors started including it in the browser setup, eliminating the deployment hit. Flash also evolved to include a scripting language (Lingo) and richer multimedia capabilities. Finally, the legal battles between Sun and Microsoft over adherence to licensing requirements (or the right to extend the language, depending on who you ask) effectively killed client-side Java use. So Liquid Motion didn't last long in the marketplace.

That's not to say that Liquid Motion wasn't without its own faults.

Behavior-based vs. Frame-based

The fundamental animation paradigm in Liquid Motion was behavior-based, rather than Flash's frame-based model. In a behavior-based model, animations and events are produced by objects called behaviors which apply to another class of objects called actors. For example, an image is an actor, and a motion path is a behavior. A motion path applied to an image is what makes an image move over time. The motion path contains information about the actual path (typically a series of points which are interpolated) and timing (when does the behavior start and stop, does it repeat, etc).

The behavior interacts with the actor by changing the actor's properties based on the current time. For example, a motion path behavior simply changes the actor's position over time; a transition behavior changes the visual filter applied to an actor over time. Behaviors can be very complex - for example, a behavior can be implemented which sets the actor's position to the current position of the mouse pointer - which will cause the actor to follow the mouse. This behavior can then be used to implement a range of effects from custom cursors to drag-and-drop of objects within an animation.

This leads to a concept called a scene graph, where the state of the animation at any point time (e.g. for playing the animation, or interactively editing it) can be determined by evaluating the contribution of all of the behaviors and actors. This is extremely elegant, in that multiple behaviors can cooperate to create very complex interactive and non-interactive effects.

This is in contrast to frame-based animation, in which the actor's state is defined at multiple points in time, called key frames. Typically, the animation author positions an actor at a starting position at time t=0, and then sets a different position for the actor at some later time, say t=10. When the animation runs, the position is interpolated over that time range. To handle interactivity like following the mouse, user specified code is usually required.

I'll go out on a limb and hypothesize that, in most cases, frame-based animation is actually simpler to author. A frame-based animation authoring environment corresponds closely to the WYSIWYG paradigm - as an animator, you go to frame X and specify how everything will appear. Then you go to frame Y and specify how everything should appear. You let the tool figure out frames X+1, X+2 ... Y-2, Y-1 and tweak any if necessary. In contrast, a time-based animation tool typically requires the animator to select the behavior and parameters in a somewhat abstract sense. Either the tool needs to emulate the ability to tweak intermediate frames (by generating additional behaviors) or the animator must understand the runtime mechanism at a deep level.

Since most animators are graphical artists rather than software developers, providing an easy to use canvas for creating animations - even if it's a more tedious canvas - is a clear advantage. In the same way, direct formatting (swipe text and make it bold) is used far more commonly than named styles in Word; the latter requires a structured approach, background knowledge and planning.

Runtime Size and Capabilities

The Liquid Motion runtime (JACK) was designed in conjunction with the authoring platform - which was also a Java application. Indeed, the same object-oriented classes existed for the same objects (in slightly different flavors) at both design-time and runtime. This greatly eased the extensibility story - for example, to add a new type of multimedia element (called an actor) to Liquid Motion you simply created a new Java class, say FormButton. This FormButton would implement design-time properties and participate with design-time features (selection UI, property dialogs, serialization, etc.) as well as runtime features (rendering, animating, de-serializing, etc.).

This object-oriented nature extended into the rest of the application and runtime - for example, there were serialization and de-serialization classes, base classes for actors and behaviors as well as classes for each actor and behavior type, objects for handling vector shapes, etc. While this made the development of the application extremely rapid and simple, the runtime suffered.

The JACK runtime came in around 270k - which was comparable to the size of the Flash runtime. During the mid-1990's - pre-broadband - that was a substantial download size relative to an animated GIF. While not all of the class files were necessarily downloaded each time it was still a large hit. Much of the size was due to the approximately 50 class files - each Java class file has an approximate 2k overhead, before any of the specific code is accounted for.

One of my tasks as a junior developer on the team was to look at optimizing the size of the runtime. I probably shaved 50k off with various approaches - for example, detecting methods that were not necessary at runtime or could be eliminated, better use of inheritance, etc. 220k is better than 270k, but not by much. The size of the runtime limited adoption during the early period before Flash reached critical mass, since prospective users had to justify the substantial download for site visitors. Although I think the other factors mentioned above were an order of magnitude more important to the demise of Liquid Motion, this is still something that has bothered me throughout the years.

Fast Forward - JACK2002

Several years later (Fall 2002) after having several more years of development experience under my belt, I suddenly realized (4 years too late) that I knew how to shrink the runtime size down dramatically. I hadn't done any Java development in a few years but I dusted off my copy of VJ++ and got to work. I followed a handful of key mantras:

• Don't get fancy - you're developing an animation runtime, not an application
• Have a goal - in this case, small execution size; verify every change to avoid size regressions
• Count on Moore's Law - re-parsing a string into an float every frame seems like a bad idea, but when considering the tradeoff (larger class size) the performance cost is minimal
• Be prepared to compromise - you may not get 100% of your design, but 90% may be good enough

The end result was a runtime comparable to the JACK format in ~20k, with a few new features in some cases. This was achieved with the following high-level approaches:

• A single polymorphic class file. Rather than one class file per object type, it's old-school - a single class with an enumerated type, and lots of switch statements. Saves 2k per type.
• A simplified, generalized parser. Instead of having dedicated parsers for each data type, an XML-like syntax is used which generates an object tree (which corresponds to the runtime tree).
• Make use of the class library. The attributes for each parsed element are just dumped into a standard Java Hashtable class.
• Little or no validation. Assume the authoring tool will output valid files, and developers who hand-tweak the file will fix buggy animations. This flies in the face of conventional wisdom, but for a sandboxed runtime it's perfectly acceptable. (Note that if this was not sandboxed there would be tons of security issues.)
• Re-use everything. For example, rather than using new member variables specific to each actor or behavior-type, the attribute hash table is used to store everything in named slots, which cuts down on the size of the class file.
• Go big. The Java 1.0 image APIs didn't support rotating images, but the code is pretty simple so I slipped it in. And once you have that, you might as well do text - it was just a little extra work. I didn't get as far as making it anti-aliased, though.
• Keep it flexible. I was able to keep the extensibility mechanism; you simply give an attribute pointing at your replacement class implementation, declare the interface, and divest yourself of responsibility. A developer is likely to test their work so you don't have to hold their hand by adding extra validation code.
• Know when enough is enough. I almost added a scripting language to handle events. Then I realized that start & stop are enough; anyone who needs more can write new classes.
• Standards are (mostly) a good thing. I based the timing model loosely on SMIL 2.0 - which was based on the HTML+TIME which came out of the Vizact work which evolved from Liquid Motion. Hmmm, that's kind of sneaky, isn't it? Size trumps everything, though, so it's not anywhere close to standards compliant.
• No assumptions. After every change to the code, recompile (in Release mode) and check the file size in bytes. If there are two ways to accomplish the same thing (e.g. caching a value versus recalculating it) try both - which will result in the smaller class file is not always obvious.

Features

^* The particle effects ("Magic") were one of my first additions to Liquid Motion. The idea came from a storyboard for animated advertisements. At one point the team started working on a product called "Liquid ProMotion" which would output banner ads. The ads were Java applets specific to the animation (rather than being a generic runtime) with the class files plus media in the 10-20k range permitted at the time. The storyboard included Disney-esque sparkles; I came up with a simple particle system algorithm with mouse movements triggering particle generation. We decided to roll the effect into Liquid Motion and added several other variants, such as bubbles, clouds and smoke.

^** SMIL uses generic animation primitives (animate and set) which act on arbitrary properties (another idea from Vizact). So instead of manipulating a scale property you animate both width and height.

^*** ColorCycle (which begat SMIL's animateColor) was another early addition of mine to Liquid Motion inspired by color palette animation, a popular animation trick in the early 1990s.

File Format

Grammar

Pseudo-XML Syntax:

Grammar:

element ::= ws* ( (opentag element* closetag ) | emptytag )?

emptytag ::= '<' tagname (ws* attribute)* '/' '>'

opentag ::= '<' tagname (ws* attribute)* '>'

closetag ::= '<' '/' ws* '>'

attribute ::= attrname ws* '=' ws* '"' [^"]* '"'

tagname ::= alphanumeric+

attrname ::= alphanumeric+

alphanumeric ::= [a-zA-Z0-9]

ws ::= space | tab | newline | linefeed

Some of the many differences from real XML:

• The text in close tags is ignored - it is assumed that the tags match.
• Comments (<!-- comment -->) are not supported.
• Text nodes are not supported (it's all attribute-based content).
• Processing instructions are not supported (and the XML PI is not required).
• Only double quotes (") can be used for attribute values.
• No namespaces.
• No encoding declaration. This is a bug - it relies on the default system encoding!
• No escaping mechanism. As a consequence, double quotes (") aren't supported in strings!

Common Attribute Sets

Timing Elements

Actor Elements

Behavior Elements

Detailed Attribute Syntax

Source

• Anim.java - the full animation runtime
• Extension.java - a sample extension class - it draws an oval

Examples

These suck, as they were hand-coded. I suppose that's why we thought they was money in creating authoring tools. Note that the applet these reference is actually compiled with debugging output, symbols, and no optimization, so it's actually closer to 30k in size than 20k in size.

• Basic sample (source), showing scale, rotation, and path behaviors on an image
• Extension sample (source), showing an extension class (renders an oval)
• Obnoxious sample (source), with pretty much one of everything including an annoying cat sound
• Interactive sample (source), with roll-over effects

Performance

The current implementation has performance issues when rotating images and text. This is done by allocating new pixel arrays and images. In the case of images, one new pixel array and new image is allocated for each rotated frame. In the case of text, three new pixel arrays and images are allocated - the first to render text to a surface, the second to produce an image with an alpha channel, the third to produce the rotated image. This takes the time to render a test frame from 30-40ms to 100-200ms (a reduction from 25fps to 5fps).

Ways to improve performance:

• Don't allow rotations on text, images, or both
• Document the performance problem and let authors decide
• Cache the pixel arrays - this is half of the memory allocation. For images this is half the allocation time. For text this will be problematic if scaling or color shifting occurs since alpha image generation outputs the scaled text using native text rendering.
• For text, combine the alpha channel generation and rotation into a single step.
• Switch the entire render channel to use MemoryImageSource and take over rendering of all pixels. This will allow for higher fidelity animations (e.g. anti-aliasing) but at the cost of having to render text, images and vector shapes pixel by pixel, which implies more code.
• Since scaling is handled by AWT, and we know the max bounds of a rotated image (sqrt(w*h) x sqrt(w*h)) the allocations (the slow parts) could be eliminated. For text that isn't scaled or color-cycled the same optimization could be used.
• For text: render text at some huge size and use image processing and scaling to render it.
• Potentially keep a large "working bitmap" for rendering scaled/rotated text, to avoid allocations. Difficult to know how (e.g.) text will be "zoomed" and what part might be rendered
• Idea: keep a couple of offscreen buffers; allocate them to 1.5 times the size needed on use. Each time they are used, reallocate if they're not large enough. So if there is no text scaling or text/image rotation, they'll never allocate. Otherwise they'll grow to the necessary size. The down side is that the coordinates will be offset, and they need to be cleared before use.
• There's a potential fork to be decided on - should the animation pipeline be primarily vector driven or pixel driven?
  • When pixel driven, each scene would render to a surface. When the scene is scaled, rotated, etc. there is a pixel copy operation. This would make filters easy, but scaled content looks rougher.
  • When vector driven, there is a transform matrix in place at all times; rendering occurs to a single surface with scaling of the points (etc) happening before any pixel hits a buffer. Filters are much harder to render, but the overall quality of nested, scaled content is much higher.
  • Currently, the pipeline avoids this; scaled scenes are not supported so scaling happens per media type, and no transforms are supported. (This is on par with the Java-based Liquid Motion runtime; the DirectAnimation-based runtime was (I believe) pixel driven)

Future Work / ToDo

• Specify encoding (UTF-8)
• Define escaping mechanism for double-quotes in strings.
• Full SMIL timing model. An important animation trick is to animate backwards - e.g. instead of trying to have the letters of a word fly together and magically arrive at the same time, start with the letters together, explode them apart, and then run the whole thing in reverse. So we need time scaling, including negative time.
• Move to the SMIL model of having generic animate and set behaviors. While this would make hand-authoring of animation files more complex (to scale an image implies animating both width and height properties; to scale text you instead animate a fontSize property), it will probably save on code size, which is the whole point of this exercise.
• Thread control uses deprecated suspend/resume/stop paradigm; update to a state machine
• Add a particle actor as an extension (optional class dynamically loaded) as proof-of-concept for add-ins.
• Add some decent demos
• Write a Liquid Motion of Vizact translator so I can get some decent demos without having to hand-code them!
• Extend applet model to decouple the Applet from the root content element, and allow chaining - e.g. applet.href="newfile.xml"