Let’s Make a Street View! Part 0

In what was once timely news (sorry), Google announced version 3 of the Google Maps API at their last I/O, designed specifically—but not exclusively—with the mobile space in mind. Since it’s not used on the main Maps page, I didn’t notice that the new API includes an embeddable “HTML5″ version of Street View until the news started making the Twitter rounds.

What’s especially interesting about this new Street View—aside from the fact that it’s now available without Flash—is that it takes a divide-and-conquer approach to ensuring good performance across browsers. There are at least three different rendering “modes” within the new library, triggered by different browsers and operating environments. I’m going to take a brief look at each of these before moving on to the main event: building our own Street View app. Though I might call it StreatVue.

Continue reading

Posted in Uncategorized | Tagged , , , , , , , , , | 3 Comments

We Can Do Anything But Keep Things In Perspective

I’ve been playing around with using 2D transformations to enable 3D effects, even (or especially) in browsers without support for the latter. There have been a few examples of this kind—putting text and images and even video on isometric cubes, for the most part—but I think we, as a community, have not gone far enough in the pursuit of the hideous. I’m here to help.

This demo does border on abuse of the format—probably better done with SVG or not at all—but as an outputted, interactive image it (vaguely) suggests something important. Before a particular problem, before identifying the needs of our users, before creating elegant interfaces, and before good visual design, I believe we need to ensure the delivery of that first promise of techno-utopianism: putting whatever the hell we want to on the screen.

To be perfectly clear, all the things I listed first are far more important than that ability, but we shouldn’t be limited before we begin. Limits inspire creativity, but they also circumscribe what is possible; I want to be in charge of that. If I want a giant blinking triangle covering half my text, then by golly, there should be a giant blinking triangle there. Just because something shouldn’t be done doesn’t mean it shouldn’t be doable.

Demo: Isometry

Interaction

Browser interaction techniques have historically been so poor that we are still astounded when the simplest of interfaces match some intuitive sense of physicality. Luckily, things are changing. There is an increasing belief in a kind of manifest destiny for the web, the idea that we can make any kind of interface that we set our mind to, but we’re just now getting around to it (we might also have been waiting for the right local storage api).

Deluded or not, I’m of this mind as well, and I was interested in experimenting with the mechanics of interacting with three-dimensional objects, specifically through the web browser. All interfaces are abstractions and therefore lossy (surjective, if you’re into that), but the 2D interface to a 3D scene ends up serving two masters, since the whole point is to mimic a physical interaction through a mouse and keyboard. A qualitatively-optimized mapping still needs to be made between data and interface, something as expressive as possible within the parameters available, but here it has to not only convince us that those degrees of freedom are sufficient for the task at hand, it has to simultaneously match our sense of the way things actually move in the real world. Otherwise: frustration.

Which is all a long way of saying that most 3D interfaces are terrible. If it’s just decoration, fine, but if you’re trying to perfectly map a physical task to the mouse, you’re doing computers wrong, and if mouse movements relate inconsistently to what’s happening on screen, your users hate you.

This demo just plays around with some ideas—there is no real task to accomplish—but it’s easy to see where it is not entirely successful. However, meaningful criticism would require a program that was meant to do something, which this one wasn’t. I will say, though, that the technical side of things is easier than you might think. So embrace our collective destiny, don’t be afraid of creating the next blink tag or flash banner ad, and stop waiting for the perfect version of local storage.

Also, those native-app developers told me you guys look like dorks, so you’d better get on that.

Projection

While waiting for the WebGL promised land to arrive, I wanted a reliable way of making 3D transformations work at interactive rates across all the major browsers, even accounting for those aging software renderers. The process turns out to be pretty simple and without any real hacks; every frame created dynamically here can be recreated statically with some plain, vendor-prefixed CSS. Now, “real hacks” is relative—especially depending on how you feel about “-ms-filter”—and I don’t explicitly support IE versions prior to 8, but it’s a start.

The key is the orthographic projection, one way of projecting a three-dimensional scene onto a two-dimensional surface (in this case, the screen). Since orthographic projections map 3D parallel lines to 2D parallel lines, objects do not shrink as they recede from a viewer. This can cause a somewhat unsettling effect—a pretty cool testament to the sophistication of our visual system—but is also one more tool to be exploited by artists. In a nutshell, the animation system keeps track of the 3D transformations per element, then does a simple projection before applying it to that element’s style property. I’ll post more on the math and implementation details shortly.

Known Issues

I want to make the core feature set work across browsers as reliably as possible. If you have an issue, please let me know about it and the platform it happened on.

[IE]

  • Internet Explorer’s matrix filter has proven surprisingly stable and performant, but this demo appears to be pushing it to its limit. Transformations nested three or four deep, especially on elements with untransformed siblings, cause background colors to bleed through in some circumstances. This is not an issue in IE9 Preview 2.

  • Unfortunately, the nested transform necessary for the recursive side is ignored altogether. This is not corrected in IE9. The transform is there, but unperformed, even though transforms nested within that transform are executed correctly. I’m hopeful that a knock in the right place will get it working.
  • Like most of us, IE9 Preview 3 isn’t a big fan of filters and won’t run this. If filters are removed from IE9 permanently (when in standards mode) and CSS3 transforms are not added, I don’t believe there is a replacement.

[Embedacube™]

  • The Embedacube™ feature saves the state of the cube just before you press the button; press it again to save the state after the first press (but before the second press).
  • An Embedacube™ cube currently only works in the browser version (Firefox, IE, etc) in which it was created.
  • Some style information is currently lost in the Embedacube™ process.

[Webkit]

  • Safari on the Mac and Chromium on Windows (with startup flag) both support hardware accelerated 3D transforms. Unfortunately, this demo interacted with -webkit-transform-style and the subsequent stacking order differently than I anticipated, so I’ve forced them back to 2D for now. Performance suffers a bit, but their software renderer is pretty good, especially for such a simple scene. I am investigating how deeply I have to change things in order to get accelerated 3D transforms working properly, and hope to have the change pushed soon.
  • CSS transitions are another feature currently unused that should map well to this demo and provide a good performance boost.

[Mobile]

  • In a few tests, mobile performance on this page has been good. However, my testing equipment is limited, so touch events, for instance, are not currently acted upon.
Posted in Uncategorized | Tagged , , , , , , , , , , , , , | 4 Comments

Transform Rewrite: Dancing Trees and Errant Pixels


Two months later, I’ve finally had some time to finish a rewrite of my “CSS3″ transform/transformed element GWT module. I’m pretty pleased with its new state and don’t anticipate the need for any more major changes in the near future. It’s still definitely in beta, but I’m sufficiently confident in it that I’ve checked in the code, generated javadocs, written a sample app, and uploaded a zipped jar which includes all the aforementioned. Though checking out the code via Subversion doesn’t seem to have been too high a barrier before now for some, hopefully this will encourage others to take a peak.

It’s so bad

Once again, I needed to write a simple demo for a module, something obvious in effect, intent, and execution. I’m usually terrible at these—hemming and hawing over details irrelevant to the point of a piece—so I gave myself two hours to come up with something and code it. I didn’t quite make it (there was some hemming and hawing), but it was close.

I don’t remember exactly how some initial thoughts on nested spinning boxes lead to this, but I think it turned out pretty well, if a little crude. Conveniently, it ended up using the major parts of the library that most need demonstrating—creation, element transformation, and the use of standalone transforms—which I’ll talk about below. There was also only a single browser workaround needed: current versions of IE don’t support border-radius, so the shadows needed some slight repositioning.

But I’ll get to all that. Go look. Firefox ends up winning for rendering quality (and was used for most of these screenshots), but all the browsers do pretty well keeping up:

Demo: Transforming Trees

If it doesn’t really strike a chord, try running this in the background at the same time. Hey, I never said it was going to change someone’s life or scrub oil off the wetlands. Two (and a half) hours, remember?

Overview

The source for the demo is over in the usual place, but I’ll go through the important parts here, concentrating on the actual application of transforms. Continue reading

Posted in Uncategorized | Tagged , , , , , , , , , , , , | Leave a comment

I Guess In Theory That Needs Fixing

If you pay close attention, you’ll notice that the “demos” or “experiments” that people like us inflict on the world tend to be most intriguing just before you actually see them run, when they’re just a thumbnail, in a list, on some gallery page.

I have to add a corollary, of sorts: apparently my work is a lot more interesting when I’m using matrix operations in one way but the browser thinks I’m using them in another.

There’s a lesson in there somewhere.

Just about every little application I’ve written using transforms needs to be recompiled and rereleased, but these are the dangers of using still-developing standards. In the meantime, you’ll get some entertaining results (perhaps more so than before) in a nightly WebKit or nightly/dev Chrome release. They recently switched the order of WebKitCSSMatrix multiplication to the “right” way (unless you did computer graphics in the 80s), but kept matrix entry access the same (e.g. “m14″ is the entry in the fourth row, first column).

The result is that a simple transform is unaffected (I*M = M*I), a more complex transform is kind of backwards but recognizable, and a transform where I get tricky—manipulating a matrix directly and then mixing it with one of the first two—goes satisfyingly nuts.

Some back story: about two and a half weeks ago, I sent an email to a W3C mailing list asking for clarification on what the SVG- and CSSMatrix notation/order “should” be. I got no response on the list. Too long winded? Probably, but it turns out it was because some WebKit developers recognized the issue as a bug in their implementation, filed it, and—before I even thought to check Bugzilla a few days later—had landed the change.

Here we should all thank Simon Fraser and Chris Marrin over at Apple, not only for fixes like this, but for their involvement (with others, of course) in making CSS transforms exist in the first place. But not only are they quick, they have also been incredibly open (if a bit busy) to discussing how to get the standard right. I may be the only developer using this object outside of a mobile browser (it’s probable), but I’m still a nobody; it’s nice for that not to matter.

If you’re actually using my library and are worried about the unstable behavior: existing code will continue to work perfectly with all officially released browsers. As of yesterday, the latest version also takes care of pre-release browsers, on top of adding a bunch of new and faster features you’ll want to use anway. There is one last WebKit bug I’m waiting on, which will change either the spec or their implementation. CSSMatrix is a pretty small interface—there’s not much left to clarify—so I’m hopeful that things will settle down after that.

This is the price we pay for riding the bleeding edge, but don’t discount the fact that your project really might be better a little broken. Thumbnails help, too.

Posted in Uncategorized | Tagged , , , , , , , , , | Leave a comment

Firefox Ethology

Barely related: I’ve written a design document (of sorts) for my implementation of web workers in GWT. Like the library itself, the document is fairly rough, but it does give a pretty good overview of the current state of things. It’s actually in a Wave, for better formatting and the hope of some feedback, so if you’d like to take a look and can’t, drop me a line and I can send you an invite. You can find the link to the wave in this thread.

On to the actual topic of the day.

That Crashing Thing

[Update added at bottom of post.]

I’m happy to report that I have been unable to crash Firefox 3.6.4 beta 1 [only once] using my Blocks worker test. I originally noticed the change in one of the 3.7 nightlies, but it’s nice that it landed earlier. Unfortunately, I’ve been unable to track down the bug that fixed it, even after spending way too long with Bugzilla yesterday.

Part of the problem is that the stack traces from the crashes were never definitive (to my eyes, at least). All of them were on account of an “EXCEPTION_ACCESS_VIOLATION,” but it sometimes happened in TraceMonkey’s TraceRecorder, indicating my earlier guesses were correct, and sometimes in the JS string code, indicating not so much.

One factor that I actually forgot to mention in my last post is that the worker object running the physics simulation is actually discarded and reloaded from file every time the test level is changed or reset. In a real application, you’d want to avoid this if at all possible (depending on the browser, the script file might have to be re-parsed, re-compiled, or—serious horrors—re-downloaded). In this case, however, it was a great way to put the worker emulation code through its paces: the old, “frustrated-user-quadruple-button-click” stability test. It actually helped me find a nice little NPE that only occurred in the unlikely event that a terminate() call came after an emulated worker was created but before it actually started running.

I’m going to wildly speculate—with the very real chance that I’ve missed something on Bugzilla—that something similar was happening with Firefox. The facts:

  1. an “EXCEPTION_ACCESS_VIOLATION”
  2. the page would only crash after the worker was reloaded at least once (though often, when restoring, it would crash again immediately)
  3. the crash frequently occurred with a call to JS_MakeStringImmutable(), which, MDC tells us, “applications must call…before sharing a JSString among threads”

Thinking what I’m thinking?

The weird thing is that nothing in the complete list of changes really matches the situation. The closest I can find is Bug 547399, “Don’t let worker messages run if the worker is suspended,” but the patch doesn’t touch any of the files I was combing through, nor do the developers involved think that the bug could trigger something as serious as a crash. It does, however, prevent a possible race condition dealing with passing strings between threads, so, who knows? Maybe my crash didn’t get enough hits and the fix is just a side effect of something else.

Still, it’s a welcome development. Thanks, Mozilla.

In Their Slowness

When I wrote the Blocks test, Firefox 3.6.0 really was as fast as the second test level would indicate, though there were hints that it was losing a lot of speed due to the way it got along with TraceMonkey. What I didn’t realize was that TraceMonkey wasn’t involed at all; JIT compilation was disabled in web workers until 3.6.2 and this bug. Imagine my surprise a month later: I finally have the time to post the test and Firefox can’t even manage the IE8 level. There was some frustration.

Our clues:

  1. It’s happened before. I think. Some code just doesn’t take to the way TraceMonkey traces, so the engine needs a set of heuristics to determine when to just give up and interpret. Scripts will sometimes be misclassified, and this seems to be just such a case. There was a really interesting example of this discovered and diagnosed in front of a live audience (I love you, open development) and the bug is still open. The code involved is rather different than this case, but the connection is too nice to think too much about confirmation biases.
  2. I mentioned this before, but just disable JIT compilation and try the test. This can be done in about:config, but it’s a lot easier to just turn on the console or script panels in Firebug. While it’s not as fast as it once was (Firebug is running, after all), you’ll usually see the blocks speed up rather comically.

The really humorous part of this whole situation is that I got exactly what I wanted: a performance update rolled out in a point release. I still agree with the sentiment, and this is undoubtedly an on-average win for performance. But, seriously, can I just have my interpreter back?

[Update: Almost immediately after publishing this post, I was able to crash 3.6.4; let's just say it will crash only rarely. Notably, the crash had nothing to do with JSString and occurred, once again, in TraceRecorder.

In addition, turning off JIT compilation in about:config—originally mentioned above—will not work to get the speedier Firefox back in action. Using Firebug to accomplish the same thing only works at first (and generally stops if you reset the test), so you'll have to extrapolate from that first few seconds of speediness to imagine what once gloriously was.

There is a bug: Bug 562455, but I am a total Bugzilla noob, so expect it to be duped or something in 3...2... =]

Posted in Uncategorized | Tagged , , , , , , , , , , | Leave a comment

On Web Workers, GWT, and a New Physics Demo

That original proof of concept gwtBox2d demo has slowly evolved along with the rest of the GWTns library. Though only a small portion of the original code remains, it has ended up being a pretty great stress test for new and tweaked library functionality; no matter how much performance improves, there can always be more blocks.

While the program itself doesn’t really do anything useful in a meaningful sense, or even help move Web Workers beyond a useful-for-simulated-annealing-while-animating-a-png demo reputation, I think it does have its own demonstrative value. And really, I have enough fun poking at it that I figured I might as well just gussy up the styling a bit and let other people have a turn.

Apologies First

  • If you’ve run the worker version of the demo without reading this, and were running (pre-3.6.4) Firefox, I probably crashed your browser. And on reopening, promptly did it again. Sorry about that (hi SUMO!). This isn’t a problem inherent to workers or this library (see TraceMonkey tear it up in The Rationals) but (probably) due to branch heavy/poorly traceable physics code. I’ll write more about Firefox below.
  • Stacking boxes turns out to be a good stressor, but Box2d can do much cooler things than this, often requiring more forgiving levels of computing power. For more examples, see the JBox2d demos or some of the many Flash/Alchemy demos out there.

The Demo

Now that you’ve been warned, here are the links. Return (or keep reading) for more on the application, your favorite browser, and some thoughts on using web workers in GWT.

Continue reading

Posted in Uncategorized | Tagged , , , , , , , , , , , | 4 Comments

On the Behavior of 2d Transformations in Internet Explorer, Part 2

Unsurprisingly, the last post ran overlong. I’ll endeavor to stick to code and equations wherever possible, as they are generally faster for me to write and you to parse. Amateur-hour Illustrator work takes a little longer.

All previous provisos still apply.

Implementation

If you’ve waded through that enormity of a post, you now know everything you need to get transformations working in Internet Explorer. To review:

  1. Keep an internal augmented matrix to represent the current transformation. Manipulate it through some combination of programmatic access and the parsing of CSS-like transform functions.
  2. Apply the linear portion of the affine transformation (the top 2×2 entries of the matrix) to the element through IE’s Matrix Filter
  3. Use “auto expand” as the SizingMethod so the element can venture beyond its original confines.
  4. Determine the shift that IE has automatically applied through its “bounding box behavior.”
  5. Translate the element by increasing its left and top attributes by the translation from the internal matrix, less the shift found in step (4).
  6. Work around minor problems like the limits of integer translations and problems due to altering layout via repositioning.

I’ll run through each of these items in turn, sharing implementation details and some of the interesting behavior I ran across in testing. If you’re the impatient type (or the “I don’t care” type), source is here. Continue reading

Posted in Uncategorized | Tagged , , , , , , , | 3 Comments

On the Behavior of 2d Transformations in Internet Explorer

In my work on the box2d and web worker modules in gwtns, I’ve needed the ability to put things up on the screen. To really make sure I was doing the word “overkill” justice, I decided to use my old TransformedElement module for that purpose. This has given me the opportunity to go back and ruminate over my original implementation.

Though it was only a few months ago, I originally took it on as more of a way to get to know the GWT deferred binding system than to write the perfect transformation library, and I wrote it in definite haste. I would probably not structure it the same way a second time. It’s not bad, but the differences between the underlying IE and CSS3 implementations are enough to make the unifying API a little less straightforward than it should be.

What was terrible, though, was the speed of the Internet Explorer version. The trial and error process to get IE’s matrix filter fully operational was annoying enough that I didn’t spend much longer on it than I had to. It helped that I only needed to position elements, not animate them, but that’s no longer true. As a result, I took a day last week to profile and revamp the IE implementation. The performance is actually now on par with the Firefox version, since the majority of execution time is taken up by DOM access.

But I’ll write more about the changes I made in my next post. For now, I really want to address how to get Internet Explorer transforming elements in almost the exact same way as all the “modern” browsers that support the CSS 2D Transforms (draft) specification. I’ve been surprised at the lack of support for IE as plugins and tools have started popping up to let authors easily transform elements. In fact, many explicitly state that IE support seems possible, but it hasn’t been implemented yet due to the pain of figuring out the different format.

But there’s no reason to repeat ourselves as a community. Supporting IE is actually pretty straightforward, you just have to be a little tricky to get around a few problems. With this done—if you’re willing to forsake Firefox users prior to 3.5 and Opera users prior to 10.5—essentially every browser in use is capable of applying geometric transformations to HTML elements. In writing this, I’m hoping that I can at least set a baseline of understanding of how transformations work in IE. That way, those that don’t want to figure it out from scratch won’t need to, and those that do will be able to concentrate on creating tools more elegant and sophisticated than I have here. Continue reading

Posted in Uncategorized | Tagged , , , , , , , | 3 Comments

“We never jit DOM workers”

This bug is very interesting in light of some assumptions I was making. Need to investigate.

It’s great to see so much work going into workers and the tracing engine, though, and to hear that changes will start rolling out in point releases. I feel for corporate deployments, and I hate when it’s done through ever-present update services, but in terms of browser security and iterating/testing standards-to-be, this seems to be the release model to follow. It’s a huge relief to not have to work around Chrome 2′s Worker problems, and it will be interesting to see if something similar happens with Chrome 3 in the next month or two. But I worry constantly about Firefox 3.0 vs 3.5 vs 3.6.

I don’t have a clear idea of where the balance should be between AUTO-UPDATE and developers waiting 5 years for implementations to converge (or at least settle). “Make all things easier for the user” is a good starting ideal—as long as “declining our help” is first on the list—but the effect in aggregate would not be linear.

Rambling finished. Flash 9 it is.

Posted in Uncategorized | Tagged , , , | 1 Comment

Web Worker Example: The Rationals

The Web Worker library I’m writing for GWT is getting to the point that it needs some examples. At least one of them needs to be simple enough that what is being done doesn’t distract from how it’s being done. You can see various examples like this in the official (draft) specification and over at the MDC. The problem with these examples, I think, is that they don’t really motivate the use of Workers; the task is usually completely unrealistic, intentionally implemented naively, and the gain so small that maintaining two separate versions of a codebase—one for browsers with Worker support and one for those without—ends up on the wrong end of the cost/benefit analysis.

But it might work for us. So, here’s a ridiculously unrealistic task, coded in the most straightforward way possible, and that runs perfectly well in Internet Explorer (8) without any Worker support. The reality is that instructive examples are of course necessary, so I’m going to trust to other demos to inspire excitement. In the meantime, I get another chance to play around with hyperbinary numbers. I’ve mentioned this sequence and its connection to the rational numbers before, but to recap: I first came across it on Division by Zero (still waiting for the rest of that series) and the original paper by Calkin and Wilf is here.

The example application calculates and displays a succession of rational numbers (fractions), as ordered by the Calkin-Wilf enumeration of the rationals. That sequence has several very interesting properties, but notably it contains every single (positive) rational number, in reduced form, exactly once. The example page doesn’t quite live up to the “every” part, but it does go on for a while. You can see it here: An Enumeration of the Rational Numbers.

By default, it actually starts at the 10-millionth rational number to make the demonstration a little more computationally taxing, but that can be changed if you want to see how it all begins (for the less motivated: “1″).

The GWT-y details

Right… Anyway, the “Rationals” GWT project is divided into two modules. The Worker module (RationalsWorker) extends a special Worker EntryPoint. It repeatedly calculates a new hyperbinary number, combines it with the last, and returns the result to the parent script:

Continue reading

Posted in Uncategorized | Tagged , , , , , , , , , , , , , | 2 Comments