Sabtu, 25 Agustus 2012

Mozilla Researcher Takes Apart Google's Octane JavaScript Benchmark


Earlier this week Google unveiled a big upgrade to its V8 JavaScript benchmarking suite, dubbed Octane. Octane is not a completely new benchmark, actually it incorporates V8, but adds four "real-world" apps and scenarios that should provide a better gauge for performance as it affects regular users and actual apps.

The move is in the right direction and benchmarks need to incorporate more real code, but not everyone is convinced. Mozilla's Alon Zakai for example has a few problems with a couple of the choices Google made when creating Octane.

"One of the new benchmarks in Octane is 'Mandreel', which is the Bullet physics engine compiled by Mandreel, a C++ to JS compiler. Bullet is definitely interesting code to include in a benchmark," he writes.

"However the choice of Mandreel’s port is problematic. One issue is that Mandreel is a closed-source compiler, a black box, making it hard to learn from it what kind of code is efficient and what should be optimized," he explains.

The problem, he believes, is that the compiler itself is closed-source, so it's hard to tell whether any performance issues are caused by the actual code or the compiling process. It also makes it hard to test the process with a different C++ starting code.

A much better choice for a C++ compiler, he believes, would have been Emscripten, an open source project to which he contributes, incidentally. Being an open source project it's much easier to analyze the entire process.

What's more, he argues that Mandreel is not that popular online, it's just popular with games in the Chrome Web Store. It's easy to understand why would Google want to test and optimize for something used by plenty of Web Store apps, but it's not representative of the entire web, necessarily.

The Mozilla researcher also explains that different C++ code compiled into JavaScript can yield very varied results when it comes to performance, a piece of code may do great in Chrome but not in Firefox, but other code may perform the other way around.

Zakai also has a problem with the choice of Box2DWeb, the JavaScript port of the popular Box2D physics engine. For one, Box2DWeb is based on an older version of Box2D. Furthermore, it performs significantly poorer than JavaScript version of Box2D compiled with Emscripten from the original C++ version. He goes into more details in his post.

All of this is not meant to denigrate Octane. The problem is, as always, that the benchmark fails to provide a comprehensive view on performance, just like every other benchmark before it. It's still better than many other tests and it's still better than V8, but it could have been even better.

Via: Mozilla Researcher Takes Apart Google's Octane JavaScript Benchmark

Tidak ada komentar:

Posting Komentar