Browser Benchmarks: Auditing Microsoft’s Performance Claims for Edge

Whenever you set up or upgrade a Windows 10 PC and launch the Edge browser for the first time, Microsoft displays a prominent ad extolling the performance advantages of the company’s web browser. Microsoft boldly claims that Edge is “faster than both Chrome and Firefox,” with speedometer graphics showing that the competing browsers are 22 and 16 percent slower, respectively.

Clicking on the tiny “see details here” button in the lower-right corner reveals the basis for Microsoft’s claim. The company’s claim is based on the JetStream 1.1 browser benchmark, a series of tests that evaluate a browser’s JavaScript performance.

The results of Microsoft’s tests are based on a specific hardware configuration: a PC with an Intel Core i5-3475S CPU, 4GB of RAM, and Windows 10 Enterprise version 17134. The Windows version is up-to-date (build 17134 is also known as the “April 2018 Update” that just shipped out to users this month), but the choice of processor is a bit unusual. While not everyone is running the latest hardware, the i5-3475S is a six-year-old part, first introduced in the second quarter of 2012.

While we don’t doubt that Microsoft’s reported numbers are accurate for this specific configuration, we wanted to perform the tests ourselves on some more modern hardware. That doesn’t make this a pure “audit” of Microsoft’s Edge benchmark, but rather an expansion in the hopes of discovering more relevant results.

The Hardware and Software

For our tests, we’re using two hardware configurations. Our “high-end” configuration is a custom-built PC running an Intel Core i7-6950X clocked at 4.0GHz with 64GB of DDR4 memory. Our “mid-range” option is an Intel NUC D54250WYK, powered by an Intel Core i5-4250U with 8GB of DDR3 memory.

Both systems were configured with clean installs of the Windows 10 April 2018 Update (Version 1803, Build 17134). For our browsers, we used the latest version of each as of the time of testing:

• Microsoft Edge 42.17134.1.0
• Google Chrome 66.0.3359.139
• Mozilla Firefox 59.0.3
• Opera 52.0.2871.99

Microsoft’s tests only include Edge, Chrome, and Firefox, but we opted to throw Opera into the mix as well, because everyone always forgets about Opera.

The Browser Benchmarks

As mentioned, Microsoft based their test only on JetStream 1.1, but in previous builds of Windows 10 prior to the April Update, Microsoft also touted results from the Octane 2.0 test. That test is now retired, however, so we chose to expand Microsoft’s testing suite to also include the Speedometer 2.0 benchmark, which some recommend as a replacement for Octane for more “real-world” browser benchmarking.

Each Test was run on each system three times, and the results reported in the charts below are the average of the three runs. For both the JetStream and Speedometer tests, a higher score equals better performance.

Benchmark Results: JetStream

Turning first to Microsoft’s test of choice, Edge actually wins the JetStream benchmark test on both our high-end and mid-range systems.

On the high-end system, Edge is 25 percent faster than Chrome, and about 17 percent faster than Firefox and Opera.

On the mid-range system, the lead is even greater, with Edge besting Chrome by 35 percent, Opera by 31 percent, and Firefox by 23 percent.

So it seems, at least thus far, that Microsoft was not only accurate in its performance claims for Edge, it may have even been a bit too conservative. But this is only one benchmark, JetStream, and while it’s a respected browser benchmark, it may not be the best indicator of real-world performance for consumers browsing the web.

Benchmark Results: Speedometer

We therefore turn to the Speedometer benchmark, which is intended to more closely address this “real-world” scenario. Unfortunately for Microsoft, Edge’s performance advantage disappears in this test, and it falls to a distant last place among the tested browsers.

On our high-end system, Edge is about 35 percent slower than Chrome and Opera, and about 22 percent slower than Firefox.

That trend worsens on our mid-range system, with Edge nearly 40 percent slower than Chrome and Opera, while maintaining the same 22 percent deficit with Firefox.

Conclusion

It should come as no surprise that Microsoft cherry-picked a browser benchmark that showed Edge in the best possible light next to its competitors. And in terms of the JetStream test, it seems that users at all ends of the hardware spectrum can expect good performance from Edge in terms of the specific JavaScript workloads that JetStream relies on.

But real-world usage might not be so rosy for Microsoft’s browser with its significant performance deficit in the Speedometer test, a test which prides itself on measuring the kinds of highly interactive online experiences that are increasingly common.

As of today, there may be other benefits to using Microsoft’s Windows 10 web browser — integration with Cortana, improved text rendering, smooth scrolling, longer battery life, etc. — but Microsoft’s blanket claims regarding its performance advantages are, of course, not so simple.