More Than Throughput: Next Generation Wi-Fi Testing with Ixia's WaveDevice
by Joshua Ho on March 25, 2016 8:00 AM EST
It’s been quite some time since I first started writing for AnandTech, and without question there’s been a lot of changes that have happened to our testing methodologies in the past few years. One of the main issues that I’ve always been thinking about while working through reviews is how we could improve our testing methodology in a meaningful way outside of simply updating benchmarks to stay current. Internally we’ve been investigating these issues for quite some time now, and these changes have included the addition of SoC power efficiency comparisons and display power efficiency measurements. There are a lot of other changes here and there that I still want to make, but one of the major unexplored areas has been wireless radio performance.
Wireless performance testing is probably one of the hardest things that we could test, and for a time I had almost given up hope on deploying such testing within AnandTech. However life has a way of playing out differently than I expect, and in the past few months we’ve been working with Ixia to make Wi-Fi testing a reality. Ixia, for those of our readers who aren't familiar with the company, is a traditional player in the networking test space. They are perhaps best known for their Ethernet test products, and more recently have been expanding into wireless and security testing with the acquisition of companies like VeriWave and BreakingPoint Systems.
We have done Wi-Fi testing before, but in the past we were mainly focused upon a relatively simple and arguably not particularly interesting test case: maximum throughput in ideal conditions. It was obvious that Wi-Fi in many devices is still not perfect, as subjective differences in reception and reliability can feel obvious. However, without any data or methods of replication it was hard to really prove that what we felt about wireless performance was really the case.
A few months ago, Ixia brought me into their offices to evaluate their WaveDevice system, which a Wi-Fi testing device uniquely suited to solving our testing needs. This system is effectively integrates a number of tools into a single chassis, including: Wi-Fi access points, traffic generators, programmable attenuators to set path loss, channel emulators to simulate a certain kind of RF environment in terms of interference and bandwidth, packet sniffers and analyzers, and signal/spectrum analyzers. These tools are implemented such that each layer of the Wi-Fi protocol can be analyzed, from the physical link layer of raw bits encoded at the carrier frequency of 2.4 of 5 GHz, to the application level where everything is neat bitstreams transmitted or received from specified addresses.
Of course, hardware is just one half of the equation. In order to provide a full solution, Ixia also has a full suite of software to make it possible to run common tests without spending excessive amounts of time developing custom tests. While WaveDevice supports custom tests through its API, WaveDevice out of the box supports a simple throughput test, which is effectively like iPerf but with more advanced statistics. There's also a general data plane test to evaluate throughput when varying the traffic direction, traffic type, frame size, and frame rate. Other than these basic tests, WaveDevice also has tests for rate vs range, roaming latency, and traffic mix testing. In the case of rate vs range, it's possible to run an automated sequence of throughput tests while varying frame rate, transmit power, and physical link rates. Meanwhile in the interesting case of roaming latency, we can test how long it takes for a Wi-Fi device to hop from one access point to another when fading out the signal of one access point and fading in the signal of another. Finally, the traffic mix test allows for a test of throughput when faced with competing traffic that is also transmitting.
Ixia has also made sure to support a range of OSes in order to make sure that almost any device can be tested against WaveDevice. In addition to maintaining iOS and Android applications, it seems that WaveAgent is a simple C application at its core that can be easily used for embedded systems like wearables and Wi-Fi cameras, so it seems that the "device" in WaveDevice really refers to any client with a Wi-Fi chipset.
While at first glance it might seem like these tests are pretty simple, it turns out that there's a huge amount of nuance to them. To try and understand the nuance I'm talking about, it's best to start with a basic primer on how Wi-Fi works.
Facebook
Twitter
RSS
44 Comments
View All Comments
Ryan Smith - Friday, March 25, 2016 - link
Correct, it's the 12.9" iPad Pro.plext0r - Friday, March 25, 2016 - link
Excellent article! Thanks for the write-up using professional-level WiFi testing.jardows2 - Friday, March 25, 2016 - link
Very interested to see this in use. In my work, I daily have to deal with people's wi-fi problems, and to see some of the insights this tool can use will be very enlightening. Trying to fix people's wi-fi over blind phone support is an excercise in frustration!Ravn - Friday, March 25, 2016 - link
Excellent stuff Anandtech. WiFi is usually in the specifications described by : WiFi: Yes, Bands: 2.4/5GHz a/b/c/g/n, Maximum throughput: xxx Mbps. And it says just about nothing about the quality in the WiFi unit. Finally some relevant RF data that describes how the WiFi performs in real life. Thank You!An additional test that could broaden the relevans of the WiFi testing, could be how the WiFi unit performs with a lot of BlueTooth units in the same area. BT's nasty frequency hopping nature in the whole WiFi band results in a lot of problems in WiFi setups. How the WiFi units handles this could be very interresting to include.
zodiacfml - Friday, March 25, 2016 - link
Awesome and powerful testing machine you have there. One "Small" Wi-Fi testing website that I read regularly would be interested too. Yet, too powerful that only electrical engineers would use most of its functions.If I'm correct, you posted before that this device can also test for MU-MIMO performance without too much difficulty. Wi-Fi AP and routers reviews on Anandtech in the future wouldn't hurt? :)
On a second thought, I think there is a brighter future for 802.11ad than say, MU-MIMO. As long it is line of sight or no obstruction, 1 Gbps is easy for this standard.
name99 - Friday, March 25, 2016 - link
You've left out some interesting aspects of the physical layer.An essential part of this layer is the Forward Error Correction (FEC), which augments the transmitted data with additional data in such a way that if a few bits in the stream are in error, they can be recreated from the remaining bits (think parity on steroids).
These error correcting codes have been improved over the years in successive specs as it's become feasible to throw more computation at them, with the current state of the art being so-called LDPC (low density parity code). [These same codes are currently used by a lot of flash vendors, but have a theoretical problem(trapping sets) that limits their effectiveness above certain noise levels, so better alternatives have been proposed (but as far as I know are not yet in production) for flash, and likely will follow in the next big WiF spec.]
The specifically interesting thing about these codes, in the context of this article, is that it's not THAT useful to simply say that a chipset implements LDPC (or some other FEC). Implementing the encoding is a fixed algorithm that you can't really get wrong, but there are many ways of implementing a decoder (in other words, ways of attempting to construct the correct data stream from a corrupted data stream). These methods, of course, differ in the power they require, how much computation they utilize, how long they take to correct errors, and how complicated they are to implement.
The real difference in performance of different chipsets (at the L1 level) is in how well their FEC decoders work. That's where the magic lives.
At the next level up (the MAC level) its is crazy how much performance is lost because of the decentralized/unco-ordinated nature of the media access protocol.(This is the CSMA/CA that the article mentions.)
Even in the simplest real world case of one base station and one device, you're losing 35% or so of your goodput to the MAC protocol, and it rapidly drops to 50% and then worse as you add just a few devices. The successive specs have tried various schemes (primarily using the logical equivalent of very long packets) to limit the damage, but all this has done is really keep things standing still so that the situation in each successive spec is not worse than in the previous spec. LTE can be vastly more efficient because it provides for a central intelligence that co-ordinates all devices and so does not have to waste time on guard intervals where everyone is looking around making sure that no-one else is talking or getting ready to talk.
I don't understand why 802.11 has been so slow to adopt this model; putting the controlling intelligence in the base station (or base station equivalent in a peer network) and having every other device act as a slave. They're going to HAVE to go there at some point anyway --- they've pretty much run out of every other performance option --- and avoiding doing so in 802.11ac just means five more years of sub-optimal performance.
[You can see this in the iPad Pro numbers. The BCM4355 supports 80MHz channels, and so a maximum PHY rate of 866Mbps, But the best you see is just under 600Mbps (and that performance is only available when transmitting extremely large packets in only one direction); the gap between this and the PHY rate is because of time wasted doing nothing but sitting around following the MAC protocol. This disappointing goodput compared to PHY rate is not due to the mythical "interference" that is blamed for every random radio issue; it is due to the design of the MAC protocol.
You also see performance fall as the received signal strength falls. This performance drop is ALSO not due to interference, unless you want to make that word meaningless. The correct term for this performance drop is that it is the result of noise, or more precisely a falling SNR (signal to noise ratio). As the signal to noise ratio falls, you can pack fewer bits into each fragment of time (ie you have to switch from using QAM256 down to QAM64 down to QAM16) and you have to use more of those bits as error correction bits rather than as actual data bits.]
Denbo1991 - Friday, March 25, 2016 - link
802.11ad and 802.11ax will have some centralized scheduling features to cut down on the overhead you talk about, especially in the context of many devices on one AP or many overlapping APs.zodiacfml - Saturday, March 26, 2016 - link
There's way a lot more to talk about here than possible.alanore - Saturday, March 26, 2016 - link
Definitely some good points that should be covered. It might be worth covering how older low speed devices can consume a large proportion of the air time and thus performance.Also in the article it might be worth calling out spatial streams and how they effect performance. In the article it was an apples for apples comparison (2x2 Vs 2x2) but I guess soon we might see a poorly performing 3x3 laptop getting similar results to the iPad pro.
Ratman6161 - Monday, March 28, 2016 - link
Interesting but....for most of us does it really mean anything? So an iPad can achieve 600 Mbps throughput. How does this help me when wireless is used nearly exclusively for accessing the internet and my ISP provides 60 Mbps? For home use, I'm more interested in how well are things working when I have my Android TV streaming an HD Netflix movie while in the other room my wife is doing the same on Amazon and we are also both web surfing on either a tablet or a laptop...and that's more about the router than the individual devices, isn't it?Even at the office, no one is doing anything that requires 600 Mbps or even the 300 of the Pixel C (and the connection in/out of our building is only 20 Mbps). Its more a question of how many devices we can get connected simultaneously at a reasonable/usable speed.