Assessing Cavium's ThunderX2: The Arm Server Dream Realized At Last
by Johan De Gelas on May 23, 2018 9:00 AM EST- Posted in
- CPUs
- Arm
- Enterprise
- SoCs
- Enterprise CPUs
- ARMv8
- Cavium
- ThunderX
- ThunderX2
Memory Subsystem: Bandwidth
Measuring the full bandwidth potential of a system with John McCalpin's Stream bandwidth benchmark is getting increasingly difficult on the latest CPUs, as core and memory channel counts have continued to grow. As you can see from the results below, it not easy to measure bandwidth. The result vary wildly depending on the setting you choose.
| Memory: STREAM Bandwidth | ||
| Mem Hierarchy |
Compiler & OS settings | Result |
| Cavium ThunderX2 Gcc 7.2 binary |
-O2 -mcmodel=large -fopenmp -DVERBOSE -fno-PIC" OMP_PROC_BIND=spread |
241 GB/s |
| Cavium ThunderX2 Gcc 7.2 binary |
-Ofast -fopenmp -static OMP_PROC_BIND=spread |
157 GB/s |
| Cavium ThunderX2 Gcc 7.2 binary |
OMP_PROC_BIND not configured | 118 GB/s |
| Intel ICC Binary | -fast -qopenmp -parallel KMP_AFFINITY=verbose,scatter |
183 GB/s |
| Intel gcc Binary | Ofast -fopenmp -static OMP_PROC_BIND=spread |
151 GB/s |
| Intel gcc Binary | Ofast -fopenmp -static OMP_PROC_BIND not configured |
150 GB/s |
Theoretically, the ThunderX2 has 33% more bandwidth available than an Intel Xeon, as the SoC has 8 memory channels compared to Intel's six channels. These high bandwidth numbers can only be achieved in very specific conditions and require quite a bit of tuning to avoid reaching out to remote memory. In particular, we have to ensure that threads don't migrate from one socket to the other.
We first tried to achieve the best results on both architectures. In case of Intel the ICC compiler always produced better results with some low level optimizations inside the stream loops. In case of Cavium, we followed the instructions of Cavium. So strictly speaking these are not comparable, but it should give you an idea of what kind of bandwidth these CPUs can achieve at their respective peaks. To be fair to Intel, with ideal settings (AVX-512) you should be able to achieve 200 GB/s.
Nevertheless, it is clear that the ThunderX2 system can deliver between 15% and 28% more bandwidth to its CPU cores. This works out to 235 GB/sec, or about 120 GB/sec per socket. Which in turn is about 3 times more than what the original ThunderX was capable off.
Memory Subsystem: Latency
While Bandwidth measurements are only relevant to a small part of the server market, almost every application is heavily impacted by the latency of memory subsystem. To that end, we used LMBench in an effort to try to measure cache and memory latency. The numbers we looked at were "Random load latency stride=16 Bytes". Note that we're expressing the L3 cache and DRAM latency in nanoseconds since we don't have accurate L3-cache clockspeed values.
| Memory: LMBench Latency | |||
| Mem Hierarchy |
Cavium ThunderX DDR4-2133 |
Cavium ThunderX2 DDR4-2666 |
Intel Skylake 8176 DDR4-2666 |
| L1-cache (cycles) | 3 | 4 | 4 |
| L2-cache (cycles) | 40/80 (*) | 8-9 | 12 |
| L3-cache 4-8 MB (ns) | N/A | 27-30 ns | 24-29 ns |
| Memory 384-512 (ns) | 103/206 (*) | 156-157 ns | 89-91 ns |
The L2-cache of the ThunderX2 is accessed with very little latency, and with a single thread running, the L3-cache is competitive with the Intel's complex L3 cache. Once we hit the DRAM however, Intel offers significantly lower latency.
Memory Subsystem: TinyMemBench
To get a deeper understanding of the respective architectures, we also ran the open source TinyMemBench benchmark. The source code was compiled with GCC 7.2 and the optimization level was set to "-O3". The benchmark's testing strategy is described rather well in its manual:
Average time is measured for random memory accesses in the buffers of different sizes. The larger the buffer, the more significant the relative contributions of TLB, L1/L2 cache misses, and DRAM accesses become. All the numbers represent extra time, which needs to be added to L1 cache latency (4 cycles).
We tested with single and dual random read (no huge pages), as we wanted to see how the memory system coped with multiple read requests.
One of the major weaknesses of the original ThunderX was that it did not support multiple outstanding misses. Memory level parallelism is an important feature for any high-performance modern CPU core: using it it avoids cache misses that would starve the wide back end. A non-blocking cache is thus a key feature for wide cores.
The ThunderX2 does not suffer from that problem at all, thanks to its non-blocking cache. Just like the Skylake core in the Xeon 8176, a second read causes the overall latency to increase by only 15-30%, and not 100%. According to TinyMemBench, the Skylake core has tangibly better latencies. The datapoint at 512 KB is of course easy to explain: the Skylake core is still fetching from its fast L2, while the ThunderX2 core has to access its L3. But the numbers at 1 and 2 MB indicate that Intel's prefetchers offer a serious advantage as the latency stays is an averag of the L2 and the L3-cache. Around 8 to 16 MB, the latency numbers are close, but once we go beyond the L3 (64 MB), Intel's Skylake offers lower memory latencies.
Facebook
Twitter
RSS
97 Comments
View All Comments
imaheadcase - Wednesday, May 23, 2018 - link
I really think Anandtech needs to branch into different websites. Its very strange and unappealing to certain users to have business/consumer/random reviews/phone info all bunched together.Ever since anand actually left it really did venture into more a business/insider based website with random stuff thrown in. It is in no way a bad thing, its just like this review for instance would not appeal to %95 of readers normally. Everyone likes technology naturally that comes to this website, but its a fine line between talking about high end server components that are out of reach to people who just read the article on the mini-itx gaming motherboard. lol
Andrei Frumusanu - Wednesday, May 23, 2018 - link
You're always free to skip articles, nobody's forcing you to read it.boeush - Wednesday, May 23, 2018 - link
I guess he'd prefer the site content to be grouped in some manner roughly mirroring market segmentation. For instance: consumer, professional, enterprise, exotic/HPC. As opposed to jumbling everything together. Personally, I don't mind - but then, I'm not known for obsessive-compulsive organizing, either :)BurntMyBacon - Thursday, May 24, 2018 - link
Given the large differences in tech, focus, needs, and trends, I wouldn't mind breaking out Phones and perhaps servers into their own sections. I think there is more than enough overlap to keep consumer and professional desktop/laptop/workstation together, but that is entirely up to how deeply you want to divide things up. On the other hand, you'll want all of it to show up on the front page in some form, or it'll look like the site doesn't have much activity. Perhaps separate pipelines for each category could work. That all said, I don't really mind just skipping over articles that don't interest me. :)imaheadcase - Thursday, May 24, 2018 - link
Please, that is just lazy excuse. Even news websites have catagory based on the news you interested in. Anandtech literally had a review of a gaming motherboard then a high end server thing, and newz feed gets filled with phone and other news.name99 - Thursday, May 24, 2018 - link
God, you must REALLY hate Twitter then...I argue with Andrei a lot, but every so often he writes a sentence like "You're always free to skip articles, nobody's forcing you to read it" that makes me want to clap him on the back and say "yes, YOU get it" :-)
Threska - Sunday, May 27, 2018 - link
Taken to it's logical extreme the front page could be a dumping ground cesspool and the retort would be "you don't have to wade through any of it" which sounds witty but doesn't solve anything, but over time would lead to the predictable outcome of people leaving.imaheadcase - Sunday, May 27, 2018 - link
I do hate twitter, but because it has no valid purpose other than to get customer service done faster with companies because it reflects more on them because public venue. Its mostly just a rant inducing place, or a place that is basically just texting anyways since everyone just wants you to send a DM.The whole idea of saying "you are free to skip it" is kinda silly thing to say on the internet now. Especially since more and more you can filter things according to what you want. Not only that, but with the tight competition with views from tech websites its in best interest to have more options.
Even the layout of website never changed. I mean have you ever been to website without a adblocker on? They don't even advertise tech related stuff on it. Its just stupid clickbait stuff.
Keep in mind, this is not a complaint about articles itself, its just how they are posted. I love this site, been coming to it ever since i built first pc when i was a kid. But its focus is all over the place now vs years ago out what its posting. I'm half thinking one day i will see a review of electronic toothbrush then next day new CPU.
GreenReaper - Monday, June 4, 2018 - link
I'd be fine with that, as long as it was the best darn toothbrush in town!Threska - Sunday, May 27, 2018 - link
Accessing through RSS might be a better solution especially with a good reader. Just needs accurate tags to match.