PCIe 4.0

As the first commerical x86 server CPU supporting PCIe 4.0, the I/O capabilities of second generation EPYC servers are top of the class. One PCIe 4.0 x16 offers up to 32 GB/s in both direction, so each socket offers up to 256 GB/s in both directions, for a full 128 PCIe 4.0 lanes per CPU. 

Each CPU has 8 x16 PCIe 4.0  links available which can be split up among up to 8 devices per PCIe root, as shown above. There is also full PCIe peer-to-peer support both within a single socket and across sockets.

With the previous generation, in order to enable a dual socket configuration, 64 PCIe lanes from each CPU were used to link them together. For EPYC, AMD still allows for 64 PCIe lanes to be used, but these are PCIe 4.0 lanes now. There is also another feature that AMD has here - socket-to-socket IF link bandwidth management - which allows OEM partners to design dual-socket systems with less socket-to-socket bandwidth and more PCIe lanes if needed. 

We also learned that there are in fact 129 PCIe 4.0 lanes on each CPU. On each CPU there is one extra PCIe lane for the BMC (the chip that controls the server). Considering we are living in the age of AI acceleration, the EPYC 7002 servers will be great as hosts for quite a few GPUs or TPUs. Density has never looked so fun.

Zen 2 and Rome: SMILE For Performance The BIG LIST of Rome CPUs: Core Counts and Frequencies
Comments Locked

180 Comments

View All Comments

  • JoeBraga - Wednesday, August 14, 2019 - link

    It can happen if Intel uses the new archtecture Sunny Cove and MCM/Chiplet design instead of Monolithic Design
  • SanX - Thursday, August 15, 2019 - link

    7zip is not a legacy test, it is important for anyone who sends big data over always damn slow network. Do you know all those ZIPs, GZs and other zippers which people mostly use, compress with turtle speeds as low as 20 MB/s even on supercomputers ? The 7Zip though parallelizes that nicely. So do not diminish this good test calling it "legacy"
  • imaskar - Friday, August 16, 2019 - link

    7zip is a particular program, doing LZMA in parallel, that's why it is faster that lets say gzip. But on server you often do not want to parallel things, because other cores are doing other jobs and switching is costly. There are a lot of compressing algorithms which are better in certain situations. LZMA rarely fits. More often it is it's LZ4 or zstd for "generate once, consume many" or basic gzip (DEFLATE) for "generate once, consume once". Yes, you would be surprised, but the very basic 30 years old DEFLATE is still the king if you care for sum of compress, send, decompress AND your nodes are inside one datacenter (which is most of the times).
  • SanX - Thursday, August 15, 2019 - link

    What you can say about Ian's own test he developed to demonstrate avx512 speed boost which shows some crazy up to 3-4x or more speedups ? Does your test of Molecular Dynamics tell that Ian's test mostly irrelevant for such huge improvement of speed of the real life complex programs?
  • imaskar - Friday, August 16, 2019 - link

    Probably because you can't use ONLY avx512. You still need regular things like jumps and conditions. And this is only the best case. Usually you also need to process part of the vector differently. For example, your vector has size 20, but your width is 16. You either do another vector pass, or 4 regular computations. Often second thing is faster or just the only option.
  • realbabilu - Sunday, August 18, 2019 - link

    Most of finite element software use Intel mkl to get every juice power spec of processor.it works for Intel ones not for amd
    Amd math kernel not heavily programmed, otnwaa just for Linux.
    Other third party like gotoblas openblas still trying hard to detect cache and type for zen2.
    I mean for workstation floating point still hard for amd.
  • peevee - Monday, August 19, 2019 - link

    Prices per core-GHz:
    EPYC 7742 $48.26
    EPYC 7702 $50.39
    EPYC 7642 $43.25
    EPYC 7552 $38.12
    EPYC 7542 $36.64
    EPYC 7502 $32.50
    EPYC 7452 $26.93
    EPYC 7402 $26.53
    EPYC 7352 $24.46
    EPYC 7302 $20.38
    EPYC 7282 $14.51
    EPYC 7272 $17.96
    EPYC 7262 $22.46
    EPYC 7252 $19.15

    Value in this 7282 is INSANE.
  • peevee - Tuesday, August 20, 2019 - link

    "Even though our testing is not the ideal case for AMD (you would probably choose 8 or even 16 back-ends), the EPYC edges out the Xeon 8176. Using 8 JVMs increases the gap from 1% to 4-5%."

    1%? 36917 / 27716 = 1.3319...

    33%. Without 8 JVMs.
  • KathyMilligan - Wednesday, August 21, 2019 - link

    University of Illinois Urbana-Champaign is very good university. I am too poorly prepared for this level of education. But I'm getting ready. I read a lot of articles and books, communicate with many smart former students of this university. I also buy research papers on site and this gives me a lot of useful information, which is not so easy to find on the Internet.
  • YB1064 - Wednesday, August 28, 2019 - link

    Looks like Intel has been outclassed, out-priced and completely out-maneuvered by AMD. What a disaster!

Log in

Don't have an account? Sign up now