Quick Look: Vulkan Performance on The Talos Principle
by Daniel Williams & Ryan Smith on February 17, 2016 10:00 AM EST
Following yesterday’s hard launch of Vulkan 1.0 – drivers, development tools, and the rest of the works – also released alongside Vulkan was the first game with Vulkan rendering support, The Talos Principle. Developer Croteam has a history of supporting multiple rendering paths with their engines, and the 2014 puzzle-em-up is no different, supporting DirectX 9, DirectX 11, and OpenGL depending on which platform it’s being run on. Now with Vulkan’s release Croteam has gone one step further, implementing early Vulkan support in a beta build of the game.
Since this is the first game with any kind of Vulkan support, we wanted to spend a bit of time looking at what Vulkan performance was like under Windows. Games with full support for Vulkan are still going to be some time off, as even with game dev participation in the standardization process it takes time to write a solid and high efficiency rendering path for these new low-level APIs, but none the less it gives us a chance to at least take a peek at the state of Vulkan on day 1.
To be very clear here this is an early look at Vulkan performance; Croteam admits from the get-go that their current implementation is very early, and is not as fast as their now highly tuned DirectX 11 implementation. Furthermore The Talos Principle is not a title that’s designed to exploit the CPU utilization and draw call improvements that are central to Vulkan (unlike say Star Swarm when we first looked at DX12). So with that in mind, it’s important to set reasonable expectations of what’s to come.
On the driver side of matters, both AMD and NVIDIA released Vulkan drivers yesterday. As is common with new API releases, both drivers are developer betas and either lack features or are based on older branches than current consumer drivers, however the NVIDIA driver has passed Vulkan conformance testing. AMD and NVIDIA will be integrating Vulkan into their release consumer drivers in the future as they improve on driver quality and catch up with the latest driver branches.
Finally, for our testing we’re using our standard GPU testbed running Windows 8.1, in part to showcase Vulkan on a platform that can’t receive DirectX 12. As the release of AMD’s drivers was unexpected – we had already begun preparing for this article earlier in the week – we don’t have results for very many AMD cards, but as this is a quick look it gets the point across.
| CPU: | Intel Core i7-4960X @ 4.2GHz |
| Motherboard: | ASRock Fatal1ty X79 Professional |
| Power Supply: | Corsair AX1200i |
| Hard Disk: | Samsung SSD 840 EVO (750GB) |
| Memory: | G.Skill RipjawZ DDR3-1866 4 x 8GB (9-10-9-26) |
| Case: | NZXT Phantom 630 Windowed Edition |
| Monitor: | Asus PQ321 |
| Video Cards: | NVIDIA GeForce GTX 980 Ti NVIDIA GeForce GTX 960 NVIDIA GeForce GTX 760 AMD Radeon R9 Fury X AMD Radeon R9 285 AMD Radeon R9 370 |
| Video Drivers: | NVIDIA Release 361.91 (DX11 & OpenGL) NVIDIA Beta 356.39 (Vulkan) AMD Radeon Software Crimson 16.1.1 Hotfix (DX11 & OpenGL) AMD Radeon Software Beta for Vulkan (Vulkan) |
| OS: | Windows 8.1 Pro |
The Talos Principle: Performance
Update 02/19: By request, I've also added Fury X numbers to our comparison to showcase high-end AMD performance
We’ve gone ahead and run our full collection of cards with Ultra settings at both 1080p and 720p to showcase a typical gaming workload and a lighter workload that is much more unlikely to be GPU limited. We’ve also gone ahead and run our two most powerful cards, the GeForce GTX 980 Ti and Radeon R9 Fury X, at 1440p to also showcase a more strictly GPU-bound scenario.
As expected from Croteam’s comments, at no point here does Vulkan catch up with DirectX 11. This is still an early rendering path and there’s no reason to expect that in time it won’t get up to the speed of DX11 (or even surpass it), but that’s not the case right now.
The real reason we set about to run these tests was not to compare early Vulkan to DX11, but rather to compare Vulkan to the API it succeed, OpenGL. OpenGL itself isn’t going anywhere – it is the DirectX 11 to Vulkan’s DirectX 12, the API that will remain for non-guru programmers who don’t need the power but need easier access – but as OpenGL suffers from many of the same performance bottlenecks as DX11 (plus some whole new ones from a 24 year legacy), there’s clear room for improvement with Vulkan.
To that end the results are more promising. As compared to The Talos Principle’s OpenGL renderer, the Vulkan renderer is not all that different in performance in clearly GPU-bound scenarios. But once we start looking at CPU-bound scenarios, even in a somewhat lightweight game like The Talos Principle, Vulkan pulls ahead. This is especially evident on the GTX 980 Ti and R9 Fury X at 1080p, and across a few different cards at 720p. This offers our first sign that Vulkan will indeed be capable of bringing its desired CPU performance benefits to games, perhaps even in games where they’re not explicitly pushing the draw calls limits of a system.
These performance results do also highlight some performance issues as well. The two slower AMD cards – both of which have 2GB of VRAM – see some unusual performance regressions. Based on our experience with DX12 and Mantle, it seems likely that on these settings The Talos Principle is approaching full VRAM utilization, leading to the occasional drop in performance. Just as with DX12, developers have near-full control of the GPU, and will need to manage VRAM usage carefully.
Radeon R9 285 Running via Vulkan
As for image quality, the rendering path that Croteam has implemented appears to be every bit as good as their existing paths. Both AMD and NVIDIA cards exhibited great image quality that was comparable to the baseline DX11 rendering path. And admittedly we weren’t expecting any differently, but it means there are no image quality affecting bugs that we’ve picked up on in our testing.
That said, these Vulkan drivers are classified as betas by both AMD and NVIDIA, and this is not a misnomer. We encountered issues with drivers for both parties, particularly in NVIDIA’s case where we couldn’t successfully run a Talos benchmarking session twice without rebooting, otherwise the game would crash. So coupled with the known limitations for these drivers, it goes without saying that these drivers are really only for testing and development purposes, and that AMD and NVIDIA will need to knock out some more bugs before integrating Vulkan support into their release drivers.
Overall with this being the third low-level API release in the past two years (and a rebirth of sorts for Mantle), for our regular readers there aren’t any great surprises to be found with Vulkan as implemented on The Talos Principle. Still, the results do show promise. Khronos has set about creating a new cross-platform low-level API, and this early preview of Vulkan shows that they have achieved their basic goals. Now it will be a matter of seeing what developers can do with the API with more developer time and in conjunction with further driver improvements from AMD, NVIDIA, and the other GPU vendors.
Facebook
Twitter
RSS
43 Comments
View All Comments
nathanddrews - Thursday, February 18, 2016 - link
Last I checked, Khronos will not be developing any games. If I had any other metrics from which to judge Vulkan, I would. At each step along the way of Mantle (pre-Vulkan) and DX12 (ongoing), we were treated to demonstrations, real world games, and beta benchmarks that showed us dramatic performance improvements over DX11 (which, in turn, pushed game/driver devs to make DX11 implementations better). This implementation of Vulkan - being the first out the gate - is not great PR for Vulkan when viewed against previous API releases. This wasn't just some random developer pushing out a random build either, this was planned release by a major studio hyped across the blogosphere by AMD, NVIDIA, and Khronos. I go back to my original post, that if everyone knew it was going to perform poorly, they should have waited for a few more nightly builds before unveiling it.Look, I just want Vulkan to be awesome - more awesomer than DX12. Microsoft just announced that Quantum Break will be DX12/Win10 Store exclusive and rumors suggest that Forza, Gears of War, etc. will also be going that way. We need a powerful cross-platform API so that Windows XP-10 gamers, Linux gamers, etc. can all play together. DX12 is great, but locked to W10 is terrible for everyone. Croteam has shaken my confidence in Vulkan. If you don't feel that way, then you're more optimistic than me.
BurntMyBacon - Thursday, February 18, 2016 - link
@nathanddrews: "If I had any other metrics from which to judge Vulkan, I would."An excellent point. That's why I'm reserving judgment for now. Interestingly, I had the opposite problem with DX12. The earliest benchmarks were extremely targeted and though they performed well, they were not representative of real world performance. So, I still had to reserve judgment.
@nathanddrews: "At each step along the way of Mantle (pre-Vulkan) and DX12 (ongoing), we were treated to demonstrations, real world games, and beta benchmarks ..."
Mantle and DX12 demonstrations and the likes were funded (at least in part) by AMD and Microsoft respectively. I'm not sure what kind of funding Khronos has to work with. Of course that is also something to consider before putting all our hopes in groups like Khronos.
@nathanaddrews: "This implementation of Vulkan - being the first out the gate - is not great PR for Vulkan when viewed against previous API releases."
Can't argue with that. If they wanted better PR, they definitely should have put a little more work into it.
@nathanaddrews: "This wasn't just some random developer pushing out a random build either, this was planned release by a major studio hyped across the blogosphere by AMD, NVIDIA, and Khronos."
Croteam isn't an indie studio to be sure, but they aren't really in the same league as Valve, EA, and Ubisoft either. IIRC they only produce the Serious Sam games and Talos Principle. I'd consider them a smaller studio more in line with developers like GSC Gameworld than Ubisoft. That said, the hype really should have waited until they saw some preliminary results. It is not encouraging that this is the demo Khronos chooses to hype. Though, I suppose they may not have expected it to be released in this state. Lack of options is also probable, though neither of these options are confidence inspiring either.
@nathanaddrews: "Croteam has shaken my confidence in Vulkan. If you don't feel that way, then you're more optimistic than me."
With DX12, I had to contain my optimism lest I be disappointed. In hindsight, that was a good approach to take. I'd suggest the opposite here. I'd contain my disappointment as there is some evidence to suggest it may not be warranted.
RobATiOyP - Sunday, February 21, 2016 - link
Also, Vulkan SDK is multi-platform, with many vendors and architectures. They need to get the API out early, have drivers implement it, find the specification mistakes early before there's a large software base; for me given there were obvious driver issues it is quite encouraging that Vulkan showed a small improvement on OpenGL.Unlike the Star Swarm demo, which was an example of an engine which had been bottle necked by OpenGL & DX11 APIs, this "Puzzle em Up" isn't throwing draw calls at a maximum rate at the API.
This is very much a notice to software developers that Vulkan is real, not vapour.
BurntMyBacon - Thursday, February 18, 2016 - link
@nathanddrews: "... the point is that performance at present is severely lacking. Personally, I find that very disappointing."Just to be clear, what do you find disappointing? Croteam/Talos Principle or Khronos/Vulkan.
I can understand the first as one could argue that they should have held the Vulkan release until it at least outperformed its OpenGL counterpart universally. Sure it is a beta and should be viewed as such, but there is something to be said for first impressions. ; ' )
The second I don't agree with (at least not yet). Putting aside API comparisons, there are plenty of examples of games with the same API that perform wildly differently. The Half-Life series, for instance, generally has excellent performance for the visual quality it brings to the table. Doom 3 was often compared to HL2 and while it was technically superior and the engine had the potential for far superior graphics (full dynamic lighting, etc.) it was held back to what was considered comparable graphics by the lack of performance. For more modern DX11 examples, you can look to Batman: Arkham Knight and Assassin's Creed Unity. For their performance, they should be leagues ahead of Crysis 3 in image quality and that isn't even considered a well optimized title. If you don't like these examples, there are many others to choose from. The point is, you can't judge an API based on one game (particularly from a smaller developer) that may not be well optimized (explicitly stated here).
RobATiOyP - Sunday, February 21, 2016 - link
This is dead wrong, "they should have held the Vulkan release until it at least outperformed its OpenGL counterpart universally." this is NOT production optomised code, this is early implementations of a new API, which needs developers to use it, use it NOW, not wait.Large applications relying on Vulkan, have to wait for API & driver maturity, there's risks & costs in early adoption.
Manch - Thursday, February 18, 2016 - link
Did you? 2nd paragraph after the graphs "The real reason we set about to run these tests was not to compare early Vulkan to DX11, but rather to compare Vulkan to the API it succeed, OpenGL."extide - Wednesday, February 17, 2016 - link
I think you are confused here.Vulkan is release 1.0 NOT a beta.
The Drivers for AMD and nVidia are both in beta state right now.
The only reason it's slower than DX11 is because this specific game's engine is not as fast when running in vulkan, not because of vulkan itself. It's a brand new API, the developers need time to learn it to tune it for speed. Plus this was not an implementation even designed for speed at all, it is more of a proof of concept simply to show it works. That's all.
RobATiOyP - Sunday, February 21, 2016 - link
Is this game even something that is bottlenecked by OpenGL/DX11 API? It looks like typical game graphics, not the crazy busy Star Storm demo, with an engine able to send 60,000 draw calls a second at the hardware. if the DX11 version is "highly tuned", would they see any benchmark improvement in DX12?Zingam - Wednesday, February 17, 2016 - link
I bet Vulkan will become relevant in 5 years when most people would have bought modern graphics cards.bcronce - Wednesday, February 17, 2016 - link
AMD released GCN 1.0 5 years ago, and they support Vulkan back to 1.0.