After the fragment overdraw improvements didn’t fix my framerate, I was stumped. The experiment was useful, in that it ruled out a possible cause, but it didn’t leave me with any clues for where to look next. Why did enabling MSAA tank my performance so badly? After a few cursory web searches failed to help, I turned to Claude Opus 5.5 in “mentor mode”.

“Mentor mode” instructs the LLM to to priortise my learning and development over simply solving the problem. It’s seeded with pedagogical research such as tacit knowledge extraction, because I suspect that a lot of what the model has learned about programming, debugging, etc. via reinforcment learning is tough to put into language. When I start a session, I describe my problem, what I’ve tried, what worked and what didn’t. Importantly, I omit all code. This forces the LLM to generate a search strategy, which is what I want to learn. If I included code, then the model may identify the issue by reading the code and generate leading questions.

Once I’ve set the context, the LLM generates thought-provoking questions about what I should do next. So in the case of my framerate issue, the first question was something like, “Apart from the fragment shader, what are some other things that happen per-frame which scale with MSAA?”. Reads and writes to and from the render pass attachments were the first thing that came to mind. In response, the LLM prompted me to estimate the overhead of the extra attachment memory traffic due to MSAA. A 4K colour attachment uses 3840*2160*4 = 33177600B = 33.1776MB, and when 4x MSAA is used that increase 4-fold to 132.7104MB. Assuming my RTX2070 has a memory bandwidth of 448GB/s. So I estimated that a full colour attachment write would take an extra (132.7104MB - 33.1776MB) / 448GB/s = 0.00022s under MSAA. That doesn’t seem anywhere near enough overhead to cause the ~100ms increase per frame that I observed.

After some more back and forth of me being stumped (I feel like some of the thought-provoking questions the LLM generates do not provoke useful thought), it escalated with a hint (paraphrased): “Your estimate of the memory traffic overhead used 448GB/s, which assumes traffic to device-local memory. Are you sure that you allocated device-local memory for the attachments?” Oh. I wouldn’t be surprised if I forgot to do that. I checked the code, and sure enough, I hadn’t requested device-local memory for the attachments. They were being stored in the default “host-local” heap, so the GPU had to interact with the attachments over PCIe. The best-case bandwidth for that is ~16GB/s. My earlier overhead calculation becomes (132.7104MB - 33.1776MB) / 16GB/s = 0.006s; several milliseconds extra for a full multisampled colour attachment write. I moved the attachment memory to device-local, and Bistro was suddenly rendering at 60FPS :)

This is the kind of problem that I’d usually pose to a forum, Reddit, or StackOverflow. Overall I think I got a better brain workout from consulting “mentor mode”, and certainly a better turnaround time. But I have mixed feelings about the process as a whole, because it’s a lost opportunity for collective sense-making.

Now that I was aware of the influence of memory traffic on my program’s performance, I wanted to figure out how to measure it.

NVIDIA performance counters in RenderDoc

I found this Vulkan performance sample which visualises memory throughput. I couldn’t find anywhere to measure that in RenderDoc, so I read through the sample’s source code. I found that used vendor-specific performance counters, so I decided to install NVIDIA’s profiling SDK to get more performance insights from RenderDoc.

It was easy to get working; RenderDoc gave me instructions on how to do it. I had to sign up for NVIDIA’s “developer program” to download the SDK, though, which felt yucky. The last step was to set boot.kernelParams = [ "nvidia.NVreg_RestrictProfilingToAdminUsers=0" ]; in my NixOS config (see also). This gave me access to, amongst many others, the dram__bytes_{read,write} and pcie__{read,write}_bytes performance counters in RenderDoc.

Here are the stats before my device-local fix:

That gives memory traffic costs of approximately:

• 488MB / 448000MB/s = 0.001s for DRAM
• 740MB / 16000MB/s = 0.046s for PCIe

After moving attachments to device-local memory, this became:

Memory traffic costs:

• 769MB / 448000MB/s = 0.0017s for DRAM
• 184MB / 16000MB/s = 0.012s for PCIe

I found another Vulkan performance guide about proper use of render pass attachments. I followed its advice and set VK_IMAGE_USAGE_TRANSIENT_ATTACHMENT_BIT and VK_ATTACHMENT_STORE_OP_DONT_CARE for the attachments that aren’t read by other passes. This didn’t make too much of a difference.

With VK_ATTACHMENT_STORE_OP_DONT_CARE:

With VK_ATTACHMENT_STORE_OP_DONT_CARE and VK_IMAGE_USAGE_TRANSIENT_ATTACHMENT_BIT:

But since I saw a modest saving, I left these changes in.

The remaining 100MB+ of PCIe traffic came from index and vertex buffers, which I also hadn’t requested to be device-local. After moving those over, here are the final numbers:

Memory traffic costs:

• 859/448000 = 0.0019s for DRAM
• 1.7/16000 = 0.0001s for PCIe

Nice. I don’t understand why PCIe write went down, though, because the index and vertex buffers are for reading only.

What I built

• Use device-local memory for render pass attachments
• Use device-local memory for index and vertex buffers

What I learned

• Memory type is super important

  I need to remember that programming a GPU is a multiplayer game: the host system cooperating with the graphics system. There’s no longer a single unified memory pool. Instead, there’s host memory and device (graphics) memory, some of which is host-only and device-only, and there’s host memory that’s device-visible, and device memory that’s host-visible. And they all have different capabilities.

Next steps

• Shade with physical units

← Sunstone dev log 19: less overdraw Newest