Gamemode performes worse than desktop

[GetWinDeck]

Describe the bug

For some reason every game I try has a noticeable fps drop when playing in gamemode compared the desktop. When running at 4k the difference is about a 25% drop in fps. When I set it to 1080p the difference is only about 5-10%. (I'm talking about the output resolution btw, from the settings menus in both steam and gnome, not from the game.)

If I'm being honest performance is quite poor overall actually. For example Spongebob Cosmic Shake runs at ~25fps at 1080p (in gamemode). I know I don't have the best hardware, but this seems a little low. I don't know if this is normal or not.

What did you expect to happen?

I expected gamemode to perform as well as desktop mode.

Output of rpm-ostree status

Hardware

I have an HP Elitedesk 800 g4 35W: Intel i5-8500T, 16GB RAM Dual Channel, 256 NVME Storage, AMD RX 560 4GB GPU.

Extra information or context

I've heard that polaris cards had issues with gamescope a little while back, so I wonder if this has something to do with that? I'm also wondering if this could be a vbios related issue? The vbios is dated to March 2017, which is before the card was released, so I wonder if there was an update that corrected some issues?