So I have finally found the issue and would like to share the cause: I use a KVM switch for my 2 displays. Now I have bypassed my main display (have it directly connected) and the crashes are gone. Sometimes my KVM switch would drop the displayport connection and I have to switch it back and forth. It seems that this dp drop caused the crashes