This post will be probably be very boring for everyone who doesn't have this problem. But since it cost me some time to figure out a solution, I think it is worth sharing.
<a href="https://blag.nullteilerfrei.de/2018/08/26/atomic-upsert-with-unique-constraint-on-null-able-column-in-postgresql/#more-4594" class="more-link">I care. You share. Read on!</a>