I think I might have found the culprit: I tried to generate output using DITA-OT 3.5.2. Our customized plugins rely on Oxygen plugins, so I naturally copied Oxygen plugins from their installation directory into the new DITA-OT directory along with our customized plugins. It looks like Oxygen has issues with running a newer DITA-OT than the one it was installed with and somehow that affects seemingly unrelated transformations. Your last response and the mention of DITA-OT 3.4.1 (instead of 3.5.2) made me think, so I downloaded a "naked" 3.4.1, ran a html5-transformation and things turned out mostly fine.
However, the problem described here still persists and I wonder, whether it will be taken care of in the near future or not. Using
<topicref keyref="topic" href="path/topic.dita"/> kind of defeats the purpose of using keys in the first place it seems. Is there a way to "vote" on issues to make them more visible and push them up in priority for development?