GPG is terrible software and gpg-agent pretending to be ssh-agent is very unreliable. There's also a ridiculous amount of work required to make Yubikey+GPG+SSH work.
I suppose, given I have GPG agent working for GPG stuff, it wasn't that much of a stretch to get it working for SSH too. The biggest pain was getting gnome-keyring-daemon to butt out :D