Good news is, things don’t sounds as terrible to me today, but I can still hear the issues. 
Haven’t really analyzed it all deeply, but (unsurprisingly) where it’s most blatantly obvious is with strings. What I’m hearing is similar to these speech “synthesis” solutions that simply paste full recorded words together into sentences. The individual words are (obviously) perfect and realistic, but you can tell something is off, as the flow through the sentences is not what you’d have if it was actually a person speaking whole sentences. My theory is that this is not strictly a speech thing, but will happen with instruments and other sound sources as well, once you become deeply familiar with them. For example, some experienced race drivers can recognize which track a car is on just by listening to the engine sound.
There are also (at least) two levels to this; physical limitations, and musical expression.
Violation of physical limitations is a fairly obvious case. This would be when a virtual instrument does things that are just not possible on the real instrument; playing out of range, vibrato or slides on instruments that can’t do these things; stuff like that. Gray zone: Things that are possible, but ridiculously awkward to do - things you would only do if you absolutely want that specific effect for expression.
Musical expression violations would be playing in ways that are perfectly possible on the real instruments, but just don’t sound correct and appropriate in context. This would correspond to incorrect pronunciation or emphasis in speech or singing. It’s not unrealistic, but just wrong, unless possibly when done intentionally for good reasons. (For example the final “Vinceeeeerò!” in Nessun Dorma - which is not like that in the original version BTW - or indeed, the title phrase itself. Backwards emphasis in both cases, AFAIK, but that’s how it is with lyrics vs music sometimes.)
My main and most specific beef at this point is probably with vibrato on strings, and it mostly falls in the expression category. The root of the problem is that you have virtually no control over vibrato in sample libraries, whereas on the real instruments, is practically defines the player’s personal sound and expression. In more practical terms, you only play continuous wide vibrato, or complete non-vib, when specifically intended, you don’t abruptly flip between vib and non-vib, and you certainly don’t don’t start vibrating at any random time in the middle of a note. Getting it wrong is kind of like singing a diphthong like “I” as “aiiiiiiii” or “aaaaaaai”, or drastically flipping from the “a” to the “i” at some random point - it sounds hilarious and robotic!
(I’ll point out some specific examples later. Need to get back to work now. 
)
