I am facing the same problem now. My site has been having that problem for a long time, but I just noticed it today since I was working on other projects.
It seems clearly like a sh404sef bug for me, using SEF URLs that in fact are NOT DEFINED nowhere to show content instead of giving an SH404SEF.
It is not a problem of the underlying OS and has nothing to see with it: it's the sh404SEF who receives the URL and should manage it. It is pretty clear, but anyway I have my site on an Ubuntu Linux distribution with Apache, so It is also empirical tested.
You are right on that: IT IS a potential duplicate content issue, since even when you do not use those invalid links, one mistake in one backlink its enough for Google to find them. And I have that problem exactly with one of those URLs, since I have found in my site stats visits that use uppercase and others using lowercase.
Anyway my site is in an old J!1.5 -as this post is also old- and I am planning to migrate it to J!1.7, or may be even a custom CMS that I am using for another pages.
In the meantime I am thinking in an eventual workaround forcing all urls lowercase with mod_rewrite into .htaccess, so Apache mod_rewrite will take care of that before the silly sh404sef gets the URL.
It seems not to be very difficult:http://www.chrisabernethy.com/force-low ... d_rewrite/
Probably this is not useful for you so long after your post, but I leave it here just for the record. It may be useful for someone later, even for myself.