Hi,
This may be caused by the "Parse Joomla SEO links" option being enabled in JoomSEF's Configuration on the Advanced tab.
When it's enabled, JoomSEF tries to parse non-existent URLs by standard Joomla SEF system first, so it is possible that Joomla parses it partially and JoomSEF then thinks that the URL exists and let Joomla display its page.
You can try disabling the option to see if it helps. However please note that disabling the option prevents Component's router and Default Joomla router handlers from working correctly for JoomSEF extensions (you can check if you use them on Manage Extensions page in JoomSEF).