I think the distinction is that cryptography is much closer to thought-crime than to any physical crime. It's on the continuum of banning speaking in private. There are a lot of ways to slice it, but it boils down to privacy being illegal per se. It would be unconscionable by the majority if the implications were clearly understood, but because it's "technology" somehow the fearmongers are able to spin it (as they do so many things) as a national security issue.