Also consider that if you're a musician, people are less likely to say that they dislike music to your face, out of politeness.
I don't think it's so much cultural pressure as it is unusual. It's like meeting somebody who doesn't like art. It is so common and there are so many kinds out there that categorically disliking it entirely is strange. Most people probably do not want to be seen as strange.
It's also possible that in Western culture, not liking music is traditionally associated with something sinister. For instance, in Julius Caesar, Cassius is described as follows
"...He reads much,
He is a great observer, and he looks
Quite through the deeds of men. He loves no plays,
As thou dost, Antony; he hears no music.
Seldom he smiles, and smiles in such a sort
As if he mock'd himself, and scorn'd his spirit
That could be mov'd to smile at anything.
Such men as he be never at heart's ease
While they behold a greater than themselves,
And therefore are they very dangerous."
which contrasts with the more sympathetic character, Brutus, and his love of music.
There's a more explicitly anti-tone-deaf example in The Merchant of Venice: "The man that hath no music in himself, Nor is not moved with concord of sweet sounds, Is fit for treasons, stratagems and spoils; The motions of his spirit are dull as night And his affections dark as Erebus: Let no such man be trusted."
I don't think it's so much cultural pressure as it is unusual. It's like meeting somebody who doesn't like art. It is so common and there are so many kinds out there that categorically disliking it entirely is strange. Most people probably do not want to be seen as strange.
It's also possible that in Western culture, not liking music is traditionally associated with something sinister. For instance, in Julius Caesar, Cassius is described as follows
which contrasts with the more sympathetic character, Brutus, and his love of music.