It is but you're mostly thinking of white asbestos, not blue or brown. If you go read about the history of asbestos and links to cancer you will find that while white asbestos isn't good, it's much much less likely to kill you as quickly as brown or blue.

Have a read about Wittenoom in Western Australia. It was a town on the outskirts of a blue asbestos mine, it's absolutely tragic what went on there, kids would be playing in mounds of blue asbestos, many of them tragically died.

[1] https://en.wikipedia.org/wiki/Wittenoom,_Western_Australia