They can, at least in California: as a pedestrian, you're entitled to cross the road at any intersection (crosswalk lights must be followed, but in their absence you're free to cross at any time). By law, all drivers must come to a complete stop until you have reached the curb.
You will need to explain how the fact that pedestrians are even less threatening and more vulnerable is the same as implying this gives them the right to walk on the middle of the road.
I suspect that pedestrians are less threatening because they rarely walk in traffic (both because it's illegal, and because it's obviously dangerous). But for bicyclists, it's legal, and apparently the danger isn't enough to stop a lot of people.
The way you describe it, my impression is that the real problem is that you are not considering bikes as participants of traffic, but obstacles in the way of your car.
I'm not considering automobiles as inherently more "valid" participants of traffic than bicycles. I just consider any road where two types of vehicles of very different sizes which behave very differently to be an inherently unsafe and inefficient road for all participants. Obviously, the occupants of the larger vehicles are much safer in collisions with the smaller vehicles, but that's not my primary consideration. I just think the roads, or at the very least the lanes, should be dedicated to only bikes or only automobiles. Since there are far more automobiles than bicycles on most roads, if building new infrastructure is out of the question, I think it makes sense to not allow bikes to share the lanes. This isn't a condemnation of bicycle commuting, but rather a basic appeal to safety and organization, just like pedestrians are not allowed to share lanes with automobiles. The current situation on most roads, where a small number of small slow-moving bikes occupy lanes in the middle of a sea of automobiles, is just a disaster waiting to happen (and in big cities, it happens all the time).
I'm not considering big rig trucks as inherently more "valid" participants of traffic than passenger cars. I just consider any road where two types of vehicles of very different sizes which behave very differently to be an inherently unsafe and inefficient road for all participants. Obviously, the occupants of the larger vehicles are much safer in collisions with the smaller vehicles, but that's not my primary consideration...
