On the other hand, you want to be able to throw out people who are a nuisance and it is quite difficult to codify what that constitutes.
The simple approach is to simply allow yourself to throw anyone out who annoys anyone and be the judge of when that annoyance is strong enough to enforce that.
I would go with that - the organizer can throw out whoever they want (even without reason). I think society already has a code of conduct. If conferences feel the need to add an extra one, it shows that they tend to think the worst about people, and they have been overtaken by SJW framing.
The simple approach is to simply allow yourself to throw anyone out who annoys anyone and be the judge of when that annoyance is strong enough to enforce that.