Maybe requiring a comment to downvote? Like you click the downvote and instead of it actually downvoting, it attaches the downvote to the comment box and requires you to leave a reply justifying your downvote.