If you are saying, "well, sure, they have the LEGAL right to do it, but by putting it there, they're just being provocative," then you may not be particularly evil yourself, but you're intimidated by the evil people into being one of their henchmen. People doing completely normal things like building community centers aren't being provocative. The people who you are afraid they will provoke? Those are the bad guys. And if you are saying that they should modify their behaviors to do what the bad guys want, then you are saying that you support the bad guys.
And that means that you're one of the bad guys.
And THAT means that I find you a danger to me, personally. I'm Jewish. Right now, Jews are mostly considered more or less white. But that could change. And the second it does, people will start saying things like, "Well, sure, they COULD build a synagogue there, but wouldn't that just be asking for trouble?"
Everything that people are saying about the downtown Manhattan community center, I just automatically hear people saying that about something that I might want to do someday. If you're against them, then you're against me, and you're also against justice, freedom, and every ideal that this country stands for.
Just so we're clear on that.