You're right; spectators need to be restricted to seeing non-neutral territories visible to both players. I'm glad you and others are pointing out the subtleties of implementing this idea, but if you think the overall concept is worthwhile then I'd still encourage you to vote for it :)