I was also considering having the scoring part not by selected Judges anymore, but by members of USP, like a poll voting system you know? Of course this might cause issues with like some people not playing all the map packs, and people voting for themselves, and then it would be unfair so to speak...
You've raised an interesting point here.
Concerning Architecture, Texturing, Lighting, Sound and Technical Execution I think judging shoudl be left over to people who have knowledge op map building.
But in regards to Conceptual Grandness, Story Construction, Story Implementation, Gameplay Awe and Gameplay Balance I am certain non-mappers are just as competent and may even be less subject to being influenced by the more technical imperfections (or perfections) than the Hellscrags, Mr. Prophets, Waffnufflys and other UBs (this is not a statement insinuating they are not objective, it's just a fact of life that they have more knowledge which could be of influence).
The gameplay balance does present a problem. Is 1 difficulty setting (Medium or Hard) going to be the
basic criterium, or is the implementaion of difficulty settings equally important. IMHO a map should be interesting for everyone which means that judging on 1 difficulty won't work. But this implies that the judges should play each map on all difficulty settings. I for one like to do that but I can imagine not everybody has the time nor the interest (UB doesn't strike me as a player who will embark upon the testing on Easy in the same way as I would, similarly on Unreal difficulty I might not have the gaming ability to judge that difficulty setting properly, whereas UB thrives on Unreal difficulty).
So, I am in favour of members judging, but not for all criteria (or perhaps with a lower weighting factor), and not by all members. Judging should only be allowed for members who are committed to playing all maps and on all designated difficulties. This would mean that the judges require sufficient time (which could turn out to be quite long).