Contest Judging

General discussion about Super Mario Bros. X.
Aero
Administrator
Posts: 4696
Joined: Fri Mar 28, 2014 2:51 pm

Contest Judging

Postby Aero » Thu Jun 08, 2017 2:41 am

If you haven't read the past few pages in the Extravaganza thread, please read them here as a background for this thread. Basically what I don't understand is why community contests are bottle-necked by a host and a few judges. The thing is, most of the people likely to be judges - or in any position of oversight on anything around here - are working and college aged. It makes sense for them to prioritize school and work over a Mario game; it's the responsible thing to do. On the other hand, they are aware of their obligations when signing up to judge so it's not like there is no room for questioning why things aren't as speedy. On top of that though is a few of these rituals the community has with contests:
  • Submitting levels in secrecy.
  • Having judges is a given.
  • Having hosts are a given.
  • Reviews are given metrics with wordy analyses.
  • There must be a winner of a contest.
This is a Mario game, and arguments arise more regularly than they should. Games are meant to be fun. Why does everything have to be processed and treated like it's some big deal? Honestly, I've defended not having level judges extensively and after their removal the levels forum didn't die and in some cases is a bit more active. If things were changed and we just let everyone submit levels openly and just organized them in the contest episode in an equal way so that players: can play the levels they enjoy, in the order they want, for as long as they want, and then leave it at that then there's no fuss. Reviews aren't useless (except the metrics) and they can be good feedback, but the progress of contests shouldn't be binded to the speed of a few individual users. Simply make a level, submit it, put it in a hub arranged in alphabetical/submission order, and let anyone who wants to tack on a review do so.

Thoughts?

Enjl
Cute Yoshi Egg
Cute Yoshi Egg
Posts: 8878
Joined: Mon Jan 20, 2014 12:58 pm
Flair: Orphion Egamalenitar Osmos IV, Esq.

Re: Contest Judging

Postby Enjl » Thu Jun 08, 2017 7:52 am

Completely removes the "contest" part of a contest but in general I agree that in terms of sorting the levels into an episode (which in itself is more of a ritual than having judges for a competition), SMWCentral has a much better system in place than us. The VLDCs sort their levels into a world map separated by theme, with a special world each for the top and bottom levels respectively. Every level is always accessible and focus lies on the levels, not an arbitrary plot about Elsa taking over the ice castle.

Though I can't say I'm surprised by how little progress has been made in terms of evolving the contest formula here. It perfectly lines up with the stagnant view on everything else.
Ideas are useless if you can't make them real.
Image

All my assets from packs and episodes are free to use for non-Novaverse levels and projects, as long as proper credit is given.

User avatar
Snessy the duck
Ludwig von Koopa
Ludwig von Koopa
Posts: 2699
Joined: Sat Sep 20, 2014 6:04 am

Re: Contest Judging

Postby Snessy the duck » Thu Jun 08, 2017 11:28 am

Enjl wrote:Completely removes the "contest" part of a contest but in general I agree that in terms of sorting the levels into an episode (which in itself is more of a ritual than having judges for a competition), SMWCentral has a much better system in place than us. The VLDCs sort their levels into a world map separated by theme, with a special world each for the top and bottom levels respectively. Every level is always accessible and focus lies on the levels, not an arbitrary plot about Elsa taking over the ice castle.
Yeah, I belive that contest episodes should have all levels avaliable from the start, so you don't have to play all the shit levels and can go straight to the really good stuff. I also feel like the ToB series tries too hard to make an actual game out of all the levels by having arbitrary things like boss battles and a story. I hope the next contest episodes are gonna be more like the VLDC hacks.
Image

My Switch friend code: SW-5399-6466-3493

Image

Inspirited
Pokey
Pokey
Posts: 242
Joined: Sun Oct 30, 2016 5:58 pm

Re: Contest Judging

Postby Inspirited » Thu Jun 08, 2017 12:01 pm

Some pros and cons I can think of for having judges:

Pro
- Using their scoring we have a contest
- Everyone who submitted is guaranteed to get opinions/criticism on their level

Con
- More effort
- Takes more time
Check out my levels!

User avatar
Electriking
Posts: 4533
Joined: Mon Jun 06, 2016 4:32 pm
Flair: I have NO idea what to put here
Contact:

Re: Contest Judging

Postby Electriking » Thu Jun 08, 2017 12:07 pm

At least the shitlevels make good YouTube videos. And the joke levels actually kind of give contest episodes a strong start.
SMM2 ID: JJF-P7G-FSF

Aero
Administrator
Posts: 4696
Joined: Fri Mar 28, 2014 2:51 pm

Re: Contest Judging

Postby Aero » Thu Jun 08, 2017 12:45 pm

Inspirited wrote:Some pros and cons I can think of for having judges:

Pro
- Using their scoring we have a contest
- Everyone who submitted is guaranteed to get opinions/criticism on their level

Con
- More effort
- Takes more time
You don't necessarily need judges for a contest. Instead of having some ranking system based on the opinions of a few users, you could have a category system like "Best Design," "Best Aesthetic," "Most Innovative," or "Most Replayable," and just have votes for each category that the community can participate in whilst they write reviews. Saves a lot of time still, retain effort from those who actually have something to say about a level rather than it being obligatory, and it keeps a contest model. What Enjl said about VLDCs above is also entirely accurate and is another way of doing things. The ritual is what I really don't understand entirely when the only variability in even user contests is just level themes/graphics and pretty much never structure.

User avatar
Mivixion
Grinder
Grinder
Posts: 1052
Joined: Sat Dec 20, 2014 2:44 pm

Re: Contest Judging

Postby Mivixion » Thu Jun 08, 2017 1:50 pm

I think you'll be pleasantly surprised when it comes to how GEx's results system works.

Inspirited
Pokey
Pokey
Posts: 242
Joined: Sun Oct 30, 2016 5:58 pm

Re: Contest Judging

Postby Inspirited » Thu Jun 08, 2017 4:52 pm

Community voting for a contest seems like an idea that people dont agree with on this site, but I'd like it. Just for the sake of having that more casual feeling.
Check out my levels!

Enjl
Cute Yoshi Egg
Cute Yoshi Egg
Posts: 8878
Joined: Mon Jan 20, 2014 12:58 pm
Flair: Orphion Egamalenitar Osmos IV, Esq.

Re: Contest Judging

Postby Enjl » Thu Jun 08, 2017 5:25 pm

The problem is always
-not knowing if they're bribed by a friend
-not knowing if they have any idea on what they're talking about
-not knowing if they actually played a level

With a small contest judge roster all of this is generally established and it's much harder for one to sneak past one of these, and once one of these unspoken rules is broken by a judge it's not a bad idea to invalidate their part in voting entirely.
Alas, rule 2 of these is hard to properly follow in this community in general. We should steal some judges from our affiliates sometime to get a roster that doesn't make people go "oh what the fuck why this guy he's so bad".
Ideas are useless if you can't make them real.
Image

All my assets from packs and episodes are free to use for non-Novaverse levels and projects, as long as proper credit is given.

Aero
Administrator
Posts: 4696
Joined: Fri Mar 28, 2014 2:51 pm

Re: Contest Judging

Postby Aero » Thu Jun 08, 2017 10:40 pm

Well you have to look at the consequences of each of those and design a contest around them, and it's entirely possible to do this without judges or even a small roster. When it comes to bribery, this is riskier with a scoring system and a small selection of judges. If a judge's opinion is influenced by the level designer being a friend with some bonus if the level is scored higher than it would, there's not really anything that can be proven with regular contests unless someone is careless and it leaks out. Leaving it to a vote for winning a certain category makes it much harder, because the more bribery there is for votes the harder it is to keep under the radar when it takes just one in a large group to blow the whole thing. When it comes to people not knowing what they're talking about, I agree it's hard to follow, but minimizing the weight a review has down to only advice for the level designer and not the justification for a score would leave it to level designers to discern if the reviewer knows what they're talking about which has always been the case for level reviews in general. Finally, when it comes to knowing if people actually played a level this can be handled with screenshot/video requirements that can not only give a visual proof that the level was played, but can also be used for demonstrations in reviews which can really help the designer. It'd be pretty dumb for someone to fake a screenshot of themself playing a level because they might as well actually play it if they downloaded it, loaded it up, and took the time to write a review that seemed legit.

There's a diversity of things that can be tried, and they should be explored. Whatever makes breaking those rules pointless (or high risk low reward) is a good place to start when designing a contest, and someone can probably articulate much better examples than I can to make things more open to the community while satisfying those rules without compromising timeliness. Our affiliates ought to be stolen and maybe they can bring some of their ideas for contests to integrate into our own some more too.

Knux
Nipper Plant
Nipper Plant
Posts: 738
Joined: Sat Nov 22, 2014 10:55 pm

Re: Contest Judging

Postby Knux » Thu Jun 08, 2017 10:53 pm

I like SMBX.

User avatar
Snessy the duck
Ludwig von Koopa
Ludwig von Koopa
Posts: 2699
Joined: Sat Sep 20, 2014 6:04 am

Re: Contest Judging

Postby Snessy the duck » Fri Jun 09, 2017 10:36 am

Knux wrote:I like SMBX.
Me too.

Also, I find the concept of backup judges pretty stupid. Backup judges are different than the main judges, so the judges could get an unfair advantage or disadvantage with their level depending on the judge. Having the judges not submit any levels would be a better idea to me (And probably others).
Image

My Switch friend code: SW-5399-6466-3493

Image

TDK
Kamikaze Koopa
Kamikaze Koopa
Posts: 1384
Joined: Wed Nov 11, 2015 12:26 pm

Re: Contest Judging

Postby TDK » Fri Jun 09, 2017 10:38 am

Snessy the duck wrote:
Knux wrote:I like SMBX.
Me too.

Also, I find the concept of backup judges pretty stupid. Backup judges are different than the main judges, so the judges could get an unfair advantage or disadvantage with their level depending on the judge. Having the judges not submit any levels would be a better idea to me (And probably others).
That's why there isn't any backup judges in the grand extravaganza.

User avatar
Electriking
Posts: 4533
Joined: Mon Jun 06, 2016 4:32 pm
Flair: I have NO idea what to put here
Contact:

Re: Contest Judging

Postby Electriking » Fri Jun 09, 2017 6:51 pm

Aero wrote:Well you have to look at the consequences of each of those and design a contest around them, and it's entirely possible to do this without judges or even a small roster. When it comes to bribery, this is riskier with a scoring system and a small selection of judges. If a judge's opinion is influenced by the level designer being a friend with some bonus if the level is scored higher than it would, there's not really anything that can be proven with regular contests unless someone is careless and it leaks out. Leaving it to a vote for winning a certain category makes it much harder, because the more bribery there is for votes the harder it is to keep under the radar when it takes just one in a large group to blow the whole thing. When it comes to people not knowing what they're talking about, I agree it's hard to follow, but minimizing the weight a review has down to only advice for the level designer and not the justification for a score would leave it to level designers to discern if the reviewer knows what they're talking about which has always been the case for level reviews in general. Finally, when it comes to knowing if people actually played a level this can be handled with screenshot/video requirements that can not only give a visual proof that the level was played, but can also be used for demonstrations in reviews which can really help the designer. It'd be pretty dumb for someone to fake a screenshot of themself playing a level because they might as well actually play it if they downloaded it, loaded it up, and took the time to write a review that seemed legit.
1) Remember that levels normally are annomynous, so really that kind of prevents this (unless the style clearly identifies the creator).
2) Votes would probably not be a good idea for contests since it is not possible to check for private leaks
3) Are you suggesting that contests don't have scores? Because if that is the case than whats the point? You may as well just have a level forum and not do contests at all. Contests are supposed to determine who is the best level designer(s), not to upload a level and see what you need to improve on (though you can use contests for that).
4) I don't agree that having video requirements in order to prove you have played a level works. It only bars those who don't have accounts on YouTube or another video uploading site from being a judge. Like what are you achieving by asking for proof here? A good host could judge if a judge has played a level if their reviews on gameplay are pretty shit, since in order to properly judge this aspect (which is more important than any of the others combined BTW) then you have to play the level to know it to the fullest possible extent. If you force one to upload a video, you only add an unneccesary barrier to and remove the opportunity for those who can't to be a contest judge.
SMM2 ID: JJF-P7G-FSF


Return to “General”

Who is online

Users browsing this forum: No registered users and 3 guests