skip to content

i paid $190 for this

Calamity_Annie scored best in:  Visual Art

And scored worst in:  Innovation

Here is your anonymous judge feedback!  The judges were instructed to keep it as constructive as possible; hopefully this provides some insight into how best to improve your game.  If there’s anything in here you want to use on your website for promotional purposes, you are more than welcome to.  Just credit the quote as “2009 Independent Games Festival Judge”:

Great concept and wonderful artwork and overall atmosphere and feel. The gameplay, involving infrequent, quick, short motions of the hand, seems to get really taxing on me, though, which makes it difficult to get through an entire game.

the entire two sentences of feedback i received from the – i’ve been assured – at least four igf judges who decided whether calamity annie was worthy of inclusion in the 2009 independent games festival. (mighty jill off’s is about the same.) that’s four judges out of the pool of, what, forty judges the igf had this year? four judges randomly chosen from a pool of sixty who aren’t allowed to communicate, confer, or come to any kind of consensus on which games become finalists and which don’t.

i didn’t expect to be an igf finalist, but for a $95 entry fee per game, i don’t think it’s too much to expect the judges could take a few minutes to type up why exactly they decided my games weren’t fit for the festival.

really, this post is an apology to you, readers, because it was your generous donations that paid for these games to participate in this competition. or maybe this post is me externalizing my own desire for an apology.

69 comments

  1. madamluna wrote:

    That’s it, really? That’s horrible, and I doubt it even falls into the realm of “constructive criticism” in the first place.

    What really gets me is the fact that it scored worst in Innovation. The persistent storyline and use of the “shoot” verb even in non-combat situations are two of the more innovative, and more importantly well-implemented, gameplay mechanics I’ve seen in a while. :|

    3/15/2009 at 9:28 am | permalink
  2. mark wrote:

    -interesting collection. -cowboyana: probably the best in the set. best graphics, best sound, fun, cute, most polished. Bumpy collision is very annoying though.
    -punishing: INDEED. Way too slippery to be fun. Music is repetitive and grating, animations are okay but graphics are boring.
    -municipal abortionist: did not run. on purpose? (unzip process gave and error, and when trying to run that file I get “cannot exectute”)
    -flywrench: cool, and I like how unusual and explorational it is. nice loading graphic (it should animate though), music is annoying, graphics are interesting. Overall difficult, frustrating, visual/aural feedback is not good enough to direct players sometimes. It’s not obvious what you’re supposed to do and how you’re supposed to do it).

    3/15/2009 at 9:51 am | permalink
  3. Jason Love wrote:

    Wow. That’s outrageous. I mean, just for comparison, participation in IFcomp is free, and despite it being a much lower profile event, you wind up with a lot more feedback than this. It seems like there might be a better model for the recognition and promotion of independent game design.

    3/15/2009 at 9:57 am | permalink
  4. Jazmeister wrote:

    Wow. So they didn’t really notice the way you bypass the skill barrier with that persistant plot, and scored it low for requiring “infrequent, quick, short motions of the hand”.

    “Gameplay”, too. So, just fix the, you know, the “Gameplay”, and remove the need for quick hand motions, and your wild west quick-draw shooter will be good to go. How about making it turn based? You could gain xp instead of points, and the % chance to hit would go up every level. You could pick up loot from bodies, it’d be lots more original.

    How long do you think they played it for?

    3/15/2009 at 10:02 am | permalink
  5. daphny wrote:

    wow messhoff makes it look like only one person tried to play the game, considering out of four judges the one that review that got sent back was the judge who couldnt even open the game

    hurrrhrhhffffff

    3/15/2009 at 10:08 am | permalink
  6. What was your goal in entering the IGF? Mainstream legitimacy, financial reward, PR? Same question goes for Mark, too, I guess.

    3/15/2009 at 10:13 am | permalink
  7. Alex wrote:

    their criticism of your game is lazy but it is pretty much right

    calamity annie is a quick draw shooter except every so often there is a cut scene of a love story but its with TWO GIRLS wooooah who cares

    3/15/2009 at 10:18 am | permalink
  8. increpare wrote:

    I know someone who submitted last year who didn’t get any feedback at all. Ho humm.

    3/15/2009 at 10:42 am | permalink
  9. Paul Eres wrote:

    That would be me. Immortal Defense got no feedback at all. Waste of $95 :D

    3/15/2009 at 11:10 am | permalink
  10. Ben Sironko wrote:

    Those bastards! IGF can suck it!

    3/15/2009 at 12:35 pm | permalink
  11. Mike wrote:

    Wow. This is my first comment to your site, even though I’ve been a big fan of your games ever since I discovered Calamity Annie. I just can’t believe that a game as innovative as this got such a brush off.

    3/15/2009 at 12:39 pm | permalink
  12. Tim wrote:

    Well, considering that I spent US$300 trying to get a visitor visa to the States (and getting rejected twice), $190 is only two-thirds of that sum.

    3/15/2009 at 12:47 pm | permalink
  13. WNF wrote:

    I’m disappointed. For $95 you think that, especially with only 4 people playing such a casual game, they could come up with something beyond “please come again”.
    Innovation is not the issue, I’m not sure how many quick-shooters there are or how bad the reviewers’ arthritis is, but this entire thang is ridiculous.

    3/15/2009 at 12:53 pm | permalink
  14. Taking a look at the judges, it looks like it’s a broken system to only have four look at each submission. One game might get Blow, Purho, Gabler, and Yu, and another might get the Game Attorney, the guy who programmed the Conan MMO, the Kotaku guy, and someone whose bio is forthcoming. Each group of 4 could have a vastly different perspective than another one. It’s clear you got a foursome who wasn’t willing to look at the game with any depth whatsoever.

    But then, it’s the IGF. Fundamentally, it’s a contest, and I wouldn’t blow its importance out of proportion. Sure, it’s good exposure, but it’s totally not that big a deal.

    The reviewer(s?!) was right about one thing: it’s damn difficult to make it to the end. I had to cheat.

    3/15/2009 at 1:15 pm | permalink
  15. slut! wrote:

    how do you cheat

    3/15/2009 at 1:20 pm | permalink
  16. Decompiling the game and setting every round’s time limit to like 8 seconds.

    3/15/2009 at 1:27 pm | permalink
  17. Nayon wrote:

    To be honest, I did not like Calamity Annie, but this feedback sucks. And from what I seem to gather, a lot of things about the IGF are kind of wrong. What a shame. With all that money you paid for entering your game, you surely deserve more.

    3/15/2009 at 1:31 pm | permalink
  18. mark wrote:

    @Commodity Culture
    money

    3/15/2009 at 1:32 pm | permalink
  19. Zaratustra wrote:

    dess I’m submitting eversion to PAX you can come too

    3/15/2009 at 2:25 pm | permalink
  20. Strong wrote:

    the problem with judging games like this- especially independent games- is that different games are for different people, and if you have four judges that don’t like your genre, you’re pretty much screwed

    3/15/2009 at 3:10 pm | permalink
  21. Malefact wrote:

    UGH. That is frankly despicable. These folks must *know* how much thought, time and effort needs to be put in to make even a short game, let alone a game that leaves an impression, that makes you think, that has a consistent and moving freaking *atmosphere* compared to mediocrity of so much in the mainstream.

    Are these judges pressed for time? I find it impossible to believe that they couldn’t have said something about Calamity Annie, Immortal Defense, or Abortionist. For example, I didn’t like Abortionist (it gave me a headache!) but not even a braindead cretin could say that it was average or ordinary. Even a few lines from each judge would have been something.

    Preaching to the converted, here, but – this shows a lack of professionalism and of basic morality, because most independents don’t have $95 to spend willy-nilly. It also shows a fundamental lack of respect and concern for both the artist and the craft, because if you want independents to improve, you need to give them feedback.

    Shame on you, lame IGF judges. I am (literally) [url=http://www.mspaintadventures.com/?s=4&p=001733] penning an angry missive[/url] as we speak.

    3/15/2009 at 3:11 pm | permalink
  22. kat kent wrote:

    IM GOING TO PAX. Grab your girl, there are nerf fights in the hotel.

    3/15/2009 at 3:53 pm | permalink
  23. Hm. Simon, IGF Chairman here, closely following another 20 posts of people shouting at us.

    You know what? I don’t know what to say. It’s a massively difficult logistical job for us just to get all the 400+ entered games (across Main and Student competitions) working, correctly assigned to judges, and to get those judges to submit feedback on time, let alone forcing them to write about the games. We spend our nights and weekends doing it, and this year, we seem to have got nothing but hassle for it.

    As for the judges – they are busy professionals, in most cases, and we are sorry that they don’t have time to comment on every entry. We can’t make them.

    A couple of things:

    - We’re certainly in a position to make entering free, and I wonder if that would make people less upset that they don’t get detailed feedback.

    However, it’s probably going to give us in the low thousands of entries, instead of the low hundreds, because basically anyone who has made any type of game would enter at this point, and at that point the chances of good games getting lost in the shuffle is even greater. That’s a major worry.

    - We are certainly considering massively increasing the amount of judges. Again, this is am increased logistical nightmare in terms of juggling tech problems.

    In addition, this will come with a significantly increased chance that one of the judges will leak or Torrent the unreleased games (because we know them less well – like leaks of Oscar screeners that certainly occur.)

    I’m prepared to discuss all this in public, and have mature conversations with people. But what I’m getting fed up with is people complaining loudly and then not giving us constructive criticism on how to fix it (in one case this year, not this blog owner, despite saying they would.)

    In the end, like it or not, judging such a large amount of games is a really difficult problem.

    So how about it, Auntie? How _should_ we do it? With specific examples?

    3/15/2009 at 3:57 pm | permalink
  24. Zaratustra wrote:

    Constructive criticism is in scant supply for everyone, Mr. Carless. Just look at what dess got for -her- troubles.

    If the biggest problem in increasing the number of judges by an order of magnitude is tech issues or pirating, you’re clearly not setting the bar very high as to who can be a judge. How about this? For each game a judge plays, they have to write at least 200 words on it.

    3/15/2009 at 4:13 pm | permalink
  25. slut! wrote:

    on average how many games does each judge play

    3/15/2009 at 4:15 pm | permalink
  26. kirkjerk wrote:

    I don’t pay enough attention to the general Indy Game Culture.

    From reading this and the webpage, it does seem like the trouble is expectations of the judges. I see they have high-falutin’ profiles, but if they can’t be arsed to provide a reasonable amount of decent quality feedback, then they’re just prestige window dressing.

    3/15/2009 at 4:39 pm | permalink
  27. kat kent wrote:

    Hey Simon,

    You have 40 anonymous judges. If they are getting any compensation, even in free games, and there are four judges for each game, if they have to write two sentences about each game, that gives the designers eight sentences to go on.

    Lots of publications pay writers in free games, furnished by the game companies. Independent designers are giving the contest their free goods, you distribute it to the judges, the judges work for you.

    I think that designers will be willing to pay for a critique from their peers. The entry fee is perfectly acceptable, but there has to be accountability from the people that you have hired to judge to do their jobs just as professionally as the people who have paid money to enter your contest.

    3/15/2009 at 5:08 pm | permalink
  28. Malefact wrote:

    Simon,

    I think that when entrants pay a fee, there is going to be an expectation that the judges are going to give their work a full and considered evaluation.

    Without written feedback it’s going to be difficult for most to feel confident that this has taken place.

    I don’t think that the solution is to remove the fee (and, by implication, give the judges even less of a reason to act professionally) but to find ways of making sure that the judges do deliver this feedback. I don’t know what sort of incentives you offer them at present (or if you feel these are enough), but couldn’t you make a comment of a certain number of words mandatory as part of the feedback they give you (which could then be referred to the artists)?

    These comments could then be stored publicly, giving the judges the incentive of exposure but also the onus of having to have what they say critiqued by more than just a select few, perhaps prompting them to give more reasoned (and more useful) judgement.

    Having a more transparent and interactive judging system would not only raise the profile of the awards, but would make artists more willing to put their hand in their pocket for a first (or a second) time.

    At the very least some constructive criticism would also make them feel a darn sight better, which as Zaratustra said (or should that be spake?) you can obviously sympathise with.

    3/15/2009 at 5:46 pm | permalink
  29. Paul Eres wrote:

    Hi Simon, good points. Here’s some of my constructive criticisms relating to this particular problem and the IGF:

    1. I do believe the entry fee is too high. It shouldn’t be free, but $95 is quite a lot. Even a reduction to $50 would be great. Sure, a high entry fee keeps random Game Maker Sonic fangames out, but it also keeps incredibly great games out. Why wasn’t Iji entered into this IGF for instance? Or many of Cactus’s games, or so on? My impression is that it’s the expense. Losing $50 with nothing in return hurts less than losing $95 (or $190 in the case in question). $190 can pay for food for a month for a person. Alternatively, do away with an entry fee entirely and have games have to be nominated to even be in the contest (nominated by the judges perhaps). That way you can assure a high quality without the large expense.

    2. I don’t think more judges is necessarily a solution, I feel that having more dedicated judges is a better idea. Many judges set out to review as many games as they can, while others do the minimum possible. Just replace those who only do the minimum. It shouldn’t be a honorary position to grant to people you like, regardless of how good they are at it, it should be a merit-based thing that you can be “fired” from for doing a bad job. Of course, this suggestion is unlikely to be taken since it largely does seem to be a honorary position rather than something people are expected to be especially skilled at, but you did ask for constructive criticism. I also feel that there are too many games journalists and not enough game developers, and that game developers generally are better judges of games than game journalists. There’s a lot of people from the games journalism media. This has been improving in recent years as previous winners become judges, but it still seems a bit lopsided towards mainstream games journalists, who often have no connection to indie games and know nothing about them, and often don’t play indie games except for this contest.

    Hope this helps,

    3/15/2009 at 7:44 pm | permalink
  30. Good points, various folks – and a minimum of vitriol!

    Paul, I think reducing the fee somewhat – say, to $50 – is definitely doable. We have enough sponsors now that we can afford to do this, since the contest is not majority funded by the entrants.

    Regarding the make-up of the judges, to be clear, you can and do get retired from being an IGF judge if you don’t judge sufficiently.

    We don’t keep people on just for their name, and if you look from last year to this year, you’ll see a fair amount of movement (not all of it because people didn’t judge, incidentally.)

    Paul, we only started including journalists last year, so your comment about it being a multi-year issue is wrong, I think.

    So here’s what we can try, potentially:

    - If we reduce to price to $50 next year
    - We may get 300 entries in Main Competition, let’s say.
    - We could make X word commentaries mandatory for each game rating.
    - Let’s say each judge has to rate 15 games in the first round with those comments, on average (more than current judges, even without comments).
    - However, even with carefully picked judges, we still have a realistic completion rate of perhaps 70% of all items due to work pressure, compatibility issues, etc.
    - If we want to get to 10 ratings and comments per game on average, this would be a total of 3000 ratings.
    - So, including the dropout rate at 70%, we’d need 285 judges to do that. We have 60 judges right now (already doubled from previous years.)

    Not insurmountable, but fairly hardcore – nonetheless, food for thought.

    3/16/2009 at 12:02 am | permalink
  31. Mike wrote:

    I agree with most of the comments above. Keep the entry fee, but make sure each entrant gets a sufficient amount of constructive criticism back in return.

    Based on some of the comments I’ve seen it seems like the judges either A) didn’t spend enough time on the game B) were not ludistically literate enough to “read” the game properly. As such, if the judges don’t have enough time / the knowledge necessary to provide adequate feedback, don’t ask them back the next year. Simlarly, I don’t think there’s anything wrong with having journalists judge the games, but make sure at least have some knowledge of indy gaming as a genre. Inviting some prominent indy bloggers as judges might help, too.

    3/16/2009 at 12:14 am | permalink
  32. Mike wrote:

    Aw man, didn’t refresh the page before I submitted my comment, so I missed Careless’ reply.

    Anyway, I just wanted to add that I like the idea of the judge’s comments being stored publicly. That creates some accountability, so hopefully judges will be less likely to make a decision about a game without fully testing it first.

    Also, I don’t see why each game would need 10 ratings and comments. 5 judges looking at each game should be enough, provided they are able to really critique it.

    3/16/2009 at 1:24 am | permalink
  33. slut! wrote:

    maybe instead of an entry fee you could have some kind of nomination system, people can go to the site and vote for games they’d like to see in the igf. so the community would judge the preliminary round, and you wont get 3094829384239048290384 cave story clones

    then you probably wouldnt have to increase the number of judges, but i think the judges should be allowed to communicate with eachother about the games instead of being some super secret one man jury situation which ive been led to believe is what goes on

    3/16/2009 at 2:47 am | permalink
  34. Simon Carless wrote:

    Not true, delightful ‘Slut!’. We have a full back-end where judges can leave comments for each other about titles – though they tend to be technically related. Why so much cloak and dagger implications?

    I think nominations would turn into a bit of a ‘who’s got the largest community’ popularity contest, which we already have in the Audience Award in some form, and tends to be representative of, uh, who has the most organized and largest community. Which is very cool, but different to the best game.

    Also, I do believe that the judges spent time on each game – I know they did, they are responsible folks. In some cases, they didn’t comment, because that is not compulsory right now. There’s a certain amount of demonization going on here, I think, Internet stylee.

    Indie folks, some of you guys KNOW a bunch of the judges. They are your peers and previous winners. They are not numbskulls.

    3/16/2009 at 9:25 am | permalink
  35. WNF wrote:

    As much as I hate Auntie getting a bad rating over control issues (in a game I personally enjoyed for its controls), I agree mostly with Simon.
    With so many entries some slip through the cracks and it’s easy to understand but it doesn’t make it right and there should be a more even system of checks and balances. Maybe consider widening the amount of time judges have to play through games, or maybe include honorary judges, people who will be included in the randomization of what judges will judges what game but at a lower ratio because they have a heavier workload of their own. So about three professional judges and one volunteer judge, I’m thinking. Or maybe a system that incorporates judges based on their availability instead. They can specify how much time they can put into it and they’ll get that amount put on their laps.

    While people may want a detailed analysis of their game, hopefully around two sentences per judge, I think it would odd for a game like Aquaria to get the same output as a game you’d find in the La La Land series. Equality is dandy, but when there’s such a gap between the amount of effort and love put in between games, and that gap is overwhelming, you have to consider that no matter what you pay to get in, someone with a more expansive and technically proficient game will most likely get a deeper analysis.

    That being said I love Auntie and her games so the kind of shallow, uneducated feedback she got is not only costly but demeaning (even if that wasn’t the intent). The automated intro to the criticism is longer than the criticism itself. It makes me wonder how these people get positions as judges, even if they make the games I love as well. Artists don’t necessarily make good art critics.

    3/16/2009 at 4:04 pm | permalink
  36. Seems like if designers are interested in feedback, why is an entity like the IGF required? Natural opportunity for a self-organizing community of peers, as is so common on the net, to form and fulfill that need. I’m sure that’s already happening on gamedev forums everywhere.

    If designers are interested in exposure, then it sounds like a gamble (or investment). A gamble that for many of them will not pay off. A roulette wheel is silent… no one complains about the lack of betting critique.

    If designers are interested in winning a competition and would like to use the feedback for next year’s entry – is this realistic? Feedback seems so subjective and the odds of having the same judge next year would be small unless you are submitting a barrage of games.

    With respect to Calamity Annie, I mean I love the production, but it’s primarily an exploration of aesthetics rather than gameplay, right? And who knows your aesthetic better than yourself?

    3/16/2009 at 5:43 pm | permalink
  37. Paul Eres wrote:

    re Commodity Culture: The contest advertises detailed feedback as a feature of the contest, though. Perhaps it should downplay that expectancy, but the reason developers are interested in it is because they were led to expect it.

    re Simon: Unfortunately the lists of previous sets of judges no longer seem to be on the IGF site, but I do vividly remember it being fairly journalist-heavy and or at least mainstream-games-industry heavy for at least the last few years. But you are more involved in it than I am so you’re probably right there.

    And in the past it was justified because there wasn’t that extension of an indie games community back then. But today it’s a very large and diverse community, with its own culture, and judging indie games by the arbiters or even the creators of mainstream games no longer makes as much sense as it did in the past. It’d be as if the various independent film festivals were judged by people like steven spielburg. Indie games, like indie film, tends to be much more subtle and “artsy” than mainstream games or hollywood movies, and people who are good at judging the latter are not necessarily good at judging the former. I view someone like auntie pixelante (for instance) as being orders of magnitude better-equipped to judge the quality of an indie game than some kotaku journalist.

    So part of the problem for me with this contest was always the feeling that it was being judged by the standards of mainstream games, rather than the standards the indie games community sets for itself and strives for, and its winners tend to reflect that.

    3/16/2009 at 6:04 pm | permalink
  38. XUE wrote:

    “In some cases, they didn’t comment, because that is not compulsory right now.”

    wait, what?

    just make it compulsory.

    is it that difficult?

    3/16/2009 at 8:41 pm | permalink
  39. auntie wrote:

    what really concerns me is the lack of communication among judges. one of randy balma’s judges couldn’t get the game to run; with sixty judges, it ought to be negligible for that judge to find one who can actually run the game.

    what i’d like to see is an attempt at consensus between the judges: i’d like the judges to engage in a conversation about which games become finalists. i’d there to be a way for second opinions to check really high or low scores. or to foster any conversation beyond the number scores.

    i think that judges shirking their feedback duties is unacceptable, though. here’s an alternate proposition: have me as an igf judge next year.

    3/16/2009 at 11:35 pm | permalink
  40. Cycle wrote:

    Me too.

    3/17/2009 at 2:59 am | permalink
  41. kirkjerk wrote:

    Also maybe the judge communication suite could include visibility about how much feedback each game has received at this point, maybe sorted from least to most, with encouragement for active, interested judges to take some off the top of the list and see if they can’t provide more thoughts?

    3/17/2009 at 8:26 am | permalink
  42. th15 wrote:

    Simon:
    First of all I think you and the other organisers deserve our respect and thanks for making the IGF happen. No matter how much fuss anyone kicks up, the world would surely be a darker place without the IGF. To anyone who is sore to about the results, I say that on the larger scale, the IGF has helped indie games gain far more acceptance than before. Also, I would point out that there are perfectly legitimate alternatives to IGF now such as PAX10, 2BeeGames and Indiecade.

    But there is one specific point where I find the IGF has been downright unprofessional. For any developer to receive feedback that amounts to “cannot run game” is simply inexcusable. I feel that the right action for a judge to take when a game does not work is to inform the creator and request further instructions. If a judge is still unable to get a game to work, his rating should not be entered (this is what I hope the current procedure is). I believe that it is the responsibility of the IGF judges to inform the creator of a malfunctioning game and attempt to get the game to work rather than simply writing it off after trying to run the game once.

    3/17/2009 at 10:47 am | permalink
  43. Mike wrote:

    If the independent developers don’t feel like the judges’ decisions reflect indie values, why doesn’t the community create a new system of recognizing innovation and talent?

    This wouldn’t necessarily be in competition with the IGF, either. The IGF has always been good at finding the most “marketable” independent games and giving them that little extra push into mainstream consciousness. It definitely does its job well, too, as past success stories show.

    An awards system judged by the people posting here seems like it would have a slightly different slant to it. I wouldn’t be surprised if it were more receptive to the kind of subtlety and artistry that Eres is talking about.

    3/17/2009 at 11:14 am | permalink
  44. @TH15:

    First, thank you. I think people quickly lose sight of what a logistical challenge it is to run a festival like this, and are quick to pen harsh internet comments. Comments that we read, apparently :/.

    We work extremely hard – Simon especially – to push the festival from nebulous ether to awesome reality. FYI Simon spent an entire week of his personal vacation time playing through and judging all the games. ALL the games. Think about that before you snark it up :).

    Second, on your unprofessional comment: why do you assume that the things you suggest don’t occur? This year, I personally emailed 20+ individual game creators if I or any of the judges had trouble getting their games to run. We are on top of these issues. Judges have easy access to the creator’s email addresses and contact if they encounter problems with a game. In addition, the judging backend facilitates communication between judges about how to get games running. This is a known – and well solved – problem.

    In some instances, the creators may have given us incorrect or outdated contact information, or are otherwise unresponsive. Any game that doesn’t run, we email about. If we hear back, we work with the creators to get things running.

    3/17/2009 at 3:26 pm | permalink
  45. th15 wrote:

    I refer to Mark’s comment, #2 from top:
    “-municipal abortionist: did not run. on purpose? (unzip process gave and error, and when trying to run that file I get “cannot exectute”)”

    I assume that that’s Mark Essen, aka Messhof. Care to identify yourself Mark?

    3/17/2009 at 8:01 pm | permalink
  46. Tim wrote:

    I’ll just mention that I did message messhof (or Mark Essen to some) via IM about his game not running just a day or two after it was made available for the judges to play.

    3/17/2009 at 8:11 pm | permalink
  47. Mike wrote:

    Well based on Careless and Swink’s replies, it seems that the best way to fix this problem is to make comments mandatory. That way entrants would be able to see just how much thought and consideration went into judging each game.

    I’d also really like to caution the event’s organizers against simply labeling all these complaints as “Internet demonization”. Sure, people’s opinions about the judges’ effort might not reflect reality, but look at what we have to go on here. If the only comments we see seem to indicate a lack of time spent on the review, that’s what people will assume happened.

    3/17/2009 at 10:43 pm | permalink
  48. @ Mike:

    Good point, and well taken. Obviously, we listen to everything that is said with the preconception that it is constructive criticism and that it should be incorporated into future iterations of the festival. We will, as ever, taking an in-depth look at how things went this year and how to improve everything next year. That includes what we can do to expose things more so people have a better understanding of exactly what is happening. The public facing feedback can certainly be improved, and I think that mandatory comments is certainly possible.

    3/18/2009 at 3:30 pm | permalink
  49. snapman wrote:

    What line of reasoning lead to them announcing who got the lowest score in some area? That just strikes me as a very wrong-headed thing to do.

    “Congrats! YOU SUCK!”

    3/18/2009 at 4:39 pm | permalink
  50. mark wrote:

    yep, this is mark essen. tech problem’s my fault.

    3/18/2009 at 9:27 pm | permalink
  51. Jason Love wrote:

    It seems like the IGF’s evaluation process could only stand to benefit from better documentation and more transparency. That’s a difficult request, I’m sure, but if I’m going to pay for the privilege of receiving a rejection notice, I’d just as soon have as much information as I can get about what I can do to prevent that outcome. I’m not sure lowering the fee is going to help, since that just means you’ll have even more entrants to evaluate. As it stands, though, people are free to suggest your judges don’t know what’s important, and better feedback and documentation would help limit that.

    3/19/2009 at 9:33 am | permalink
  52. th15 wrote:

    Simon & IGF Friends:
    I apologize, it seems I’ve jumped to conclusions. I guess I just didn’t know any better.

    As Jason says, maybe the solution is to communicate what the judging process is like. Perhaps an article on GameSetWatch on how the IGF comes together? I think it’d be interesting to know how the IGF is brought together. Not specifically the judging process, but what it takes to bring the whole IGF together. If it seems a little odd to be writing about yourselves, perhaps someone could interview you guys about it?

    3/19/2009 at 9:55 am | permalink
  53. Very useful files search engine. indexoffiles.com is a search engine designed to search files in various file sharing and uploading sites.

    3/19/2009 at 10:53 am | permalink
  54. Tim wrote:

    Hey messhof, hope you are attending GDC this year. If you’re not, let us know or something alright.

    3/19/2009 at 5:56 pm | permalink
  55. I entered Harpooned in this year’s IGF. It wasn’t selected as a finalist.

    Here’s the feedback I got. I was surprised to get any feedback at all and I agree with the judges suggestions.

    —————–

    Harpooned scored best in: Overall
    And scored worst in: Technical

    Very effective use of classic videogame tropes to support a theme.

    It’s short but that is good; it’s just long enough to do what it set out to do.

    The only negative mark, craft-wise, is that the music is way too repetitive. In a known-duration game experience like this, the music should be calibrated to the length of the experience, or at the very least not actively detract from the rest of the game.

    This game is very tongue in cheek but at its core is largely a straight-forward shooter. It is a little over the top in the way that it tries to communicate its message. I do appreciate that it has a message to share though :)

    3/19/2009 at 9:38 pm | permalink
  56. mark wrote:

    i’ll be at gdc, i’m flying in on sunday.

    3/20/2009 at 6:18 pm | permalink
  57. daphny wrote:

    HEY MESSHOFF DITCH THAT ANNA JERK AND HANG OUT WITH ME

    LETS PARTY

    3/20/2009 at 8:00 pm | permalink
  58. daphny wrote:

    man are all the judges going to comment here ?that would be kind of sweet, and you might get your feedback

    3/21/2009 at 12:42 am | permalink
  59. daphny wrote:

    MESSHOFF. PARTY. WOO

    3/21/2009 at 12:42 am | permalink
  60. daphny wrote:

    god i am going to yell at everyone this week

    3/21/2009 at 12:43 am | permalink
  61. auntie wrote:

    me too.

    3/21/2009 at 1:04 am | permalink
  62. Matt wrote:

    Hey, wow, awesome conversation happen from this post. I had something to say when I came here, but I’ve forgotten it after reading all the comments.

    It’s a growing experience.

    3/21/2009 at 12:28 pm | permalink
  63. Teclo wrote:

    I think when the entrants are paying this sort of money, they should get a proper review of their work. In that case, it would actually encourage people that wouldn’t normally enter (due to costs or other reasons) to take part – not necessarily to “win” but for detailed feedback and critique from experienced individuals in the field.

    Even if they don’t do a big professional-level review, if I were to enter a game, I’d personally enjoy a video review. The judge would record themselves playing the game and comment as they play it. This would potentially be more valuable feedback than a conventional review because it would be more natural and more honest – essentially, the thoughts expressed would be “live” and more likely to reflect the thoughts of the average person as they play the game. Obviously this becomes less and less practical the longer the game is.

    It’s hard for me to read the Calamity Annie “critique” as anything other than “it’s really good but I suck at it”, but then that’s how a lot of reviews sound nowadays (and I know I sound like an old bastard when I say that).

    3/25/2009 at 1:48 am | permalink
  64. tim wrote:

    Teclo: someone send me some recording equipment, because I play nearly all entries every year.

    I’m serious.

    3/27/2009 at 7:03 pm | permalink
  65. Hunter wrote:

    Hahaha, welcome to the IGF! That’s almost verbatim the feedback I got on the game I submitted three years ago.

    I said it before in anticipation of this “feedback”, and now that you’ve received the feedback, I’ll repeat it:

    FUCK THE IGF. (to the tune of “Fuck the Police”)

    I still say we should start our own game festival. With hookers, and blackjack.

    3/30/2009 at 4:47 pm | permalink
  66. auntie wrote:

    at the game over show, phil fish was pretty excited about indiecade (he is apparently on the judges panel).

    3/30/2009 at 5:36 pm | permalink
  67. @STEVE: I don’t like the “logistics for this are hard” argument. If they were so hard that you couldn’t do them properly, you should return the money to Auntie and to every other person who complains, since your own organizational problems put them in this situation.
    Also, if it’s so hard you can’t do it properly, maybe you shouldn’t do it at all. Game programming is also quite hard and so are many, many other things that are worth doing. Why didn’t you say anything about how difficult this was *before* starting the competition? That would have changed things for the participants. and I’m sure many wouldn’t have participated.

    4/9/2009 at 12:18 pm | permalink
  68. @STEVE: I’m thinking about your comments again, and they seem especially out of touch with reality: imagine what my clients would say if I told them I couldn’t finish an AI program or computer vision application because “they were too hard, as you know.” Would I still get paid? Of course not!

    4/9/2009 at 12:23 pm | permalink
  69. Matthew wrote:

    @Andres Kievsky:

    Let’s take your computer vision example. Compromises must be made in the face of complexity. Maybe your program can detect the gender of a person, but only from the front. That places limits on the use case (photos from behind must be discarded, for instance).

    In the abstract, the IGF also loses functionality as a means of logistical optimization. I don’t think Steve is simply whining that the logistics are insurmountable. However, I *do* think logistics are a valid defense against why the IGF isn’t “perfect” and why the IGF cannot absolutely satisfy all parties.

    Your point makes a very big assumption, too–that the IGF organizers tackle the IGF full-time. This is not the case.

    Frankly, and to your comment specifically, negative-only messages don’t accomplish much. Sure, the IGF can–and should, and will–improve, but there is no reason why you cannot frame your comment as “thank you for all that you have done, and here are some things I care about that I would like to see improved”, versus your slightly wordier “FU”.

    4/13/2009 at 10:25 pm | permalink

2 trackbacks/pingbacks

  1. [...] Auntie Pixelante vs. the IGF – via Auntie Pixelante [...]

  2. [...] Carless defends the IGF – via Auntie Pixelante [...]

post a comment

your email is never published nor shared. required fields are marked *
*
*