Core77

Why Design Contests Are Bad

By Don Norman - July 28, 2010

in Columns

Every year the world holds many contests for industrial designers. Lots of submissions, lots of time spent by jurors reviewing them, lots of pretty pictures afterwards. Fun to read, wonderful for the winners. What's the problem?

I have been a juror for a number of contests, including the major American yearly contest sponsored by the Industrial Design Society of America, IDSA, and BusinessWeek. Although I always enjoyed the experience and the interaction with talented, hard-working fellow jurors, I have become increasingly dissatisfied with the results.

Why are shows bad? Shouldn't we reward good design? Sure, if that's what the shows accomplish, but they don't. In fact, I believe they do harm to the profession. They reward the visible parts - styling - and ignore the most important, but hardest parts: interaction, experience, truly meeting needs, and even economic success. Oh sure, the rhetoric that accompanies the awards often heaps praise upon these other aspects of design, but that praise is not based upon solid evidence. No tests or studies, no independent evidence. As a result, the contests perpetuate the myth that industrial design is primarily about style and that brilliant styling leads to success in the marketplace. Both statements are false.

Jurors in design contests can only judge the material submitted to them. Invariably, the contest entries consist only of drawings, photographs, and videos accompanied by lush words of praise written by the design team, their company or client, or worse, their PR agency. Most of the time the actual products are not available for the jurors, nor can they be, when some products are tractors or cranes, air-conditioning units or automobiles - things far too large to transport to the jurors. As a result, jurors cannot experience them in use, they can't watch the intended audience use them, they can't assess how well they provide for graceful interaction, what pleasure or pain they provide, what benefits they provide. These problems result in fundamental limits to design contests.

I recently spoke with a founder of one of the largest and most successful design firms in the world who confessed that although his firm consistently wins multiple awards year after year, he is frequently puzzled by the choices. Entries he considered mediocre (from his own company) would at times win top prizes whereas entries he considered exemplary and superior would sometimes get passed over. What does that tell us of the contests?

Well-meaning, well-qualified judges are thwarted by the system. They are doomed to fail, for they lack the information required to make informed, intelligent choices. The only thing that can be judged is appearances. As a result, these shows perpetuate the myth that design is only about appearance. DESIGN CONTESTS JUDGE THE SUPERFICIAL Why does this happen? I do not fault the organizers or the judges: the fault lies with the complex logistics of running these contests. The organizers and judges are dedicated professionals, committed to rewarding the best aspects of design. They would prefer to consider the underlying design research, the effectiveness of the product, ergonomics, ease of use, and the ability to satisfy those who purchase and use it. Ideally, the award winning products will also be commercial successes, validating the awards through the marketplace. But in the end, jurors are unable to make the correct decisions because the structure of design contests is fundamentally incapable of providing the appropriate evidence. As a result, the judges must use the material that is provided them - images and self-serving congratulatory product descriptions, buttressed by whatever accidental experience some of the jurors might have had with the product category.

Design for the modern designer is about finding unmet needs, about the way people interact with and use the product or service, about economic and environmental sustainability, about providing utility and pleasure. Design can be applied to services and organizational structure, to financial systems and medical practice. It is not just style and appearance, although these play an important role.

But how can the jurors assess utility, understandability, sustainability, economic factors, or market viability? Style and form can be judged through photographs and videos, but economic and environmental sustainability cannot be photographed. Market success requires months or years to assess. Services, experiences, and utility could be photographed and recorded, but assessing the images would require considerable time and analysis, plus evidence that the tests were run by independent organizations using procedures that assured a lack of bias and adherence to standard testing protocols. Doing all of this would take time and money.

So, how could we make design contests better? We would have to change how they operated.

HOW COULD DESIGN CONTESTS IMPROVE? Contrast the way design contests are run with the methods used by Consumers Union, publisher of Consumers Reports, to assess the quality of products. Design contests assemble an ad-hoc team of designers, give them more material than they can readily assimilate in the brief time allowed for the judgments, do not provide the actual items under review, and never have time nor money to do the kinds of analyses that the awards require.

Consumers Union only reviews products that they have purchased themselves from stores, avoiding manufacturer's samples for fear that they might be non-representative of the items under sale. The items undergo exhaustive tests in the laboratories and with users. All of this takes considerable time and money.

Of course the goals of a design contest are different from that of Consumers Union. Design contests are intended to recognize creativity and ingenuity, to applaud design teams that moving the profession forward, pushing the envelope on form and function, environmental responsibility and efficiency, economic value and usability. The point of the design contest is to reward innovation and advancements, not to determine which of many products in a category is most worthwhile. As a result, although we can learn from Consumers Union we have no need to copy them.

I once proposed to the organizers of the yearly IDSA-BusinessWeek contest that they should prescreen all applications. Each product or service submission would have to be accompanied by an evaluation performed by an independent testing group that evaluates its functionality, usability, durability, and market impact. These would be somewhat in the spirit of the Consumers Union evaluations, although much less exhaustive and thorough, done by agencies certified as both qualified and independent by the contest organizers. The costs should be born by the submitters, but hopefully these could be reasonably low for each submission.

Jurors need to understand each product category if they are to asses the contribution of the entry in that category. Today, we hope that the knowledge of the assembled jurors will cover all categories, but in my experience this is seldom true. After all, the vary of categories covered by industrial design ranges from streetlights to train systems, from organizational structure to cooling systems for high-technology server farms. Fork lifts and tableware: the range is far too great even for a group of jurors.

Contests could prepare brief overviews of each product or service category being judged to help establish the state of the art. This review would be provided by the contest organizers, would provide a brief review of products in the category and the history of the area, thus letting judges unfamiliar with the category set appropriate expectations. Reviews of each product or service category would greatly enhance the panel's ability to judge the novelty of a submission, something that is difficult to do today when a single, relatively small panel must assess submissions from every conceivable field. These reviews could be reused year after year, usually with only simple updating, minimizing cost.

Finally, there is the problem of composition of the jury. In all the contests in which I have participated, the jurors, including the chief juror, the chair, change every year. Presumably this is done to allow for wider industry participation, but it precludes any learning or the benefits of history. I recommend rolling three-year terms, with 1/3 of the jurors appointed each year. Each jurors would have one year to learn the ropes, one year to practice what had been learned, and a final year both to practice and to mentor newer jurors. The chair should be appointed through a similar rule. One possibility is to appoint an experienced juror for a three-year term, the first as chair-in training, the second as chair, and the third, as past chair, acting as advisor and mentor to the current chair and chair in training.

One colleague suggested that perhaps we could make use of crowd-sourcing, the informed feedback of design professionals who could help assess the entries. This would require delaying the assessment of an entry until after it had been on the marketplace for long enough that there would be evidence to support the claims. The prizes would then reflect market success as well as informed expert opinion. There are obvious issues that would need to be addressed, but huge potential benefits as well.

SHOULD WE HAVE DESIGN COMPETITIONS? Yes, we should have design competitions. They help bring out the best in design. But today, they reward only a tiny part of modern design. The design profession prides itself on its ability to solve wicked, difficult problems. Well, here is a difficult problem crying out for a superior solution. Design professionals, it is time to design the reward structure for the design profession so that the designs that advance the field, that make a real difference in the world of products, services, and all the other problem areas in which modern designers work receive recognition. We should have contests, but we should do them properly.

o

Favorite This
Q
20

Comment

Comments

Sara Ludvigsson

13 years ago

Z

Z

Reply

You ask yourself "how could we make design contests better?". I would rather ask "why should design contests exist?".
In contests you always determine a winner, which also implies that there is a loser, or rather losers. How is that fact a good thing in any design discipline?
!Report as spam
RalfLippold

13 years ago

Z

Z

Reply

Excellent blog and inspiring article in particular!
!Report as spam
clippingimages

13 years ago

Z

Z

Reply

Great blog.Thanks for sharing.........
!Report as spam
Alpay Er

14 years ago

Z

Z

Reply

Well pointed and rather timely. I completely agree.
!Report as spam
Alex Debkalyuk

14 years ago

Z

Z

Reply

Hiring professionals to examine the competing design is exactly what investors would do... Right you are, judges should be in a position similar to that of an investor.
!Report as spam
Alexander Debkalyuk

14 years ago

Z

Z

Reply

Yep. They really are mere presentation design contests.
!Report as spam
David A. Parrott

14 years ago

Z

Z

Reply

Finally! Apply this point to the entire profession (not just design contests which are just a distillation of everything wrong with the profession) and someone is saying all the things about ID that need to be said.
!Report as spam
Aaron Szymanski

14 years ago

Z

Z

Reply

Enjoyable article. While the system could certainly be improved all of the designers I work with and most of our clients have already assigned a reasonably appropriate value to the major ID awards. Which is to say that they intuitively know that there are many ways to judge an object and most awards indicate success in a narrow selection of those ways.
!Report as spam
Gabriel Mathews

14 years ago

Z

Z

Reply

"Jurors need to understand each product category if they are to asses the contribution of the entry in that category." KABOOM!

!Report as spam
paul sandip

14 years ago

Z

Z

Reply

How would you judge design concepts?
!Report as spam
ary

14 years ago

Z

Z

Reply

Would really like to participate in a contest where you would have to really design something up to the point of having everything ready for production. That would be a smart arrangement between design schools, clients being able to choose from a wide spectrum of ideas meeting their growth needs, and students that acquire valuable experience and have the chance to actually make the world a better (or cooler) place.
!Report as spam
Steven

14 years ago

Z

Z

Reply

To be honest. I feel as if the design world is trying to become the art world. I don't see the ingenious designs in the functionality as I see an interesting idea with pretty packaging. I was at a portfolio review and watched someone get praised for bending a spoon. No matter how well researched, conceptualized, or modeled another project was; the thing that was ranted about was a BENT SPOON. It is sickening.
!Report as spam
-Joel-Kramka-

14 years ago

Z

Z

Reply

Really like this article as it highlights the many problems and inconsistencies of design contests. I think that the closest comparison is to judging a contemporary art show. You evaluate the piece based on looks and feeling alone. The problem is that there can be so many more variables that make a design important or useful. Maybe products should be evaluated after a certain period of time after the release however, so much has to do with concept and not actual production. I think that you've hit on the point that there needs to be more guidelines in order to make design contests more credible and useful. No system is perfect and it's nice to give designers an outlet for exposer bit I appreciate the acknowledgement of the imperfection by someone with your background. Great article, it's nice to see someone have the guts to critique the system of critiquing.
!Report as spam
Claudio J.

14 years ago

Z

Z

Reply

Thanks for this article. Nicely words for reflection
!Report as spam
Stephen Tiano

14 years ago

Z

Z

Reply

Any thoughts as to whether the whole idea of design contests is a bad idea: predatory and cynical, getting a crowd of people to work for free, while paying one, who, even according to this piece may not have done the best work?

I'm less concerned, actually, with whether or not any contest's winner deserves to win than the fact that a whole load of people would think, create, and work and then turn over all that effort to someone for free. Call me pisher, but as a book designer I'm quite tired of people offering me nothing but "exposure" for work I normally get paid thousands of dollars to do. Mighty big of them. And then such contest promoters still have access to all the losers' work for free.

Contests devalue the work of designers and make it less likely that neophytes starting out will find a climate that welcomes them as paid professionals without first giving work away.

And this is not to suggest that there aren't groups or associations, non-profits, genuinely worthy of pro bono assistance at times. In fact, I think straight out giving like that to worthy groups is far preferable than contributing enough free content to populate a paid-for issue of a sponsoring publication, say.
!Report as spam
Markus Wierzoch

14 years ago

Z

Z

Reply

Well... I guess all the above points can be applied to every award ever given by a profession to itself.
Watch Jerry Seinfeld's acceptance speech for the "HBO Comedian Awards" and replace every instance of the word "comedian" with "designer": www.youtube.com/watch?v=o_OqvUbBNA4
"The whole feeling in this room of reverence and honoring is the exact opposite of everything I have wanted my life to be about."
!Report as spam
Deepty Jain

14 years ago

Z

Z

Reply

I have also been a part in lot of design competitions ranging from architectural to product design and graphic design and also the way competition is held and designs are selected from jury to online voting systems. Firstly the competitions where judgments are made by the jury. Often the jury forgets the criteria/guidelines for designs defined by the competition. This leads to awarding the designs that do not adhere to the guidelines. At least this can be made sure by the committee. Secondly where judgment is given by the number of votes. This is true that people having maximum contacts will get maximum votes (is this fair?). Moreover, so many designs are submitted that no common man can go through them at once to give fair judgment. Probably here, the committee can screen the designs first before making them publicly available for voting.
!Report as spam
Danny

14 years ago

Z

Z

Reply

Amazing insight from the veritable Mr. Norman!
!Report as spam
ravi sawhney

14 years ago

Z

Z

Reply

Don, Good points, all of them. Winning awards is great when you get them, but as you've pointed out they are by no means a real test of a job well done. Perhaps its not the process of judging and giving out awards that is the problem. Maybe, its the hero worship that is asscoiated with them. We should be awarding designers and design but necessarily believeing in them as showing us the way. To the designer who has an ever incresing challenge of balancing the value of design, as many of us do, awards should should not result in either euphoria or disappointment. They are just another sign of simple answers to very complex questions. Somtimes simplicity wins, even when its off the mark. Just my two cents.
Ravi P.S., Check out my book Predictable Magic under the Pearson/Wharton label, it started shipping this week.
!Report as spam
Rob @ GruvDesign

14 years ago

Z

Z

Reply

Definitely some good points. I've been involved in some design contests, and there's another problem, even when you do have decent judges and juries. The sheer amount of data to be processed by each jury/judge, is quite large. This leads to "skimming". I've submitted designs to many contests, and sometimes I get back comments that are harshly critiquing my design, but that "issue" was already clearly stated in the first image, and the solution to that issue. It seems to be a case of judges just flying through the entrants, picking out what they feel is "clever or playful", not well-thought out, well-researched, well-designed, manufacturable solutions.

The other issue, is that a lot of contests are ditching the use of experienced, educated judges, and instead using "crowd sourcing". Unfortunately, this usually results in online voting, which usually picks the most outrageous, unrealistic, yet cool looking solution. Never mind manufacturing. Never mind materials. Never mind the real world. Never mind research. Just OOOOOO PRETTY, click! The simple fact is that the average person cannot accurately judge design contests, because they simply don't know enough about the process.

Another issue with the crowd sourcing/online voting aspect is that it becomes a popularity contest. As an entrant, you watch godawful designs get thousands of votes, due to school size, nationality, or other unknown factors, and you must fight back by literally whoring yourself out by writing to blogs, your friends, family, and coworkers to vote for your design.

MAYBE, just maybe, if online voting for a crowd soured idea required some sort of test in order to vote. A few quick questions about materials, human factors, etc, that made sure the people voting wasn't someone's grandmother and nursing home staff.

Anyway, nice article, and it definitely highlights a real problem with our industry.
!Report as spam