How Well Can People Identify Their Most Creative Ideas?

Reference: Silvia, P. J. (2008). Discernment and creativity: How well can people identify their most creative ideas?. Psychology of Aesthetics, Creativity, and the Arts,2(3), 139.

Abstract: How well can people judge the creativity of their ideas? The distinction between generating ideas and evaluating ideas appears in many theories of creativity, but the massive literature on generation has overshadowed the question of evaluation. After critically reviewing the notion of accuracy in creativity judgments, this article explores whether (1) people in general are discerning and (2) whether some people are more discerning than others. University students (n 226) completed four divergent thinking tasks and then decided which responses were their most creative. Judges then rated the creativity of all of the responses. Multilevel latent-variable models found that people’s choices strongly agreed with judges’ ratings of the responses; overall, people were discerning in their decisions. But some people were more discerning than others: people high in openness to experience, in particular, had stronger agreement between their decisions and the judges’ ratings. Creative people are thus doubly skilled: they are better at generating good ideas and at picking their best ideas.

Notes: Sometimes I choose to add papers here not so much because of their intellectual value, but for the clarity with which they illustrate a whole research tradition widely accepted in some community. One could talk about the many weak assumptions behind studies like this, the biases and limitations of their methods and their analysis. But here I instead just focus on the one-paragraph conclusion in this paper, which reads:

"The prevalence of tired, rewarmed ideas could make a cynic conclude that people are bad at separating their creative ideas from their unoriginal ideas. This study explored how well people can pick their best ideas. People could choose their best ideas better than chance: their choices covaried significantly with judges’ creativity ratings. But some people—such as people high in openness to experience—were more discerning. Creative people are thus doubly skilled: they are better at generating creative ideas and at discerning which ones are the best."

Wow! the wording of these claims is "worth a thousand words", particularly based on what the researchers actually did:

1. Studied 226 undergrad students of psychology at a specific US university, a large majority female
2. Asked them to respond to four divergent thinking tests
3. Each test lasts for 3 minutes, that's right, 180 seconds
4. All tasks are based on words (linguistic divergence)

So, just as an exercise, let's re-write that final paragraph to be more realistic and see how it looks:

"We don't know the processes by which young, American female students of psychology assess their responses to linguistic-based divergent tests. We also don't whether their assessment matches that of three undergraduate research assistants who we gave instructions to assess responses to such type of tests, and which we call "judges". This study explored how these students pick their best ideas when instructed to "circle the two responses that they thought were their most creative ones", without any definition of what we meant. These students chose their best ideas better than chance: their choices covaried significantly with how the three judging undergrad students rate creativity using our criteria. But some of these students -such as those high in openness to experience- were more discerning. Undergrad psychology students who generate more responses to a certain type of divergent reasoning tests which are more likely to be selected by our judges are doubly skilled: they are better at generating divergent ideas that we consider creative and at discerning which ones we consider the best."

But I guess that would read like a circular argument and isn't as interesting to close a paper, is it? But that'd be a lot more honest and aligned with the actual work and the actual findings.