Student assessment in design

Reference: Morgan, C 2011, 'In the eye of the beholder? An investigation of student assessment in the creative arts in universities', PhD thesis, Southern Cross University, Lismore, NSW.

Abstract: This thesis addresses the question of how university academics in the creative arts might assess students’ creative works fairly and appropriately. Specifically, it considers how these academics conceptualise and approach the task of student assessment, what tensions and dilemmas are experienced in student assessment, and how student assessment practices in the creative arts might be made more transparent and robust. The complexities of student assessment in the creative arts, and the uncertainties and vulnerabilities that beleaguer academics when making judgements about the quality of their students’ creative works are widely recognised, but not well understood. This investigation employed Naturalistic Inquiry to illuminate the assessment experiences in creative arts of 30 academics from across three countries. It documents how these academics conceptualised creativity in their own disciplines, how they conveyed their understandings to students, how they provided feedback on students’ creative development, how they prepared students for assessment, and how they made judgements about students’ creative works. Four interlinked dilemmas of student assessment practice in the creative arts were thus identified. The first is the lack of a shared understanding of creativity and of its manifestations in particular disciplinary settings. The second is the difficulty of explaining creativity to students. The third is the challenge of providing transparency for students about how their creative works are to be judged. The fourth is the decision about what role subjectivity should play in judgements about creativity in student works. The findings from this investigation challenge accepted wisdom about student assessment that is based largely on positivist paradigms. Outcomes-based education, for example, does not readily allow for the unexpected or unanticipated – yet these are highly prized qualities for academics assessing students’ creative works. Based on the insights obtained, a seven-phased, cyclical model of student assessment in the creative arts is proposed. This model identifies the purposes of each phase of the student assessment cycle, addresses the dilemmas of student assessment that have been identified by academics in the creative arts, and provides indicators of sound practice. The model is designed to support assessors to navigate through a complex terrain, to reflect on current practice, and to harness their activities consistently towards the central purpose of fostering creativity while at the same time assessing student work with demonstrable fairness and appropriateness.

The Art of City-making

Title: The Art of City-making

Author: Charles Landry

Abstract: City-making is an art, not a formula. The skills required to re-enchant the city are far wider than the conventional ones like architecture, engineering and land-use planning. There is no simplistic, ten-point plan, but strong principles can help send good city-making on its way. The vision for 21st century cities must be to be the most imaginative cities for the world rather than in the world. This one change of word - from 'in' to 'for' - gives city-making an ethical foundation and value base. It helps cities become places of solidarity where the relations between the individual, the group, outsiders to the city and the planet are in better alignment. Following the widespread success of The Creative City, this new book, aided by international case studies, explains how to reassess urban potential so that cities can strengthen their identity and adapt to the changing global terms of trade and mass migration. It explores the deeper fault-lines, paradoxes and strategic dilemmas that make creating the 'good city' so difficult.

A Framework for Cross-Disciplinary Team Learning and Performance

Title: A Framework for Cross-Disciplinary Team Learning and Performance
Authors: Schaffer, Scott P;Lei, Kimfong;Paulino, Lisette Reyes
Reference: Schaffer, Scott P;Lei, Kimfong;Paulino, Lisette Reyes (2008) A Framework for Cross-Disciplinary Team Learning and Performance, Performance Improvement Quarterly; 21, 3

Abstract: The construct of teamwork has been of considerable interest to researchers and practitioners across domains. The literature on teams includes many studies related to team composition, processes, and roles, but it pays much less attention to how teams learn and innovate. Studies examining how cross-disciplinary teams interact during projects are even less common. The study examined here was conceived to fill the need for a theoretical framework to describe how individuals from different disciplines evolve into a team that creates new forms of knowledge and innovative solutions or products. The framework, which was validated in a university service-learning program with over 25 teams, is a comprehensive theory merging two existing team models within a sociocultural system framework emphasizing the mediating aspects of the collective team and context. The theoretical foundations, the definitions, and dimensions of the framework are presented in this article.

Notes: Scanning the literature on creative teamwork, most papers seem to be about developing hypothetical models, and reinforcing some commonsensical and general ideas. The modified CDTL proposed here is interesting, well presented, yet fails to contribute anything new as the conclusions are that "cross-disciplinary team learning is more likely to be found when:

• Members shift from self-efficacy to collective efficacy
• They shift from individual process orientation to team goal orientation
• Shift from knowledge acquisition to creation
• Become aware of others' "functional discipline knowledge shifts"

Which to me seem like four different ways of saying that good teams are more likely when their members work more as a team than a collection of individuals, which is kind of circular and obvious. Will keep looking for more complete studies that lead to more detailed and reliable models.

Shikakeology

Title: Shikakeology: From framework to implementation
Authors: Naohiro Matsumura and Renate Fruchter
Reference: Matsumura, N., & Fruchter, R. (2014). Special issue: Shikakeology: From framework to implementation. AI & SOCIETY, 1-3.

Abstract: Shikake is a Japanese word with a wide range of meanings regarding triggers for behavior change, and using shikakes to change behaviors could be a promising and feasible approach for making the world better. However, the methodology for developing a new shikake is not well studied. To define such a methodology, Dr. Naohiro Matsumura, one of the editors of this special issue, coined the term ‘‘Shikakeology’’ in 2011 as a new academic field where the mechanism underlying a shikake as an artifact, a theory, a system, and a practice designed to change behavior can be discussed and understood.

Notes: The concept of 'shikake' seems very very similar to 'nudges' introduced by Thaler and Sunstein in their 2008 book by the same name. This is a special journal of AI&Society worth reading. Is this another case of reinventing the wheel, or using different terms to refer to the same concept -not sure.

Social simulation and cognitive models

Title: When does social simulation need cognitive models?
Author: Nigel Gilbert
Reference: Gilbert, N. (2006). When does social simulation need cognitive models.Cognition and Multi-Agent Interaction, 428.

Abstract: Contributors to this volume have explored the ways in which cognitive models or architectures may be helpful or even essential for building simulations. In this epilogue, I shall be considering whether cognitive model are always necessary – is a social simulation necessarily inadequate if it has no or only a very simple model of cognition? If not, is it possible to specify classes of simulations for which cognitive models are necessary or unnecessary?

Notes: Brief editorial with some clear ideas on multi-level agent simulations (but can't seem to be able to find this book online)...

How future CAD systems looked in 1983

Title: "Future CAD systems"
Author: K. Preiss
Journal: Computer-Aided Design, 15(4), 223-227

This paper presents a rather informal view of what the author considered to be the "changes which are clearly coming". Not surprisingly, the future as portrayed here is rather conservative, merely projecting what was being developed at the moment. The paper is worth reading for a number of reasons, including the common flaws in forecasting the future. How will CAD systems look in 30 years?

Guidelines for the future based on the past

There is a fundamental problem with management research, the whole field takes for granted that you can build the future by applying guidelines extracted from the past. Design research often tends to do the same, and the results can only be mediocre.

Here is an example from: Hayagreeva Rao and Robert Sutton: How Do You Scale Excellence?: "they interviewed business leaders, reviewed research, and studied and conducted case studies "
http://www.gsb.stanford.edu/insights/hayagreeva-rao-robert-sutton-how-do-you-scale-excellence

I won't list here everything that is wrong with such assumption, and of course it is valuable to learn from the past. Just imagine if in creativity, innovation and design people were making decisions based on what leaders say in interviews and what case studies showed to be successful in the past. Who in their right mind would think that repeating the same decisions would make sense in the future?

Analysing the writing of a paper

This entry is not about a paper, but a tool to diagnose and analyse writing. Of course, the first thing is to use it to analyse one own's writing (and improve it), but it's also quite interesting as a way of testing how 'readable' are papers in our field (written by others, including famous authors and scholars). I just did this with a few of the most influential papers (the Introduction section anyway), and the results are quite interesting (worrying or amusing, it depends on how serious you are about 'good writing' and how much you trust these tools to measure 'readability'). A couple of these papers were in the "Heart attack territory". Now, since these papers are widely cited and influential in my field, shall we conclude that writing in this community is not a particularly important criterion? See for yourself: http://www.writersdiet.com/WT.php 

Self‐reported differences in creativity across 56 domains and with hidden interpretations of the rating instrument

Title: Self‐reported differences in creativity by ethnicity and gender
Authors: Kaufman, J.C.
Source: Kaufman, J. C. (2006). Self‐reported differences in creativity by ethnicity and gender. Applied Cognitive Psychology, 20(8), 1065-1082.

Abstract: Creativity assessment has been proposed as a supplement to intellectual testing, in part because of
reduced differences by ethnicity; creativity testing might also specifically help reduce stereotype threat.
Recent trends in creativity research point to a domain-specific view challenging the more traditional
generalist view. With these trends in mind, the current study assessed creative self-perceptions of 3553
students and community members in 56 different possible domains distributed across five factors (as
determined by principal components analysis). African Americans were less likely to fall prone to
gender stereotypes in creativity. In addition, African Americans and Native Americans tended to rate
themselves as more creative than other ethnicities. Specific trends in the factors and implications for
future research are discussed.

Notes: Large scale survey asking mainly students "How creative are you?" in 56 domains ranging from travel to woodworking, physics, political science, sports, etc. Interesting to see that after complex statistical pirouettes, some conclusions are built about ethnicity, gender and self-reported creativity, only to admit in the end that: "it is unknown if people conceived of creativity as the same construct across all domains. People may have had a difficult time imagining what it meant to be creative in certain domains (such as the sciences)." Another way of viewing this is based on experience is that no, of course people did not conceive of creativity as the same construct (across people and across domains), a simple word association survey shows evidence. Then, "In addition, some people may have conflated their skill in a domain with their creativity in that domain. Another possibility is that some people or groups simply showed a tendency to use the lower or upper ends of the Likert scale." Anyone involved in grading, and particularly coordinating a group if graders (as in competitions, selection committees, classes) knows that indeed, people interpret differently values in a Likert scale... So, like in many other papers, the "science" seems to rest on the use of statistical measurements and the "validity" on the 'big data', but the essential concepts behind the experiment are incredibly limited, making any claims extremely weak.

Participatory design: Issues and concerns

Reference: Kensing, F., & Blomberg, J. (1998). Participatory design: Issues and concerns. Computer Supported Cooperative Work (CSCW), 7(3-4), 167-185.

Abstract: We characterize Participatory Design (PD) as a maturing area of research and as an evolving practice among design professionals. Although PD has been applied outside of technology design, here we focus on PD in relation to the introduction of computer-based systems at work. We discuss three main issues addressed by PD researchers; the politics of design; the nature of participation; and method, tools and techniques for participation. We also report on the conditions for the transfer of “PD results” to workers, user groups, and design professionals that have characterized PD over time and across geopolitical terrains. The topic of the sustainability of PD within an organizational context is also considered. The article concludes with a discussion of common issues explored within PD and CSCW and frames directions for a continuing dialogue between researchers and practitioners from the two fields. The article draws on a review of PD and CSCW literatures as well as on our own research and practical experiences.

Comments: This is a highly influential paper (500+ citations) and is a must-read for those interested in Participatory Design. I found particularly interesting these ideas: "Among the activities of the work groups are developing a common understanding of the current relations between technology and the organization of work, exploring new organizational forms, formulating system requirements, and prototyping new systems… Equally important to the principles of organization are the issues of resource and time allocation... Responsibilities and accountabilities also vary depending on how projects are supported and initiated... There are also a variety of ways in which PD projects are initiated... Some PD projects are undertaken to explore specific technology possibilities while others have a more open-ended technology agenda"

"'Clement and Van den Besselar (1993) note that the experimental nature of most PD projects often leads to small-scale projects which are isolated from other parts of the organization. When the researchers leave, the participatory processes seldom diffuse to other organizational entities."