A Position on Peer Reviewing in HCI, part 2

This post follows on from my previous post, in which I outline my position on peer reviewing and my reasoning for it.

In this post, I offer four observations in the form of a guide to serving as a good Associate Chair (AC).

[1]

A CHI paper submission typically represents 12-24 months of a research team’s work. That effort deserves the respect of a substantive and effortful review!

A substantive AC review includes the following:

• A summary of the key criticisms and praise of reviewers;
• A substantive articulation of the AC’s own criticisms and praise;
• Thoughtful and constructive suggestions to improve the quality and acceptance chances of future versions of the work.

It’s nearly impossible to do all of the above in 2-3 sentences, so AC reviews do need a certain length to be just and effective.

It is an honor to be selected as an AC, a reflection of your community’s esteem for you. Be worthy of that or let someone else do it who will.

A Position on Peer Reviewing in HCI, part 1

HCI is continuing a trend towards using conferences, rather than journals or books, as the premier venue for published work. The speed of the submission-decision cycle often means that decisions are fast and binary: one shot and you’re in or you’re out. And in this process, a very small number of people hold a lot of power over your paper’s fate. In this post, I offer my position on the role and responsibilities of being an AC. The short version: peer reviewing is a critical, not a scientific, activity; it ought to be acknowledged as such; and people ought to be mentored in and held to the standards to which other academic critics hold themselves.

It is possible to argue that ACs and reviewers are simply there to verify that your science is good. Antti Oulasvirta has done precisely that, in a lengthy blog about CHI rejections that lays out 8 reasons “why your paper was rejected,” as determined by an AC. His 8 reasons all deal with issues of validity, research design, replicability, etc. Implied but not stated explicitly in Oulasvirta’s post is that if your paper didn’t get in, it’s your fault for doing bad science or reporting as if you had done bad science. Also implied but not stated is that if your paper didn’t get in, it’s not your AC’s fault. Oulasvirta’s self-exculpating post concludes with a cartoon that says, “Shouting at reviewers in your rebuttal is only going to make it worse.” I think this post reflects an assumption that the data (in this case, “your paper”) is presumed to speak for itself, and the reviewer or AC simply sees what is there. If your work lacks construct validity, or your work is not replicable, or your findings are inapplicable, then your reviewer discovers that fact and rejects you. The agency for the outcome is all in your paper.

I disagree with this position. I don’t disagree that papers get rationally rejected because of scientific flaws. I disagree with the implied proposition that the data speaks for itself, that the AC and the reviewers are not responsible for their decision, because the paper itself is.

Apple Above Critique?

Salon has an article today about a controversy that happened recently at Print magazine, when it published a critique of Apple.

The original critique is here (and has some very interesting analysis beyond its basic premise):
http://printmag.com/Article/An-Anatomy-of-Uncriticism

And the Salon article on the dust-up is here:
http://www.salon.com/2012/01/12/design_critique_imprint/

One thing at stake in all this is how people perceive the role of “critique” in design. Often, people assume that critique is saying bad things about something else. That’s how we use the term “criticism” in every day life. But as the Salon article notes, the purpose of serious critique (including design criticism) is to illuminate, not to scold. The relevant quote is here:

If the criterion for what warrants design criticism is based on a level of social, cultural or political impact, then a particular work is fair game regardless of the age or virtuosity of its maker. Since criticism is not meant to be a scold, but is rather a means of illuminating — delving below the surface — finding aspects of work that benefits by explanation and analysis, nothing and no one should be exempt.

There are two gems in that quote worth pointing out. The first I already have, which is that criticism is about illuminating a community, not about attacking a person or artifact. The second point reinforces the first: the role of criticism is not simply to call attention to flaws, but rather it is about “delving below the surface” to help explain or analyze the “social, cultural or political impact” of a design or event. Since we all must live with the social, cultural, and political impacts of designs, we all have a stake in their critique, with no exemptions.

Note: This post is based on an entry I posted earlier at the Interaction Culture Class blog.

Blinding You With Science

As many of this blog’s readers know, my background is in the Humanities, which has been a good and bad thing for me as a researcher in the predominantly scientific discipline of HCI. For several years, one of my projects has been to clarify and distinguish from each other scientific and humanistic modes of knowledge production (e.g., their respective epistemologies, contribution types, expressive forms (e.g., reports versus essays, etc.)). Indeed, this blog has been my whiteboard for sketching out my ideas in that domain.

Maybe two or three years ago, I realized that I didn’t know enough about science to do what I really wanted to do, and so I started reading from the vast literature in the history and philosophy of science. I’ve really enjoyed it; among other things, it has helped this humanist come to appreciate and love science. And in the past 18 months or so, these readings have started to bubble out into my teaching and publications–and I think their introduction to my work has not only strengthened it from my own perspective, but I think readers and students have also responded to aspects of my work and teaching that takes the time out to explain some of these terms.

The positive reaction to my use of core scientific philosophy took me a bit by surprise, because my own “impostor syndrome” had me convinced that everyone but me already knew this stuff. But increasingly, I’m realizing that many people know how to do science with rigor, but they can’t always clearly articulate their own, say, epistemology–at least on short notice. Sometimes, they inadvertently treat their own ways of knowing as a normative standard, i.e., that all knowledge production should have the same indicators of “rigor” that theirs does. But when I start to talk through these issues using basic scientific technical vocabulary in a reasonably precise way, it excites them–and that really facilitates good dialogue, which is what we’re all about.

Here are some very simple examples of what I am talking about:

HCI and the Essay

Sorry I haven’t been as active on this blog of late; I’m going up for tenure soon, and blogging doesn’t seem to count for much. (Even though I have to wonder how blogs stack up to conference presentations in terms of scholarly impact.)

Anyway, I recently composed a position paper that started as a rant on this very blog. The topic is on the essay and HCI, and specifically why HCI ought to embrace–rather than marginalize–the essay as a contribution type.

Download it here. (PDF, 84 kb)

Interaction Criticism: How To Do It Handout

My 7-part series “Interaction Criticism: How To Do It” is among the most popular hits on this site. As of now, it is also spread across 7 different posts, some of them with their own comments, all of them with their own blog clutter.

As a long overdue service to readers (and also to help me get motivated to revise it and send it somewhere as a proper paper), I have compiled all seven parts, ever so lightly edited them, and now present them in a single downloadable form. Hopefully, it is more user friendly now.

Bardzell – Interaction Criticism: How To Do It

UPDATE: The above link was broken for a while, but it has since been fixed.

CHI2009 Interaction Criticism and Aesthetics

Below please find links to the paper and slides that I presented at CHI2009 in Boston on “Interaction Criticism and Aesthetics.” It was a great experience for me, and I hope this paper and these slides help us move HCI’s cultural agenda forward!

Paper (ACM digital library, subscription required)
http://portal.acm.org/citation.cfm?doid=1518701.1519063

Slides (PDF, 13 MB, no subscription–should download right away) UPDATE: This URL was dead for a while and has now been fixed.
http://dl.getdropbox.com/u/1874332/InteractionCriticismAndAesthetics.pdf

Thanks to the many people who helped me with this!

The Essay and HCI

In my recent post on discourse analysis versus close reading, I got into a discussion in the comments on the origin of the critic’s understanding and the role of subjectivity, objectivity, and so forth. In the course of that discussion (and I’d like to thank Jeremy Hunsinger for his part of the discussion that helped clarify this for me), I realized that there are really two aspects of the problem I am talking about. The first aspect is the method or set of interpretive strategies that leads the text analyst to a certain point of view, and the second aspect is the structure of the expression (i.e., paper) in which this analysis is articulated and defended.

So my original gripe in that post is that people in HCI sometimes seem to think that unless one does some form of coding, one’s textual analysis doesn’t deserve the name, and one is instead merely advancing an opinion. The point I’m advancing in this post is its corollary: Unless one writes a recognizable scientific paper (intro, lit review, methodology, results, discussion, conclusion) one likewise runs the risk of being seen as merely advancing one’s opinions or writing (as one reviewer once accused me) like a journalist.

Discourse Analysis vs. Close Reading

Introduction

Textual analysis is fundamental to many kinds of research, from psychology to literature, philosophy to information science. Not surprisingly, different strategies have emerged from within the various disciplines that do textual analysis, and naturally these strategies reflect the epistemologies of the disciplines from which they emerge. And as long as one stays insular to one’s own discipline, there isn’t a problem.

But as soon as a field claims the mantel of “interdisciplinarity,” it faces a dilemma: to protect and preserve what is known to work, or to open itself out to alternative ways of knowing. Now, both of these impulses are in themselves legitimate in themselves, but as they enter everyday life (e.g., the writing, reviewing, and editing of papers), they sometimes appear in clumsy ways. Some of these clumsy ways are as follows:

• Epistemological bigotry: This happens when someone asserts (often without meaning to) that she or he knows the right way and everything else is “fluff” or wrongheaded. In HCI, scientism is often confused for science, to the detriment of both HCI and science.
• Piecemealism: This happens when someone injects a small piece from one tradition uncritically into another, without recognizing that a piece might not represent the whole from which it is drawn, nor recognizing that that piece might be at intellectual odds with the rest. In HCI, I see this with “critical” approaches to HCI where a single concept is ripped from a complex tradition, such as poststructuralism, and applied to traditional design approaches to, say, mobile phones or Web applications.
• Equivocation: This happens when two or more groups of people use the same word in completely different ways, without seeming to be aware that their use is not “natural” or universal. In HCI, “aesthetics” seems to be a word that has almost no relationship to the 2,500 year old tradition of aesthetic theory, as I’ve ranted on before.

All of these involve a combination of dogmatism and muddled thinking. While scientism–by which I refer to as a fetish for scientific ways of knowing, placing it above other forms of intellectualism–is dogmatic and often intellectually muddled, I would stress that neither dogmatism nor muddled thinking is scientific. Scientism so-defined is bad science.

Two Takes on the Hermeneutic Circle

Lately, a lot of my intellectual energy has been devoted to issues that one way or another pertain to the notion of the hermeneutic circle. The following is an effort to develop some clarity on it.

Defining the Hermeneutic Circle

Just so everyone knows what I am going on about here, let me first attempt a definition, and then I’ll develop the concept in two different directions. The hermeneutic circle refers to the situation in which when we encounter a text (i.e., any cultural phenomenon) we can only understand it (i.e., make sense of it) with reference to other texts, and in turn our understanding of these other texts is modified by our understanding of this text.

In other words, to understand a given expression, we must understand the language in which it is written.  “Language” here means more than English or Chinese. For example, to understand an expression such as Shakespeare’s Antony and Cleopatra, one must have some competence with Western tragedy in general, Shakespearean tragedy in particular, Roman history, poetry (e.g., meter), Renaissance culture, and so forth. Yet these linguistic contexts are all made out of other texts, which can only be understood the same way. We face just such a problem in HCI when we interpret and/or evaluate an interaction: the criteria by which we evaluate or interpret an interaction are determined by concepts and theories ingrained in our field, and these in turn are derived from, among other things, interpretations and evaluations of other interactions.

In this post, I want to explore two takes on the hermeneutic circle, a pessimistic view and an optimistic view. I will also make at least token efforts to relate this discussion to HCI.