1. In Part 1 I offer a broad rationale for my position
2. In Part 2 I offer specific recommendations for ACs and reviewers

In this part, I offer my own recommendations for the research community moving forward. I stress from the outset that this is a position. I do not attempt to speak from the Voice of God, but mainly from my own experience. I just want to stimulate a conversation; I have no ambition here of offering the final word.

My recommendations begin after the fold.

Recommendation 1. Treat ACing as explicit, rather than tacit, knowledge that is distinct from peer reviewing.

This recommendation is necessary because the community is growing and the number of submissions keeps increasing. We have to bring in new people, sometimes in large numbers. Rather than leaving it up to them to figure it out, or to rely on SCs who are both overworked and unlikely to spontaneously handle this in a consistent way across the board, let us make an effort as a community to articulate what we want from ACs so junior researchers have at least a model to hold themselves to. We teach our Ph.D. students how to do peer reviews. We need to teach junior scholars how to be ACs. I put my money where my mouth is in the previous part.

Such an approach could also help undermine the “Old Boy Network” that the CHI community (like any other community) sometimes slips into, because it would offer mechanisms to support the infusion of new blood.

Recommendation 2. Each major HCI subcommunity should consider explicitly articulating (and maintaining over time) its primary peer reviewing criteria.

What makes a good contribution to the Design subcommunity is likely to differ from a good contribution in the UIST or CSCW subcommunities. Yet reviewers and ACs (for good and desirable reasons) often cross these lines–but do they bring tacit norms with them?

Such an approach would also be effective for handling emergent scholarly practices. For example, for years a number of us have been vocally complaining about “opinion papers” as a category in HCI, where “essay” is what (presumably) was meant. Good news: the language surrounding the submission process for essays is greatly improved. Bad news: it is less clear how well the HCI community understands the epistemologies and norms of effective essays.

Recommendation 3. Ensure that adequate opportunities for scholarly dialogue exist surrounding our paper decisions.

I feel very strongly about this one.

Beyond reviewer-AC dialogue, this recommendation also advocates for author-AC dialogue and ideally author-reviewer dialogue. Because papers don’t just “speak for themselves,” and given the stakes involved in the decisions we make, if authors feel confused or wronged during the process,  there should be some mechanism to express that and work it out in a reasonable way.

This one is extremely difficult to deal with, given all the constraints involved. But I can still make some recommendations here:

Recommendation 3a. Allow authors to respond to as many points (criticisms, suggestions for revision) as they feel they need to during rebuttals.

I know people disagree with me on this, and in some subcommittees it may be appropriate if all that is at stake are questions of fact. However, in my subcommittee, at stake are often judgments and interpretations and the reasoning and assumptions surrounding them. The 5,000 character rebuttal is insufficient to support the intellectual role it was intended to play. It’s also not clear to me that the persent form is better than nothing.

Journals in general and CSCW 12 have a process where authors typically submit a revision with a statement in a two-column format. In the left column is a quote or paraphrase of a particular reviewer concern, criticism, or suggestion. In the right column, the author responds to it and explains what will be/has been done about it. Such documents are arguably easier to read than rebuttals, and they structurally encourage people to stick to the issues, rather than expressing anger or frustration in a general sense. They also support quality revision, which is good for the long-term health of the discipline.

Recommendation 3b. Keep trying and iterating on the CSCW/UIST reviewing process from 2011.

In particular, the Revise and Resubmit process at CSCW was effective at supporting reasonable and, when necessary, lengthy dialogue between ACs, reviewers, and authors. And yet it was less work overall for ACs (at least, in my experience). Also: better decisions were made, and accepted papers had been much more thoughtfully revised than normal (at least in my experience). Now CSCW is having a banner year with tons of great papers that were substantially, not cosmetically, improved.

Recommendation 3c. Allow authors to communicate directly with their ACs.

Allowing this backchannel communication can alleviate many of the pressures on the rebuttal process, support productive scholarly dialogue, and lead to better decisions for now, and better papers down the line.

Recommendation 4. Find or develop mechanisms to enhance both reviewer and AC accountability–and make it part of institutional memory.

We could use both crude quantitative measures and also simple qualitative measures to evaluate reviewers and ACs and encourage high quality reviewing decisions.

Some crude quantitative measures include number of reviews done in the PC system, average word length of reviews, scoring patterns (reviewer average compared to paper average, which would help identify the Negative Nellys and the Pollyannas), number of times accept/reject recommendation agreed with final decision, timeliness of reviews, and I’m sure many others.

Some qualitative measures might include ratings of ACs by authors and SCs and ratings on reviewers by ACs and authors. These might include simple Likert scales that ask whether the reviewer understood the paper, whether actionable revision suggestions were offered, whether the rationale for the decision was explained, and so on.

Another idea is some sort of post-mortem, where SCs and ACs talk about their experiences–not just in a general way, but in an individual way. If an AC is a Negative Nelly or a Pollyanna, he or she might not even realize it, and yet I bet the SC generally knows.

—

As I get feedback and pushback, I may revise and/or expand this. Regardless, it is meant to be constructive and to start, not to finish, a conversation.

In this post, I offer four observations in the form of a guide to serving as a good Associate Chair (AC).

[1]

A CHI paper submission typically represents 12-24 months of a research team’s work. That effort deserves the respect of a substantive and effortful review!

A substantive AC review includes the following:

• A summary of the key criticisms and praise of reviewers;
• A substantive articulation of the AC’s own criticisms and praise;
• Thoughtful and constructive suggestions to improve the quality and acceptance chances of future versions of the work.

It’s nearly impossible to do all of the above in 2-3 sentences, so AC reviews do need a certain length to be just and effective.

It is an honor to be selected as an AC, a reflection of your community’s esteem for you. Be worthy of that or let someone else do it who will.

[2]

The AC is responsible for two very different jobs.

1. ACs are charged with helping make a decision about whether or not something is accepted.
2. ACs are also charged with providing constructive and worthwhile feedback to authors both to explain the decision and also to offer suggestions for improving the work (regardless of whether or not it is accepted).

Effective ACs do more of the latter than the former.

[3]

ACs have a second conflicting charge.

• They are tasked with representing the reviewers
• They are tasked with offering their own review

This double-task has some implications:

• Authors deserve to know that their ACs actually read their work and didn’t merely summarize reviews; otherwise, why have ACs?
• ACs need to respect their own reviewers’ reviews; otherwise, why have reviewers?
• If ACs push in a different direction than their own reviewers, they need to
  • Faithfully represent and acknowledge what their own reviewers said, including and especially when they (the ACs) don’t agree with the reviewer(s)
  • Be accountable to their own position by stating very clearly why they disagree with their reviewers
  • Seriously consider asking for new reviewers, a 2AC, and/or discussion

Here is one way this problem creates arbitrary problems in the current system. Let’s say that an AC disagrees with her or his own reviewers and wants to give a paper a 2, even though that AC’s reviewers on average gave it a 4. What does the AC record for her or his score? Some ACs will average their own scores into the reviewers average, in this case offering a meta-review score of 3.5. Others will offer a meta-review score of 2. There is no consistency among ACs (I know this from experience). But since AC meetings use a numeric scale as the primary guide to the preliminary ranking of papers, those who have the first kind of AC will have an advantage over those who have the second kind of AC. That’s arbitrary.

[4]

ACs are responsible for their reviewers’ reviews.

Another way to say this is that ACs must serve as critics of reviewers’ reviews. Reviewers sometimes write poor quality reviews. A poor quality review is one where the critical judgment (accept/reject) is not rationally explained or justified and/or there are no constructive recommendations about how to strengthen the research moving forward (whether or not it is accepted). It’s an AC’s job to try to catch these early and do something about them. Here are some recommendations for dealing with common types of poor quality review.

• Reviewers who rate themselves as “1 No Knowledge” should be seriously reconsidered. A reviewer is a critic, and a critic with no knowledge is not a critic at all. Instead of a defensible judgment, that person can only offer an impressionistic opinion. Imagine if a New York Times book reviewer wrote, “Well, I don’t know anything about contemporary fiction, but your library shouldn’t buy Murakami’s 1Q84, because I didn’t like it.”
• Vacuous reviews of all types should be challenged or replaced. Examples:
  • The 2-sentence review
  • The typo correction review
  • The obsess over one tiny flaw review
  • The “this should go to a different conference so I’m not going to say anything about this paper” review
  • The empty fence-sitter review
• Reviewers who have conflicts of interest are not acceptable. This is true not just when it’s known in advance, but also when it becomes clear subsequently. Conflicts are not just institutional! If a paper critiques a given scholar’s work, then there is a potential conflict in having that person as a peer reviewer.
• Scores and reviews that don’t line up (e.g., a positive review rated 2; a highly critical review rated 4.5) should be explained and/or clarification from reviewers sought.
• Anomalies deserve explanation (e.g., when reviewers give a wide distribution and the AC takes a side–again, that’s OK and our job to do–some explanation is needed so authors can understand why their ACs took that side).

When several of these sorts of low quality judgments occur in the reviews of the same paper–a short AC review, vacuous reviews, anomalous scores–it’s low quality and even meaningless feedback, and yet it has serious consequences.

———

CHI, DIS, CSCW, UIST, etc. are flagship conferences of the field; the papers submitted to them represent hundreds of hours of work; most rejected papers are going to be resubmitted in the future: therefore, out of respect for our own profession, we need to hold ourselves as ACs to high standards, especially until we can create better structural accountability.

Continue to part 3, where I make recommendations to the HCI community at-large!

It is possible to argue that ACs and reviewers are simply there to verify that your science is good. Antti Oulasvirta has done precisely that, in a lengthy blog about CHI rejections that lays out 8 reasons “why your paper was rejected,” as determined by an AC. His 8 reasons all deal with issues of validity, research design, replicability, etc. Implied but not stated explicitly in Oulasvirta’s post is that if your paper didn’t get in, it’s your fault for doing bad science or reporting as if you had done bad science. Also implied but not stated is that if your paper didn’t get in, it’s not your AC’s fault. Oulasvirta’s self-exculpating post concludes with a cartoon that says, “Shouting at reviewers in your rebuttal is only going to make it worse.” I think this post reflects an assumption that the data (in this case, “your paper”) is presumed to speak for itself, and the reviewer or AC simply sees what is there. If your work lacks construct validity, or your work is not replicable, or your findings are inapplicable, then your reviewer discovers that fact and rejects you. The agency for the outcome is all in your paper.

I disagree with this position. I don’t disagree that papers get rationally rejected because of scientific flaws. I disagree with the implied proposition that the data speaks for itself, that the AC and the reviewers are not responsible for their decision, because the paper itself is.

ACs and reviewers are chosen not on the basis that they represent (as part of a sample) a community, but because they are exceptional: they are presumed to know the topic especially well and to be in a uniquely qualified position to judge a paper. Judging a human-made work to assess its contribution to a community and the world is criticism. It entails a complex and holistic decision that weights factors ranging from the timeliness of the topic, the rigor of the science, the framing of the problem space, the relevance of the contribution, the nature of the contribution, the applicability of the contribution, and so forth. Poor rigor is seldom a winning recipe for publication, of course! At the same time, some flaw or compromise in the rigor can be found in most papers. The issue is not the ability of a reviewer to note the fact that there was such-and-such flaw in the methodology: the issue is how seriously to judge that flaw as a part of the whole contribution.

So in my view, ACs need to stop acting like a paper’s acceptance or rejection is simply causally related to whether that paper adheres to prevailing standards of scientific rigor. ACs make judgements based on their expert point of view: they should and they must. This judgment, because of its consequences, needs a rational justification. This is what the metareview is for–though it is not always effectively used that way. ACs and reviewers need to be accountable to their own position of power–and how responsibly they wield it. And if they fail in that duty, then they deserve to be yelled at in the rebuttal. And if you are an AC or reviewer that gets yelled at in a rebuttal, before browbeating disempowered authors with your copy of The Basics of Social Research, how about a little introspection? Because at CHI today no one else can or will demand it of you, thanks to a combination of (a) reviewer anonymity and (b) structural power imbalance. Let’s explore this:

• Reviewers and ACs decide what is accepted and rejected, both shaping what constitutes mainstream discourse at CHI and also influencing people’s careers (tenure, etc.). Authors have no such power over reviewers or ACs.
• Reviewers and ACs each can write as much as they want about a paper. Authors can reply via a one-time rebuttal with up to 5,000 characters (not words!) to all ACs and reviewers. Reviewers can write as much as they want in reply to rebuttals, but authors can’t see or reply to these responses. ACs and reviewers can hold secret conversations that authors are not privy to. ACs meet in a secret meeting to make final decisions, and questions and comments are raised in that meeting that authors will never hear or have any chance to respond to.
• Authors don’t know who these people are who are wielding this power over them.

Now I am aware of all the practical reasons that have led to such structural outcomes. I am not declaring a need to replace them with a new process (though personally I support both the CSCW’12 and alt.chi’08 and ’12 models). But I want to make one thing very clear: ACs and reviewers have very little structural accountability in the current CHI Papers and Notes model. If ACs or reviewers make an honest mistake, or, worse, if they are negligent through laziness, intellectual narrowmindedness, or personal conflicts, there is very little consequence for them, opportunity to correct it, or recourse for authors.

Hence my main thesis: if we continue to use the existing reviewing structure, then to ensure the most rational reviewing process, we need to work proactively to ensure that ACs and reviewers are competent and act with integrity inasmuch as they are critics (a job separate from whatever their area of subject expertise is).

In the spirit of contributing to this work, I have taken the time to lay out some of the key expectations every AC should be held to. To see them, continue on to Part 2.

The original critique is here (and has some very interesting analysis beyond its basic premise):
http://printmag.com/Article/An-Anatomy-of-Uncriticism

And the Salon article on the dust-up is here:
http://www.salon.com/2012/01/12/design_critique_imprint/

One thing at stake in all this is how people perceive the role of “critique” in design. Often, people assume that critique is saying bad things about something else. That’s how we use the term “criticism” in every day life. But as the Salon article notes, the purpose of serious critique (including design criticism) is to illuminate, not to scold. The relevant quote is here:

If the criterion for what warrants design criticism is based on a level of social, cultural or political impact, then a particular work is fair game regardless of the age or virtuosity of its maker. Since criticism is not meant to be a scold, but is rather a means of illuminating — delving below the surface — finding aspects of work that benefits by explanation and analysis, nothing and no one should be exempt.

There are two gems in that quote worth pointing out. The first I already have, which is that criticism is about illuminating a community, not about attacking a person or artifact. The second point reinforces the first: the role of criticism is not simply to call attention to flaws, but rather it is about “delving below the surface” to help explain or analyze the “social, cultural or political impact” of a design or event. Since we all must live with the social, cultural, and political impacts of designs, we all have a stake in their critique, with no exemptions.

Note: This post is based on an entry I posted earlier at the Interaction Culture Class blog.

Maybe two or three years ago, I realized that I didn’t know enough about science to do what I really wanted to do, and so I started reading from the vast literature in the history and philosophy of science. I’ve really enjoyed it; among other things, it has helped this humanist come to appreciate and love science. And in the past 18 months or so, these readings have started to bubble out into my teaching and publications–and I think their introduction to my work has not only strengthened it from my own perspective, but I think readers and students have also responded to aspects of my work and teaching that takes the time out to explain some of these terms.

The positive reaction to my use of core scientific philosophy took me a bit by surprise, because my own “impostor syndrome” had me convinced that everyone but me already knew this stuff. But increasingly, I’m realizing that many people know how to do science with rigor, but they can’t always clearly articulate their own, say, epistemology–at least on short notice. Sometimes, they inadvertently treat their own ways of knowing as a normative standard, i.e., that all knowledge production should have the same indicators of “rigor” that theirs does. But when I start to talk through these issues using basic scientific technical vocabulary in a reasonably precise way, it excites them–and that really facilitates good dialogue, which is what we’re all about.

Here are some very simple examples of what I am talking about:

• What is the difference between a “scientific explanation” and a “description”?
• We all know that astrology is not a science, but can we say exactly why that is so?
• Is positivism a good or a bad thing? How has “post-positivism” replaced “logical positivism,” and how can we tell which form of positivism a given paper today is using?
• Should the social sciences be able to achieve the same standards of clarity and progress that the natural science have, and why or why not?
• What are the relationships among laws, theories, and models?
• What is the difference between methods and methodologies?
• What are the difference among common theories of truth: correspondence, pragmatist, coherence, etc.?
• Was Kuhn right about scientific paradigms, normal science, etc.?
• What does a rigorous scientific process look like? How about a rigorous scientific product?

This is basic stuff. But it’s also really important stuff, because more often than not, scientific paper submissions and grant proposals are evaluated against a tacit understanding of how well they measure up against scientific ideals. Yet, when we ourselves are fuzzy on these ideals–a problem that I see especially in the social sciences, which have a complex relationship to natural science, quantitative statistics, etc., on the one hand, and the humanities, critical theory, and design on the other–it becomes difficult to measure submissions against scientific ideals with the rigor and intersubjective agreement they deserve.

Recently, a scientist friend of mine heard that I was an “expert” in these things. (Such is demonstrably not the case, since I was merely seeking to address a gaping deficiency in my own knowledge; I will own up now to being an enthusiastic amateur in matters pertaining to the philosophy of science but no more. Maybe when I grow up.) And he asked me for some references, and it occurred to me that other people might benefit from some of this as well.

The history and philosophy of science, like any other academic field, is huge and daunting. At times, it has its own highly technical vocabulary and ways of doing, which are hard for outsiders to get into. Of course, Wikipedia covers many of these topics, but for obvious reasons Wikipedia defines a lot of these terms without explaining them. To me, encountering a definition without really explaining it either requires considerable investment on my part to get anything from it, or it just dribbles right back out of my head (<– most common outcome). Moreover, I’ve found that sometimes a concept in Wikipedia will be expressed from the perspective of a particular science, e.g., physics, that makes it a little bit hard to generalize from.

More usefully, there are a number of introductory anthologies that either (a) survey the key ideas, in chapters written recently for that introductory volume (i.e., they are accessible, but not famous or foundational chapters), or (b) collect seminal writings, mostly from the twentieth century (i.e., they are famous and foundational, but they are not reliably accessible to non-specialist readers). Both types of introductory volume have their benefits. All of them will address the sorts of issues laid out in the bullet list above.

Here is an example of the first type:
http://www.amazon.com/Blackwell-Guide-Philosophy-Science-Guides/dp/0631221085/ref=sr_1_1?ie=UTF8&qid=1286208464&sr=8-1

Here are two examples of the second type:
http://www.amazon.com/Introductory-Readings-Philosophy-Science-Klemke/dp/1573922404/ref=sr_1_1?ie=UTF8&s=books&qid=1286208497&sr=1-1

http://www.amazon.com/Philosophy-Science-Central-J-Cover/dp/0393971759/ref=pd_sim_b_8

All of the above cover both natural and social sciences, often with an emphasis on natural, but there is a book specifically on the social sciences that I really like (which is not to say that I agree with its argument):
http://www.amazon.com/Philosophical-Foundations-Social-Sciences-Controversies/dp/0521558913/ref=sr_1_1?ie=UTF8&s=books&qid=1286208549&sr=1-1

Clearly, this post is not an earth-shattering contribution to interaction design, but if you are an empirical scientist or an aspiring one, or if, like me, you work with empirical scientists, it’s good to have a fairly explicit and articulatable facility with these kinds issues. If nothing else, a working knowledge of core issues in the philosophy of science demystifies “scientific rigor,” which may help practitioners design better user studies, and it may help researchers speak more carefully about their taken-for-granted ideals.

Anyway, I recently composed a position paper that started as a rant on this very blog. The topic is on the essay and HCI, and specifically why HCI ought to embrace–rather than marginalize–the essay as a contribution type.

Download it here. (PDF, 84 kb)

As a long overdue service to readers (and also to help me get motivated to revise it and send it somewhere as a proper paper), I have compiled all seven parts, ever so lightly edited them, and now present them in a single downloadable form. Hopefully, it is more user friendly now.

Bardzell – Interaction Criticism: How To Do It

UPDATE: The above link was broken for a while, but it has since been fixed.

Paper (ACM digital library, subscription required)
http://portal.acm.org/citation.cfm?doid=1518701.1519063

Slides (PDF, 13 MB, no subscription–should download right away) UPDATE: This URL was dead for a while and has now been fixed.
http://dl.getdropbox.com/u/1874332/InteractionCriticismAndAesthetics.pdf

Thanks to the many people who helped me with this!

So my original gripe in that post is that people in HCI sometimes seem to think that unless one does some form of coding, one’s textual analysis doesn’t deserve the name, and one is instead merely advancing an opinion. The point I’m advancing in this post is its corollary: Unless one writes a recognizable scientific paper (intro, lit review, methodology, results, discussion, conclusion) one likewise runs the risk of being seen as merely advancing one’s opinions or writing (as one reviewer once accused me) like a journalist.

This calls to mind my regularly repeated rant that in CHI, the top conference of our field, there are eight “contribution types,” and these include “Development or Refinement of Interface Artifacts or Techniques”; “Systems, Tools, Architecture, and Infrastructure”; “Methodology”; “Theory” and others. The eighth of eight is “Opinion.” Now, clearly Opinion is the slot for essays; its accompanying description uses the term “provocative essay.” Because many trained scientists seem to hold the position that subjective = opinion = lack of rigor = not knowledge, I wrote a post distinguishing between opinion and judgment, and I’ll let that do its work and not restate it all here. Suffice it to say that I don’t believe many in HCI have a robust understanding of the nature and contribution of the critical essay.

What I want to do in this post is highlight the nature and goals of the essay as a form of discourse. It is structurally, substantially, and even epistemologically distinct from a typical scientific paper. This is not to suggest that one is better than the other, but rather only to suggest that essays cannot be evaluated on the same terms as scientific papers (and vice-versa, of course). Incidentally, Wikipedia has an entry on the essay, so if you want something more comprehensive, go read that. But I am going to share some quotes from one of my favorite essayists, Phillip Lopate. I offer this not as a final word on the essay in HCI but just to help HCIers recognize one when it lands in their laps and hopefully also to be able to evaluate them fairly.

Following are a bunch of quotes pulled from the beginning of Lopate’s essay, “In Search of the Centaur: The Essay-Film,” included in his collection Totally, Tenderly, Tragically. The essay considers why the essay-film isn’t more common than it is, and it works through definitions of its key terms and contemplates both films that broadly meet the criteria and those that are like essay-films but are not. But I’m not worried about the essay-film here, and so I focus only on Lopate’s attempts to define the essay:

the essay offers personal views. That’s not to say it is always first-person or autobiographical, but it tracks a person’s thoughts as he or she tries to work out some mental knot, however various its strands. An essay is a search to find out what one thinks about something.

Often the essay follows a helically descending path, working through preliminary supposition to reach a more difficult core of honesty.

This gets right at why I think critical essays are subjective and why it is so important not to fetishize objective understanding. I am not here rehearsing the Kantian argument that there is no such thing as objective knowledge that has taken such prominence in postmodernism. Rather, I am stressing a form of knowledge production whose source is not in empirical reality at all, but rather which is fundamentally embedded in the subject’s sensemaking, interpretive, and reflective practices. This sort of knowledge cannot be found in external reality, and so it is absurd to suggest that objective, empirical, or scientific approaches would be more effective in unconvering it.

I also want to stress the temporal unfolding of such knowledge: it is not a static representation of a state of affairs; it is a process of engaging, which has emergent outcomes. The essayist is likely to disagree with herself over time; indeed, Foucault’s response to critics who pointed out the inconsistencies in his work was to say (very roughly paraphrasing): “Of course! I write in order to change myself. At the end of a book, I am not the same person that began it. It would be a boring waste of time otherwise.” Back to Lopate on the essay:

Montaigne’s “What do I know?” is a mental freedom and cheekiness in the face of fashion and authority. The essayist wears proudly the confusion of an independent soul trying to grope in isolation towards truth.

This quote underscores the non-replicability, non-objectivity of the essay. The essayist has a voice, which is cheeky, confused, groping, truth-oriented but not arrived. To dismiss the essayist as merely providing her point of view utterly defeats its purpose. One can of course demand an accounting for that point of view (and the earlier quote, which stressed the temporal working out of a knot, suggests at least what that looks like), but the cultivation and expression of a point of view is arguably the essay’s raison-d’être.

Adorno, in “The Essay as Form,” saw precisely the anti-systematic, subjective, nonmethodic method of the essay as its radical promise.

Do I really need to say this? What Adorno is obviously contrasting the essay to is science. This is by no means a rejection of science! It is, rather, an effort to develop and articulate alternative practices of knowledge production, which are both non-scientific and nonetheless rigorous and legitimate.

Note also the word “radical” in that sentence. Radical implies an effective activism that actually will lead to substantive, desirable change in the world. Criticism generally has a progressive orientation, which seeks to complement the often conservative outcomes of science (that is too rich a claim to defend here, but short version: whose agenda does science typically serve?).

Whatever twists and turns occur along its path, and however deep or moral its conclusions, an essay will have little enduring interest unless it also exhibits a certain sparkle or stylistic flourish…. Freshness, honesty, self-exposure, and authority must all be asserted in turn. An essayist who produces magisterial and smoothly ordered arguments but is unable to surprise himself in the process of writing will end up boring us. An essayist who is vulnerable and sincere but unable to project any authority will seem, alas, merely pathetic and forfeit our attention. So it is a difficult game to pull off. Readers must feel included in a true conversation, allowed to follow through mental processes of contradiction and digression, yet be aware of a formal shapeliness developing simultaneously underneath.

I have spoken about the importance of the critic’s voice, and here Lopate develops the idea and puts some flesh to it.

I now give Lopate the final word, and he, in turn, passes the final word onto the great Marxist critic György Lukács. As you read this final quote, in light of the other quotes that precede it, consider how this relates to a field like HCI, with its interface design, user research, and growing awareness of its own socio-cultural responsibility (e.g., sustainability, aesthetics, etc.):

An essay is a continual asking of questions–not necessarily finding “solutions,” but enacting the struggle for truth in full view. Lukács, in his meaty, “On the Nature and Form of the Essay,” wrote: “The essay is a judgment, but the essential, the value-determining thing about it is not the verdict (as is the case with the system) but the process of judging.”

Textual analysis is fundamental to many kinds of research, from psychology to literature, philosophy to information science. Not surprisingly, different strategies have emerged from within the various disciplines that do textual analysis, and naturally these strategies reflect the epistemologies of the disciplines from which they emerge. And as long as one stays insular to one’s own discipline, there isn’t a problem.

But as soon as a field claims the mantel of “interdisciplinarity,” it faces a dilemma: to protect and preserve what is known to work, or to open itself out to alternative ways of knowing. Now, both of these impulses are in themselves legitimate in themselves, but as they enter everyday life (e.g., the writing, reviewing, and editing of papers), they sometimes appear in clumsy ways. Some of these clumsy ways are as follows:

• Epistemological bigotry: This happens when someone asserts (often without meaning to) that she or he knows the right way and everything else is “fluff” or wrongheaded. In HCI, scientism is often confused for science, to the detriment of both HCI and science.
• Piecemealism: This happens when someone injects a small piece from one tradition uncritically into another, without recognizing that a piece might not represent the whole from which it is drawn, nor recognizing that that piece might be at intellectual odds with the rest. In HCI, I see this with “critical” approaches to HCI where a single concept is ripped from a complex tradition, such as poststructuralism, and applied to traditional design approaches to, say, mobile phones or Web applications.
• Equivocation: This happens when two or more groups of people use the same word in completely different ways, without seeming to be aware that their use is not “natural” or universal. In HCI, “aesthetics” seems to be a word that has almost no relationship to the 2,500 year old tradition of aesthetic theory, as I’ve ranted on before.

All of these involve a combination of dogmatism and muddled thinking. While scientism–by which I refer to as a fetish for scientific ways of knowing, placing it above other forms of intellectualism–is dogmatic and often intellectually muddled, I would stress that neither dogmatism nor muddled thinking is scientific. Scientism so-defined is bad science.

In this post, I will talk about discourse analysis versus close reading. Both are strategies of textual analysis. Both have disciplinary rigor. Both have legitimate benefits. And yet often when I do close reading, I am attacked on the grounds that I am not being “systematic,” not “coding,” and/or just putting forward my “opinion.” And I want to just scream out: I’ve read Virgil in Latin, Proust in French, Dante in medieval Italian, Joyce in whatever language he wrote in: I don’t need you to tell me how to read! But that is self-expression. It doesn’t solve the broader problem, which is that the rigor I bring to text analysis seems to be literally invisible to these reviewers. Instead, 12 years (!) and a doctorate in a Ph.D. program in Comparative Literature comes off as me just putting forward my “opinion.” I need to address this.

Ironically, and no doubt to the detriment of my tenure case, I think a lot more people read my blog than any of my papers, so I want to use this platform to define both discourse analysis and close reading with the hope of making very clear the following:

• They take different epistemological positions. That means that their way of knowing–their assumptions about how to derive meaning from texts, what meanings are supposedly there in texts, the approaches you use to access them–differ.
• They embody different forms of rigor, and if it is your job to evaluate rigor then it is your responsibility to know how to recognize different strategies of textual analysis and to know how to recognize and evaluate their actual or lack of rigor.
• Their outputs are different. What you learn from discourse analysis is not the same thing as what you learn from a close reading, and each approach lends itself to certain kinds of claims–and also fails to lend itself to other sorts of claims.

My Thesis Statement

All of this leads to my thesis statement, and to make sure no one misses it, or if you just skim this post and only see on thing, then let it be this:

1.  Discourse analysis and close reading are NOT interchangeable

And that implies this:

2.  If someone does a close reading, it does not follow necessarily that they should have done a discourse analysis instead.

With my thesis out of the way, I’m ready to blog-defend it. (A “blog defense” means that this is probably a half-baked and half-assed defense; your recourse is to take me on in comments.) Before I start, I want to share one other value: I try to be generous with concepts, theories, and methodologies. That means that I will attempt a fair-minded summary of both approaches, even though everyone reading this knows which one I like better and am more likely to practice (that said, I have done discourse analysis). But my personal preferences are just that: personal preferences. They do not amount to universal claims or pretenses. Stated directly: I respect discourse analysis as much as I respect close reading, even though I personally practice one more than the other.

On Discourse Analysis

Discourse analysis is a scientific and empirical strategy of textual analysis. At its most basic level, it entails a methodology along these lines:

1. Identify a phenomenon you are concerned with, whose significance is at least partially embodied in texts. Example: FOX, CNN, and MSNBC written news coverage of Obama; mommy blogs; letters to the editor published in your local paper on topic X; Amazon.com customer reviews of Y.
2. Identify the totality of texts available, and identify a significant and representative sample of the whole.
3. Develop a coding system that lets you tag instances of a significant textual feature (e.g., the presence or absence of a feature in a given unit of text).
4. Preferably with multiple people, code the texts using the framework. (I’m hereby skipping summary of establishing intra- and inter-coder reliability, but if you are curious, go read Krippendorf who lays all this stuff out nicely).
5. When you are done with step 4, you now have a numeric representation of your sample. This can now be analyzed statistically.

What this sort of analysis gets you: If you do it well, you have a bona fide empirical snap-shot of your phenomenon. You are in a position to claim what has been said in those texts. You are in a position to observe patterns that are explicitly present, but which may have been hard to see just by reading all the texts. You are also in a position to discover relationships among those patterns: female writers were more likely to A, while male writers were more likely to B; MSNBC coverage was more friendly to liberals, FOX coverage to conservatives, CNN coverage to lipstick celebrities.

Limitations of this analysis: Strongly empirical approaches such as this are very good at exposing what is there. They are less successful at exposing what is “between the lines,” because in a literal way, what is between the lines is not “there” to be found or represented. Now, obvious stuff between the lines is easy enough to unearth–FOX is conservative, MSNBC is liberal, CNN is vapid–but the deeper, juicier stuff can’t be accessed this way. Discourse analysis alone cannot also get at context very well; who said it and why? I’m sure discourse analysis practitioners will contest me on this, but I mean context in much broader and more radical way than I typically see in these sorts of papers: psychoanalytic, ideological, and other complex cultural and/or subcultural contexts are extremely difficult to see using a positivist strategy like discourse analysis.

[Update]: See comments below for a discussion of whether this is a good or fair summary of discourse analysis.

On Close Reading

The term “close reading” is descriptive, not exactly technical. I might have said “humanistic reading” or “interpretive reading” or something like that. Examples of what I am talking about are acts of criticism. Here I don’t mean critical theory but rather close interpretations of single “texts” (“text” here understood as any cultural artifact): Sontag’s interpretation of photographic portraits of herself; Butler’s interpretation of the ethics of torture photographs in the Bush years; Bloom’s interpretation of Plato; Bazin’s interpretation of de Sica’s Bicycle Thieves; Barthes’ interpretation of a photo on the cover of Paris Match; and so on.

A close reading doesn’t involve a set methodology and as such it is very hard to describe. Foolishly perhaps, I nonetheless attempted it here. But the gist of this sort of approach is that an expert (which I will leave undefined here) engages with a text with great care. This engagement typically entails a number of activities: multiple readings/viewings of the text; situating the text in its social and historical contexts; deconstructing the text using a variety of critical strategies (e.g., from Marxism, feminism, poststructuralism, postmodernism, structuralism, reception theory, psychoanalysis); bringing to bear what, if anything, everyone else has said about that text, including interviews with the author/creator, its critical tradition, similar texts (e.g., by the same author/creator); and so forth. Note that this sort of approach is holistic and relies for its success on the expertise of the expert doing it; it is unique, individual, and subjective; it does not follow any disembodied abstract methodology but rather the logic of the scholar-expert in whose hands it is being executed.

What this sort of analysis gets you: A close reading of this sort explores and exposes far more sensitively the complex cultural embeddedness of the text. It gets at matters of aesthetics, craft, and ethics in profound ways. It is capable of revealing much about a text and a community that is neither explicit in the text nor even known to its community. A spectacular example of this is Dick Hebdige’s Subculture: The Meaning of Style, whose analysis of the punk subculture explores the specific historical and operational details of the xenophobic working class underpinnings of punk’s emergence and war on mainstream mass culture. At no point does Hebdige claim that his analysis represents the conscious expressed point of view of a subculture; rather, he explores and reasons about what the unconscious, unspoken point of view seems to be, where it comes from, and what evidence justifies this line of thinking.

Limitations of this analysis: Close readings are strongly inductive and speculative in nature, so what it won’t get you is confidence that you have an objective and correct representation of external reality as it is. Rather, a close reading situates the text against a network of complex ideas and reflections, with the hope of cultivating our capacity to appreciate and understand the source text. Close readings of aesthetic works often call attention to theorization of art to help expose (or even create) its cultural significance–in the most robust possible sense, and for better and/or worse–in the critic’s society. As I have said elsewhere, a critic often models the act of reading, not to reproduce a static understanding in the reader’s head of what is in the critic’s head, but rather to encourage the reader to use similar interpretive strategies both in the original text and in subsequent texts of interest to the reader.

Summary

If you interview 1,000 people coming out of a theatre and transcribe the interviews, you can use discourse analysis to get a real sense of how that film was liked, understood, perceived, etc.

If you read a critical essay about that same film (and here I don’t so much mean newspaper movie reviews but rather scholarly film critiques), you will understand that film’s participation in film, mass media, and everyday culture: its craft, its ideology, its construction of concepts that matter: love, social justice, freedom, sexual liberation, identity, politics, beauty, and so on.

It should be obvious at the very least that both of these kinds of knowledge are legitimate and important, if not always to the same people. If I am a film investor, I absolutely want to understand how moviegoers perceive, experience, and evaluate movies. That is fundamentally an empirical question, and empirical strategies are entirely appropriate. If I am a prospective director, a concerned citizen, a film student or teacher, a film buff, someone who makes decisions about which films should be shown as a part of a community film festival,  and so on, then the film critic’s message is much more likely to resonate with me.

In HCI, we combine all of these audiences. We want to design stuff for commercial success. We want to design things that do what they are supposed to do. Our scientific and empirical approaches are already very good at helping us achieve these goals.

But now we also care about sustainability, felt experience, quality of life, social justice. We have Web 2.0 communities whose emergent behavior literally changes the “meaning” of a system over time. As battles between Web 2.0 communities and their software “owners” (e.g., Facebook, Second Life) have shown, it is not even clear who does or should be responsible for these systems. Thanks to APIs and SDKs, software developers from Adobe and Blizzard to Twitter and Yahoo allow users to redesign interfaces. The emergent UI results are sometimes cannibalized and implemented in future releases of the software. What, then, is an “interface” now, anyway?

These broader questions are much more complex than whether a system is usable or whether users prefer this color scheme to that one. Their complexity in large part lies in the un-articulated and often unseen relationships between and among vastly complex phenomena, from human identity practices to social behavior, from globalisation to the history of art, from emergent user-created interfaces to the incomprehensibility of information produced by user-content creators. These issues cannot be adequately described by scientific reductionism, the way predicting task performance can be. This is not at all to say that scientific reductionism can’t contribute to our understandings in powerful ways–of course it can! But drop the scientism, HCI! It’s not going to meet our needs and it’s lousy science anyway (all dogmatism is). Good science and good critique should complement and reinforce each other. But as long as we categorically dismiss non-scientific strategies, we’re only fake-interdisciplinary and we’re going to botch our work.

And today, bad HCI is more than an unusable Web page–it is unsustainable, socially unjust, culturally irresponsible–and a significant majority of our thousand best users just might miss it.