In this post, I offer four observations in the form of a guide to serving as a good Associate Chair (AC).

[1]

A CHI paper submission typically represents 12-24 months of a research team’s work. That effort deserves the respect of a substantive and effortful review!

A substantive AC review includes the following:

• A summary of the key criticisms and praise of reviewers;
• A substantive articulation of the AC’s own criticisms and praise;
• Thoughtful and constructive suggestions to improve the quality and acceptance chances of future versions of the work.

It’s nearly impossible to do all of the above in 2-3 sentences, so AC reviews do need a certain length to be just and effective.

It is an honor to be selected as an AC, a reflection of your community’s esteem for you. Be worthy of that or let someone else do it who will.

[2]

The AC is responsible for two very different jobs.

1. ACs are charged with helping make a decision about whether or not something is accepted.
2. ACs are also charged with providing constructive and worthwhile feedback to authors both to explain the decision and also to offer suggestions for improving the work (regardless of whether or not it is accepted).

Effective ACs do more of the latter than the former.

[3]

ACs have a second conflicting charge.

• They are tasked with representing the reviewers
• They are tasked with offering their own review

This double-task has some implications:

• Authors deserve to know that their ACs actually read their work and didn’t merely summarize reviews; otherwise, why have ACs?
• ACs need to respect their own reviewers’ reviews; otherwise, why have reviewers?
• If ACs push in a different direction than their own reviewers, they need to
  • Faithfully represent and acknowledge what their own reviewers said, including and especially when they (the ACs) don’t agree with the reviewer(s)
  • Be accountable to their own position by stating very clearly why they disagree with their reviewers
  • Seriously consider asking for new reviewers, a 2AC, and/or discussion

Here is one way this problem creates arbitrary problems in the current system. Let’s say that an AC disagrees with her or his own reviewers and wants to give a paper a 2, even though that AC’s reviewers on average gave it a 4. What does the AC record for her or his score? Some ACs will average their own scores into the reviewers average, in this case offering a meta-review score of 3.5. Others will offer a meta-review score of 2. There is no consistency among ACs (I know this from experience). But since AC meetings use a numeric scale as the primary guide to the preliminary ranking of papers, those who have the first kind of AC will have an advantage over those who have the second kind of AC. That’s arbitrary.

[4]

ACs are responsible for their reviewers’ reviews.

Another way to say this is that ACs must serve as critics of reviewers’ reviews. Reviewers sometimes write poor quality reviews. A poor quality review is one where the critical judgment (accept/reject) is not rationally explained or justified and/or there are no constructive recommendations about how to strengthen the research moving forward (whether or not it is accepted). It’s an AC’s job to try to catch these early and do something about them. Here are some recommendations for dealing with common types of poor quality review.

• Reviewers who rate themselves as “1 No Knowledge” should be seriously reconsidered. A reviewer is a critic, and a critic with no knowledge is not a critic at all. Instead of a defensible judgment, that person can only offer an impressionistic opinion. Imagine if a New York Times book reviewer wrote, “Well, I don’t know anything about contemporary fiction, but your library shouldn’t buy Murakami’s 1Q84, because I didn’t like it.”
• Vacuous reviews of all types should be challenged or replaced. Examples:
  • The 2-sentence review
  • The typo correction review
  • The obsess over one tiny flaw review
  • The “this should go to a different conference so I’m not going to say anything about this paper” review
  • The empty fence-sitter review
• Reviewers who have conflicts of interest are not acceptable. This is true not just when it’s known in advance, but also when it becomes clear subsequently. Conflicts are not just institutional! If a paper critiques a given scholar’s work, then there is a potential conflict in having that person as a peer reviewer.
• Scores and reviews that don’t line up (e.g., a positive review rated 2; a highly critical review rated 4.5) should be explained and/or clarification from reviewers sought.
• Anomalies deserve explanation (e.g., when reviewers give a wide distribution and the AC takes a side–again, that’s OK and our job to do–some explanation is needed so authors can understand why their ACs took that side).

When several of these sorts of low quality judgments occur in the reviews of the same paper–a short AC review, vacuous reviews, anomalous scores–it’s low quality and even meaningless feedback, and yet it has serious consequences.

———

CHI, DIS, CSCW, UIST, etc. are flagship conferences of the field; the papers submitted to them represent hundreds of hours of work; most rejected papers are going to be resubmitted in the future: therefore, out of respect for our own profession, we need to hold ourselves as ACs to high standards, especially until we can create better structural accountability.

It is possible to argue that ACs and reviewers are simply there to verify that your science is good. Antti Oulasvirta has done precisely that, in a lengthy blog about CHI rejections that lays out 8 reasons “why your paper was rejected,” as determined by an AC. His 8 reasons all deal with issues of validity, research design, replicability, etc. Implied but not stated explicitly in Oulasvirta’s post is that if your paper didn’t get in, it’s your fault for doing bad science or reporting as if you had done bad science. Also implied but not stated is that if your paper didn’t get in, it’s not your AC’s fault. Oulasvirta’s self-exculpating post concludes with a cartoon that says, “Shouting at reviewers in your rebuttal is only going to make it worse.” I think this post reflects an assumption that the data (in this case, “your paper”) is presumed to speak for itself, and the reviewer or AC simply sees what is there. If your work lacks construct validity, or your work is not replicable, or your findings are inapplicable, then your reviewer discovers that fact and rejects you. The agency for the outcome is all in your paper.

I disagree with this position. I don’t disagree that papers get rationally rejected because of scientific flaws. I disagree with the implied proposition that the data speaks for itself, that the AC and the reviewers are not responsible for their decision, because the paper itself is.

ACs and reviewers are chosen not on the basis that they represent (as part of a sample) a community, but because they are exceptional: they are presumed to know the topic especially well and to be in a uniquely qualified position to judge a paper. Judging a human-made work to assess its contribution to a community and the world is criticism. It entails a complex and holistic decision that weights factors ranging from the timeliness of the topic, the rigor of the science, the framing of the problem space, the relevance of the contribution, the nature of the contribution, the applicability of the contribution, and so forth. Poor rigor is seldom a winning recipe for publication, of course! At the same time, some flaw or compromise in the rigor can be found in most papers. The issue is not the ability of a reviewer to note the fact that there was such-and-such flaw in the methodology: the issue is how seriously to judge that flaw as a part of the whole contribution.

So in my view, ACs need to stop acting like a paper’s acceptance or rejection is simply causally related to whether that paper adheres to prevailing standards of scientific rigor. ACs make judgements based on their expert point of view: they should and they must. This judgment, because of its consequences, needs a rational justification. This is what the metareview is for–though it is not always effectively used that way. ACs and reviewers need to be accountable to their own position of power–and how responsibly they wield it. And if they fail in that duty, then they deserve to be yelled at in the rebuttal. And if you are an AC or reviewer that gets yelled at in a rebuttal, before browbeating disempowered authors with your copy of The Basics of Social Research, how about a little introspection? Because at CHI today no one else can or will demand it of you, thanks to a combination of (a) reviewer anonymity and (b) structural power imbalance. Let’s explore this:

• Reviewers and ACs decide what is accepted and rejected, both shaping what constitutes mainstream discourse at CHI and also influencing people’s careers (tenure, etc.). Authors have no such power over reviewers or ACs.
• Reviewers and ACs each can write as much as they want about a paper. Authors can reply via a one-time rebuttal with up to 5,000 characters (not words!) to all ACs and reviewers. Reviewers can write as much as they want in reply to rebuttals, but authors can’t see or reply to these responses. ACs and reviewers can hold secret conversations that authors are not privy to. ACs meet in a secret meeting to make final decisions, and questions and comments are raised in that meeting that authors will never hear or have any chance to respond to.
• Authors don’t know who these people are who are wielding this power over them.

Now I am aware of all the practical reasons that have led to such structural outcomes. I am not declaring a need to replace them with a new process (though personally I support both the CSCW’12 and alt.chi’08 and ’12 models). But I want to make one thing very clear: ACs and reviewers have very little structural accountability in the current CHI Papers and Notes model. If ACs or reviewers make an honest mistake, or, worse, if they are negligent through laziness, intellectual narrowmindedness, or personal conflicts, there is very little consequence for them, opportunity to correct it, or recourse for authors.

Hence my main thesis: if we continue to use the existing reviewing structure, then to ensure the most rational reviewing process, we need to work proactively to ensure that ACs and reviewers are competent and act with integrity inasmuch as they are critics (a job separate from whatever their area of subject expertise is).

In the spirit of contributing to this work, I have taken the time to lay out some of the key expectations every AC should be held to. To see them, continue on to Part 2.

The original critique is here (and has some very interesting analysis beyond its basic premise):
http://printmag.com/Article/An-Anatomy-of-Uncriticism

And the Salon article on the dust-up is here:
http://www.salon.com/2012/01/12/design_critique_imprint/

One thing at stake in all this is how people perceive the role of “critique” in design. Often, people assume that critique is saying bad things about something else. That’s how we use the term “criticism” in every day life. But as the Salon article notes, the purpose of serious critique (including design criticism) is to illuminate, not to scold. The relevant quote is here:

If the criterion for what warrants design criticism is based on a level of social, cultural or political impact, then a particular work is fair game regardless of the age or virtuosity of its maker. Since criticism is not meant to be a scold, but is rather a means of illuminating — delving below the surface — finding aspects of work that benefits by explanation and analysis, nothing and no one should be exempt.

There are two gems in that quote worth pointing out. The first I already have, which is that criticism is about illuminating a community, not about attacking a person or artifact. The second point reinforces the first: the role of criticism is not simply to call attention to flaws, but rather it is about “delving below the surface” to help explain or analyze the “social, cultural or political impact” of a design or event. Since we all must live with the social, cultural, and political impacts of designs, we all have a stake in their critique, with no exemptions.

Note: This post is based on an entry I posted earlier at the Interaction Culture Class blog.

Maybe two or three years ago, I realized that I didn’t know enough about science to do what I really wanted to do, and so I started reading from the vast literature in the history and philosophy of science. I’ve really enjoyed it; among other things, it has helped this humanist come to appreciate and love science. And in the past 18 months or so, these readings have started to bubble out into my teaching and publications–and I think their introduction to my work has not only strengthened it from my own perspective, but I think readers and students have also responded to aspects of my work and teaching that takes the time out to explain some of these terms.

The positive reaction to my use of core scientific philosophy took me a bit by surprise, because my own “impostor syndrome” had me convinced that everyone but me already knew this stuff. But increasingly, I’m realizing that many people know how to do science with rigor, but they can’t always clearly articulate their own, say, epistemology–at least on short notice. Sometimes, they inadvertently treat their own ways of knowing as a normative standard, i.e., that all knowledge production should have the same indicators of “rigor” that theirs does. But when I start to talk through these issues using basic scientific technical vocabulary in a reasonably precise way, it excites them–and that really facilitates good dialogue, which is what we’re all about.

Here are some very simple examples of what I am talking about:

• What is the difference between a “scientific explanation” and a “description”?
• We all know that astrology is not a science, but can we say exactly why that is so?
• Is positivism a good or a bad thing? How has “post-positivism” replaced “logical positivism,” and how can we tell which form of positivism a given paper today is using?
• Should the social sciences be able to achieve the same standards of clarity and progress that the natural science have, and why or why not?
• What are the relationships among laws, theories, and models?
• What is the difference between methods and methodologies?
• What are the difference among common theories of truth: correspondence, pragmatist, coherence, etc.?
• Was Kuhn right about scientific paradigms, normal science, etc.?
• What does a rigorous scientific process look like? How about a rigorous scientific product?

This is basic stuff. But it’s also really important stuff, because more often than not, scientific paper submissions and grant proposals are evaluated against a tacit understanding of how well they measure up against scientific ideals. Yet, when we ourselves are fuzzy on these ideals–a problem that I see especially in the social sciences, which have a complex relationship to natural science, quantitative statistics, etc., on the one hand, and the humanities, critical theory, and design on the other–it becomes difficult to measure submissions against scientific ideals with the rigor and intersubjective agreement they deserve.

Recently, a scientist friend of mine heard that I was an “expert” in these things. (Such is demonstrably not the case, since I was merely seeking to address a gaping deficiency in my own knowledge; I will own up now to being an enthusiastic amateur in matters pertaining to the philosophy of science but no more. Maybe when I grow up.) And he asked me for some references, and it occurred to me that other people might benefit from some of this as well.

The history and philosophy of science, like any other academic field, is huge and daunting. At times, it has its own highly technical vocabulary and ways of doing, which are hard for outsiders to get into. Of course, Wikipedia covers many of these topics, but for obvious reasons Wikipedia defines a lot of these terms without explaining them. To me, encountering a definition without really explaining it either requires considerable investment on my part to get anything from it, or it just dribbles right back out of my head (<– most common outcome). Moreover, I’ve found that sometimes a concept in Wikipedia will be expressed from the perspective of a particular science, e.g., physics, that makes it a little bit hard to generalize from.

More usefully, there are a number of introductory anthologies that either (a) survey the key ideas, in chapters written recently for that introductory volume (i.e., they are accessible, but not famous or foundational chapters), or (b) collect seminal writings, mostly from the twentieth century (i.e., they are famous and foundational, but they are not reliably accessible to non-specialist readers). Both types of introductory volume have their benefits. All of them will address the sorts of issues laid out in the bullet list above.

Here is an example of the first type:
http://www.amazon.com/Blackwell-Guide-Philosophy-Science-Guides/dp/0631221085/ref=sr_1_1?ie=UTF8&qid=1286208464&sr=8-1

Here are two examples of the second type:
http://www.amazon.com/Introductory-Readings-Philosophy-Science-Klemke/dp/1573922404/ref=sr_1_1?ie=UTF8&s=books&qid=1286208497&sr=1-1

http://www.amazon.com/Philosophy-Science-Central-J-Cover/dp/0393971759/ref=pd_sim_b_8

All of the above cover both natural and social sciences, often with an emphasis on natural, but there is a book specifically on the social sciences that I really like (which is not to say that I agree with its argument):
http://www.amazon.com/Philosophical-Foundations-Social-Sciences-Controversies/dp/0521558913/ref=sr_1_1?ie=UTF8&s=books&qid=1286208549&sr=1-1

Clearly, this post is not an earth-shattering contribution to interaction design, but if you are an empirical scientist or an aspiring one, or if, like me, you work with empirical scientists, it’s good to have a fairly explicit and articulatable facility with these kinds issues. If nothing else, a working knowledge of core issues in the philosophy of science demystifies “scientific rigor,” which may help practitioners design better user studies, and it may help researchers speak more carefully about their taken-for-granted ideals.

Anyway, I recently composed a position paper that started as a rant on this very blog. The topic is on the essay and HCI, and specifically why HCI ought to embrace–rather than marginalize–the essay as a contribution type.

Download it here. (PDF, 84 kb)

As a long overdue service to readers (and also to help me get motivated to revise it and send it somewhere as a proper paper), I have compiled all seven parts, ever so lightly edited them, and now present them in a single downloadable form. Hopefully, it is more user friendly now.

Bardzell – Interaction Criticism: How To Do It

UPDATE: The above link was broken for a while, but it has since been fixed.

Paper (ACM digital library, subscription required)
http://portal.acm.org/citation.cfm?doid=1518701.1519063

Slides (PDF, 13 MB, no subscription–should download right away) UPDATE: This URL was dead for a while and has now been fixed.
http://dl.getdropbox.com/u/1874332/InteractionCriticismAndAesthetics.pdf

Thanks to the many people who helped me with this!

So my original gripe in that post is that people in HCI sometimes seem to think that unless one does some form of coding, one’s textual analysis doesn’t deserve the name, and one is instead merely advancing an opinion. The point I’m advancing in this post is its corollary: Unless one writes a recognizable scientific paper (intro, lit review, methodology, results, discussion, conclusion) one likewise runs the risk of being seen as merely advancing one’s opinions or writing (as one reviewer once accused me) like a journalist.

This calls to mind my regularly repeated rant that in CHI, the top conference of our field, there are eight “contribution types,” and these include “Development or Refinement of Interface Artifacts or Techniques”; “Systems, Tools, Architecture, and Infrastructure”; “Methodology”; “Theory” and others. The eighth of eight is “Opinion.” Now, clearly Opinion is the slot for essays; its accompanying description uses the term “provocative essay.” Because many trained scientists seem to hold the position that subjective = opinion = lack of rigor = not knowledge, I wrote a post distinguishing between opinion and judgment, and I’ll let that do its work and not restate it all here. Suffice it to say that I don’t believe many in HCI have a robust understanding of the nature and contribution of the critical essay.

What I want to do in this post is highlight the nature and goals of the essay as a form of discourse. It is structurally, substantially, and even epistemologically distinct from a typical scientific paper. This is not to suggest that one is better than the other, but rather only to suggest that essays cannot be evaluated on the same terms as scientific papers (and vice-versa, of course). Incidentally, Wikipedia has an entry on the essay, so if you want something more comprehensive, go read that. But I am going to share some quotes from one of my favorite essayists, Phillip Lopate. I offer this not as a final word on the essay in HCI but just to help HCIers recognize one when it lands in their laps and hopefully also to be able to evaluate them fairly.

Following are a bunch of quotes pulled from the beginning of Lopate’s essay, “In Search of the Centaur: The Essay-Film,” included in his collection Totally, Tenderly, Tragically. The essay considers why the essay-film isn’t more common than it is, and it works through definitions of its key terms and contemplates both films that broadly meet the criteria and those that are like essay-films but are not. But I’m not worried about the essay-film here, and so I focus only on Lopate’s attempts to define the essay:

the essay offers personal views. That’s not to say it is always first-person or autobiographical, but it tracks a person’s thoughts as he or she tries to work out some mental knot, however various its strands. An essay is a search to find out what one thinks about something.

Often the essay follows a helically descending path, working through preliminary supposition to reach a more difficult core of honesty.

This gets right at why I think critical essays are subjective and why it is so important not to fetishize objective understanding. I am not here rehearsing the Kantian argument that there is no such thing as objective knowledge that has taken such prominence in postmodernism. Rather, I am stressing a form of knowledge production whose source is not in empirical reality at all, but rather which is fundamentally embedded in the subject’s sensemaking, interpretive, and reflective practices. This sort of knowledge cannot be found in external reality, and so it is absurd to suggest that objective, empirical, or scientific approaches would be more effective in unconvering it.

I also want to stress the temporal unfolding of such knowledge: it is not a static representation of a state of affairs; it is a process of engaging, which has emergent outcomes. The essayist is likely to disagree with herself over time; indeed, Foucault’s response to critics who pointed out the inconsistencies in his work was to say (very roughly paraphrasing): “Of course! I write in order to change myself. At the end of a book, I am not the same person that began it. It would be a boring waste of time otherwise.” Back to Lopate on the essay:

Montaigne’s “What do I know?” is a mental freedom and cheekiness in the face of fashion and authority. The essayist wears proudly the confusion of an independent soul trying to grope in isolation towards truth.

This quote underscores the non-replicability, non-objectivity of the essay. The essayist has a voice, which is cheeky, confused, groping, truth-oriented but not arrived. To dismiss the essayist as merely providing her point of view utterly defeats its purpose. One can of course demand an accounting for that point of view (and the earlier quote, which stressed the temporal working out of a knot, suggests at least what that looks like), but the cultivation and expression of a point of view is arguably the essay’s raison-d’être.

Adorno, in “The Essay as Form,” saw precisely the anti-systematic, subjective, nonmethodic method of the essay as its radical promise.

Do I really need to say this? What Adorno is obviously contrasting the essay to is science. This is by no means a rejection of science! It is, rather, an effort to develop and articulate alternative practices of knowledge production, which are both non-scientific and nonetheless rigorous and legitimate.

Note also the word “radical” in that sentence. Radical implies an effective activism that actually will lead to substantive, desirable change in the world. Criticism generally has a progressive orientation, which seeks to complement the often conservative outcomes of science (that is too rich a claim to defend here, but short version: whose agenda does science typically serve?).

Whatever twists and turns occur along its path, and however deep or moral its conclusions, an essay will have little enduring interest unless it also exhibits a certain sparkle or stylistic flourish…. Freshness, honesty, self-exposure, and authority must all be asserted in turn. An essayist who produces magisterial and smoothly ordered arguments but is unable to surprise himself in the process of writing will end up boring us. An essayist who is vulnerable and sincere but unable to project any authority will seem, alas, merely pathetic and forfeit our attention. So it is a difficult game to pull off. Readers must feel included in a true conversation, allowed to follow through mental processes of contradiction and digression, yet be aware of a formal shapeliness developing simultaneously underneath.

I have spoken about the importance of the critic’s voice, and here Lopate develops the idea and puts some flesh to it.

I now give Lopate the final word, and he, in turn, passes the final word onto the great Marxist critic György Lukács. As you read this final quote, in light of the other quotes that precede it, consider how this relates to a field like HCI, with its interface design, user research, and growing awareness of its own socio-cultural responsibility (e.g., sustainability, aesthetics, etc.):

An essay is a continual asking of questions–not necessarily finding “solutions,” but enacting the struggle for truth in full view. Lukács, in his meaty, “On the Nature and Form of the Essay,” wrote: “The essay is a judgment, but the essential, the value-determining thing about it is not the verdict (as is the case with the system) but the process of judging.”

Textual analysis is fundamental to many kinds of research, from psychology to literature, philosophy to information science. Not surprisingly, different strategies have emerged from within the various disciplines that do textual analysis, and naturally these strategies reflect the epistemologies of the disciplines from which they emerge. And as long as one stays insular to one’s own discipline, there isn’t a problem.

But as soon as a field claims the mantel of “interdisciplinarity,” it faces a dilemma: to protect and preserve what is known to work, or to open itself out to alternative ways of knowing. Now, both of these impulses are in themselves legitimate in themselves, but as they enter everyday life (e.g., the writing, reviewing, and editing of papers), they sometimes appear in clumsy ways. Some of these clumsy ways are as follows:

• Epistemological bigotry: This happens when someone asserts (often without meaning to) that she or he knows the right way and everything else is “fluff” or wrongheaded. In HCI, scientism is often confused for science, to the detriment of both HCI and science.
• Piecemealism: This happens when someone injects a small piece from one tradition uncritically into another, without recognizing that a piece might not represent the whole from which it is drawn, nor recognizing that that piece might be at intellectual odds with the rest. In HCI, I see this with “critical” approaches to HCI where a single concept is ripped from a complex tradition, such as poststructuralism, and applied to traditional design approaches to, say, mobile phones or Web applications.
• Equivocation: This happens when two or more groups of people use the same word in completely different ways, without seeming to be aware that their use is not “natural” or universal. In HCI, “aesthetics” seems to be a word that has almost no relationship to the 2,500 year old tradition of aesthetic theory, as I’ve ranted on before.

All of these involve a combination of dogmatism and muddled thinking. While scientism–by which I refer to as a fetish for scientific ways of knowing, placing it above other forms of intellectualism–is dogmatic and often intellectually muddled, I would stress that neither dogmatism nor muddled thinking is scientific. Scientism so-defined is bad science.

In this post, I will talk about discourse analysis versus close reading. Both are strategies of textual analysis. Both have disciplinary rigor. Both have legitimate benefits. And yet often when I do close reading, I am attacked on the grounds that I am not being “systematic,” not “coding,” and/or just putting forward my “opinion.” And I want to just scream out: I’ve read Virgil in Latin, Proust in French, Dante in medieval Italian, Joyce in whatever language he wrote in: I don’t need you to tell me how to read! But that is self-expression. It doesn’t solve the broader problem, which is that the rigor I bring to text analysis seems to be literally invisible to these reviewers. Instead, 12 years (!) and a doctorate in a Ph.D. program in Comparative Literature comes off as me just putting forward my “opinion.” I need to address this.

Ironically, and no doubt to the detriment of my tenure case, I think a lot more people read my blog than any of my papers, so I want to use this platform to define both discourse analysis and close reading with the hope of making very clear the following:

• They take different epistemological positions. That means that their way of knowing–their assumptions about how to derive meaning from texts, what meanings are supposedly there in texts, the approaches you use to access them–differ.
• They embody different forms of rigor, and if it is your job to evaluate rigor then it is your responsibility to know how to recognize different strategies of textual analysis and to know how to recognize and evaluate their actual or lack of rigor.
• Their outputs are different. What you learn from discourse analysis is not the same thing as what you learn from a close reading, and each approach lends itself to certain kinds of claims–and also fails to lend itself to other sorts of claims.

My Thesis Statement

All of this leads to my thesis statement, and to make sure no one misses it, or if you just skim this post and only see on thing, then let it be this:

1.  Discourse analysis and close reading are NOT interchangeable

And that implies this:

2.  If someone does a close reading, it does not follow necessarily that they should have done a discourse analysis instead.

With my thesis out of the way, I’m ready to blog-defend it. (A “blog defense” means that this is probably a half-baked and half-assed defense; your recourse is to take me on in comments.) Before I start, I want to share one other value: I try to be generous with concepts, theories, and methodologies. That means that I will attempt a fair-minded summary of both approaches, even though everyone reading this knows which one I like better and am more likely to practice (that said, I have done discourse analysis). But my personal preferences are just that: personal preferences. They do not amount to universal claims or pretenses. Stated directly: I respect discourse analysis as much as I respect close reading, even though I personally practice one more than the other.

On Discourse Analysis

Discourse analysis is a scientific and empirical strategy of textual analysis. At its most basic level, it entails a methodology along these lines:

1. Identify a phenomenon you are concerned with, whose significance is at least partially embodied in texts. Example: FOX, CNN, and MSNBC written news coverage of Obama; mommy blogs; letters to the editor published in your local paper on topic X; Amazon.com customer reviews of Y.
2. Identify the totality of texts available, and identify a significant and representative sample of the whole.
3. Develop a coding system that lets you tag instances of a significant textual feature (e.g., the presence or absence of a feature in a given unit of text).
4. Preferably with multiple people, code the texts using the framework. (I’m hereby skipping summary of establishing intra- and inter-coder reliability, but if you are curious, go read Krippendorf who lays all this stuff out nicely).
5. When you are done with step 4, you now have a numeric representation of your sample. This can now be analyzed statistically.

What this sort of analysis gets you: If you do it well, you have a bona fide empirical snap-shot of your phenomenon. You are in a position to claim what has been said in those texts. You are in a position to observe patterns that are explicitly present, but which may have been hard to see just by reading all the texts. You are also in a position to discover relationships among those patterns: female writers were more likely to A, while male writers were more likely to B; MSNBC coverage was more friendly to liberals, FOX coverage to conservatives, CNN coverage to lipstick celebrities.

Limitations of this analysis: Strongly empirical approaches such as this are very good at exposing what is there. They are less successful at exposing what is “between the lines,” because in a literal way, what is between the lines is not “there” to be found or represented. Now, obvious stuff between the lines is easy enough to unearth–FOX is conservative, MSNBC is liberal, CNN is vapid–but the deeper, juicier stuff can’t be accessed this way. Discourse analysis alone cannot also get at context very well; who said it and why? I’m sure discourse analysis practitioners will contest me on this, but I mean context in much broader and more radical way than I typically see in these sorts of papers: psychoanalytic, ideological, and other complex cultural and/or subcultural contexts are extremely difficult to see using a positivist strategy like discourse analysis.

[Update]: See comments below for a discussion of whether this is a good or fair summary of discourse analysis.

On Close Reading

The term “close reading” is descriptive, not exactly technical. I might have said “humanistic reading” or “interpretive reading” or something like that. Examples of what I am talking about are acts of criticism. Here I don’t mean critical theory but rather close interpretations of single “texts” (“text” here understood as any cultural artifact): Sontag’s interpretation of photographic portraits of herself; Butler’s interpretation of the ethics of torture photographs in the Bush years; Bloom’s interpretation of Plato; Bazin’s interpretation of de Sica’s Bicycle Thieves; Barthes’ interpretation of a photo on the cover of Paris Match; and so on.

A close reading doesn’t involve a set methodology and as such it is very hard to describe. Foolishly perhaps, I nonetheless attempted it here. But the gist of this sort of approach is that an expert (which I will leave undefined here) engages with a text with great care. This engagement typically entails a number of activities: multiple readings/viewings of the text; situating the text in its social and historical contexts; deconstructing the text using a variety of critical strategies (e.g., from Marxism, feminism, poststructuralism, postmodernism, structuralism, reception theory, psychoanalysis); bringing to bear what, if anything, everyone else has said about that text, including interviews with the author/creator, its critical tradition, similar texts (e.g., by the same author/creator); and so forth. Note that this sort of approach is holistic and relies for its success on the expertise of the expert doing it; it is unique, individual, and subjective; it does not follow any disembodied abstract methodology but rather the logic of the scholar-expert in whose hands it is being executed.

What this sort of analysis gets you: A close reading of this sort explores and exposes far more sensitively the complex cultural embeddedness of the text. It gets at matters of aesthetics, craft, and ethics in profound ways. It is capable of revealing much about a text and a community that is neither explicit in the text nor even known to its community. A spectacular example of this is Dick Hebdige’s Subculture: The Meaning of Style, whose analysis of the punk subculture explores the specific historical and operational details of the xenophobic working class underpinnings of punk’s emergence and war on mainstream mass culture. At no point does Hebdige claim that his analysis represents the conscious expressed point of view of a subculture; rather, he explores and reasons about what the unconscious, unspoken point of view seems to be, where it comes from, and what evidence justifies this line of thinking.

Limitations of this analysis: Close readings are strongly inductive and speculative in nature, so what it won’t get you is confidence that you have an objective and correct representation of external reality as it is. Rather, a close reading situates the text against a network of complex ideas and reflections, with the hope of cultivating our capacity to appreciate and understand the source text. Close readings of aesthetic works often call attention to theorization of art to help expose (or even create) its cultural significance–in the most robust possible sense, and for better and/or worse–in the critic’s society. As I have said elsewhere, a critic often models the act of reading, not to reproduce a static understanding in the reader’s head of what is in the critic’s head, but rather to encourage the reader to use similar interpretive strategies both in the original text and in subsequent texts of interest to the reader.

Summary

If you interview 1,000 people coming out of a theatre and transcribe the interviews, you can use discourse analysis to get a real sense of how that film was liked, understood, perceived, etc.

If you read a critical essay about that same film (and here I don’t so much mean newspaper movie reviews but rather scholarly film critiques), you will understand that film’s participation in film, mass media, and everyday culture: its craft, its ideology, its construction of concepts that matter: love, social justice, freedom, sexual liberation, identity, politics, beauty, and so on.

It should be obvious at the very least that both of these kinds of knowledge are legitimate and important, if not always to the same people. If I am a film investor, I absolutely want to understand how moviegoers perceive, experience, and evaluate movies. That is fundamentally an empirical question, and empirical strategies are entirely appropriate. If I am a prospective director, a concerned citizen, a film student or teacher, a film buff, someone who makes decisions about which films should be shown as a part of a community film festival,  and so on, then the film critic’s message is much more likely to resonate with me.

In HCI, we combine all of these audiences. We want to design stuff for commercial success. We want to design things that do what they are supposed to do. Our scientific and empirical approaches are already very good at helping us achieve these goals.

But now we also care about sustainability, felt experience, quality of life, social justice. We have Web 2.0 communities whose emergent behavior literally changes the “meaning” of a system over time. As battles between Web 2.0 communities and their software “owners” (e.g., Facebook, Second Life) have shown, it is not even clear who does or should be responsible for these systems. Thanks to APIs and SDKs, software developers from Adobe and Blizzard to Twitter and Yahoo allow users to redesign interfaces. The emergent UI results are sometimes cannibalized and implemented in future releases of the software. What, then, is an “interface” now, anyway?

These broader questions are much more complex than whether a system is usable or whether users prefer this color scheme to that one. Their complexity in large part lies in the un-articulated and often unseen relationships between and among vastly complex phenomena, from human identity practices to social behavior, from globalisation to the history of art, from emergent user-created interfaces to the incomprehensibility of information produced by user-content creators. These issues cannot be adequately described by scientific reductionism, the way predicting task performance can be. This is not at all to say that scientific reductionism can’t contribute to our understandings in powerful ways–of course it can! But drop the scientism, HCI! It’s not going to meet our needs and it’s lousy science anyway (all dogmatism is). Good science and good critique should complement and reinforce each other. But as long as we categorically dismiss non-scientific strategies, we’re only fake-interdisciplinary and we’re going to botch our work.

And today, bad HCI is more than an unusable Web page–it is unsustainable, socially unjust, culturally irresponsible–and a significant majority of our thousand best users just might miss it.

Defining the Hermeneutic Circle

Just so everyone knows what I am going on about here, let me first attempt a definition, and then I’ll develop the concept in two different directions. The hermeneutic circle refers to the situation in which when we encounter a text (i.e., any cultural phenomenon) we can only understand it (i.e., make sense of it) with reference to other texts, and in turn our understanding of these other texts is modified by our understanding of this text.

In other words, to understand a given expression, we must understand the language in which it is written.  “Language” here means more than English or Chinese. For example, to understand an expression such as Shakespeare’s Antony and Cleopatra, one must have some competence with Western tragedy in general, Shakespearean tragedy in particular, Roman history, poetry (e.g., meter), Renaissance culture, and so forth. Yet these linguistic contexts are all made out of other texts, which can only be understood the same way. We face just such a problem in HCI when we interpret and/or evaluate an interaction: the criteria by which we evaluate or interpret an interaction are determined by concepts and theories ingrained in our field, and these in turn are derived from, among other things, interpretations and evaluations of other interactions.

In this post, I want to explore two takes on the hermeneutic circle, a pessimistic view and an optimistic view. I will also make at least token efforts to relate this discussion to HCI.

Take 1: The Pessimistic View

The hermeneutic circle seems to interfere with, if not absolutely preclude the possibility of, objective knowledge. This is so because our reasoning would seem to be circular. The only way to justify a reading (by which I mean “serious act of sensemaking”) is to point to other readings, which themselves need justification in the same way, leading to an infinite regress argument.

Another practical implication is that it is hard to enter one of these circles, since every possible entry point seems to take for granted the whole. You can’t understand Shakespeare without being in a position to understand Shakespeare; how can you get in such a position without reading Shakespeare?

Since the hermeneutic circle, on this view, is bad, we should try to break out of it. Philosopher Charles Taylor, in his essay “Interpretation and the Sciences of Man” explores this argument. He writes that one way people have tried to break out of it is by taking recourse to empirical science. Empiricism is, he writes,

a genuine attempt to go beyond the circle of our own interpretations, to get beyond subjectivity. The attempt is to reconstruct knowledge in such a way that there is no appeal to readings or judgments which cannot be checked further. That is why the basic building block of knowledge on this view is the impression, or sense datum, a unit of information which is not the deliverance of a judgment, which has by definition no element in it of reading or interpretation, which is a brute datum. The highest ambition would be to build our knowledge from such building blocks by judgments which could be anchored in a certainty beyond subjective intuition…. By “brute data,” I mean … data that cannot be questioned by offering another interpretation or reading, data whose credibility cannot be founded on or undermined by further reasoning.

On this view, empiricism (“let the data speak for itself”) offers an alternative to the hermeneutic circle, a way of knowing that is not entangled in it. We see an example of this sort of reasoning in Csikszentmihalyi & Robinson’s The Art of Seeing:

An analysis of any sort begins with a description of the phenomena under study. Yet a thoroughgoing and empirically grounded description of the aesthetic experience has been conspicuously absent from aesthetic theory–of whatever stripe–in the past. For the most part, aesthetic study has proceeded from a priori assumptions concerning what the aesthetic experience must be or the basis of the analyst’s own experiences.

So the authors are promising to do what 2,500 years of aesthetic theory have failed to do, namely, offer a rigorous description of the aesthetic experience (because apparently only empirical study is “thoroughgoing” and gets past “assumption”). They promise to replace empty theorizing with good, empirical evidence.

But pure empiricism has problems. As Wittgenstein, Quine, and other philosophers of science in the twentieth century have demonstrated, pure empiricism does not work, even in physics, let alone the human sciences. A hypothesis cannot exist without a theory already in place to give it coherence: How is the phenomenon studied? How is its data analyzed? On what basis is this hypothesis deemed important in the first place?

Additionally, Taylor notes, the brain simply doesn’t work that way to begin with, that is, pure empiricism itself entails a theory of perception that is outmoded. Finally, Taylor observes that for this kind of empiricism to work, reality itself must be structured in a certain way for it to be available to this kind of perception and knowledge building, i.e., it must be made available in a comprehensive and truthful way to human perception.

So how did Csikszentmihalyi & Robinson do? Using a grounded theory approach (note that grounded theory is based on a priori assumptions about what empirically informed theory must be), they interview nearly 70 museum curators (this “population” was “selected” based on established social science methodologies). So before they interviewed a single subject, their study itself is situated within the hermeneutic circle of empirical social science. How about their data? Their analysis of the data revealed that the curators express themselves according to four interrelated dimensions of aesthetic experience: perceptual, emotional, intellectual, and communicative. This may be surprising, inasmuch as it looks more like the table of contents of a psychology textbook than a work on aesthetics; the authors have minimally delivered on their promise of not revealing aesthetics as usual, but it seems fair to question whether they instead delivered psychology as usual.

How about the “brute data” itself? The authors offer quotes about how the curators perceived art. Here is an example: one curator describes the “color and forms,” “paint manupulation,” “dry, chalky lines,” and “stuccolike surfaces” of a painting; that sounds perceptual. The problem is interlaced among these terms were references to this work as “a perfect Cubist picture” and “that you associate with Cubist paintings.” In short, perception itself, by Csikszentmihalyi & Robinson’s own account was mediated by the hermeneutic circle: the painting was rendered perceptible to the curator on the basis of its participation in the Cubist tradition. Of course, the “Cubist tradition” refers to a bunch of subjective theorizing by artists, critics, and aesthetic philosophers; it certainly isn’t an empirically derived category. In short, Csikszentmihalyi & Robinson have not gotten out of the hermeneutic circle, as promised in their rhetoric; all they have done was empirically demonstrate that museum curators are themselves inside of a hermeneutic circle (i.e., art history), suggesting that all those dusty “a priori assumptions” that they were seeking to circumvent in fact are the basis of aesthetic experience.

Now, HCI is an empirical field, and its rhetoric sometimes resembles that of Csikszentmihalyi & Robinson. But as I have argued here, empiricism does not exempt anyone from the hermeneutic circle. Perhaps HCIers will resist this, but evidence of the presence of the hermeneutic circle in our field abounds. First, the idea that HCI is a “field”–what does that mean, if not that we all share a common general understanding (i.e., theory) of what HCI is? Such a shared theory is necessary for degrees in HCI to have meaning, for job descriptions to recruit the right sorts of people, and so on. We prefer certain kinds of research processes: empirical research is privileged over critical essays, for example. Equally, we as a field prefer certain research methods and epistemologies over others. Second, imagine a CHI paper with no references. With a pure model of empiricism, references would not only be superfluous, they would be corrupting! Yet how many of us when we submit are castigated for not having this-or-that paper in our references? This is an assertion of the hermeneutic circle (along with a second assertion that our work is outside of it). Or consider a criticism about an inappropriate reference: such a criticism amounts to a claim that the cited reference or definition doesn’t belong in the hermeneutic circle, or at least not in the way presented.

So it seems that the pessimistic view is well named; it views the hermeneutic circle in negative terms, and it is unable to get out of it.

Take 2: The Optimistic View

The optimistic view would seem to start with a few basic positions. First, pure empiricism is impossible and the hermeneutic circle is inevitable, and yet science has not faltered and dissolved. It seems that scientific progress is possible even if it is stuck within the hermeneutic circle. So, it may be that Csikszentmihalyi & Robinson’s claims to a privileged understanding of the aesthetic experience were delusional, but nonetheless they obviously offer genuine insight on how curators experience art, and in particular they reveal (even if inadvertently) how important aesthetic theory is to art appreciation, and in which sorts of ways. Above all, they help us see how a model audience appreciates art, a contribution to knowledge that benefits art educators, museum exhibit designers, art industry marketers, art critics, and so on.

A second benefit of a given field’s hermeneutic circle is that it renders itself visible. Let me explain that. Anyone remotely familiar with ubiquitous computing knows that Mark Weiser’s “The Computer in the 21st Century” was a seminal essay; everyone cites it and many people define themselves in relation to it. Because we know that this paper is a formative contributor to ubicomp, (a) we know that we need to teach this essay to graduate students in HCI; and (b) we know that anything obsolete, wrong, or brilliantly right about that essay are likely to have repercussions in ubicomp. In short, recognizing the hermeneutic circle(s) in which you operate de-naturalizes your work and renders it visible to reflection and critique. That in turn may prevent you from making dumb claims like you are going to fix the error made by 2,500 of years of philosophy going back to Plato by having your graduate students interview 67 people, and instead making claims about what your contribution actually is (e.g., help making visible how model or expert art viewers experience art).

The essence of the optimistic view, though, goes back to Gadamer (and was inadvertently given support in Csikszentmihalyi & Robinson). For Gadamer, our participation in the hermeneutic circle extends our perceptual horizons. Not only can I perceive what my senses make visible, but I can perceive the world making use of concepts that I have acquired by participating in a hermeneutic circle. We see this in the Csikszentmihalyi & Robinson paper where a curator expresses what she or he perceived in terms of Cubism. Without knowledge of Cubism, this curator may not have even noticed the chalkiness or stucco-texture of the paint. The perceptual qualities of the paint are only visible, significant, and interesting when viewed with an understanding of Cubism.

This line of reasoning obviously applies to HCI. I don’t have to reinvent “usability”; I can read about it, understand what it is, why it matters, how it has been evaluated, and it becomes a part of the repertoire with which I view, interpret, and experience a design. The concept of usability, therefore, is like an extra set of eyes, a new sense. Using Dewey’s and later anthropological notions of “experience,” I am able to reflect more thoughtfully, precisely, and substantially on my own experiences with interaction designs. This also means that HCI does not have to reinvent aesthetics, but rather can avail itself of aesthetics’ existing tradition, using it to extend our capacity in HCI to engage with beautiful, artistic interaction.

In short, the hermeneutic circle makes it possible for me to see and to participate in HCI as a field. It enables me to leverage the contributions of thousands of people in my own work, amplifying–or rather creating the very possibility of–my capacity to produce knowledge in the field. It also offers my colleagues a basis on which to critique and thereby help me improve my work. Finally, it offers all of us the opportunity to critique not just seminal papers, but the rippling of their effects across the field. In the philosophy of science, the realization that objective empiricism does not work (at least in a pure sense) led to the development of theorizing around scientific practices (such as triangulation) that seek to address the intrinsic weaknesses of pure empiricism in a way that allows empirical approaches to move forward on surer footing.

The hermeneutic circle is not a trap to avoid, but rather an opportunity to participate in the constructive development of our field. Our field, like all scientific fields, is hermeneutic, but that does not mean it cannot continue to rely on empirical strategies. It merely means that we should not be delusional about how this knowledge is produced.