As the CHI deadline approaches I thought it might be handy to share a heuristic I find useful while writing. Note I’m not particularly successful at getting into CHI, but I have a lot of experience of failure. Bearing that in mind, here’s my tip:
- Try and channel your inner CHI psycho-reviewer
I’m sure we’ve all had reviewers’ comments that make you doubt the sanity of the writer. There are those that are almost willfully obtuse in failing to see what you are talking about. There are the egomaniacs who only read anything in terms of how it relates to their research and rate everything accordingly. And there are the members of the Methodological Inquisition who sniff out any hints of experimental heresy. With most of these there is little you can do other than to hope you don’t get too many.
But there are some slightly more normal people reviewing, whose reactions can be predicted and where a bit of rewriting and reframing can help in the process of reviewer-proofing. It can help to use the lawsuit metaphor that we already borrow from in talking about a thesis ‘defense’. As the defending counsel in a case, our job is to guess what the prosecution will say in advance and pre-empt it, rather than relying just on a rebuttal.
So what are some classic attacks in reviewing? Here are some. You may have others.
- I don’t see what’s new here. Hasn’t this all been done and said before? What’s the value-add?
- Isn’t this just a reapplication of work X in context Y? I’m prepared to concede you are the first to do it in Y, but is it really that different?
- This is all good and worthy usability leading to better user experiences – but where’s the research contribution? Aren’t you just Doing The Right Thing? I know that’s rare, but that doesn’t make it research. Is there some theoretical contribution or generalization beyond the particular setting worked on?
- Very nice, but will it generalize? Please replicate your study with an n of 7 billion in all possible settings where computers are or might be. Then, maybe, I’ll believe you.
- Yes you’ve pointed something out. I agree that you’ve shown it happens. But so what? What can we do with that knowledge?
- Isn’t that just obvious?
- Won’t the problem go away if you just use {insert latest gizmo / business fad here}
- But what about all these other confounding factors that you ignore, or even state that you are ignoring? Don’t they swamp the effect?
- That looks like Smith & Jones (2009), a paper I’ve never read but has similar words in the title. It’s about computers too. Why didn’t you cite it?
Now on first glance we may think we’ve already addressed those issues in our paper. But sometimes they need to be made more explicit.
Why am I interested in this approach? Well because it is part of what I do when analyzing learning, confusions and misconceptions in my research. I think it’s possible to predict classic misconceptions about an application that some users can have that can be caused by or encouraged by an interface to that application. Then a bit of redesign can prevent or lower the odds of that confusion happening. Likewise I think it’s possible to predict classic misconceptions about a piece of research that some reviewers can have that can be caused by or encouraged by a paper about that research. Then a bit of rewriting can prevent or lower the odds of that confusion happening. So just like user models and student models we can have reviewer models.
Think of peer review as a rather irritating kind of user test of your paper. It’s so much less fun than doing real user tests. It must be what it feels like when I test other peoples’ apps.
It can be hard to look at your own paper in this light. Just like it can be hard to look at your own interface in this light. What I do is draw on the set of loony CHI reviewer personas I keep locked away in my head. If you try that, be careful that they don’t take you over, especially if you ever review anything I’ve written.
