Monday, February 14, 2011

Rebooting Computer Science Publications: Organic Processes

A recent ACM Communications viewpoint by Jonathan Grudin (login required) summarizes the reasons for the current emphasis on archival conferences over journal articles as computer science publication venues. His final paragraphs suggest some possible new directions for conference publications that could potentially alleviate the challenges he identifies as the result of low-acceptance conference venues (loss of community, deleterious effects on reviewing, perceived emphasis on incremental results). Certainly, there has been much discussion of these topics of late in our field, including a CHI workshop in 2010 on precisely these issues.  However, Grudin's most radical suggestion is as follows:

"A more radical possibility is inspired by the revision history and discussion pages of Widipedia Articles. Authors could maintain the history of a project as it progresses through workshop, conference, and journal or other higher-level accreditation processes."
 I believe this organic process is the future of all academic publishing. A process that begins with the concept for a research project, publishing the work in progress to a website that supports commentary and complementary projects and ending with an archival journal publication with associated commentary and linked supporting research. It behooves us, as computer scientists, theoretically at the forefront of technology, to be the early adopters of a process that can revolutionize the way that we manage the publication cycle. Below, I've identified a few arguments for why an organic approach to publishing is important.

  1. Improved collaboration: Despite the byword of collaboration in many research projects, research remains fundamentally competitive. The next steps in many solutions (especially Master's-level projects) are clear to most people in a field and the race to publish that next step is part of the process. Organic processes would recognize this and improve it by permitting wider collaborations with less competition. Work in progress would be clearly visible to anyone interested and rather than protecting and "scooping", we could encourage our students and labs to find the holes in the work in progress and fill them with our own research.  This doesn't invalidate the ongoing research, but complements it, clearly providing spaces for various sizes of contributions, from undergraduate honours-sized projects to full Ph.Ds.
  2. Replication of Results: We often need to replicate a study in part, either to juxtapose the results with our own work or as a piece of practice research for students. Organic publishing would include much more transparent methodologies and results, permitting easier replication or even the re-use of data. Concerns about privacy and permission are important, but not insurmountable.
  3. Publication of Negative Results: One great loss in our field is that we fail to publish our negative results.  Trying and failing is almost as important a piece of research as success. Not having these results handy is inefficient for two reasons: 1) It's likely that if I tried and failed, you will too and 2) if you could see my results, you might be able to help me fix the problem. Organic publishing would necessarily include all of the failed attempts at research, since the publication process would start at the beginning of the project rather than at the end.
There are plenty of other reasons for moving to an organic publishing process, and I'll likely talk about them further, but these three reasons alone are enough to convince me.  Are you convinced?

Of course, there are barriers.  Tenure, promotion, and hiring would change considerably.  Evaluation metrics would become different.  Arguably, however, we could have a better idea of how effective researchers are by observing the entire process of research rather than the end result. We could also see how researchers interact with collaborators and critics, notice how agile they are in including ongoing research on similar paths, and the rate at which they are able to complete new research. We could certainly establish metrics acceptable to funding agencies and hiring institutions, but the real benefit would be at the departmental level in academic institutions or at the lab level in industry. The current model for publication would also wither somewhat, and our acceptance metrics for conferences would have to change, but I believe it would be for the better.

No comments:

Post a Comment