Tuesday, August 24, 2010

Journal-isms: What Would It Take To Reform Scholarly Publishing?

Well bust my britches, if the paper of record didn't put we scholars on the front page this morning! Reporting on the decision of the Shakespeare Quarterly decision to experiment with posting articles on line for open review, the New York Times reports that:

a core group of experts — what [Katherine] Rowe called “our crowd sourcing” — were invited to post their signed comments on the Web site MediaCommons, a scholarly digital network. Others could add their thoughts as well, after registering with their own names. In the end 41 people made more than 350 comments, many of which elicited responses from the authors. The revised essays were then reviewed by the quarterly’s editors, who made the final decision to include them in the printed journal, due out Sept. 17.

This process of online review, the Times argues,

goes to the very nature of the scholarly enterprise. Traditional peer review has shaped the way new research has been screened for quality and then how it is communicated; it has defined the border between the public and an exclusive group of specialized experts.

Well, not quite, but let's pick this ball up and run with it, shall we? While I think this is an interesting and productive shift, and that opening up the review process is a bold thing to do because it puts a dent in the Bell of Silence that scholars erroneously believe honesty requires, the practice --as envisioned by the editors and utilized by those truly brave people who participated -- adhered to tradition in important ways. First, the journal obtained promises from a "core group" of scholars that they would participate; and second, if you read down to the bottom of the article, one of the participants still felt it was necessary to secure a promise from a dean that the article would still count for tenure. (Let's give a round of applause to this young person, shall we, for participating in something new and untested? I hope you do get tenure: we need more people like you in this profession.)

My point is that traditional gatekeepers are still in place -- even though the process has become more open and, importantly, more public. Jennifer Howard's in-depth piece about Shakespeare Quarterly last month in the Chronicle of Higher Education makes this, and other, good points.

The Times also notes that peer review, although the Rosetta Stone of the tenure and promotion process, is deeply flawed. I concur. There are numerous examples one could cite of plagiarism, or poor practice, that seem to slip right through the peer review process. Add to this the fact that many, if not most, journals are famous for vetting processes that are as slow as Cream of Wheat going down the kitchen drain. Graduate assistants and faculty editors who lose track of manuscripts; readers who are given six months to complete the review and have to be pushed to complete it anyway; and the capacious use of "revise and resubmit" rather than bluntly saying the article is poor and needs to be completely rewritten -- all of these things and more are acknowledged problems with the academic publishing process that make many people reluctant to send work to journals.

Another outcome of cumbersome journal review mechanisms that many, if not most, scholars in the humanities and social sciences think are flawed, is that readers often receive manuscripts that are in horrible shape. Graduate students and young scholars are often counseled to send work out for review to -- well, to put it bluntly, get free advice from top people in the field, and to get their work "in the pipeline" in hopes that a journal will commit to it at an early stage. This is particularly true of dissertations and dissertation chapters. Dissertations are not, or are very rarely, books; and dissertation chapters are not articles. And yet, they are often sent out to readers as if they were, and the privacy of the process -- while it doesn't seem to stop readers from hemming and hawing and recommending that it be "revised and resubmitted" -- discourages the authors from being embarrassed about sending out work that isn't ready for review yet.

So I see what the Shakespeare Quarterly is doing as an important step in reforming the process. Even if other humanities and social science journals do not care for the experiment as it was conducted, they need to find some way to move towards the following reforms:

The period of time from submission to acceptance or rejection, should be dramatically shortened. Eight weeks is really sufficient; and actually, for many of us, looking at our calendar might mean spotting a free day for doing the review that is even sooner. Reviewers should be held to this date, and the date should be conveyed to the author.

Eliminate revise and resubmit. There should be two categories: accept and reject. One can give cogent reasons for rejecting a piece that do not prevent it from being revised and submitted elsewhere. One could recommend accepting an article pending revision of even serious flaws because it makes a real contribution that is as yet unrealized.

Journals should not accept articles they are not ready to put into production in the next year. Having a piece fully accepted and then delaying publication for a year to eighteen months is idiotic, and a drag on the system. It means that the value of article for the bean counters (those who are "counting" publications for merit raises, tenure, promotion) is often greater than the value of he article to scholars, or at least those scholars in the field who ought to be reading it. If it is worth reading, it is worth reading now.

All journals should begin enhancing their web presence immediately. Paper journals, at least in the humanities and social sciences, will eventually be dead -- you know it, I know it, and it is just a matter of time. Cuts in library budgets are damaging journals, but the problem is larger than that. Newspapers who spend around 80% of their gross revenue actually getting the newspaper-as-object to the reader, and I suspect this is true for journals as well. And what happens to those objects? I belong to three professional associations (two history, one interdisciplinary) who, between them, send me twelve journals a year. I just weighed the pile of journals pictured at the top of this post, and now know that 13.8 pounds of extremely good quality paper comes into my house on an annual basis in the form of journals, paper that is even more expensive because it has to be printed, transformed into a book-thing and mailed. Within the next twelve months, these high-quality and very aesthetic objects, sadly, will end up in the recycling bin, because who wants to accumulate over a foot a year of journals when they are searchable and readable on line? And when much of the material in them is not in one's field?

As an example, I would pay the same American Historical Association dues to not get a paper copy of the American Historical Review. This is not because the AHR isn't good, although there are entire issues that pass filled with beautifully researched and written articles that are so tangential to my work that I can't prioritize them in an already over-taxed reading life. But even if I did read them cover to cover, it is ecologically unsound and an utter waste of the organization's money. I have adapted to doing a significant portion of my professional reading on line and really, I wish they would use my dues some other way, like lobbying state legislatures to restore cuts in higher education and hire faculty full-time. The AHR might even consider paying the people who they engage to do peer review so they would do it in a timely manner.

Finally, moving to an all-web presence over time would permit articles and book reviews to be published in a more timely manner. They could go up when they -- or a cluster of like articles -- was ready. A regularly updated book review section could review books (gasp!) when they come out as opposed to, say two to four years later. Journals could respond to political and cultural developments in a more timely manner -- and perhaps even become relevant to a broader, educated audience.


Since you are too busy getting ready for the new students to check out my constantly updated toolbar, before you stop reading today, check out this brilliant post on cultivating "beginner's mind" at Roxie's World. It's especially aimed at veteran teachers who might be taking too much for granted at the beginning of the semester -- and missing the joy.


Roxie Smith Lindemann said...

Thanks for the link and the kind words, TR, and the smart (as always) thoughts on scholarly publishing. Hey, what is that small, grinding sound I hear? Why, it's the wheels of academic tradition trying tentatively to shift into a new gear!

Katrina said...

I completely agree re. the slowness of journals, I wrote about it here http://katrinagulliver.posterous.com/article-submissions-and-journal-response-time

In a recent conversation with some scientist friends, they were shocked when I said it could take years for an article of mine to appear in print (they are also used to journals with a publication frequency of monthly, even weekly).

This was in a discussion of citation factors, where I was explaining that the long lead time on humanities publications means nothing has been cited yet by the time people go up for tenure or promotion, and that the impact of significant articles in history isn't evident perhaps for more than a decade after publication.

But this made me realise that we are MORE beholden to journal reputation than other fields - search committees and others looking at a cv will care whether the article was in the AHR, or Past & Present, because they have a good reputation, and that serves as "proof" of the article's value. But the article's own merits are hard to demonstrated. Whereas for someone in the sciences, of course publishing in Nature or Cell is highly prized. But an article in a less-well-known journal can still make a splash, and the author can demonstrate its importance through citation rankings.

While I don't advocate a shift to such indices for history, the current system does just reinforce the hegemony of the big name journals (the ones you don't have to justify publishing in for tenure...), which of course increases the pressure - and the backlog - at those journals as everyone submits to the same few venues. The AHR is published quarterly, and prints perhaps 15 research articles per year (of what - 300+ submissions?)

If there were monthly (or, heaven forfend, WEEKLY) history journals, reviewers would really have to pick up the pace!

To be fair to the AHR though, they're one of the quickest on the history journal wiki http://scratchpad.wikia.com/index.php?title=History_Journal_Response_Times

Tenured Radical said...

Thanks for the links Katrina: if you put them in html next time, you will make it even easier for the next readers to access them (write me off-line if you don't know how to do this.)

Roxie: riddle me this from the lit world-- why are Shakespeare scholars and early modernists often responsible for doing edgy things? Is it because they are used to having to fight to have their relevance recognized? What could the rest of us learn from this?

Notorious Ph.D. said...

I like most of these recommendations, TR. I agree that R & R is the refuge of the too-nice reviewer. But I also agree that there does need to be something between "accept" and "reject" that acknowledges that there are almost-ready pieces out there; maybe "accept with revisions", or something similar?

Can I add one more recommendation that goes to the point of something you say in the body of your post? Journal editors should conduct a preliminary review before sending out a MS to readers, and not send out pieces that are obviously wrong for the journal, or just not ready for prime time. Having just been the victim of one such submission (albeit at the hands of a first-year editor, so I can't really fault her for not knowing the ropes yet), I think this is something we need to think about.

Little Midwestern College said...

The Times article notes that peer review was developed as “a way of keeping eager but uninformed amateurs out” and quotes one scholar who says that “evaluating originality and intellectual significance. . .can be done only by those who are expert in a field.” Academe is reluctant to address the fact that a very small cadre of “experts” have a firm control on what’s published--and they’ve created a hierarchy of topics/approaches to subjects controlling what is let through the gate. This Bell of Silence, as you term it, allows gatekeepers to replicate approaches similar to the ones they already occupy (or have trained their students to occupy). And because academe is a small town, everybody knows everybody else’s approaches, thereby creating a closed loop that is difficult to break. Crazy revise and resubmit policies force authors to capitulate to that culture in order to publish instead of producing truly cutting-edge work.

This is compounded by another elephant in the room of academic culture that privileges a long list of accepted articles on one’s cv over a smaller number (one or two?) of high-quality pieces. This results in the fact that a good idea that can be addressed in one article or a portion of one’s book is reprised in multiple versions of the same thing, thus contributing to the backlog of publishing and reinforcing the culture of ideas already created by the problems I noted above.

I think what Shakespeare Quarterly is doing is a good effort, because it gets at the heart of what I think publishing should be about: sharing ideas and generating discussion and broader feedback in order to further the work of everyone.

Mr. Gunn said...

It's so great to see this important topic being discussed via the New York Times. It's a big issue on my side of scholarship, the science side, as well.

I'd say revise-and-resubmit should stay, because it does serve a very valuable function in iteratively improving the final work, especially for beginning researchers.

Katrina - the time lag to publication in many areas of science is also two years - WAY TOO LONG! Especially when you consider that much of what we publish isn't finely-crafted prose but rather a this shell of explanatory text around datasets. The data could go online as it's being generated. That would really accelerate things if we could encourage that behavior, but instead making traditional publishing the end-all practically prevents this by engendering a fear of "being scooped".

TR - re: links in text - some blog platforms automatically spamfilter posts with too many links. I often paste plain text links if I'm worried about this.

Comrade PhysioProf said...

This shitte gets trotted out in the science publishing worlde all the fucken tyme: "Hey! Let's just start a journal where we publish everything submitted online, and then the peer review will all be via post-publication commentary on the Web ELVELNTYSS!!!!11!11!"

Journals can do whatever the fucke they wante, but it ain't gonna amount to a hille of beanes unless and until the hiring, promotion, tenure, and grant review committees change *their* practices to appropriately allocate academic credit for such activities.

Anonymous said...

A very tiny, only sort of relevant comment: you can totally email the AHR and ask them to stop sending you paper copies, and they will actually do it! I have found this to be life-changing - no more AHR piled up around my house! No more guilt because I see them there, collecting dust, totally unread for years at a time!

As for the actual content of your post - thought-provoking and excellent as always.

Susan said...

The last article I reviewed (on time, a 6 week turnaround) had three categories -- accept, reject, reject with possibility of resubmit. I've also had journals that had an "accept with mandatory revision" category, which I think deals with Notorious's "almost ready" pieces.

shaz said...

I hear you -- I've repeatedly had articles in the pipeline for 2-3 years before publication. One piece, on historiography, went from up to the minute to less-than-timely.

I'd love to see creative solutions. For instance, in Computer Science, they often submit full conference papers for peer review, the reviewers see each others' comments and if they don't match up, have to reconcile mixed messages. I'd love to interact with other reviewers, and it would likely encourage good reviewing habits. This would be an easy thing to do that could really help editors and authors.

Emily said...

Well, as a graduate student who just had her first article accepted, and another one out for review, and another one getting ready to be sent out in advance of job season, I'd be very interested in getting things speeded up.

I have had articles bumped back, one for being not-ready-for-primetime, and one for being out of scope. So it does happen.

Speaking as a social scientist, I wish that SSRN could be more like the arXiv, as in, useful and highly respected and how you publish first. The conference-paper distribution role it fulfills is useful, but it could be better. Is there something similar to this that historians use?

trixie dang said...

holy dang TR, i am once again impressed. post is insightful and neat (duh.) but moreover, this thing of yours has some of the most intelligent, on-topic comment threads in all of Tubedom.

it's like the post and the comments following it are all part of one engaging, coherent conversation. trippy.

a semantic nitpick, in line with what you (and roxie and a number of other folks) are talking about-

"a core group of experts — what Ms. Rowe called 'our crowd sourcing' (sic) — were invited to post their signed comments on the Web site,"

i don't know Katherine Rowe, but i have a hunch this is one of those cases where someone makes a joke, and the Grey Lady quotes them like they were serious. or someone describes a multi-part dynamic process, and the Paper Of Record decides that a substantial part of that process is just an incidental postscript, and totally mangles the concept and the quote.

By my understanding, the core group of experts aren't The Crowd Sourcing, they're the catalyst to kick off participation by everyone else who participated in the open call. the open call is sorta the important part. crowdsourcing as a term was coined to describe web-enabled systems of structured open participation, once people started realizing how surprisingly speedy and effective such systems could be at generating useful stuff.

but without the open participation, it's just an expert panel critiquing an article on the internet. which is decidedly web 1.0.

so this seems like a really exciting development. but i fear that other academic journals and institutions, (the ones who might still spell out their website with every backslash, colon, and "dot oh are gee") who have never heard of crowdsourcing, will read this article, invite 5 experts to submit response papers to a journal piece, publish the whole mess on their website, and go "look, we have Crowd Sourced It!"

it'll kinda make my head hurt when they do that. and roxie's wheels of academic tradition might just take the opportunity to let out the clutch and return to 5th gear, and when/if the Big Truck makes it up the hill without stalling, the transmission will be totally shot. but at least the peer-reviewed articles on What That Funny Smell Is will have had time to catch up.

Anonymous said...

Want to speed up that lag time to publication? Hire more people to work on the production side of things! If not more people, hire better-trained people, or pay the ones you have a decent wage, or hire people with new ideas on how to get things done.

Most journals get by on a wing and a prayer in terms doing the actual work of getting things to the printer. As brilliant as many scholars are, their articles still need editing, and a lot goes on behind the scenes. Also, authors are sometimes great about responding to queries; at other times it's like pulling teeth.

green mountain girl in dixie said...

I like the categories that Susan proposes. In the recent olden days, I submitted an article based on my master's thesis to a good journal, and it was accepted with revisions. It was published a good chunk of years later after I had a)completed a second master's degree and Ph.D.; b)started a tenure-track job; c)developed a maturity in my research and voice, not to mention distance from the published topic, and felt like the article was very foreign to my identity and capacity as a scholar. And yet, it came out the year after the copyright on my dissertation. I am appreciative of the journal's willingness to publish the work of a young scholar, but it came out so many years after I had submitted it (but a year after the copyright date on my dissertation) that I wonder about whether or not it was a liability or misrepresentation of the state of my work.

Anonymous said...

Relevant? In the scientific disciplines as I understood them, peer review was never part of the scientific method, which revolved around hypothesis, testing, falsifiability. Peer review was quality control for a journal. Add to that the role of peer review in the promotion process.

Shake, stir.

Get one Anthropogenic Global Warming Theory that may or may not be accurate and has become highly politicized and untouchable.