Begin typing your search above and press return to search. Press Esc to cancel.
“Current dissatisfaction with peer review is such an opportunity for change, so we call for taking advantage of this opportunity as fully as we can. We build our recommendations on the idea that mutual critical engagement is a skill developed through ongoing practice and actual engagement with each other’s ideas.”
In the following guest post, Samantha Copeland and Lavinia Marin (Delft University of Technology) put forward the case for an “open forum” model of peer review in philosophy. The post is based on the fuller discussion and argument in their “‘It takes a village to write a really good paper’: A normative framework for peer reviewing in philosophy” which was published in Metaphilosophy earlier this year.
This is the second in a series of weekly guest posts by different authors at Daily Nous this summer.
(Posts in this series will remain pinned to the top of the homepage for several days following initial publication.)
We recently published an article on peer review and reforming the way that publications are produced, reviewed, and disseminated in academic philosophy. We had been discussing this issue for a while, and more and more was being written on the problem by philosophers on social media and in print. Good points were being brought up about the dominance of the English language and its effect on how we evaluated philosophical arguments, and proposals for reform included one we found particularly interesting: the “open forum”. This format advocates for a public, iterative, and community-driven peer review process. What would this look like in academic philosophy?
Pre-publication and post-publication forums have been proposed, whether run by journals or societies, often similar to arXiv (hosted by Cornell). We propose that peer review should also happen in a similarly open context. On our proposal, an archive would host not only papers, but also comments and advice from others alongside it, as part of a discussion and improvement process initiated by authors who post their work. The system would not be anonymized.
It’s a risky idea. Take, for example, the concerns expressed over the Matthew Effect in the ongoing discussion in the literature. Heesen and Bright (2021) suggest that their proposal for a prepublication peer review of the “forum” type would not necessarily prevent cadres of well-known philosophers from “running the show” and effectively preventing outsiders from joining the discussion. We think, however, that this problem is present in systems which emphasize anonymity, even when anonymity is in place, in part, with the aim of protecting those vulnerable to bullying or unfair treatment. Exclusion of outsiders by strict adherence to guidelines can happen as easily as exclusion through the obvious bias of the reviewers. One would hope that, at least, a public, non-anonymous forum would increase pressure on commenters to justify their criticisms and increase their accountability for mistakes and ignorance.
But even more important than this is that we might have a chance to make a positive change with open forum peer review by providing more opportunities for people to create and to critically reflect on philosophy. There is also potential for educational and professional benefits for those who would be given opportunities they do not currently have or get recognition for. Changing the nature of peer review in philosophy so that it takes place in a public forum built on the norms of transparency and democracy could help spread these opportunities and benefits more widely.
Current dissatisfaction with peer review is such an opportunity for change, so we call for taking advantage of this opportunity as fully as we can. We build our recommendations on the idea that mutual critical engagement is a skill developed through ongoing practice and actual engagement with each other’s ideas.
In the hard sciences, peer review is often seen as a check on quality and a floodgate against bad science, yet many publications still get retracted, which is just one reason some say that peer review is a scientific experiment that failed. One of the arguments for reform in philosophy has been similar, that peer review is failing to save us from reading bad philosophy.
In our recent paper, we acknowledge that not all gatekeeping is bad. We also argue that relying on anonymity alone to prevent bias is not enough—it tends, instead, to reinforce the wrong kind of gatekeeping: keeping people out instead of instructing them on how to get “in”. While reviewers may not know where an anonymized author studied or what their gender is, other markers like writing style or citation practices can still indicate who already “belongs” and who doesn’t to those reviewers looking for such indicators and who see their role as preventing bad philosophy from being rewarded with a publication.
We think it’s more valuable to have practices and institutions in which the emphasis is on helping people get in. An open forum for reviewing, wherein the review process is an explicit and public part of developing a paper and its arguments, fits better with this than current practices.
Our proposal is inspired in part by the pedagogical value of peer review. When it is at its best, peer review even now provides philosophers with professional and disciplinary training whenever editing moves into critical engagement with the argumentation or discusses the paper’s very worth. In conferences and university hallways, we seek out each other’s reflections on our philosophical ideas and for advice on how to take them further—good peer review is good philosophical practice (You can find further details about this in the full paper.) We wouldn’t want a peer review system that, as in the sciences, simply checks for methodological soundness and conformity. Indeed, good philosophy engages even its own methodology critically, so the boundaries between acceptable and unacceptable methodologies cannot and ought not to be drawn too clearly, nor should the process be so refined and minimized. But that is what we fear will happen if reforms take the route of entrenching anonymity and focusing primarily on improving the prevention of bias and bullying. Our paper presents a different view of reform than normally taken, insofar as we want to emphasize this moment of increased dissatisfaction as an opportunity to enact a positive change in our profession.
So, we imagined instead what an ideal peer review system for academic philosophy would be like.
A lot of what goes into the making of “quality” philosophical work is, obviously, skills: at writing, argumentation, reading, creativity, imagination, and other tasks involved in the production of a written text. Such skills develop only with training. Training typically means trainers, and so the development of such skills also needs someone willing to spend their time training us: skilled instructors. Further, it is in and through the very practice of these skills that new ideas can be crafted and tested and thus, philosophical progress can be made. In an ideal system, then, there would be means for the recognition and reward for the use of one’s own skills to develop those of others (such as is done in what we think of as good peer review, aimed at strengthening rather than tearing down each other’s arguments). A bonus would be a wider acknowledgment that practicing these skills through deliberation is needed to keep honing those skills, even in our most senior years of philosophical training (i.e., no one is beyond the stage of needing peer review, we think).
Inclusion is still a serious problem in academic philosophy, and we think putting the pedagogical above the gatekeeping aims of peer review offers a chance to make that better. Speaking from the experience of one of the authors, coming from Eastern Europe, it can be quite difficult to find someone to teach you those necessary skills. The most respected philosophers usually learned to write by getting a Western-style education. But what about those who cannot leave their home country easily to get an education elsewhere? Should we accept that one is condemned to never publish in philosophy because one cannot polish the words enough to compete with more stylish writers? If you come from a disadvantaged background, then often peer review feedback may be your primary chance to get an education in how to improve your professional philosophical writing. So it’s important that the helpful, instructional elements of peer review are encouraged, while the gatekeeping role is deemphasized.
Worries about bullying and bias will still exist in an open forum model. But anonymity has not helped in the past, and there is some reason to think it can make things worse (see, for instance, here or here). And there will be monitoring work to be done. But making the reviews of our peers a part of the general educational duty of an academic philosopher would result, we think, in a fairer system overall. For example, a public forum would provide evidence for teaching and supervising effectiveness that published articles cannot now show—engaging professionally with philosophers who are more junior than oneself is itself a skill that should not only be demonstrated but needs to be honed through practice and exemplified by those who have such skills.
The philosophical community has shown that it can reflect on its own weaknesses. As we remark in our article, there have been noted improvements in philosophy: in the way that philosophy conferences are held, in increased fairness and improved norms in post-talk Q&A sessions, in more thoughtful approaches to hiring, and in more self-aware citation practices, for example. So change is possible. And as the very discussion we have waded into with our paper on peer review suggests, change is desirable for many.
We hope that our article’s normative stance shifts the burden of proof onto those who would prevent such a radical change and the burden of justification onto those who would argue against rather than for such changes in philosophy’s processes of peer review. More importantly, we hope to shift the discussion toward what is important about peer review in philosophy, and how transforming our publication and peer review processes can create opportunities for positive change in academic philosophy itself.
The standard sequence is: say something in article, then once published people may discuss it, perhaps reaching a consensus. If we start crowdsourcing reviewer feedback, we risk ending up with this less desirable sequence: say something in a draft, people must discuss it, and then the published article essentially becomes a report of the consensus that was required. I would stick to the former approach, since — despite it’s flaws — it leaves more room for dissent and less room for groupthink.
The problem (according to some, I did not read the linked paper so not sure if this is their view exactly) is that with the current model bad papers get published. For instance, even if I recommend a paper to be rejected, it will likely appear in another journal anyway. So the solution to the problem is to abandon peer-review and use some kind of collective post-review instead? So then everyone could upload their half-finished drafts anywhere and expect others to fix their mistakes? Then we would just have even more bad and unfinished papers to be read?
There are two problems in peer-review (of course you can disagree with me on these). First, people do not do their fair share, for example, people submit their work to a journal but refuse to review for the same journal.
Second, in philosophy, referees often miss the point of peer-review. The point is not to evaluate whether the author has managed to prove something or convince the reader. The point is to check whether the paper meets some minimum requirements regarding writing and methods and whether the author is saying something new on an important topic that people would be interested to discuss about.
There has already been some trials in this direction. An article published to explain this method on Colombia University Website. The journal is called Qeios. it has ‘an AI-based system finds peer reviewers suited to each submission’. Article gets published after getting three favourable reviews. ‘Reviews are then published alongside the article as they come in. Authors have the chance to respond and update their article, as needed. The reader sees the whole history of comments and changes. The entire process of vetting an article is transparent—and fast. Qeios receives a median of 10 peer reviews per article and the time from submission to publication is roughly 10 days.’ I personally this this is a much better way than the gate keepers for the journals as we have now that actively block new and alternative ideas in the field. This way we hold both the authors and the reviewers accountable.
https://www.publichealth.columbia.edu/news/new-journal-seeks-reduce-bias-scientific-publishing
The missing part in all of this is, unfortunately, prestige. In a job market in which self-promotion is, again unfortunately, something of a prerequisite, it is unclear how replacing the old broken system would work. Perhaps one could implement a kind of “points” or “grade” system in the forum, just like other non-academic forums do? Readers can have the option of liking or upvoting strong preprints, with the reward being that the algorithm would then favour strongly liked preprints with more exposure. This could help avoid the issue of authors submitting bad manuscripts in the hopes of getting others to help them make them better. And reviewer comments could also get liked or upvoted by readers who find them to be helpful and of high quality. Not only would this tell authors which reviews are worth paying attention to, but the reviewers themselves could receive some kind of reward as well – a higher ranking, and with it, increased exposure for their own work? Finally, were this to be widely adopted, higher rankings could translate to better job prospects, if hiring committees could be convinced to pay attention to such rankings. Just spit-balling here
I think this is an interesting suggestion. But it also sounds a little exhausting and not great for those of us (early career folk) who like to do our philosophy offline if we can. Starting to appreciate the status quo!
These are interesting suggestions, although one potential issue is that if there is no bar to participation for readers, then the process could be driven by non-experts more than we might like.
(The quality of the average reddit comment comes to mind…).
This is a fear of mine as well, but I think if it was done well, the forum would be a bit more like those dedicated forums for video games. When very specific questions are asked, such as how best to respond to Cartesian scepticism in light of X’s defence of it, for example, you will naturally attract more specific answers. I think that what would produce a Reddit-like forum would be asking questions and making claims that everyone feels more or less qualified to address.
“Where is the argument that normativity in grammar or usage must come down to post-hoc rationalization, etc.?”
See, the problem is that you understand neither the […]
Thanks for your reply, Jamie.
No, I don’t think that descriptivism about grammar somehow implies that Tim Williamson speaks Old English. I wasn’t speaking there of […]
Agreed that administrative bloat is a problem that faculty need to be more proactively addressing. But declining student enrollments is an underlying problem as well, […]
This is a fear of mine as well, but I think if it was done well, the forum would be a bit more like those […]
I see.
Well, right, it’s possible that the problem is that I am very uncharitable. And possible that, as you said, others who disagree with you about […]
Thanks to everyone who weighed in. I’ll ruminate and talk with my colleagues. If we come up with anything I think would be useful, I
Julian, you claim that I have “not given any indication of what… [I] would give as evidence” that something is grammatical.” That is simply false. […]
These are interesting suggestions, although one potential issue is that if there is no bar to participation for readers, then the process could be driven
I think this is an interesting suggestion. But it also sounds a little exhausting and not great for those of us (early career folk) who
The missing part in all of this is, unfortunately, prestige. In a job market in which self-promotion is, again unfortunately, something of a prerequisite, it […]
Enter your email address to subscribe to this blog and receive notifications of new posts by email.