The Peer Reviewers’ Openness Initiative, Year 1

The Peer Reviewer’s Openness (PRO) initiative is a grass-roots movement in which reviewers agitate for new standards of transparency at the journals who depend on them. I’m a signatory to this initiative and a co-author on the paper announcing it, and since 1 January 2017 I have been enforcing PRO as a reviewer. I’ll reflect on how that is going.
First, what does a PRO reviewer do? This depends a bit on interpretation. PRO guidance indicates that the reviewer should withhold providing their full review to the editor until the manuscript has met the minimum standards for transparency that PRO endorses: all papers should state whether the data are publicly available or not and include a link to a stable online repository if the data are available. If the data are not available, a reason (any reason at all) should be explicitly given. Most of the manuscripts that I’m sent to evaluate do not meet these standards. That’s why we’re doing this: we want a new normal that is much more transparent than the current status quo. However, no one wants to needlessly slow down peer review by waiting two months to announce to the editor that all we’re going to say about the manuscript is that the data transparency is inadequate. To avoid this, I adopt the proactive approach of contacting the action editor as soon as I discover that the manuscript doesn’t meet the PRO standard, and ask the editor to request an agreement from the authors that they will address this in review. Editors could also promise to enforce my request themselves at this stage. Either way works: if resubmission is encouraged, information about how to access the data and materials will be there when the paper is published. One more paper will provide access to data, and a few more authors will learn that this is becoming the new normal.
I’ve had mixed reactions in my interactions with editors of various journals. Many have relayed my query to the authors. Those that did got a response quickly. In every case, the authors were happy to provide the information I wanted, and there was no delay whatsoever in the peer review process. A few others have dropped me as a reviewer. These editors gave several reasons:
  • One called my request “ahistoric” – that’s right! We are trying to change how things are done.
  • Others argued that they could not spare the time to relay a query to the manuscript author. Fine: in that case, I don’t have time to voluntarily write a review for you.
  • At APA journals, I meet with the consistent argument that they will never ask one author to do something that isn’t asked of all authors. This is an absurd policy. Since every paper is unique and reviewers may already ask for anything at all –  they may ask you to remove or add analyses, to change or omit arguments, really, anything they think matters – I don’t see how editors could ever enforce this counterproductive view of fairness. I’ll venture to say that actually, they don’t.
It is satisfying to know that the papers I endorse for publication now all meet at least the PRO standard of transparency. It has also been satisfying to see that many of the journals I work most closely with have now signed on to the Transparency and Openness Promotion guidelines. Level 1 of these guidelines is all the PRO initiative requires. The best outcome those of who signed the PRO initiative could hope for is that in another couple of years, describing data accessibility is so commonplace that we no longer need to request it.

PhD positions at Cardiff University School of Psychology!

There are several opportunities to undertake PhD study at the School of Psychology at Cardiff University. The deadline is 23 February 2018, to begin in October 2018. One of the positions is in my lab!

I’m really looking forward to working with a new student. I loved being a PhD student myself, despite the widely-noted downsides of working in academia (which you should be aware of going in!). I want my PhD students to look back fondly on their training too, and not to regret their choice. Here’s what went into my decision to become a PhD student:

What career are you aiming for, and do you need a PhD to get it? You need a PhD to enter tenure-track academia, but in other research careers, you may not need a PhD, and having one might not increase your income. If you want to try for an academic career, that’s great! Be aware that getting the PhD is just necessary for that, not sufficient. But though the academic job market is weak, the market for competent researchers is not. If you would be happy to work outside a university, the skills you acquire as a PhD student in cognitive psychology (writing, critical thinking, organisation, statistical analysis, programming) will make you more employable (though a PhD is not the only way to acquire these skills). Your prospects in academia are also better if you can afford not to limit yourself to a particular geographic region. I moved ~1000 miles for my PhD position. Since then, I’ve worked in three countries, but that wasn’t a drawback – I enjoy living abroad (and so does my partner, which is also something you may have to consider). So, keep in mind that getting your PhD is just one of the challenges you will face pursuing a career in academia.

Studying for a PhD is worth your time and effort only if you enjoy the process. You want to be confident about this – your happiness for the next few years depends on it. What makes you think you will enjoy research?  Did you enjoy doing research during your bachelor’s or master’s study? I’m not asking whether you excelled at it – did you actually enjoy thinking about the research idea, reading about it, analysing the data? If you did not actually enjoy it, you will struggle with your PhD no matter how smart or motivated you are. Your undergraduate research occupied just a portion of your working time. You got plenty of breaks from it to work on other things. PhD work won’t be like that: it will be your full-time job for three years. Let’s be honest: it will probably be even more work than a full-time job. If you have spent time working in a lab, voluntarily while you were a student or as a job, that will give you a better basis for deciding whether research work is for you. When I decided to join Nelson Cowan’s lab as a PhD student, I had worked part-time 1) at a library helping people with research for 3 years, 2) in a clinical developmental lab for undergraduate course credit for 2 years, 3) in a cognitive lab for undergraduate course credit for 1 year, and 4) as a part-time research assistant on a project investigating the effectiveness of preschool literacy programs for 2 years. These experiences were really important. Because of them, I knew I enjoyed working in labs and doing research. Even the work I enjoyed least was alright.

I had also tried out a few topics, and knew before I began looking for a PhD position which of these topics appealed most to me. I cannot overemphasise the importance of this. Your PhD will be a deep dive into a narrow topic. You are going to examine a very specific problem in more depth than anyone has ever examined it before. If you are not already interested in this problem, you will not enjoy this process. Don’t start down a three-year journey to expertise in one topic of psychology thinking that when it is over, you can easily pivot to something very different. The opportunities available to you in my lab will be focused on working memory and the methods we can use to study it that you can master in <3 years. The extended network you will join as a junior member will be full of people obsessed with working memory. If my PhD student is thriving and wants to pursue an academic career, s/he would be well-positioned to get a postdoctoral research job in a lab investigating working memory, executive functioning, multi-tasking, or something in that ballpark. If my PhD student wants a post-doc job in a different ballpark, s/he will be competing with other thriving PhDs who are already playing over there. You don’t want to dive into a PhD project before you are sure it is something you can get happily obsessed with.

When it works right, PhD study is brilliant. It works right when the supervisor and student share a vision and excitement about the project. I had that as a PhD student and I want you to have it too. Check out our opportunities and see if something we are offering has that potential for you.

 

ESCoP preliminary satellite event: The new Journal of Cognition

In December 2016, ESCoP made the momentous decision to stop working on behalf of a publishing company and instead control our own journal. This presents the opportunity for us to create the communication platform for cognitive psychology that researchers want. We envision a journal that is free for all to access, with high-quality content that has been curated by experts and improved by constructive peer review. We are among the first learned societies in psychology to lead in publishing in the fair open-access model, and  I will be speaking and leading discussion about the new Journal of Cognition at the upcoming ESCoP meeting in Potsdam. I will be hosting a discussion about the new journal just before the conference begins on Sunday 3 September from 13:00-14:00. I plan to give an introduction to the innovative content we seek (including Registered Reports and data reports) and the benefits of the fair open access model for researchers and their universities. I will also be seeking feedback from delegates on our plans, including our plans for promoting the journals content and increasing the impact of cognitive psychology generally. All delegates are invited to attend this forum. I hope to see you there!     

Use Mac OS to generate speech files from text

Here’s a free (if you’ve got access to a Mac), quick method for creating spoken experimental stimuli, which I’ve implemented in this project and now use regularly in my lab. Hat tip to Richard Morey for suggesting this and writing a few lines of code that saves many hours of tedious voice recording.

  1. Create a plain text file with the texts you want to create .wavs for entered on separate lines. Line break after each word or phrase that you want to generate a sound file for. Name your text file “words.txt”. It should look something like this one we recently used to generate audio files for a digit span task: 
  2. Open iTerm, and navigate to the folder where your “words.txt” file is located. Here’s a lesson on how to navigate via command line. TLDR: the “cd” (current directory) command is what you need. cd .. will move you to the directory one above where you are. The “ls” command lists the folders in the current directory.
  3. Type the following into iTerm:

In a few seconds, the directory containing “words.txt” will also include a .wav file for each entry from “words.txt”.

This code uses the MacOS voice “Kate” (a female voice with a British accent). You can replace “Kate” with any voice available on your system by removing “Kate” from the code and typing your preferred voice instead.

Text-to-speech isn’t  perfect. You want to check that your .wav files sound like the text you meant for them to sound like. Different voices sometimes pronounce the text in different ways, so one solution to weird-sounding speech is to try another voice. You can also try alternative spellings to get the right pronunciation.

Who benefits from for-profit academic publishing?

If you’ve never considered how academic publishing works, start here. If you’re interested in learning more about the forces at work, a group of scholars, librarians, and publishing experts have written a white paper detailing where the money in publishing goes, especially in open-access publishing. We can’t for a minute forget that traditional publishers’ primary aim is to sell content freely provided by scholars for a profit. At the moment, publishers make money selling journal subscriptions to libraries while at the same time charging fees for making select content publicly available. Because widespread open access would threaten their ability to sell subscriptions, the fees charged for open access at for-profit journals do not reflect only the work needed to publish an OA article, but also the anticipated loss involved in not keeping that article behind the paywall. But while most content remains behind the paywall, libraries cannot stop paying for those subscriptions. Publishers collect money for the subscriptions, while also collecting gold-OA fees. Universities are stuck paying both.

Publishing needn’t work this way; there are journals owned and controlled by non-profit groups, and more content should be. When the European Society for Cognitive Psychology recently decided to break with its publisher, we were motivated by the realization that we could not do much to influence our journal’s policies, because we didn’t own and control the journal’s title. We decided to remedy this by launching a new journal adhering to fair OA principles. A mass shift away from traditional publishing to a fair OA model would save universities an enormous sum without reducing the quality of scholarship at all.

Though the biggest traditional publishers talk about embracing the principles of OA, they have little motivation to work on an OA journal if they do not own it. When ESCoP invited publishers to help us launch our new OA journal, we were greeted enthusiastically until we made clear that ESCoP would retain ownership of the journal. Only one of the top-five publishers was interested in working on a society-owned open access journal, and only if ESCoP would financially contribute a substantial amount to the production costs. Fortunately we also received competitive bids from publishers specializing in online OA publishing. None of these publishers requested that ESCoP pay money to mitigate their risks. They provided straightforward estimates of what costs they incurred from taking on the work of publishing our journal, along with a plan for per-article fees to meet those costs. We ultimately chose Ubiquity Press from these options (more about that next week).

The big publishers all have official positions about open access, describing how they want to “empower” scholars or help us “innovate”. But their exorbitant gold OA fees cannot be justified by any of these principles. I suspect we are never going to get around the fact that the traditional publishers are only committed to offering OA if they can do so while protecting their lucrative subscription model. 

II. What’s in a fair article processing charge?

A companion post explaining why I think scholars should switch to publishing via non-profit open-access is here.

Ideally, a scholar would know before choosing an open-access journal how the article processing charge (APC) is reckoned. It should always be clear what we are paying for. APCs for open-access publishing vary enormously. A few non-profit open-access journals are able to offer their authors gratis publishing, while other journals charge thousands of £/€/$ for ostensibly the same service. Why the differences, and what services are worth paying for?

The most expensive APCs I’ve heard of are charged by traditional, for-profit journals to make individual articles available on a case-by-case basis. Scholars supported by universities or funding agencies that insist on open access might pay these fees to comply with institutional requirements. The APC does not reflect any superior service. It reflects the loss to the publisher incurred when the paper is not exclusively available by subscription. The gratis options I’ve heard of are from non-profit open access journals who have either 1) obtained financial backing from non-profit institutions (such as national science organizations or scholarly societies) that pay the essential fees incurred in publishing for the authors or 2) recruited disinterested, idealistic editors who volunteer their services, and additionally perform the essential professional services usually provided by a publisher (e.g., hosting a journal’s website, maintaining databases of reviewers, sending reminder emails, ensuring articles are securely archived). When open-access publishing is free for the author, that doesn’t imply that there are not costs involved. It means that someone else is paying them. 

Can we decompose an APC to evaluate its fairness? Some of the costs involved in publishing an article are really incurred when the article is accepted and complete when the processing is finished. For instance, we can calculate the exact cost of the hours of expert labor needed to copy-edit and typeset a manuscript. However, many other expenses are less straightforward. Some costs may be bulk-distributed across all the content a publisher handles, and may scale depending on how much content the publisher distributes. These include things like costs of dedicated staff, and server space for housing the journal content. A small journal independently publishing 20 papers a year might approach these costs differently than a larger journal publishing 200 papers per year. Whatever the scale, these costs may still be estimated per paper accepted. You want a professional to handle these issues. There are not going to be lots of academics willing to honorarily proofread content or take responsibility for the integrity of a server (and do a solid job of it). Handling these tasks requires professional skill and we should value people with those skills enough to reward their good work.

Editorial evaluation of submitted content cannot be valued on the basis of accepted articles, because offering differential rewards for accepted papers would create a perverse incentive. Often, editorial work goes uncompensated but I think this is a tactical mistake that we should strive to avoid. Evaluating a manuscript requires skill and effort, and we want the people doing this to be motivated to do this thoroughly and quickly. At the journals I am familiar with where editors are paid (note that frequently they are not), they are either paid an honorary annual lump sum or a handling fee per paper. This sort of compensation does not introduce bias because editor income is not contingent on the decision. So even non-profit article processing charges are not a simple break down of editorial fees plus production fees: the processing charge must be larger than the real cost in order to account for the editorial labor involved in rejecting papers. Assuming that editors are paid a handling fee of €40 per paper and the journal rejects on average 50% of submissions, the APC due to content evaluation is €160: With a 50% rejection rate, each accepted article covers the €40 fee for editor-in-chief and action editor for the accepted article and one rejected paper. Obviously, if the rejection rate is higher, this changes. Obviously, this is in addition to the other costs incurred when a paper is accepted. It also assumes that that peer reviewers work for free; in my ideal world, they would be compensated too. Thinking about this, it is becoming easier for me to suppose that a fair APC will be many hundreds of €.         

Where could costs be cut? If enough scholars and their universities fully endorsed the non-profit open-access publishing model, then one excellent step would be for universities to factor staff’s editorial duties into their workload models. Though academics are currently expected to accept editorial responsibilities and peer review regularly, these activities are not (usually? ever? I would love to hear if any of you are given time each week at work for this) explicitly included in a workload model. If editorial work were not an extra task but replaced other work, then the real cost of a fair APC would be appropriately subsidized by universities, who have a stake in this, and the charged APC would decrease. Rejection rates are another big factor. Reconsidering policies contributing to high rejection rates (e.g., closed editorial decision systems that discourage sharing opinions between journals, placing arbitrary limits on journal size to create scarcity, poor self-evaluation of manuscripts and feedback leading to resubmission of the same rejected content to journal after journal, etc.) would also reduce the need to budget as much for editorial evaluation.

To summarize, decomposing the costs in a fair APC depends on a lot of factors: the size of the journal, its rejection rate, whether editorial work is compensated, whether there are funding sources absorbing some of the costs, etc. The real costs in labor of evaluating and producing an article and ensuring that it remains accessible easily reach many hundreds of €s. We should expect non-profit open access journals to periodically justify its APC to authors, proving that it is a fair rate that allows the journal to properly support its mission.

I. Why non-profit, open-access publishing is better for scholars

Scholars, who do the bulk of the work involved in academic publishing, are faced with a choice: 1) give away your work to a for-profit journal to publish, or 2) pay a non-profit journal an article processing charge. Let’s look at the costs of publishing, both in terms of money and effort, incurred at every stage from the start of a research project until it finally appears in a journal, to discover where the costs come from, and who benefits most. 

The initial costs of producing a scholarly article are borne by scholars and the universities or funding agencies supporting them. Scholars conceived the idea for the project, made sure that the work necessary for writing the article was accomplished, which can involve hours of management, pain-staking data coding, costly data collection, and/or many hours of data analysis, all on top of actually writing the article. On the day you finally submit your article for publication, you and the institutions supporting you have incurred all the costs for producing that work. What next? Well, we all know that we can’t sell scholarly writing. If we charged for it, few people would ever read it. But scholars and their universities want the work to be available to interested readers in perpetuity, and they also want some indication about its quality. So we give the article to a journal capable of determining its quality, and if they accept it, the journal handles its distribution. Ideally, scholars get what they want from this arrangement: the assurance that their work will be forever available to a wide audience, and that its quality will be certified by the editorial process.

Whether we chose a for-profit publisher or a non-profit publisher, the article must now be evaluated by one or two of the journal’s editors plus an additional 2-4 peer reviewers. Assuming it is eventually accepted, staff at the journal copy edit it, check that all the references are right, apply their house style so that it doesn’t look like a term paper, and distribute it. If it is rejected, the author will submit it to another journal, where the evaluation process recommences. So, in order for the scholar to get the credit and exposure they hope for, hours of skilled labor are required from at least 4 additional people. How is that labor paid for? At a traditional for-profit journal, this comes from the earnings the publisher makes from selling subscriptions to the journal. The publisher makes as much as 37% profit, which you might naturally assume enables them to pay a fair rate for the skilled labor that made the whole process possible. But, just as authors give their work to a journal for the honor of letting that journal publish it, scholars evaluate articles for the journal for the honor of doing so, or possibly for an honorarium that does not amount to a fair rate for the labor needed to do a thorough job. For the effort of recruiting some honorary scholar labor (a task that is also usually farmed out to scholars), proofreading, and type-setting your paper, the publisher takes control of your work so that they can charge your university money so that your students have the opportunity to read it. Honorary scholarly labor is the backbone of this system, and for providing it, your university has to pay for access to the final product. 

The emerging alternative is to instead choose an open-access journal that does not operate by selling subscriptions. In this model, authors (or preferably their universities) typically pay an article processing fee when their paper is accepted for publication. Paying to give something away may seem suspicious, but the value of this fee depends on what the fee covers and what you get in exchange. Suppose you just want your paper to be publicly available, perhaps because it is a stipulation of your funding source or a rule of your institution. For-profit publishers charge huge fees to for a la carte open access to offset the decrease in income they would see from paid subscriptions if many author chose open access. This justification of the fee does not benefit the author at all. Non-profit open access journals charge lower rates designed to reflect the actual expenses incurred from evaluation and production of the articles. The published article is made freely available, so the university will never need to pay anything further for its students and staff to access the final product.

In terms of giving scholars what they want (i.e., credit and exposure), how do these publishing models compare? First, let’s agree that it is not really the publisher who provides the lovely feeling of esteem we get from having a paper accepted: that’s the editor and peer reviewers, who are respected colleagues. The process of editorial review is identical at for-profit and non-profit journals. The for-profit model adds nothing essential here. In terms of distribution of your article, there is no comparison between models: the non-profit open-access model makes your paper freely available to anyone, forever. The for-profit publishers will try their hardest to ensure that only the interested readers with a current university ID can possibly access your paper without paying for it. Anyone else will be charged a ludicrous fee, a fee which probably exceeds what your editor received for handling your paper! 

So to avoid paying for the necessary services to evaluate and distribute your article represented by a non-profit article processing charge, you (with your effort) and your university (with their money) are willing to perform most of the work needed to produce an article, sign away control of it, and then buy the copy-edited and typeset product at a marked-up rate. Universities are arguably better off paying a non-profit publisher an article processing charge to procure the services they want to ensure access to an unrestricted finished product. This financial benefit for the university increases as more scholars and their universities choose the open-access model and less new content is subscription-restricted. 

Next up: What’s in a fair article processing charge?

What do scholars want in a journal?

Members of the European Society for Cognitive Psychology (ESCoP) have just learned that their executive committee unanimously took the courageous step of abandoning the society’s journal to its publisher. Instead, we are focusing our efforts on launching a new journal, one that is designed to appeal to and benefit scholars. You can find Sebastiaan Mathot’s summary of why we took this decision here. In short, it was frustrating for us to imagine that we could spend the coming years working hard to make our journal better, but that ESCoP would not necessarily derive any additional benefits from our extra effort. If scholars are going to invest so much, we should make sure that scholars are the principle beneficiaries of our work.

We now have the opportunity to create a cognitive psychology journal that serves scholars, that puts our priorities first. How do we want journals to function? This is what I want:

  1. Rigor: I want editors and peer reviewers to hold me to high standard. When I submit a paper, I hope that reviewers suggest ways I could improve it. If my research design has a fatal flaw that limits the conclusions that I can draw too much, I want them to catch that and stop me from publishing it. I do not want the editors to reject my otherwise excellent paper merely because they do not imagine that it will incite a lot of publicity.
  2. Transparency: I want to have access to data and materials of the papers I read. Sometimes this is the only way to get the details necessary to evaluate the methods. This may not be crucial for a casual reader, but it is absolutely crucial for the scientist thinking of planning an extension of the published work, or the scholar trying to meta-analyze some aspect of the data that the original authors did not thoroughly describe. Much time is wasted on the misunderstandings that arise from relying merely on narrative text for conveying every detail that was important for running or analyzing a study. If we leave this to the authors’ discretion, it is all too likely that the data and materials will not be preserved and curated properly. I want scholars to encourage each other to pre-emptively curate their work.
  3. Accessibility: I want my work to be accessible to everyone. I believe that accessibility will increase its impact. I don’t want to pay a huge fee to a publisher to compensate for the loss it takes when making my work accessible.
  4. Promotion: I want my work to spark discussion and new ideas. I want the journal I publish in to match the efforts I take to let interested readers know about my work, and to take steps that encourage interested readers to discuss the work.

A scholarly society is in the ideal position to create a journal that prioritizes scholarship first. ESCoP’s motives are aligned with the motives of the scholars seeking to publish with us. We don’t want to hide your work behind paywalls to anyone who hasn’t paid for access. We don’t need to own your work and generate profits from it; though we haven’t settled on a fee yet, the margin we assume beyond the necessary fees for the labor involved in processing an article will be thin. If any profit is realized, it will be spent by ESCoP on things valuable to scholars, like affording opportunities for summer schools and conference attendance to PhD students. Most importantly, ESCoP wants to take pride in the quality of the work it publishes. We do not need to fill a specific number of pages, nor do we need to limit ourselves to some arbitrary number of papers. We can publish and promote as much high-quality cognitive psychology as scholars want to entrust us with.

ESCoP has entrusted me and my editorial board to design an open-access journal that serves the scholar. We’re excited about this. Are you? What policies would you appreciate in such a journal? Please tweet them to me @CandiceMorey.

Individual differences in experiencing visual imagery

One frustration with studying visual memory is that we have no way to directly experience what someone else sees in their mind. When trying to understand individuals’ memories about uncontrolled events, we rely mainly on verbal descriptions of visual memories and individuals’ feelings of confidence in the clarity and vividness of those memories. There is a lot of variability here. Some people believe that their visual memories are as clear and high-fidelity as a photograph (though there are very few examples of this fidelity being put to a convincing objective test; see this article for an overview). Other people report experiencing no visual imagery at all (see here and here for descriptions and discussion), even when asked to imagine something that should be extremely familiar. Most of our experiences probably fall somewhere in between these extremes.

My student Gintare Siugzdinyte is currently studying individual differences in visual imagination experiences, and hoping to attract a large and diverse sample to respond to a short questionnaire. Later in 2017 after we have analyzed these data, I can provide an update about what we learned about visual imagery from this survey. You can find our survey here. Thanks very much for your help!

It’s exam time: Do your best by trying empirically-confirmed advice

Our Learning Technologist Nick Daniels asked me how students should approach studying (or “revising”, if you’re British) so I explained Karpicke and Roediger’s (2008) crystal-clear testing effect results. The four other things I always remind my students:

  1. Start early (ideally you already started, but if not, now)
  2. Space out study sessions, taking breaks in between
  3. Elaborate: don’t read over your texts repeatedly, think about relationships between concepts, talk about them with other students
  4. Take care of yourself: Eat healthy food, exercise regularly, go to sleep every night

Below is the short video Nick produced in which I explain the testing effect in depth and mention these other important tips. Maybe you are looking for more advice or more detail about why we think these techniques work? Check out The Learning Scientists blog, an excellent source for links between basic cognitive psychology research and education practice. Good luck with your exams!