Interpreting nulls, even surprising ones, is not trivial

Sometimes I design an experiment really wondering what will happen, but that wasn’t the case when I first decided to compare proactive interference effects for verbal and visual memoranda. Proactive interference occurs when some information you have previously memorized disrupts your ability to learn new information. For example, having studied Spanish in high school could impair learning Italian vocabulary now: as you try to retrieve the now-relevant Italian word, you risk retrieving the previously-learned Spanish word instead. Learning new information also conflicts with remembering older information (that’s retroactive interference). Proactive interference is particularly interesting for me though because it helps address what happens to briefly presented and retained information like the arbitrary word lists or color patterns we ask participants to try to remember in short-term memory tasks: you can only observe proactive interference if the earlier stimuli remain in mind. If new stimuli overwrite these arbitrary lists or patterns, as is expected in many short-term memory models, then proactive interference wouldn’t occur. Testing whether proactive interference occurs for both verbal and visual memoranda seemed like an important way to assess whether parallel short-term memory systems operate for both kind of material.

I had recently conducted a series of studies including a verbal recognition memory task designed to be analogous to visual recognition memory tasks, and I realized that it would be great to use these materials to estimate the size of proactive interference for verbal and visual information. There were two possible outcomes, each of which would be interesting: 1) proactive interference would be observed for verbal, but not visual memoranda or 2) proactive interference would be observed for both kinds of materials. We designed a release-from-proactive-interference experiment. Participants observed a string of trials in which stimuli were drawn from the same set of similar materials. In the verbal task, this was a closed set of phonologically similar words (for instance, words that were close phonological neighbors to the word “cat”) and in the visual task, it was a closed set of similar colors (e.g., varying from blue to yellow). After drawing stimuli from this same set several times, we switched to a different set constructed in the same way. For verbal stimuli, this was phonological neighbors of “men” and for visual stimuli, it was colors varying between blue and red. If we hold lingering memories of the early trials that interfere with our ability to remember precisely which sequence of words that sound like cat we heard most recently, then performance should grow worse and worse as we keep drawing from the “cat” set. Then, when we switch to the “men” set, performance should spike because the fading memories of cat-like combinations cannot be confused for these novel men-like combinations. Lots of previous literature predicts that this pattern would be observed for verbal memoranda, and our study would establish whether something similar or different occurs with comparably constructed visual memoranda.

Only we didn’t find much release from proactive interference with verbal memoranda. The graph below give estimates of verbal task capacity averaged across 18-trial blocks. The switch to a new stimulus set occurred on trial 13. In our first experiment, we did observe performance in the right direction: on average, performance got worse from Trial 1 to Trial 12, and then got a little better again after trial 12. But not by much at all.

fig3_verbAna_k

This picture of proactive interference accumulation and release with verbal memoranda, though detectable in our modeling, didn’t provide a compelling comparison for the utterly null effects we found with visual memoranda:

fig2_visAna_k

These figures came from our first attempt. Though the evidence for proactive interference in the verbal task was underwhelming, at first we believed that we had a small but legitimate effect that was simply being masked by something trivial about our task or design, and with tweaking, we could make it emerge more strongly. At this point, I’ve conducted several follow-ups, and it never gets clearer. Sometimes we find no release of proactive interference with verbal stimuli, sometimes we find a very small effect. We don’t see it when participants have to reconstruct the stimuli rather than recognize them (at least, not yet). We don’t see it for verbal materials that are recalled via speech (at least, not yet). We never see it with visual memoranda, but that finding is difficult to interpret without the clear, expected verbal effect as a foil. A summary of the results of our series of studies on this, along with the data, analysis scripts, and experimental paradigms is available on OSF. This summary document outlines what we’ve done so far.

 

I’m happy to #BringOutYerNulls and let anyone examine our progress with this; in fact, these results have been available on OSF for quite a while and I’ve discussed them with a few colleagues. But interpreting a null effect, even in light of the strong expectation that an effect should have occurred, isn’t easy. There are many reasons why this might not have worked, and many of them are really boring. Possibly, these null results are really important: maybe there are boundary conditions on proactive interference that limit its generality, and if we firmly established these conditions we would know something novel and fundamental about memory.

Share this post:

A visit from the Ghost of Research Past

A request for an old data set recently afforded me the opportunity, much like Ebenezer Scrooge, of revisiting my Past-Self when I was a brand-new post-graduate student, and allowing Past-Self and Future-Self to help me critique how my lab curates our data and materials in the present day. Both Past-Self and Future-Self are compelling agitators for a proactive approach to opening data, especially implementing a Data Partner scheme. 

Openness about our work is consistent with believing that the work is important and excellent. Being asked for access to your work is an acknowledgement that it is valuable, and sharing it is an expression of your confidence in its value. I’ve found openness to be rewarding, leading to additional citations, gracious acknowledgments, and sometimes new collaboration opportunities. 

However, requests for data or materials fluster us, arriving out-of-the-blue. It always seems necessary to perform fresh checks: Is the code understandable and functional? Data may need to be explained and possibly tidied: what do the column headings mean again? Could there be identifying details in any of the responses? I might spend hours performing these checks before complying with a request. 

Waiting until the request arrives to open up data and materials can be seen as a tacit judgment on the expected impact of the data. Why, if I believe the work I do is worthwhile, am I not preparing it for public consumption before I publish it? When did I start imagining that no one was likely to be interested in re-analyzing my data or using my experimental code?  

Recently I was asked for data from the first paper I ever published, part of my master’s research project, which were collected in autumn 2002 and published in 2004. Possibly, sharing data that has been untouched for more than 10 years is asking too much. It wouldn’t have been strange if I had lost it in institutional moves and computer crashes, or if it proved impossible to adequately document. But if found, going through these data would give me an opportunity to pay a visit to my Past-Self, recall what it was like to begin a research project for the first time, and maybe learn something from her.

One thing that struck me as I examined Past-Self’s data is that Past-Self organized it expecting that other people would be looking at it. Past-Self inserted comments explaining what numeric codes meant. Past-Self wrote summaries of the purpose of experiments, and Past-Self organized files into hierarchical directories with sub-folders for data files, analyses, and experimental stimuli. I think it would have surprised Past-Self that no one would ask to look at this information until 2015. Past-Self thought this work was important and documented it accordingly.

Though Past-Self began as a data-sharing idealist, she had minimal skills for curating data and materials. Some organization elements improved drastically in the later experiments in her project. Past-self learned it is better to make category codes self-explanatory (e.g., why assign “male” or “female” to arbitrary numeric codes instead of just entering the words?). Past-self developed sensible conventions for naming files. Past-self reduced redundancies in data recording. 

But though some practices improved, it also became clear that Past-Self abandoned the expectation that anyone apart from her and her supervisor would ever see these raw data and materials. As the project drew on, the helpful comments disappeared, and the summaries for subsequent experiments were unchanged from the earliest ones. The whole directory was organized around an 8-experiment master’s project, which eventually resulted in the publication of three experiments in two separate papers. Past-Self never re-organized these materials so that it would be immediately obvious how to locate the materials pertaining to each paper specifically.

Altogether I interrogated Past-Self for about 5 hours: we located the data sets requested, established through re-analysis that they did in fact include the same data that were published, saved them in an accessible non-proprietary format, documented what the data sets contained and how these variables were coded, and published the data and guidance on Open Science Framework. On the one hand, that isn’t terrible. My Future-Self, who checked in throughout this process, insists that 5 hours of work accomplished now is a sound investment. It enables a colleague on the other side of the planet to do a meaningful new analysis, from which we might all learn something novel. Furthermore, those data are now available to anyone else who might have other ideas for how our data can be useful. Future-Self insists that this will lead to glory. On the other hand, this 5 hours of work entirely replicated work that Past-Self did more than 10 years ago in her haphazard manner. If Past-Self had carried on carefully documenting her data, if she had considered that materials should be available in commonly accessible formats, and if she had updated her personal repository to reflect the published record, then these materials would have been ready for sharing upon request in minutes, not hours. Future-Self is anxious to know how I am going to prevent this waste of time. Past-Self wonders whether I can do more to help my trainees learn good habits.

What, if any, are the constraints to proactively curating lab work? Proactive curation is obviously desirable for Future-Self: it saves her time and effort and it increases the impact and utility of the work. It is arguably good for trainees and PIs alike. Because I work with many short-term trainees, I have handled most data curation myself, but this is a valuable skill that Past-Self needed to learn better, and that Future-Self wants delegated. The Data Partner scheme is ideal for this: my trainees can be paired with trainees from a colleague’s lab, and these two students will help each other curate data by seeing whether their partner’s work is clear, self-explanatory, and reproducible. They do this independently of me. When the data are shown to me, they have already been vetted by one other person, providing an additional chance to catch mistakes. My trainees get the practice that Past-Self lacked, and Future-Self will never wonder whether data and materials are ready to be shared.

Are you at Psychonomics 2015? Come to our talk, Open Science: Practical Guidance for Psychological Scientists, Friday at 10:40 am, in the Statistics and Methodology II session.

Update: Check out Lorne Campbell’s thoughts on this too.

 

Share this post: