Introduction

Lately I’ve been spending a lot of time with Experiment Factory (expfactory) to build and run experimental cognitive tasks. For those who don’t know, expfactory is a framework for building reproducible behavioral experiments, using the jsPsych JavaScript library. But it goes a step further than jsPsych by not only allowing the researcher to write code using handy functions, but allowing them to write entire experiments that can be “containerized” using Docker. Containers are critical for software reproducibility for a variety of reasons that Carl Boettiger discusses in “An introduction to Docker for reproducible research, with examples from the R environment”. One of these reasons is what Boettiger calls dependency hell: the problem of recreating the same computational environment for which an experiment was originally designed.

Just this week I had the frustrating experience of dealing with this issue. A colleague shared code from an experiment they wrote only a few days ago in PsychoPy. But because the code was written for an old version of the software (PsychoPy2 Experiment Builder v1.84.2, more than 2 years old!), I had trouble getting it to run. I could not get it to work in PsychoPy3, the latest version, and it would only sort of run in the latest version of PsychoPy2 (v1.9; it would start, but crash after the instructions). The only way I could get it to run was to install the exact same version of PsychoPy with exactly the same dependencies. It was very fortunate that the code included comments that specified the version of PsychoPy. If it had not, I may have never gotten it to run! (As Boettiger discusses, poor documentation is another major barrier to reproducibility).

Since I don’t want the same fate to befall the experiments I code, I’ve been using expfactory to write containerized experiments and publishing them to the expfactory github repo. There’s a bit of a learning curve to getting started, especially for someone who is not familiar with some of these technologies (Docker, GitHub).

I thought I would share how I begin forking an expfactory task. This will be useful for anyone who wants to take an existing expfactory task and modify it. For example, maybe you want to run a Stroop task, but you want to increase the number of trials, or change the instructions text. You can also take a base task and use that to create something entirely new! Because many cognitive paradigms follow similar procedures, this can really speed up the development time.

Checklist

To begin, you will need:

  • A Linux server with Docker installed
  • An expfactory task that you want to use as your base, to fork.
    (Take a look at the library of tasks.)
  • The ability to SSH into the web server (this is how we access the terminal/console to do stuff; I use PuTTY)

I like to use DigitalOcean for my web servers, and for this tutorial I’ll be forking the kirby task. My linux user is bitnami, so you will also be seeing that in the code below.

Recipes

Set shell variables

First we’ll set some shell variables to make the other recipes more streamlined. HOME should point to the home directory for a user on your web server. Since I’m running as the bitnami user, I’ll put it all there. TASK should refer to the task that you want to fork.

TASK=kirby
HOME=/home/bitnami

Make directories

Like me, for the sake of convenience you’ll probably want to access the data and the logs from outside the Docker container. This doesn’t impact the containerization. It just lets you access output from the experiment more easily once you have everything up and running.

mkdir -p $HOME/expfactory/$TASK
mkdir -p $HOME/expfactory/$TASK/data
mkdir -p $HOME/expfactory/$TASK/logs
sudo chmod -R 777 $HOME/expfactory/$TASK/

Generate Dockerfile

Now we’ll change to the task directory we created above and build the Dockerfile in that directory.

# go to directory
cd $HOME/expfactory/$TASK/

sudo docker run -v $HOME/expfactory/$TASK:/data \
  vanessa/expfactory-builder \
  build $TASK

Great! Now if you issue a dir command you will see the following folders/files: data Dockerfile logs startscript.sh

Build container

Next we’ll build the docker container, using the Dockerfile we just created.

sudo docker build -t expfactory/experiments .

Clone task files to host directory

Now, while we are developing the task, we want to make it as easy as possible to modify “on the fly”, even while it is running in a container. To do this, first we must clone the task files from github to a local directory

cd /home/bitnami
sudo git clone https://github.com/expfactory-experiments/$TASK.git
sudo chmod -R 777 $TASK
cd $TASK

Run container

Now we run the container. When we run the container, we pass it all of the folders that we created before.

sudo docker run -d \
   -d -p 80:80 \
   -v $HOME/expfactory/expfactory:/opt/expfactory \
   -v $HOME/expfactory/$TASK/data:/scif/data \
   -v $HOME/expfactory/$TASK/logs:/scif/logs \
   -v $HOME/$TASK:/scif/apps/$TASK \
   expfactory/experiments

Ready to go!

Now it should be up and running. By default, expfactory runs over port 80, so you should be able to access it by typing the URL of your server into a web browser.

Because of the configuration we’ve used in these recipes, the task files are served out of /home/$TASK on the host-side. This makes it really easy to work with during the development process. The file that you probably want to start hacking away at is /home/$TASK/experiment.js, and the data will be stored in $HOME/expfactory/$TASK/data. After you’ve modified the task, you can reload the site in your browser to see the changes live.

When you finally complete your task, you should make it official by contributing it to the github repo. Happy coding!