non-default spaces fails under 20.1.0 singularity container (but not 20.0.7)

[dmd]

/home/fmriprep/.cache/templateflow is missing from the Singularity image for 20.1.0.

Therefore, under the 20.0.7 singularity container, if I specify a nondefault space like:

--output-spaces MNI152NLin6Asym:res-2 anat fsaverage

it works fine, and does not attempt to download anything (which won't work in the container, and won't work on a cluster where nodes can't reach the internet).

The exact same dataset, submitted with the exact same arguments but using the 20.1.0 container, fails:

Downloading https://templateflow.s3.amazonaws.com/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/util/connection.py", line 79, in create_connection
    raise err
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/util/connection.py", line 69, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

[mgxd]

I just checked the 20.1.0 image:

$ singularity shell fmriprep_20.1.0.simg 
Singularity> ls /home/fmriprep/.cache/templateflow/
tpl-MNI152Lin		 tpl-MNI152NLin2009cSym  tpl-MNI152NLin6Sym  tpl-MNIPediatricAsym  tpl-OASIS30ANTs  tpl-WHS   tpl-fsaverage
tpl-MNI152NLin2009cAsym  tpl-MNI152NLin6Asym	 tpl-MNIInfant	     tpl-NKI		   tpl-PNC	    tpl-fsLR
Singularity> ls -lh /home/fmriprep/.cache/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz
-rw-rw-r-- 1 root root 31M May 28 10:04 /home/fmriprep/.cache/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz

What does your command look like?

[dmd]

Huh. It looks like that. I could have sworn when I checked a few minutes ago it wasn't there.

Anyway, so that's not the reason, so what is the reason for this failure?

[effigies]

Do you have SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow defined?

[dmd]

No, but I didn't on 20.0.7 either.

[dmd]

This works:

/usr/bin/singularity run \
    --contain \
    --cleanenv \
    -B /tmp -B /data -B /data1 -B /data2 -B /data3 -B /cm/shared \
    /cm/shared/singularity/images/fmriprep-20.0.7.simg \
    /data/pipeline/PIPE_20200528_201740194 \
    /data/pipeline/PIPE_20200528_201740194/derivatives \
    participant \
    --fs-license-file /cm/shared/freesurfer-6.0.1/license.txt \
    --participant_label e001baseline \
    --output-spaces MNI152NLin6Asym:res-2 anat fsaverage \
    --n_cpus 8 \
    --mem-mb 16384 \
    --notrack \
    --dummy-scans 2 \
    --use-aroma \
    --use-syn-sdc \
    -vv \
    -w  /data/pipeline/fmriprep-working/PIPE_20200528_201740194

The exact same thing submitted with the same environment but with 20.1.0.simg fails.

[dmd]

Adding that environment variable fixed it. Not sure why it wasn't needed under 20.0.7, though.

[oesteban]

That's weird, the Dockerfile of both tags seems the same.

[mgxd]

I'm not sure why the 20.0.7 image worked, but the behavior of the 20.1.0 image is expected - singularity users have to explictily set the TEMPLATEFLOW_HOME envvar to /home/fmriprep/.cache/templateflow to take advantage of the pre-cached templates

[effigies]

Was there a reason we didn't put TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow in the Dockerfile? I can't quite remember.

[mgxd]

Yes, the reason is Singularity users won't have write permission to that directory. So by default, users requesting non-cached templates would experience failures