FIRST segmentation throws traiterror

[RiaanZoetmulder]

Summary

I am writing code to create several segmentations, one of which is segmentations using FIRST. This seems to create some issues however. It throws this error and no crash log:

                Currently running:
                   * _segmentation1
                   * _segmentation0

190203-13:17:07,238 nipype.workflow WARNING:
[Node] Error on "_segmentation1" (/mnt/e/Projects/medical-taskonomy/testdata/example_preprocess/segmentation/mapflow/_segmentation1)
190203-13:17:08,681 nipype.workflow ERROR:
Node _segmentation1 failed to run on host 18-012665.
190203-13:17:08,684 nipype.workflow ERROR:
Saving crash info to /home/rzoetmulder/crash-20190203-131708-rzoetmulder-_segmentation1-57d356fa-459c-44c2-aa7c-a08fc2626ff8.pklz
Traceback (most recent call last):
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
result['result'] = node.run(updatehash=updatehash)
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 473, in run
result = self._run_interface(execute=True)
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 557, in _run_interface
return self._run_command(execute)
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 637, in _run_command
result = self._interface.run(cwd=outdir)
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 371, in run
outputs = self.aggregate_outputs(runtime)
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 472, in aggregate_outputs
raise error
TraitError: The trait 'vtk_surfaces' of a FIRSTOutputSpec instance is an existing file name, but the path '/mnt/e/Projects/medical-taskonomy/testdata/example_preprocess/segmentation/mapflow/_segmentation1/segmented-R_HippL_Accu_first.vtk' does not exist.

190203-13:17:08,705 nipype.workflow INFO:
[MultiProc] Running 1 tasks, and 0 jobs ready. Free memory (GB): 57.12/57.32, Free processors: 3/4.
Currently running:
* _segmentation0
190203-13:17:18,970 nipype.workflow WARNING:
[Node] Error on "_segmentation0" (/mnt/e/Projects/medical-taskonomy/testdata/example_preprocess/segmentation/mapflow/_segmentation0)
190203-13:17:20,688 nipype.workflow ERROR:
Node _segmentation0 failed to run on host 18-012665.
190203-13:17:20,691 nipype.workflow ERROR:
Saving crash info to /home/rzoetmulder/crash-20190203-131720-rzoetmulder-_segmentation0-6b6a4de0-b131-4dfe-9e1f-916d4c8c79b2.pklz
Traceback (most recent call last):
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
result['result'] = node.run(updatehash=updatehash)
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 473, in run
result = self._run_interface(execute=True)
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 557, in _run_interface
return self._run_command(execute)
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 637, in _run_command
result = self._interface.run(cwd=outdir)
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 371, in run
outputs = self.aggregate_outputs(runtime)
File "/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 472, in aggregate_outputs
raise error
TraitError: The trait 'vtk_surfaces' of a FIRSTOutputSpec instance is an existing file name, but the path '/mnt/e/Projects/medical-taskonomy/testdata/example_preprocess/segmentation/mapflow/_segmentation0/segmented-R_HippL_Accu_first.vtk' does not exist.

190203-13:17:20,704 nipype.workflow INFO:
[MultiProc] Running 0 tasks, and 0 jobs ready. Free memory (GB): 57.32/57.32, Free processors: 4/4.
190203-13:17:22,664 nipype.workflow INFO:
***********************************
190203-13:17:22,667 nipype.workflow ERROR:
could not run node: example_preprocess.segmentation
190203-13:17:22,670 nipype.workflow INFO:
crashfile: /home/rzoetmulder/crash-20190203-131708-rzoetmulder-_segmentation1-57d356fa-459c-44c2-aa7c-a08fc2626ff8.pklz
190203-13:17:22,674 nipype.workflow ERROR:
could not run node: example_preprocess.segmentation
190203-13:17:22,676 nipype.workflow INFO:
crashfile: /home/rzoetmulder/crash-20190203-131720-rzoetmulder-_segmentation0-6b6a4de0-b131-4dfe-9e1f-916d4c8c79b2.pklz
190203-13:17:22,679 nipype.workflow INFO:
***********************************

RuntimeError Traceback (most recent call last)
in ()
105 process.write_graph(graph2use='colored', format='pdf', simple_form=True)
106
--> 107 process.run('MultiProc', plugin_args={'n_procs': 4})

/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/engine/workflows.pyc in run(self, plugin, plugin_args, updatehash)
597 if str2bool(self.config['execution']['create_report']):
598 self._write_report_info(self.base_dir, self.name, execgraph)
--> 599 runner.run(execgraph, updatehash=updatehash, config=self.config)
600 datestr = datetime.utcnow().strftime('%Y%m%dT%H%M%S')
601 if str2bool(self.config['execution']['write_provenance']):

/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/plugins/base.pyc in run(self, graph, config, updatehash)
189
190 self._remove_node_dirs()
--> 191 report_nodes_not_run(notrun)
192
193 # close any open resources

/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/plugins/tools.pyc in report_nodes_not_run(notrun)
80 logger.debug(subnode._id)
81 logger.info("***********************************")
---> 82 raise RuntimeError(('Workflow did not execute cleanly. '
83 'Check log for details'))
84

RuntimeError: Workflow did not execute cleanly. Check log for details

Actual behavior

All the other methods (canny edge detector, BET etc.) work. So my installation of FSL/Nipype should be correct.
I have set FIRST to only segment one structure namely: 'R_HippL_Accu'. It throws the error shown above, I think it just doesn't save the surface mesh in the correct folder. It doesn't work with different structures either.

I have already gone to https://miykael.github.io/nipype_tutorial/notebooks/basic_error_and_crashes.html

to figure out the crashes, though I couldn't fix it.

Script/Workflow details

Link to code

Platform details:

Using windows with linux subsystems.
Version of Linux: Ubuntu
version of windows: windows 10

Execution environment

Anaconda with jupyter lab.

[satra]

this is strange as we have a very similar workflow that we used here with FSL 5.0.9 and nipype 0.12: https://github.com/ReproNim/simple_workflow/blob/master/run_demo_workflow.py#L103

could you please let us know which version of FSL you are using?

[RiaanZoetmulder]

'which fsl' give:

/usr/lib/fsl/5.0/fsl

So version 5.0

[satra]

we will need the subversion to check. could you please paste the output of this command?

python -c "from nipype.interfaces.fsl import Info; print(Info.version())"

[RiaanZoetmulder]

This yields: 5.0.9

[satra]

thanks @RiaanZoetmulder - the only things i can think of are that your FIRST data files are missing:

you can check this with:

ls $FSLDIR/data/first/models*

and when you install FSL in your windows subsystem, you can do:

apt-get install fsl-core fsl-first-data

other than these two, i'm at a loss as to why those files would not be produced by the same version of FSL.

[mgxd]

@RiaanZoetmulder this may be related to #2813 - could you check the type of the transformation matrix generated? it should be in the /mnt/e/Projects/medical-taskonomy/testdata/example_preprocess/segmentation/mapflow/_segmentation1 folder

[RiaanZoetmulder]

thanks @RiaanZoetmulder - the only things i can think of are that your FIRST data files are missing:

you can check this with:

ls $FSLDIR/data/first/models*

and when you install FSL in your windows subsystem, you can do:

apt-get install fsl-core fsl-first-data

other than these two, i'm at a loss as to why those files would not be produced by the same version of FSL.

Hey,
Thanks for your response. I have run the command apt-get install fsl-first-data .
fsl-core should already be installed otherwise BET would already not be working right?

I have checked the directory, I can find the files here:

/usr/share/fsl/data/first/models_336_bin

There is a bunch of .bmv files there, with the names of anatomical parts of the brein before the dot.

Should I link these files somewhere?

[RiaanZoetmulder]

@RiaanZoetmulder this may be related to #2813 - could you check the type of the transformation matrix generated? it should be in the /mnt/e/Projects/medical-taskonomy/testdata/example_preprocess/segmentation/mapflow/_segmentation1 folder

Do you mean the .mat file?

It is named: age0002_ABI50032_brain_to_std_sub.mat

[RiaanZoetmulder]

Just to make sure you have the overview of what I am doing. Before first I rotate, resample and interpolate and skull strip. Might this influence it?

[satra]

@RiaanZoetmulder - the files do not have to be linked. as long as FSL_DIR is defined it should work (in principle)

a couple of other ways to debug:

1. use the cli

nipypecli crash crashfile.pklz -r -i

this will run the node in an isolated directory and drop you into the ipython debugger (if ipython is available in your python environment). then you can check if first generated the files anywhere at all.

2. go into the node directory. you will find a command.txt and just execute the command inside it and check if the surface files are created.

3. run the same thing inside a docker container to check if it's specific to your OS. i should say we don't have good testing on windows because of it's various options and interactions with the windows subsystem for unix.

[RiaanZoetmulder]

I have run first in a different way. Firstly, the command line tool seems to work. When I run the python code like this however (different from my current implementation):

from nipype.interfaces import fsl
first = fsl.FIRST()
first.inputs.in_file = '/mnt/e/Projects/medical-taskonomy/testdata/data/original/age0001_ABI50030.nii'
first.inputs.out_file = 'segmented.nii'
res = first.run()

What happens when out_file is set to something in my windows environment in the mnt folder is that it saves the temporary outputs there, but when it looks for the vtk files it doesn't look in the output folder.

I can run it however in my home folder in the linux subsystem. Very strange behavior.

[satra]

@RiaanZoetmulder - if you run it directly it will not do several things to isolate the computation. you can instead run it inside a Node object.

from nipype import Node
from nipype.interfaces import fsl
first = Node(fsl.FIRST(), name='FIRST')
first.inputs.in_file = '/mnt/e/Projects/medical-taskonomy/testdata/data/original/age0001_ABI50030.nii'
first.inputs.out_file = 'segmented.nii'
res = first.run()

[RiaanZoetmulder]

@RiaanZoetmulder - if you run it directly it will not do several things to isolate the computation. you can instead run it inside a Node object.

from nipype import Node
from nipype.interfaces import fsl
first = Node(fsl.FIRST(), name='FIRST')
first.inputs.in_file = '/mnt/e/Projects/medical-taskonomy/testdata/data/original/age0001_ABI50030.nii'
first.inputs.out_file = 'segmented.nii'
res = first.run()

I have done that to check what happens:

from nipype import Node
from nipype.interfaces import fsl
first = Node(fsl.FIRST(), name='FIRST')
first.inputs.in_file = '/mnt/e/Projects/medical-taskonomy/testdata/data/original/age0001_ABI50030.nii'
first.inputs.out_file = 'segmented.nii'
res = first.run()

This puts the output in the tmp/tmpdOjL15/FIRST and works

However if I want to change my output folder to:

first.inputs.out_file = '/mnt/e/Projects/medical-taskonomy/testdata/data/segmented.nii'

It puts all of the .vtk and .bvar files in that folder, but when it collects all of these to make the final segmentation file it looks in the /tmp/tmpdOjL15/FIRST/ folder.

This is the error it throws:

190208-16:04:19,11 nipype.workflow INFO:
[Node] Setting-up "FIRST" in "/tmp/tmpdOjL15/FIRST".
190208-16:04:19,44 nipype.workflow INFO:
[Node] Running "FIRST" ("nipype.interfaces.fsl.preprocess.FIRST"), a CommandLine Interface with command:
run_first_all -m auto -i /tmp/tmpdOjL15/FIRST/age0001_ABI50030.nii -o /mnt/e/Projects/medical-taskonomy/testdata/data/segmented.nii
190208-16:14:32,157 nipype.interface INFO:
stdout 2019-02-08T16:14:32.157138:23403
190208-16:14:32,319 nipype.workflow WARNING:
[Node] Error on "FIRST" (/tmp/tmpdOjL15/FIRST)

TraitError Traceback (most recent call last)
in ()
4 first.inputs.in_file = '/mnt/e/Projects/medical-taskonomy/testdata/data/original/age0001_ABI50030.nii'
5 first.inputs.out_file = '/mnt/e/Projects/medical-taskonomy/testdata/data/segmented.nii'
----> 6 res = first.run()

/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.pyc in run(self, updatehash)
471
472 try:
--> 473 result = self._run_interface(execute=True)
474 except Exception:
475 logger.warning('[Node] Error on "%s" (%s)', self.fullname, outdir)

/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.pyc in _run_interface(self, execute, updatehash)
555 self._update_hash()
556 return self._load_results()
--> 557 return self._run_command(execute)
558
559 def _load_results(self):

/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.pyc in _run_command(self, execute, copyfiles)
635 logger.info(message)
636 try:
--> 637 result = self._interface.run(cwd=outdir)
638 except Exception as msg:
639 result.runtime.stderr = '%s\n\n%s'.format(

/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/interfaces/base/core.pyc in run(self, cwd, ignore_exception, **inputs)
369 runtime = self._run_interface(runtime)
370 runtime = self._post_run_hook(runtime)
--> 371 outputs = self.aggregate_outputs(runtime)
372 except Exception as e:
373 import traceback

/home/rzoetmulder/anaconda3/envs/nipype/lib/python2.7/site-packages/nipype/interfaces/base/core.pyc in aggregate_outputs(self, runtime, needed_outputs)
470 "'%s'." % (val, self.class.name, key))
471 raise FileNotFoundError(msg)
--> 472 raise error
473
474 return outputs

TraitError: The trait 'vtk_surfaces' of a FIRSTOutputSpec instance is an existing file name, but the path '/tmp/tmpdOjL15/FIRST/segmented-L_Hipp_first.vtk' does not exist.

[satra]

@RiaanZoetmulder - ah then we know the issue here. i would file this as a fixable FIRST bug (would love a PR if you have the bandwidth to do so - i'm sure we can guide you).

in general when Node is used, we don't expect the workflow developer to specify an absolute output path. in general Nipype suggests that this should happen with a DataSink. however, we can consider the scenario in FIRST, where such a file is specified and adjust the outputs appropriately.

[RiaanZoetmulder]

@RiaanZoetmulder - ah then we know the issue here. i would file this as a fixable FIRST bug (would love a PR if you have the bandwidth to do so - i'm sure we can guide you).

in general when Node is used, we don't expect the workflow developer to specify an absolute output path. in general Nipype suggests that this should happen with a DataSink. however, we can consider the scenario in FIRST, where such a file is specified and adjust the outputs appropriately.

Sent you an email regarding that! Let's continue the conversation there!

[mgxd]

Hi Riann - I went ahead and made a PR that addresses this bug. You can try it out by installing that branch:
pip install -U https://github.com/mgxd/nipype/archive/fix/first.zip