Appending to an existing netCDF file fails with scipy==1.0.1

[shoyer]

https://travis-ci.org/pydata/xarray/builds/359093748

Example failure:

_____________________ ScipyFilePathTest.test_append_write ______________________
self = <xarray.tests.test_backends.ScipyFilePathTest testMethod=test_append_write>
    def test_append_write(self):
        # regression for GH1215
        data = create_test_data()
>       with self.roundtrip_append(data) as actual:
xarray/tests/test_backends.py:786: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../miniconda/envs/test_env/lib/python3.6/contextlib.py:81: in __enter__
    return next(self.gen)
xarray/tests/test_backends.py:155: in roundtrip_append
    self.save(data[[key]], path, mode=mode, **save_kwargs)
xarray/tests/test_backends.py:162: in save
    **kwargs)
xarray/core/dataset.py:1131: in to_netcdf
    unlimited_dims=unlimited_dims)
xarray/backends/api.py:657: in to_netcdf
    unlimited_dims=unlimited_dims)
xarray/core/dataset.py:1068: in dump_to_store
    unlimited_dims=unlimited_dims)
xarray/backends/common.py:363: in store
    unlimited_dims=unlimited_dims)
xarray/backends/common.py:402: in set_variables
    self.writer.add(source, target)
xarray/backends/common.py:265: in add
    target[...] = source
xarray/backends/scipy_.py:61: in __setitem__
    data[key] = value
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.io.netcdf.netcdf_variable object at 0x7fe3eb3ec6a0>
index = Ellipsis, data = array([0. , 0.5, 1. , 1.5, 2. , 2.5, 3. , 3.5, 4. ])
    def __setitem__(self, index, data):
        if self.maskandscale:
            missing_value = (
                    self._get_missing_value() or
                    getattr(data, 'fill_value', 999999))
            self._attributes.setdefault('missing_value', missing_value)
            self._attributes.setdefault('_FillValue', missing_value)
            data = ((data - self._attributes.get('add_offset', 0.0)) /
                    self._attributes.get('scale_factor', 1.0))
            data = np.ma.asarray(data).filled(missing_value)
            if self._typecode not in 'fd' and data.dtype.kind == 'f':
                data = np.round(data)
    
        # Expand data for record vars?
        if self.isrec:
            if isinstance(index, tuple):
                rec_index = index[0]
            else:
                rec_index = index
            if isinstance(rec_index, slice):
                recs = (rec_index.start or 0) + len(data)
            else:
                recs = rec_index + 1
            if recs > len(self.data):
                shape = (recs,) + self._shape[1:]
                # Resize in-place does not always work since
                # the array might not be single-segment
                try:
                    self.data.resize(shape)
                except ValueError:
                    self.__dict__['data'] = np.resize(self.data, shape).astype(self.data.dtype)
>       self.data[index] = data
E       ValueError: assignment destination is read-only

[jhamman]

this may be related to this recent PR: scipy/scipy#8483

[shoyer]

The only other netCDF related PR between scipy 1.0.0 and 1.0.1 is scipy/scipy#8068, which also looks like a plausible culprit.

[stale]

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically

[mathause]

This was fixed in scipy 1.1.0 (scipy/scipy/pull/8633). Currently xarray requires

xarray/ci/requirements/py36-min-all-deps.yml

Line 45 in 18e34cc

- scipy=1.3

so I think these can be un-xfailed.

xarray/xarray/tests/test_backends.py

Line 1969 in 18e34cc

class ScipyWriteBase(CFEncodedBase, NetCDF3Only):

[mathause]

This was fixed in scipy 1.1.0 (scipy/scipy/pull/8633). Currently xarray requires

xarray/ci/requirements/py36-min-all-deps.yml

Line 45 in 18e34cc

- scipy=1.3

so I think these can be un-xfailed.

xarray/xarray/tests/test_backends.py

Line 1969 in 18e34cc

class ScipyWriteBase(CFEncodedBase, NetCDF3Only):