Skip to content

local test failure - pandas/tests/io/test_parquet.py::TestBasic::test_compression[fastparquet-snappy] #32327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
simonjayhawkins opened this issue Feb 28, 2020 · 4 comments
Closed

local test failure - pandas/tests/io/test_parquet.py::TestBasic::test_compression[fastparquet-snappy] #32327

simonjayhawkins opened this issue Feb 28, 2020 · 4 comments
Labels
IO Parquet parquet, feather Testing pandas testing functions or related to the test suite Unreliable Test Unit tests that occasionally fail Windows Windows OS

Comments

@simonjayhawkins
Copy link
Member 

$ conda list snappy
# packages in environment at C:\Users\simon\Anaconda3\envs\pandas-dev:
#
# Name                    Version                   Build  Channel
python-snappy             0.5.4            py37hd25c944_1    conda-forge
snappy                    1.1.8                he025d50_1    conda-forge

pandas/tests/io/test_parquet.py::TestBasic::test_compression[fastparquet-snappy] Windows fatal exception: code 0xc000001d

Current thread 0x00002074 (most recent call first):
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\snappy\snappy.py", line 92 in uncompress
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\compression.py", line 44 in snappy_decompress
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\compression.py", line 145 in decompress_data
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\core.py", line 25 in _read_page
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\core.py", line 99 in read_data_page
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\core.py", line 245 in read_col
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\core.py", line 331 in read_row_group_arrays
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\core.py", line 354 in read_row_group
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\api.py", line 228 in read_row_group
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\api.py", line 399 in to_pandas
  File "C:\Users\simon\pandas\pandas\io\parquet.py", line 195 in read
  File "C:\Users\simon\pandas\pandas\io\parquet.py", line 300 in read_parquet
  File "C:\Users\simon\pandas\pandas\tests\io\test_parquet.py", line 168 in compare
  File "C:\Users\simon\pandas\pandas\tests\io\test_parquet.py", line 174 in check_round_trip
  File "C:\Users\simon\pandas\pandas\tests\io\test_parquet.py", line 317 in test_compression
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\python.py", line 167 in pytest_pyfunc_call
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 86 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 92 in _hookexec
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\python.py", line 1445 in runtest
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 134 in pytest_runtest_call
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 86 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 92 in _hookexec
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 210 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 237 in from_call
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 210 in call_runtest_hook
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 185 in call_and_report
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 99 in runtestprotocol
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 84 in pytest_runtest_protocol
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 86 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 92 in _hookexec
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 271 in pytest_runtestloop
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 86 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 92 in _hookexec
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 247 in _main
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 197 in wrap_session
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 240 in pytest_cmdline_main
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 86 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 92 in _hookexec
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\config\__init__.py", line 93 in main
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\Scripts\pytest-script.py", line 10 in <module>
Fatal Python error: Illegal instruction

Current thread 0x00002074 (most recent call first):
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\snappy\snappy.py", line 92 in uncompress
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\compression.py", line 44 in snappy_decompress
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\compression.py", line 145 in decompress_data
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\core.py", line 25 in _read_page
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\core.py", line 99 in read_data_page
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\core.py", line 245 in read_col
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\core.py", line 331 in read_row_group_arrays
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\core.py", line 354 in read_row_group
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\api.py", line 228 in read_row_group
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\fastparquet\api.py", line 399 in to_pandas
  File "C:\Users\simon\pandas\pandas\io\parquet.py", line 195 in read
  File "C:\Users\simon\pandas\pandas\io\parquet.py", line 300 in read_parquet
  File "C:\Users\simon\pandas\pandas\tests\io\test_parquet.py", line 168 in compare
  File "C:\Users\simon\pandas\pandas\tests\io\test_parquet.py", line 174 in check_round_trip
  File "C:\Users\simon\pandas\pandas\tests\io\test_parquet.py", line 317 in test_compression
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\python.py", line 167 in pytest_pyfunc_call
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 86 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 92 in _hookexec
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\python.py", line 1445 in runtest
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 134 in pytest_runtest_call
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 86 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 92 in _hookexec
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 210 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 237 in from_call
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 210 in call_runtest_hook
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 185 in call_and_report
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 99 in runtestprotocol
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 84 in pytest_runtest_protocol
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 86 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 92 in _hookexec
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 271 in pytest_runtestloop
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 86 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 92 in _hookexec
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 247 in _main
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 197 in wrap_session
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 240 in pytest_cmdline_main
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 86 in <lambda>
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 92 in _hookexec
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\config\__init__.py", line 93 in main
  File "C:\Users\simon\Anaconda3\envs\pandas-dev\Scripts\pytest-script.py", line 10 in <module>

INSTALLED VERSIONS

commit : bd2ee0f
python : 3.7.6.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : None.None

pandas : 1.1.0.dev0+631.gbd2ee0fa1
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.2.0.post20200209
Cython : 0.29.15
pytest : 5.3.5
hypothesis : 5.5.4
sphinx : 2.4.3
blosc : None
feather : None
xlsxwriter : 1.2.8
lxml.etree : 4.5.0
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.1
IPython : 7.12.0
pandas_datareader: None
bs4 : 4.8.2
bottleneck : 1.3.2
fastparquet : 0.3.3
gcsfs : None
lxml.etree : 4.5.0
matplotlib : 3.1.3
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.1
pandas_gbq : None
pyarrow : 0.16.0
pytables : None
pytest : 5.3.5
pyxlsb : None
s3fs : 0.4.0
scipy : 1.3.1
sqlalchemy : 1.3.13
tables : 3.6.1
tabulate : 0.8.6
xarray : 0.15.0
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.2.8
numba : 0.48.0
@simonjayhawkins simonjayhawkins added the IO Parquet parquet, feather label Feb 28, 2020
@ShaharNaveh
Copy link
Member

ShaharNaveh commented Feb 28, 2020 •
edited
Loading

I guess this only relevant for Windows users as I can't reproduce this on master

Test log:

(venv-pandas) $ pytest pandas/tests/io/test_parquet.py 
============================================ test session starts =============================================
platform linux -- Python 3.8.1, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /home/user/Documents/Github/Community/Python/Projects/pandas-MomIsBestFriend, inifile: setup.cfg
plugins: cov-2.8.1, forked-1.1.3, xdist-1.31.0, asyncio-0.10.0, hypothesis-5.5.4
collected 56 items                                                                                           

pandas/tests/io/test_parquet.py ................s...s........................s..........               [100%]

============================================== warnings summary ==============================================
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_filter_row_groups
pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_partition_cols_supported
pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_partition_cols_string
pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_partition_on_supported
  /home/user/Documents/Github/Community/Python/Venvs/venv-pandas/lib/python3.8/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py:1929: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
    iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec])

pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::test_cross_engine_pa_fp
pandas/tests/io/test_parquet.py::TestParquetFastParquet::test_filter_row_groups
  /home/user/Documents/Github/Community/Python/Venvs/venv-pandas/lib/python3.8/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py:975: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
    iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec])

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================ 53 passed, 3 skipped, 20 warnings in 13.36s =================================

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 4e5e73e
python : 3.8.1.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.19.a-1-hardened
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.0.dev0+635.g4e5e73e23
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.2.0
Cython : 0.29.15
pytest : 5.3.5
hypothesis : 5.5.4
sphinx : 2.4.3
blosc : 1.8.3
feather : None
xlsxwriter : 1.2.8
lxml.etree : 4.5.0
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.1
IPython : 7.12.0
pandas_datareader: None
bs4 : 4.8.2
bottleneck : 1.3.2
fastparquet : 0.3.3
gcsfs : None
lxml.etree : 4.5.0
matplotlib : 3.1.3
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.1
pandas_gbq : None
pyarrow : 0.16.0
pytables : None
pytest : 5.3.5
pyxlsb : None
s3fs : 0.4.0
scipy : 1.4.1
sqlalchemy : 1.3.13
tables : 3.6.1
tabulate : 0.8.6
xarray : 0.15.0
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.2.8
numba : 0.48.0

python-snappy : 0.5.4

@simonjayhawkins simonjayhawkins added the Windows Windows OS label Feb 28, 2020
@simonjayhawkins
Copy link
Member Author

Thanks @MomIsBestFriend It occurred since recreating my local conda environment. This issue is a place holder until investigate which version of what package broke it. (I don't have a record of the previous versions installed)

not a pandas issue but may need to add skips to the test.

@ShaharNaveh
Copy link
Member

I guess the package snappy broke it, since this is the only package that's installed on your machine that I don't have it installed on mine.

@simonjayhawkins simonjayhawkins mentioned this issue Mar 4, 2020
Closed
@kendricng kendricng mentioned this issue Mar 4, 2020
Closed
@mroeschke mroeschke added the Unreliable Test Unit tests that occasionally fail label Apr 5, 2020
@mroeschke mroeschke added the Testing pandas testing functions or related to the test suite label Apr 19, 2020
@mroeschke
Copy link
Member

Looks like this test is running OK in our Windows CI so going to close for now. We can reopen if you are still seeing this with more recent packages

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
IO Parquet parquet, feather Testing pandas testing functions or related to the test suite Unreliable Test Unit tests that occasionally fail Windows Windows OS
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mroeschke @simonjayhawkins @ShaharNaveh