Skip to content

ValueError: Big-Endian buffer not supported on little-endian compiler #920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kahemker opened this issue Feb 29, 2020 · 4 comments · Fixed by #921
Closed

ValueError: Big-Endian buffer not supported on little-endian compiler #920

kahemker opened this issue Feb 29, 2020 · 4 comments · Fixed by #921

Comments

@kahemker
Copy link
Contributor

Describe the bug
Sometimes when using the process_data function in forecast.py (usually by invoking get_processed_data), the script would crash with the error:

ValueError: Big-Endian buffer not supported on little-endian compiler

I found the bug by using forecast.get_processed_data() function. I can pull forecast data consistently without errors using the only the get_data() functions.

ValueError: Big-endian buffer not supported on little-endian compiler

To Reproduce
I creating this bug on Windows 10-64 Bit OS with Python 3.8.2. You can follow the script written in this StackOverflow post and occasionally you should see this error.

Expected behavior
Data retrieved should not create runtime errors during processing

Versions:

  • pvlib.0.7.1
  • pandas.1.0.1
  • python: 3.8.2

Additional context
I believe I can fix this bug. I pushed my modified forecast.py here.

@wholmgren
Copy link
Member

Thanks for digging into this @kahemker. Can you also post the versions of netcdf and hdf5 that you're using?

@dopplershift, do you know anything about the byte-order behavior for netcdf subset on THREDDS? This issue and the SO post linked to above suggest that the returned data is sometimes big-endian and sometimes little-endian. This causes a problem when trying to convert to a pandas.Series. I thought netcdf was supposed to make all this just work. Thanks in advance!

@kahemker
Copy link
Contributor Author

kahemker commented Mar 2, 2020

I am using netcdf4 version 1.5.3 and I am not using hdf5. It does not appear in the setup.py list of dependencies for pvlib-python (required or optional).

Here is a pastebin that has my entire conda environment.yml file.

@dopplershift
Copy link

I'm pretty sure netCDF is supposed to be dealing with this. @WardF can you shed any light on netCDF endian handling?

Can someone point to the data in question?

@WardF
Copy link

WardF commented Mar 2, 2020

NetCDF handles endianness under the hood transparently; however, we are dealing with an issue currently (see Unidata/netcdf-c#1338) where there appears to be a problem. We recently got some Big Endian hardware in-house, to debug this for our upcoming 4.7.4 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants