build failure of master on windows

[ruidc]

the nightly windows builds don't appear to be produced anymore since mid january, and as we need some fixes in master i tried to build amd64 seems to work fine, but that's using windows SDK 7.0, when using VS 2008 Express or mingw I get different problems:

c:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ipandas/src/klib -Ipandas/src -IC:\Python27
\lib\site-packages\numpy\core\include -IC:\Python27\include -IC:\Python27\PC /Tcpandas/src/datetime/np_datetime.c /Fobuild\temp.win32-2.7\Release\pandas/src/datetime/np_datetime.obj
np_datetime.c
c:\python27\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by
 #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
pandas/src/datetime/np_datetime.c(229) : error C2143: syntax error : missing ')' before '('
pandas/src/datetime/np_datetime.c(229) : error C2143: syntax error : missing ')' before '('
pandas/src/datetime/np_datetime.c(229) : error C2091: function returns function
pandas/src/datetime/np_datetime.c(229) : error C2143: syntax error : missing ')' before 'string'
pandas/src/datetime/np_datetime.c(229) : error C2091: function returns function
pandas/src/datetime/np_datetime.c(229) : error C2143: syntax error : missing '{' before 'string'
pandas/src/datetime/np_datetime.c(229) : error C2059: syntax error : '<Unknown>'
pandas/src/datetime/np_datetime.c(229) : error C2059: syntax error : ')'
pandas/src/datetime/np_datetime.c(229) : error C2059: syntax error : ')'
pandas/src/datetime/np_datetime.c(229) : error C2059: syntax error : ')'
pandas/src/datetime/np_datetime.c(267) : warning C4244: '=' : conversion from 'npy_int64' to 'npy_int32', possible loss of data
pandas/src/datetime/np_datetime.c(852) : warning C4244: '=' : conversion from 'npy_datetime' to 'npy_int32', possible loss of data
pandas/src/datetime/np_datetime.c(866) : warning C4244: '=' : conversion from 'npy_datetime' to 'npy_int32', possible loss of data
pandas/src/datetime/np_datetime.c(881) : warning C4244: '=' : conversion from 'npy_datetime' to 'npy_int32', possible loss of data
pandas/src/datetime/np_datetime.c(897) : warning C4244: '=' : conversion from '__int64' to 'npy_int32', possible loss of data
pandas/src/datetime/np_datetime.c(914) : warning C4244: '=' : conversion from '__int64' to 'npy_int32', possible loss of data
pandas/src/datetime/np_datetime.c(931) : warning C4244: '=' : conversion from '__int64' to 'npy_int32', possible loss of data
pandas/src/datetime/np_datetime.c(949) : warning C4244: '=' : conversion from '__int64' to 'npy_int32', possible loss of data
pandas/src/datetime/np_datetime.c(959) : warning C4244: '=' : conversion from '__int64' to 'npy_int32', possible loss of data
pandas/src/datetime/np_datetime.c(976) : warning C4244: 'function' : conversion from 'npy_datetime' to 'int', possible loss of data
pandas/src/datetime/np_datetime.c(1002) : warning C4013: 'add_seconds_to_datetimestruct' undefined; assuming extern returning int
error: command '"c:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\cl.exe"' failed with exit status 2

with mingw, it builds but cannot import:

DLL load failed: The specified module could not be found.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\pandas-0.13.1.dev-py2.7-win32.egg\pandas\__init__.py", line 6, in <module>
    from . import hashtable, tslib, lib
ImportError: DLL load failed: The specified module could not be found.

dependency versions are:

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 32
OS: Windows
OS-release: XP
machine: x86
processor: x86 Family 6 Model 15 Stepping 2, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.13.1.dev
Cython: 0.20
numpy: 1.8.0
scipy: 0.13.2
statsmodels: None
IPython: 1.1.0
sphinx: None
patsy: None
scikits.timeseries: None
dateutil: 2.2
pytz: 2013.9
bottleneck: 0.8.0
tables: 3.1.0
numexpr: 2.3
matplotlib: 1.3.1
openpyxl: 1.6.1
xlrd: 0.9.2
xlwt: 0.7.5
xlsxwriter: None
sqlalchemy: None
lxml: None
bs4: None
html5lib: None
bq: None
apiclient: None

mingw build output files are here:
https://github.com/ruidc/files/blob/master/build_output.txt
https://github.com/ruidc/files/blob/master/install_output.txt

run using:

python setup.py build --compiler=mingw32 > build_output.txt 2>&1
python setup.py install > install_output.txt 2>&1

and directory listing here:
https://github.com/ruidc/files/blob/master/dir.txt

[jreback]

see this issue on how to setup builds: #5561
its a bit non-trivial to build windows. I run them nightly (but don't post them...maybe at some point).

everything on master passes.

what issues are you encountering?

[ruidc]

i'm not sure which issues we were after that have been fixed in master, but i do know that if you could provide x32 bit windows builds, we will use them! I'm using the suggested for x64 but there seems to be a lot of contradictory info around windows builds. I'll try with SDK v7 as that worked on x64 but have not seen that referred to for x32.

[ruidc]

results of trying to do this from the x64 machine where x64 works (having removed previous build and dist directories):

C:\Program Files\Microsoft SDKs\Windows\v7.0>set DISTUTILS_USE_SDK=1
C:\Program Files\Microsoft SDKs\Windows\v7.0>setenv /x86 /release

c:\Dev\src\pandas>python setup.py build_ext --inplace
setup.py:220: UserWarning: WARNING: Couldn't get git revision, using existing pandas/version.py
  warnings.warn("WARNING: Couldn't get git revision, using existing pandas/version.py")
running build_ext
skipping 'pandas\index.c' Cython extension (up-to-date)
skipping 'pandas\algos.c' Cython extension (up-to-date)
skipping 'pandas\lib.c' Cython extension (up-to-date)
skipping 'pandas\tslib.c' Cython extension (up-to-date)
skipping 'pandas\parser.c' Cython extension (up-to-date)
skipping 'pandas\hashtable.c' Cython extension (up-to-date)
skipping 'pandas\src\sparse.c' Cython extension (up-to-date)
skipping 'pandas\src\testing.c' Cython extension (up-to-date)
skipping 'pandas\msgpack.cpp' Cython extension (up-to-date)

any help appreciated. I'll try on the x32 machine when i can physically boot it in the office tomorrow.

[jreback]

Here's an example of a python 27 x64 scripts
I find that you have to explicity path everything on windows....PITA

@echo off
echo "starting 27-64"

setlocal EnableDelayedExpansion
set MSSdk=1
CALL "C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\SetEnv.cmd" /x64 /release
set DISTUTILS_USE_SDK=1

title 27-64 build
echo "building"
cd "c:\users\Jeff Reback\documents\github\pandas"
C:\python27-64\python.exe setup.py build > build.27_64.log 2>&1

echo "testing"
C:\python27-64\scripts\nosetests -A "not slow" build\lib.win-amd64-2.7\pandas > test.27_64.log 2>&1

echo "versions"
cd build\lib.win-amd64-2.7
C:\python27-64\python.exe ../../ci/print_versions.py > ../../versions.27_64.log 2>&1

exit

[ruidc]

thanks, that gets further, but I think I'll need to switch to an all 32bit machine to get this working this morning, as adjusting the above for x86 building from my 64bit machine gives:

C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\Bin\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python27\libs /LIBPATH:C:\Python27\PCbuild\amd64 /EXPORT:initindex build\temp.win-amd64-2.7\Release\pandas\index.obj build\temp.win-amd64-2.7\Release\pandas/src/datetime/np_datetime.obj build\temp.win-amd64-2.7\Release\pandas/src/datetime/np_datetime_strings.obj /OUT:build\lib.win-amd64-2.7\pandas\index.pyd /IMPLIB:build\temp.win-amd64-2.7\Release\pandas\index.lib /MANIFESTFILE:build\temp.win-amd64-2.7\Release\pandas\index.pyd.manifest
python27.lib(python27.dll) : fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'X86'

[jreback]

no, just use x86 instead of x64....I dropped using visual studio....SDK 3.5 works very nicely (you need 4.0 to build 3.3 FYI)

[ruidc]

thanks, that's what I did but on this machine I only have python 32bit installed, so will put SDK on the machine where i was trying VS2008 and mingw32, I only need to build 2.7 on both x32 and x64 until we can migrate our users to full x64 - and x64 is working already as far as pandas is concerned.

[jreback]

ok....yah you have to install completely separate environments to build. It is not fun...upside is when it works it DOES work.....going to close this issue, but periodically ping me back to put up windows binaries....I have them computing on-demand now....next step would prob be posting them

[jreback]

jus put up a 27-64, 27-32 and 33-64 build off of master

lmk if you need more (and if these work!)

http://pandas.pydata.org/pandas-build/dev/

[ruidc]

Thanks will do and will see if I can get my x32 build env working too so we don't bug you.

.

[ruidc]

the bad news is i'm still getting linker errors on my x32 machine after having removed mingw32 and VS2008 and installed SDK 7. The good news is your binaries work for me - and resolve the issue in #6254 so thanks!

[jreback]

gr8!

[jreback]

@cpcloud see here for my windows build script (I have posted binaries too)

[cpcloud]

nice thanks

[arnicas]

I just had a truly awful afternoon with this very error on my attempt to get a 2.6 pandas update on a 64 bit windows machine. I had the same import error after all my pip attempt, but these binaries worked. Thank you so much @jreback !

[Yensan]

I searched to this page, and then, edit another .bat for VScode.
if you do not want use VScode, just copy to the .bat file to the setup.py dir, then edit python(python path) and platform(amd64 OR X86) , then run .bat.
${workspaceRoot}/.vscode/build_pyd.bat

@echo off
rem if installed VC, "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat"
rem if installed Microsoft Visual C++ Build Tools ONLY
set vcbuildbat="C:\Program Files (x86)\Microsoft Visual C++ Build Tools\vcbuildtools.bat"
rem set logpath, or just use this bat file path
if "%1"=="" ( set logfile=buildpyd_log.txt ) else ( set logfile="%1/debug/buildpyd_log.txt")
rem set python path, OR use python in PATH
if "%2"=="" ( set python="python" ) else ( set python=%2 )
rem set setup.py path, OR use this bat file path
if "%3"=="" ( set dir=%cd% ) else ( set dir=%3 )
rem set platform, OR just X86
if "%4"=="" ( set platform="X86" ) else ( set platform=%4 )

echo "%date% %time% -------------------------------------------" >> %logfile%
echo python version info: >> %logfile%
%python% -V -V
%python% -V -V >> %logfile%
echo Setting ENV >> %logfile%

title setting MSVC ENV
CALL %vcbuildbat% %platform%

title platform build
echo building %platform%
echo building %platform% >> %logfile%
cd /d %dir%
%python% setup.py build_ext --inplace >> %logfile% 2>&1
echo build done
echo build done >> %logfile%
echo ------------------------------------------------------------ >> %logfile%
exit

tasks.json

    {"label": "build_pyd", // build_pyd.bat "C:/Python3686/python.exe", "${fileDirname}", "X86"
        "windows": {"command": "${workspaceRoot}/.vscode/build_pyd.bat"},
        "args": ["${workspaceRoot}", "C:/Python3664/python.exe", "${fileDirname}", "amd64"], //
        "type": "shell", "problemMatcher": ["$tsc"],
        "presentation": {"echo": true,"reveal": "never","focus": false,"panel": "shared"}
    },