Building Scipy on Windows

Author: Pearu Peterson <pearu@cens.ioc.ee>
Last changed:2004-02-11
Revision: 1.5
Discussions to:scipy-dev@scipy.org

Contents

Foreword

I am a dedicated linux user that explains the approach used in this document: from a Linux box login via SSH to a Windows machine serving Cygwin and build Scipy win32 port using MinGW/MSYS.

To setup a SSH server under Windows is easy, see Howto install OpenSSH on a Windows 2000/XP/Server 2003. Here is a quick howto for Win98 users: install OpenSSH provided by Cygwin, run ssh-host-config, copy Linux RSA public key to .ssh/authorized_keys (so that you can login from linux box without password asked), and finally run /usr/sbin/sshd.

However, if you are already sitting behind a Windows box or can use Remote Desktop, there is no need to install SSH (though it is a very useful tool in general), having a Cygwin shell is enough to build Scipy. In fact, one can build Scipy using MinGW/MSYS only but I'll use Cygwin because it provides much more powerful shell capabilities (command history, auto-completion of commands, etc.) than MSYS.

Setting up environment

Installing Cygwin

Click on http://cygwin.com/setup.exe to install or upgrade Cygwin. This will launch an installer. Then follow the steps until Select Packages where you should make sure that the following components are checked for install:

  • Devel/swig to construct SWIG wrappers, used by Scipy setup scripts.
  • Devel/cvs to download sources from CVS repositories.
  • Interpreters/python, always useful to have but needed for msys.py script (see below).

And optionally:

  • Devel/{gcc, gcc-g++, gcc-g77} to build programs under Cygwin.
  • Editors/{emacs, mc}, an editor and a file-system browser.
  • Web/wget to download files from Internet.
  • Shells/tcsh needed to run full ATLAS test site.
  • Admin/cygrunsrv when you indent to set up an SSH server.
  • Net/openssh contains SSH programs.
  • System/procps, useful when something hangs and you are not close to the Windows box (there would be a dialog box waiting for you to press Close button). ps command would show process IDs so that you could remotely kill troublesome programs using kill -9.
  • Archive/{unzip, zip}, Devel/make, Graphics/gnuplot, Interpreters/perl, Utils/{cpio, file, rpm, rpm-build}.

Installing MSYS and MinGW

MSYS and MinGW can be downloaded from MinGW download page. There is a table of File Lists. Look at the Current section where you can download MSYS and MinGW executables. At the moment of writing, the required files are MSYS-1.0.9.exe and MinGW-3.1.0-1.exe.

First install MSYS, say, to directory C:\msys\1.0. Then install MinGW to C:\msys\1.0\mingw directory.

Note:

If you choose to install MinGW to C:\mingw as the installer suggests by default then you have to create a file C:\msys\1.0\etc\fstab containing

c:/mingw /mingw

and then restart your computer so that /mingw will be mounted.

Using MSYS from Cygwin

To run MSYS from Cygwin prompt, you could use

$ /cygdrive/c/msys/bin/sh.exe --login -i

The downside in this approach is that the PATH has Cygwin components before Mingw components and will cause problems when Cygwin program is used instead of the corresponding Mingw program.

As a workaround, save the following Python script, msys.py, to Cygwin /usr/local/bin directory and read the installation notes in the header of this file (if you are using MSYS version 1.0.x, Python 2.3.x, and all software are installed on C: drive then msys.py should be ready to use as it is). Now, when you want to execute a MSYS command from a Cygwin prompt, then run

msys.py <MSYS command>

msys.py will take care of fixing PATH environment variable, changing to the current working directory, etc.

For example, when you have installed Python both for Cygwin and Win32, then we have

$ python -i
Python 2.3.3 (#1, Dec 30 2003, 08:29:25)
[GCC 3.3.1 (cygming special)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

$ msys.py python -i
Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; sys.exit()

Installing prerequisites

At minimum, Python, Numeric, F2PY, Atlas (or BLAS), and LAPACK is required to build Scipy.

Installing Python and Numeric for Cygwin [can be skipped]

Cygwin provides Python and it is easy to build Numeric using gcc provided by Cygwin. At the moment of writing, recent Numeric source file is Numeric-23.1.tar.gz. Download and unpack it to ~/src/Numeric-23.1, for instance. Change to this directory and run

python setup.py install

that will build and install Numeric to /usr/lib/python2.3/site-packages/.

Installing Python and Numeric for MSYS

Easiest way to get Python installed for Win32 port is to download its installer from Python homepage. At the moment, recent Python installer is Python-2.3.3.exe. This will install Python to C:\PYTHON23\ directory (note that if you choose a different directory then you should modify msys.py script accordingly).

And the easiest way to install Numeric is to get its Win32 installer from Numpy homepage. At the moment, recent Numeric installer is Numeric-23.1.win32-py2.3.exe which will install Numeric to C:\PYTHON23\Lib\site-packages.

Failures in building Numeric [can be skipped]

Unfortunately, with the current version of distutils (of Python version 2.3.3) I have not been able to build any extension module for Win32 either using Cygwin or MinGW tools. For example, when trying to build Numeric using distutils mingw32 compiler, I get the following failures under Cygwin:

$ python setup.py build --compiler=mingw32
...
In file included from /usr/include/python2.3/Python.h:48,
                 from Src/ufuncobject.c:14:
/usr/include/python2.3/pyport.h:118:24: sys/select.h: No such file or directory
...

and under MSYS, after installing Numeric with the following command

$ msys.py python setup.py build --compiler=mingw32 install

importing Numeric fails:

$ msys.py python -i
Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import Numeric
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "C:\PYTHON23\lib\site-packages\Numeric\Numeric.py", line 91, in ?
    import multiarray
ImportError: DLL load failed: The system cannot find the file specified.
>>> 

These issues are fixed in scipy_distutils package (if I figured it correctly from reading scipy_distutils sources then Python distutils's mingw32 compiler cannot be used from MSYS) and, in principle, one can modify Numeric's setup.py script to use scipy_distutils instead of distutils and so getting a working Numeric built for MSYS. This is relevant, for example, when you want that Numeric linear algebra modules use Atlas libraries. However, to build Scipy this is not important and I shall skip this step as it would require installing scipy_distutils separately.

Installing F2PY

F2PY is a pure Python package and its installation for Cygwin and MSYS is the same. First, get F2PY from its CVS repository as follows (type guest when password is asked):

$ mkdir ~/cvs; cd cvs
$ cvs -d :pserver:anonymous@cens.ioc.ee:/home/cvs login
Logging in to :pserver:anonymous@cens.ioc.ee:2401/home/cvs
CVS password: 
$ cvs -d :pserver:anonymous@cens.ioc.ee:/home/cvs -z7 checkout f2py2e

Or you could download and unpack its latest source file as F2PY-2-latest.tar.gz (most of the time using CVS version of F2PY is recommended).

To install F2PY from sources, run

$ cd f2py2e  # or `cd F2PY-2.??.???-????` when using tar-ball
$ python setup.py install
$ msys.py python setup.py install

Here is fairly recent Win32 installer for F2PY: F2PY-2.38.234_1605.win32.exe.

Note that f2py2e module is not fully functional until scipy_distutils package is installed, scipy_distutils is part of Scipy_core, see below.

Getting BLAS/LAPACK sources

If for some reasons you cannot use prebuilt ATLAS/LAPACK libraries then it is possible to build Scipy using BLAS and LAPACK Fortran sources from Netlib. You can download them as follows

$ mkdir -p ~/src/blas/; cd ~/src/blas
$ wget http://www.netlib.org/blas/blas.tgz
$ tar xzf blas.tgz
$ cd ~/src/
$ wget http://www.netlib.org/lapack/lapack.tgz
$ tar xzf lapack.tgz

and later, before running Scipy setup.py script, define

$ export BLAS_SRC=~/src/
$ export LAPACK_SRC=~/src/

environment variables. Then Scipy setup.py script (scipy_distutils/system_info.py to be specific) will pickup the sources and compiles them. However, it is highly recommended to build BLAS/LAPACK libraries, see below.

Building BLAS/LAPACK libraries

Run

$ cd ~/src/blas/
$ msys.py g77 -fno-second-underscore -O2 -c *.f
$ msys.py ar r libfblas.a *.o
$ msys.py ranlib libfblas.a
$ export BLAS=~/src/blas/libfblas.a  # required when running scipy setup.py

$ cd ~/src/LAPACK/
$ wget http://cens.ioc.ee/~pearu/scipy/make.inc.MINGW
$ mv make.inc.MINGW make.inc
$ msys.py make lapacklib
$ cp lapack_MINGW.a liblapack.a
$ export LAPACK=~/src/LAPACK/liblapack.a  # required when running scipy setup.py

Building ATLAS Win32 libraries

Download Atlas library sources tar-ball (currently this is atlas3.6.0.tar.gz) to ~/src directory. Now run

$ cd ~/src
$ tar xzf atlas3.6.0.tar.gz
$ cd ATLAS

Note for SSH users:

ATLAS configure program is not working properly when run through an SSH session to Windows box. The underlying reason is that Windows system() command requires that fflush() is called on any stream before calling system(), see Run-Time Library Reference about system(). As a fix, download atlas3.6.0.ssh2cygwin.patch file to ~/src and apply it inside the ATLAS directory as follows

$ patch -p1 < ../atlas3.6.0.ssh2cygwin.patch

Next run

$ make xconfig
$ ./xconfig -t c:/cygwin`pwd`

Edit the generated Make.<arch> (in the following <arch>=WinNT_P4SSE2 is used for example) file as follows:

F2CDEFS = -DAdd_ -DStringSunStyle
               ^_____________________ note the single underscore!
F77 = /cygdrive/c/mingw/bin/g77.exe
F77FLAGS = -fomit-frame-pointer -O -fno-second-underscore
GCCdir = /cygdrive/c/mingw/bin
GOODGCC = /cygdrive/c/mingw/bin/gcc.exe
CC = /cygdrive/c/mingw/bin/gcc.exe
MCC = /cygdrive/c/mingw/bin/gcc.exe
# Leave XCC as it is, it must refer to Cygwin gcc compiler.
RANLIB = ranlib
BLASlib = $(TOPdir)/../blas/libfblas.a

Now run:

$ make install arch=WinNT_P4SSE2

ATLAS libraries are now built to lib/WinNT_P4SSE2 directory. Here is a list of prebuilt Win32 ATLAS libraries:

To run sanity tests, execute

$ make sanity_test arch=WinNT_P4SSE2

To run full tests, download atlas_test.tar.gz, unpack it to ~/src, download atlas_test1.0.1.mingw32.patch to ~/src, and apply it as follows:

$ cd ~/src/AtlasTest
$ patch.exe -p1 < ../atlas_test1.0.1.mingw32.patch

Fix TSTdir in Makefile file to TSTdir = $(HOME)/src/AtlasTest and finally execute

$ make TOPdir=c:/cygwin/$HOME/src/ATLAS ARCH=WinNT_P4SSE2
Warning:Running ATLAS full tests may take several days to complete.

From another terminal you can use

$ cd ~/src/AtlasTest/
$ ./scope.sh WinNT_P4SSE2

to check the testing progress.

Building a complete LAPACK library

ATLAS does not provide a full LAPACK library. Here follow basic steps how to build a complete LAPACK/ATLAS library that gives the best performance:

$ cd ~/src/ATLAS/lib/WinNT_P4SSE2
$ cp liblapack.a liblapack_orig.a # make a backup
$ mkdir tmp; cd tmp
$ ar x ../liblapack.a
$ cp ~/src/LAPACK/lapack_MINGW.a ../liblapack.a
$ ar r ../liblapack.a *.o
$ cd ..; rm -rf tmp

Building Scipy

Get Scipy from its CVS repository as follows (press Enter when password is asked):

$ cd ~/cvs
$ cvs -d :pserver:anonymous@scipy.org:/home/cvsroot login
Logging in to :pserver:anonymous@scipy.org:2401/home/cvsroot
CVS password:
$ cvs -d :pserver:anonymous@scipy.org:/home/cvsroot -z7 checkout scipy

To build Scipy, Scipy_core must be installed (or at least scipy_distutils package that is used by Scipy setup.py scripts). When checking out scipy, also scipy_core is checked out and will be used by Scipy setup.py script. However, building Scipy can be a very long running task and to be sure that it will be successful, it is recommended to install Scipy_core first and check that it is working properly. If positive, then next time you can skip building Scipy_core as it is included also in Scipy building process.

Important note:

You'll need build_src branch of scipy_distutils before continuing:

$ cd ~/cvs/scipy/scipy_core/scipy_distutils/
$ cvs -z7 update -r build_src

This is temporary step and will be removed in future.

Installing Scipy_core

To install Scipy_core under Cygwin, run

$ cd ~/cvs/scipy/scipy_core
$ python setup.py install

and under MSYS, run

$ rm -rf build/
$ msys.py python setup.py install

Note:

scipy_distutils overrides distutils MingW32CCompiler class and sets mingw32 compiler as default when calling MSC compiled Python from MSYS environment. If you need distutils defaults, then use --compiler=msvc option.

Scipy_core -- that actually consists of four packages: scipy_distutils, scipy_testing, scipy_base, and weave -- is working properly if the following Python sessions work:

$ python
Python 2.3.3 (#1, Dec 30 2003, 08:29:25) 
[GCC 3.3.1 (cygming special)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy_base
>>> scipy_base.test()
<snip>
>>> ^D

$ msys.py python -i                    
Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy_base
>>> scipy_base.test()
<snip>
Ran 122 tests in 0.600s
OK
>>> import sys; sys.exit()

Note for Scipy developers: at the moment scipy_base.test() gives 8 failures related to NaN/InF stuff under Cygwin.

Building Scipy

First make sure that all environment variables that Scipy setup.py uses are set:

$ export ATLAS=~/src/ATLAS/lib/WinNT_P4SSE2  # fix <arch> part if needed

To verify that they are detected correctly, run

$ cd ~/cvs/scipy/
$ msys.py python scipy_core/scipy_distutils/system_info.py

The output should contain

atlas_info:
  FOUND:
    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['c:/cygwin/home/pearu/src/ATLAS/lib/WinNT_P4SSE2']

for instance. To start Scipy building process, execute

$ cd ~/cvs/scipy
$ msys.py python setup.py build

Note that when building Lapack from source (that contains huge number of files to be compiled) then you may get the following failure:

error: g77.exe: C:/MSYS/1.0/mingw/bin/../lib/gcc-lib/mingw32/3.2.3/specs: Too many open files

Just ignore it and rerun setup.py script again.

To create a Win32 installer, run

$ msys.py python setup.py bdist_wininst

This will create dist/Scipy.xxx.exe and scipy_core/dist/Scipy_core.xxx.exe files.