The tool FPIG has been successfully used in wrapping large number of Fortran codes and libraries. For example, it is used in the application to Large Aero-Structural Computations of wing design optimization.
Nowadays in many fields of Science and Engineering computers are used to model and analyze various problems. In fact, computers have made it possible to tackle many important problems and to discover new phenomena that would be impossible to achieve without an ability to process huge amounts of data in a reasonably short time. Therefore, computers have become one of the basic tools for the contemporary scientist and engineer. On the other hand, the diversity of problems in science and engineering has left its mark also in computer programs that are developed in different programming languages, and that even create domain-specific languages that are suitable to translate/describe the problems in to the computer's native language.
In interdisciplinary fields of Science and Engineering it is not rare that a scientist/engineer is faced to problems of different fields that have been solved and the corresponding tools (programs) have been developed, already, but in different programming environment. Unfortunately, that scientist/engineer may not have time (or will) to learn a new programming language so that s/he will start to develop the corresponding tools in languages that s/he is familiar with. This procedure has several drawbacks: it actually may take more time to develop and test a sophisticated tool; and it is very unlikely that a non-specialist in a field can produce more efficient codes.
To avoid the situations like one described above, one solution would be to provide interfaces between different programming languages. (Another possibility is to provide language translators, but they obviously require more work than interface generators - a translator must understand all language constructs while for an interface generator only a subset of a language needs to be understood.) Having an interface between two languages, a scientist or engineer can effectively use programs written in another programming language without even learning it.
It is clear that it is not possible to interface all programming languages. Luckily, there is also no reason for doing that. Namely, roughly speaking, each programming language is suitable for certain tasks. For example, low level languages such as C, Fortran are well known for they speed and suitable for programs where performance issues are critical. On the other hand, high level scripting languages are, in principle, slower but they are much easier to learn and use, especially for doing interactive analysis. Therefore, in general, it makes sense to create interfaces in one direction: from lower level language to higher level language.
In ideal world, scientists and engineers would use higher level languages for manipulating the mathematical notations of the problem rather than struggling too much with various programming issues. For tasks that are computationally demanding, they would use transparently interfaces to high performance routines that are actually written in some other lower level language for speed.
This paper addresses a tool that is developed for creating an interface between Fortran and Python.
The Fortran language is widely used programming language in Scientific Computing, mostly in applications that use extensively matrix manipulations (e.g. Linear Algebra). Since Fortran has been the standard language among many scientists and engineers for at least three decades, there is a huge number of codes available that implement various tasks using very sophisticated algorithms (see e.g. [1]).
The Python language [2], on the other hand, is relatively young programming language. It is very high level scripting language that supports Object-Oriented Programming. What makes Python language especially appealing is its very clear and natural syntax that makes it easy to learn and use. In Python one can implement relatively complicated algorithms and tasks in a very short time with a very source code.
Although there are active development projects going on for extending Pythons usage to Scientific Computation, it still lacks reliable tools that are common in scientific and engineering practice. For example, ODE integrators, equation solvers, tools for using FEM, etc. Implementing all these tools in Python would be too time consuming and also inefficient. On the other hand, these tools are already developed in other, computationally more efficient, languages such as Fortran or C. So, in the context of Scientific Computing, the perfect role for the Python language would be the role of a ``glue'' language. That is, it would provide interfaces to different (C, C++, Fortran,...) libraries from its very high level and unique environment.
There are number of widely used tools available that can be used for interfacing software libraries to Python. For binding C libraries with various scripting languages, including Python, the most used tool is SWIG [5]. Wrapping Fortran routines to Python is less popular, mainly because there are number of platform or compiler specific issues that need to be addressed. Nevertheless, there is a great interest for interfacing Fortran libraries because they provide invaluable tools for Scientific Computing. For example, in LLNL a tool PyFort is developed for ease connecting Fortran and Python [4].
All these tools mentioned above require some sort of input file describing signatures of functions to be interfaced. To create these input files, one needs to have a good knowledge of C or Fortran languages. In addition, to bind libraries that have thousands of routines, will be certainly a very tedious task even with these tools.
FPIG (Fortran to Python Interface Generator) [6], introduced in this paper, is a tool for auto-generating interfaces to bind Fortran and Python languages. Different from the interface tools mentioned above, FPIG can create signature files automatically by scanning the source codes of libraries and from that it constructs Python C/API extension modules. Note that the user need not to be experienced in Fortran or even less in C language. In addition, FPIG is designed to wrap large Fortran libraries containing a number of routines in only one or two commands. This process is very flexible. One can always modify the generated signature files to insert additional attributes in order to achieve more sophisticated interface functions that can take care of optional arguments, predict the sizes of array arguments, perform various checks on the correctness of the input arguments, etc.
The organization of this paper is as follows. First, a simple example of FPIG usage is given. Then FPIG basic features are described and solutions to platform/compiler specific issues are discussed. Unsolved problems and future work on FPIGs development are aligned. Finally, an application to Large Aero-Strucural Computations is presented as an example of FPIGs usage.
To get acquainted with FPIG, let us consider a simple Fortran 77 subroutine as shown in Fig. 1.
subroutine exp1(l,u,n)
C Input: n is number of iterations
C Output: l,u are such that
C l(1)/l(2) < exp(1) < u(1)/u(2)
C
Cf2py integer*4 :: n = 1
Cf2py intent(out) l,u
integer*4 n,i
real*8 l(2),u(2),t,t1,t2,t3,t4
l(2) = 1
l(1) = 0
u(2) = 0
u(1) = 1
do 10 i=0,n
t1 = 4 + 32*(1+i)*i
t2 = 11 + (40+32*i)*i
t3 = 3 + (24+32*i)*i
t4 = 8 + 32*(1+i)*i
t = u(1)
u(1) = l(1)*t1 + t*t2
l(1) = l(1)*t3 + t*t4
t = u(2)
u(2) = l(2)*t1 + t*t2
l(2) = l(2)*t3 + t*t4
10 continue
end
In the sections that follow, two ways of creating interfaces to this Fortran subroutine are described. The first and simplest way is suitable for Fortran codes that are developed in connection with f2py . The second method, not really more difficult, is suitable for interfacing existing Fortran libraries possibly developed by others.
You need Numerical Python [7] in order to compile extension modules generated by FPIG.
In order to call the Fortran routine exp1 from Python, let us create an interface to it by using f2py (the front-end program of FPIG). For that, we issue the following command
sh> f2py -m foo exp1.f
where option -m foo sets the name of a Python C/API extension module to foo that f2py is going to construct. To learn more about f2py command line options, run f2py without arguments.
The output messages in Fig. 2 illustrate the main features of f2py program: (i) it scans Fortran source codes specified in command line, (ii) it determines/analyses routine signatures, (iii) it constructs the corresponding Python C/API extension modules, (iv) it writes full documentation to LaTeX file, (iv) and it creates a GNU Makefile for building the shared modules.
Reading fortran codes... Reading file 'exp1.f' Post-processing... Block: foo Block: exp1 Creating 'Makefile-foo'... Linker: ld ('GNU ld' 2.9.5) Fortran compiler: f77 ('g77 2.x.x' 2.95.2) C compiler: cc ('gcc 2.x.x' 2.95.2) Building modules... Building module "foo"... Constructing wrapper function "exp1"... l,u = exp1([n]) Wrote C/API module "foo" to file "foomodule.c" Documentation is saved to file "foomodule.tex" Run GNU make to build shared modules: gmake -f Makefile-<modulename> [test]
So, let us build the foo module:
sh> make -f Makefile-foo
Figure 3 illustrates a sample session of calling Fortran routine exp1 from Python.
>>> import foo,Numeric
>>> print foo.exp1.__doc__
exp1 - Function signature:
l,u = exp1([n])
Optional arguments:
n := 1 input int
Return objects:
l : rank-1 array('d') with bounds (2)
u : rank-1 array('d') with bounds (2)
>>> l,u = foo.exp1()
>>> print l,u
[ 1264. 465.] [ 1457. 536.]
>>> print l[0]/l[1], u[0]/u[1]-l[0]/l[1]
2.71827956989 2.25856657199e-06
>>> l,u = foo.exp1(2)
>>> print l,u
[ 517656. 190435.] [ 566827. 208524.]
>>> print l[0]/l[1], u[0]/u[1]-l[0]/l[1]
2.71828182845 1.36437527942e-11
Note the difference between the signatures of the Fortran routine exp1(l,u,n) and the corresponding wrapper function l,u=exp1([n]). Clearly, the later one is more informative to the user: exp1 takes one optional argument n and it returns l, u. This exchange of signatures is achieved by special comment lines (starting with Cf2py) in the Fortran source code - these lines are interpreted by f2py as normal Fortran code. So, in the given example the line Cf2py integer*4 :: n = 1 informs f2py that the variable n is optional with a default value equal to 1. And the line Cf2py intent(out) l,u informs f2py that variables l,u are to be returned to Python after calling Fortran function exp1.
In our example the Fortran source exp1.f contains f2py specific information, though only as comments. When interfacing libraries of other parties, it is not recommended to modify their sources. Instead, one should use special auxiliary file to collect the signatures of all Fortran routines and insert f2py specific declaration and attribute statements in that file. This auxiliary file is called signature file and identified by the extension .pyf.
We can use f2py for generating these signature files by using -h <filename>.pyf option. In our example case, f2py should be called as
sh> f2py -m foo -h foo.pyf exp1.f
where option -h foo.pyf informs f2py to read the routine signatures, save them to the file foo.pyf, and then exit. In the case where the file exp1.f in Fig. 1 contains no lines starting with Cf2py, the corresponding signature file foo.pyf is as shown in Fig. 4.
!%f90 -*- f90 -*-
python module foo
interface
subroutine exp1(l,u,n)
real*8 dimension(2) :: l
real*8 dimension(2) :: u
integer*4 :: n
end subroutine exp1
end interface
end python module foo
! This file was auto-generated with f2py
! (version:2.298).
! See http://cens.ioc.ee/projects/f2py2e/
For obtaining exchanged (more convenient) signature l,u=foo.exp1([n]), we edit foo.pyf as shown in Fig. 5
!%f90 -*- f90 -*-
python module foo
interface
subroutine exp1(l,u,n)
real*8 dimension(2) :: l
real*8 dimension(2) :: u
intent(out) l,u
integer*4 optional :: n = 1
end subroutine exp1
end interface
end python module foo
! This file was auto-generated with f2py
! (version:2.298) and modified by pearu.
! See http://cens.ioc.ee/projects/f2py2e/
Python C/API extension module foo can be now constructed by applying f2py only to the signature file:
sh> f2py foo.pyf
Building the corresponding shared module and using it in Python is identical as described in the previous section.
In conclusion, we note that the syntax of the signature files is an extended syntax of Fortran 90/95. It means that only some new constructs are introduced for f2py in addition to all standard Fortran constructs, you can write signature files even in fixed form. A complete set of constructs that are used when creating interfaces, is described in the f2py User's Guide [8].
In this section a short overview of f2py features is given.
integer[ | *1 | *2 | *4 | *8 ] logical[ | *1 | *2 | *4 | *8 ] real[ | *4 | *8 | *16 ] complex[ | *8 | *16 | *32 ] double precision, double complex character[ |*(*)|*1|*2|*3|...]In addition, they all can be in the kind-selector form (e.g. real(kind=8)) or char-selector form (e.g. character(len=5)).
Fortran to Python interface can be thought as a three layer ``sandwich'' of three languages: Python, C, and Fortran. Such a sandwich has two interfaces: Python-C and C-Fortran. Since Python itself is written in C then there are no principle difficulties to implement Python-C interfaces (see [3]). On the other hand, the C-Fortran interface offers many platform/compiler specific issues to be dealt. In the following, we will discuss these issues in some detail and describe how they are solved in FPIG.
Table 1 defines how Fortran types are mapped to C types in f2py .
| Fortran type | C type |
| integer *1 | char |
| byte | char |
| integer *2 | short |
| integer[ | *4] | int |
| integer *8 | long long |
| logical *1 | char |
| logical *2 | short |
| logical[ | *4] | int |
| logical *8 | int |
| real[ | *4] | float |
| real *8 | double |
| real *16 | long double |
| complex[ | *8] | struct {float r,i;} |
| complex *16 | struct {double r,i;} |
| complex *32 | struct {long double r,i;} |
| character[*...] | char * |
Users may redefine these mappings by creating .f2py_f2cmap file in the working directory. This file should contain a Python dictionary of dictionaries, e.g. {'real':{'low':'float'}} that informs f2py to map Fortran type real(low) to C type float (here PARAMETER low = ..., for example).
In general, when mixing Fortran and C codes, one has to know how function names are mapped to low-level symbols in their object files. Different compilers may use different convention for that. For example, gcc appends the underscore _ to a Fortran routine name. Other compilers may upper the name cases or/and prepend/append different symbols to Fortran routine names. Anyway, if the low-level symbols corresponding to Fortran routines are valid for C language specification, compiler specific issues can be solved by using CPP macro facilities.
Unfortunately, there exist Fortran compilers that use symbols in constructing low-level routine names that are not valid for C. For example, the (IRIX64) MIPSpro 7 Compilers use `$' character in the low-level names of module routines which makes it impossible (at least directly) to call such routines from C when using MIPSpro 7 C Compiler.
In order to overcome this difficulty, FPIG introduces an unique solution: instead of using low-level symbols for calling Fortran module routines from C, the references to such routines are determined in run-time by using special wrappers.
For example, consider the following Fortran 90 module with a subroutine:
Figure 6 illustrates how one should implement the Python C/API extension module in order to access F90 module subroutine bar from Python.module fun subroutine bar() end end
#include "Python.h"
...
char *bar_ptr;
void init_bar(char *bar) {
bar_ptr = bar;
}
static PyObject *
bar_capi(PyObject *self,PyObject *args) {
...
(*((void *)bar_ptr))();
...
}
static PyMethodDef
foo_module_methods[] = {
{"bar",bar_capi,METH_VARARGS},
{NULL,NULL}
};
extern void finitbar_; /* GCC convention */
void initfoo() {
...
finitbar_(init_bar);
Py_InitModule("foo",foo_module_methods);
...
}
subroutine finitbar(cinit)
use fun
extern cinit
call cinit(bar)
end
So, when Python module foo is loaded, finitbar is called. finitbar calls init_bar by passing the reference of Fortran 90 module subroutine bar to C where it is saved to variable bar_ptr. Now, when one executes foo.bar() from Python, bar_ptr is used in bar_capi to call the F90 module subroutine bar.
Clearly, with these hooks the mix of C and F90 modules becomes as portable and compiler independent as mixing C and F77 codes.
Note that f2py actually generates a bit different code than shown above but the idea remains exactly the same. Extension modules generated by f2py use PyFortranObject for holding references of Fortran objects, see Section 5.5 below for more information.
Recall that Fortran language has two types of routines: subroutines and functions. When a Fortran function returns a composed type such as COMPLEX or CHARACTER-array then direct calling it from C may not work for all compilers as C functions are not supposed to return such references.
To guarantee compiler independence of Python C/API extension modules, FPIG constructs an additional Fortran wrapper subroutine for each such Fortran function. These wrappers just call the corresponding functions in Fortran layer and return the result to C through its first argument.
In Fortran language one can use COMMON blocks and Fortran module variables to save data that are accessible from other Fortran routines. Using FPIG, one can access these data containers also from Python. For that FPIG uses special wrapper functions (similar to ones for wrapping Fortran module routines) to save the references of these data containers so that they can be later used from C.
FPIG can also handle allocatable arrays. For example, if a Fortran array is not yet allocated then by assigning it in Python, the Fortran to Python interface will allocate and initialize the array. To guarantee compiler independence, f2py generates a sophisticated wrapper function (in Fortran) for accessing allocatable arrays from C. In every attempt to access an array from Python, this wrapper function checks whether the array is allocated or not, when requested then allocates it, and when requested with different size then deallocates-allocates the array.
For example, the F90 module allocatable array bar defined in
can be allocated from Python as followsmodule fun integer, allocatable :: bar(:) end module
For implementation details about this method, see the following section.>>> import foo >>> foo.fun.bar = [1,2,3,4]
In general, we would like to access from Python the following Fortran objects:
Assuming that we have Fortran sources, we can determine the signatures (full specification of routine arguments, layout of Fortran data) of these objects. In fact, f2py gets this information while scanning the Fortran sources.
To access these objects from C, we need to determine the references of these objects. Here FPIG uses various wrapper functions in order to guarantee portability and compiler independence of the Python C/API extension modules. Note that these wrapper functions are especially needed when accessing objects from F90 modules and that is because the direct access is extremely compiler dependent and not always even possible. The basic reason why wrapper functions make things work is that they are ordinary F77 subroutines that (i) can easily access objects from F90 modules and (ii) are easily accessible from C. So, they are just used for passing the references of Fortran objects to C variables.
In order to store the references of Fortran objects, f2py generated Python C/API extension modules use PyFortranObject type introduced in FPIG. In addition to the storing functionality, PyFortranObject also provides methods for accessing/calling Fortran objects from Python in a user-friendly manner. For example, the item a in COMMON /bar/ a(2),b is accessed in Python as foo.bar.a.
For detailed example usages of PyFortranObject in Python C/API modules, see [9].
Fortran routines may have arguments specified as external. These arguments are functions or subroutines that the Fortran routine will call from its body as a Fortran routine. For such arguments FPIG constructs a call-back mechanism (originally contributed by Travis Oliphant) that allows Fortran routines to call Python functions. This is actually realized using C layer between Python and Fortran, of course. Currently, the call-back mechanism is compiler independent unless a call-back function should return a composed type (e.g. COMPLEX).
The signatures of call-back functions are determined when f2py scans Fortran source codes. To illustrate this, consider the following example code:
subroutine foo(bar, fun, boo)
integer i
real r
external bar,fun,boo
call bar(i, 1.2)
r = fun()
call sun(boo)
end
f2py recognizes successfully the signatures of user routines
bar and fun. It uses lines call bar(i,
1.2) and r = fun() for guessing the routine signatures to be
as follows
But note that f2py cannot determine the signature of the user routine boo because the source contains no information at all about boo specification. Here user needs to provide the signature of boo manually.subroutine bar(a,b) integer a real b end function fun() real fun end
Currently, FPIG can be used to wrap (almost) any Fortran code. However, there are still issues that can be improved. Some of them are listed below:
As a demonstration of the power and usefulness of FPIG, we will present work that has been done by the Aircraft Computing Laboratory at Stanford University. The focus of the research group is on aircraft design optimization using high-fidelity analysis tools such as Computational Fluid Dynamics (CFD) and Computational Structural Mechanics (CSM) [11].
The group's analysis codes are mainly in Fortran and represent the result of many years of development. Until now, any researcher that needed to use these tools, would necessarily have to learn a less than user-friendly interface and become relatively familiar with the inner workings of the code itself. The need to couple analysis of different disciplines revealed the additional inconvenience of gluing and scripting the different codes with Fortran.
It was therefore decided that the existing tools should be wrapped using an object-oriented language in order to improve their ease of use and versatility. The use of several different languages such as C++, Java and Perl was investigated but Python seemed to provide the best solution for us. The fact that it combines scripting capability with those of a fully-featured object-oriented programing language as well as its clean syntax contributed to our choice. The introduction of tools like FPIG that greatly facilitate the task of wrapping Fortran with Python provided the final answer to our need to realize our objective.
The first objective of this effort was to design the objects for each type of analysis, each representing an independent module. In our case, we are interested in performing aeroelastic analysis and optimization of aircraft wings and therefore we need an analysis tool for the flow (CFD), another tool for analyzing the structure (CSM), as well as a geometry database. In addition, we need to interface these two tools in order to analyze the coupled system. The object design for each of these modules should be general enough so that the underlying analysis code in Fortran can be changed without changing the Python interface. Another requirement is that the modules can be used on their own.
The Geometry class provides with a database for the outer mold geometry of the aircraft. This database needs to be accessed by both the flow and structural solvers. It contains a parametric description of the aircraft's surface as well as methods that extract and update this information.
The flow solver was wrapped in a class called Flow. The class was designed so that it can wrap any type of CFD solver. It contains two main objects: the computational mesh and a solver object. A graph showing the hierarchy of the objects in Flow is shown below:
+---- Flow
|
+---- Mesh
| |
| +---- Block
| |
| +---- Surface
|
+---- Solver
|
+---- Parameters
Methods in the flow class include those used for the initialization of all the class' components as well as methods that write the current solution to files.
The Structure class wraps a structural analysis code. The class stores the information about the structure itself in an object called Model which also provides methods for changing and exporting its information. A list of the objects contained in this class can be seen below.
+---- Structure
|
+---- Model
| |
| +---- Node
| |
| +---- Element
| |
| +---- Group
| |
| +---- Material
|
+---- LoadCase
Since the Structure class contains a
dictionary of LoadCase objects, it is able to store and solve
multiple load cases, a capability that the original Fortran code
does not have.
The aero-structural analysis module uses the three modules described above to form the Aerotsructure class, which contains a Geometry, a Flow and a Structure. In addition, the class defines all the functions that are necessary to translate the aerodynamic loads to the structural loads and the structural displacements to geometry surface deformation.
One of the main methods of this class is the one that solves the aeroeslastic system. This method is printed below:
def Iterate(self, load_case):
"""Iterates the aeroelastic solution."""
while 1:
self.flow.Iterate()
self._UpdateStructuralLoads()
self.structure.CalcDisplacements(load_case)
self.structure.CalcStresses(load_case)
self._UpdateFlowMesh()
break
return
This is indeed a very readable script, thanks to Python, and any
high-level changes to the solution procedure can be easily
implemented.
The Aerostructure class also contains methods that export all
the information of the current solution for visualization, an example
of which is shown in the next section.
For visualizing our results, since we needed to view results from multiple disciplines simultaneously, we selected OpenDX. Output files in DX format are written at the Python level and the result can be seen in the figure below for the case of a transonic business jet configuration.
The figure illustrates the multi-disciplinary nature of the problem. The grid pictured in the background is the mesh used by the flow solver and it is colored by the pressure values computed at the cell centers. The wing in the foreground is clipped to show its internal structural components and each of the components is colored by its stress value.