Rob Hetland’s web journal

A small adventure in compiling the netcdf operators (with netcdf4 support) uses these things

First, the shared netcdf4 libraries must be used. And despite the advice on the web, all of the include and library directories must be specified… see

./configure --help

for the variables that must be set. Then, I needed to modify

nco/src/nco/nco_netcdf.h

so that

#define HAVE_NETCDF4_H

was set at the beginning of the file. This variable should be set when using netcdf4, but it was not. Then configure and make like this:

./configure --enable-netcdf4 --disable-shared
make -j 8

I linked to a version of hdf and netcdf4 that did not use shared libraries, thus the –disable-shared option. I also had success linking to shared versions of these libraries.

The build fails on the math operators, like usually, but it makes ncks. I noticed that it was searching for an old version of libnco, so delete that first. You may also need to do a ldconfig to find the right libraries after the install.

The short answer is that Things wins… Now for the long answer, after the break.. (more…)

Based on directions found here, I created a working build of VTK with Python bindings.

First, download the source from CVS, here. You will need to make sure you have a recent version of CMake installed as well. Then, go into the VTK directory and type:

mkdir build; cd build
ccmake ..

Press c to do the initial configuration. Set:

BUILD_SHARED_LIBS=ON
VTK_WRAP_PYTHON=ON

Press c again. Press t to look at the advanced options. This is where my directions diverge slightly from the enthought directions. I used the system python, which ccmake seems to recognize right away, so I only changed:

PYTHON_LIBRARY=-framework Python

Press c, then press g to generate the files. Then

make
make install

The make can take forever, so be patient.

One final thing to get vtk working with python is to go into the Build/Wrapping/Python directory and do a python build and install:

cd Wrapping/Python
python setup.py build
sudo python setup.py install

Done. Now you should be able to import vtk in python.

For my intro to Oceanography class, I made animations of the barotropic tide sea surface height. You can find them here. Trust me, they are very slick.

I found this command on macupdate:

sudo find / -not -name English.lproj -name "*.lproj" -exec rm -r {} \;

and ran it. It seems to have saved about 3 Gb from the Application directory, and about 5 GB total.

In a recent post on Daring Fireball, John Gruber describes how some new small changes in Spaces in Mac OS 10.5.3 have made it usable.

I use the terminal quite a bit, and I have noticed this small tip. If you use ⌘-` you will cycle through the windows within one space/screen (i.e., you won’t switch virtual screens). ⌘-→ or ⌘-← will cycle through all of the terminal windows, even on other spaces screens.

I was suffering delays logging in between the head node and cluster nodes. I tried all the usual tricks — mainly editing /etc/hosts. I tracked the problem down with

ssh -vvv c0-0

which showed that the delay was in xauth. Fine. What I needed to do was turn on ForwardX11Trusted, it was the Trusted part that was missing. This can be done with ssh -Y, or by editing /etc/ssh/ssh_config and adding a line

ForwardX11Trusted   yes

This is needed to get tools like cluster-fork to work with the Trusted forwarding.

I’ve seen this interview with Donald Knuth linked a few times, and I also found it very interesting. In particular, I found it interesting that he:

• uses EMACS/TeX — after writing everything by hand.
• does not particularly care for unit tests
• is a big fan of (i.e., invented) Literate programming
• is not a big fan of multicore processing (he does note, however, that physics and biology as exceptions where multicore programming makes sense..)

I have to admit, that I am on his side in terms of the unit testing and literate programming issues. I find that my biggest chalange in programming is not getting the code right (i.e., unit tests), but rather designing the way the function/class will behave for the user. Writing down what you expect the program to do in different cases (i.e., the API documentation) seems to be the best way to write a good, solid program.

I have recently written down the steps I needed to build netcdf3, hdf5, and netcdf4 on my Mac using both ifort and gfortran. Details follow (more…)

I wrote a build script for ROMS in python based on some of the other build scripts (in bash and Cshell) I have seen. The usage is as follows:

Create a directory with a header file called case_name.h in it. This should have the CPP defs for your particular case. Then run the build script like this:

build-roms.py case_name

Options, as seen from the help are:

-h, --help        Print this message

--fort=[ifort]    Specify fortran compiler
--netcdf=[4, 3]   Version of NetCDF to use [default is NetCDF4]
-j=n              Parallel make [n = number of procs, default is 4]

--mpi             Compile using mpi
--clean           Delete Build scratch directory before build
--debug           Compile in debug mode
--root            Specify ROMS_ROOT_DIR

This script creates a Build directory in the local directory, where the actual compilation happens. Analytical things can be defined in an optional Functionals directory, just like with the other build scripts. To see the script, continue…

(more…)

I have been looking for ways to interpolate big model results to other big mode results.. again. This seems to be an unsolved problem in ocean modeling. However, I have stumbled on a new method for finding nearest neighbors that seems quite promising.

As long as you are interpolating from something that has a relatively uniform grid with a relatively smooth field, you don’t need to bring out the big guns, like optimal interpolation. In other words, you may often ignore data error in your interpolation – especially going from one model to another to generate initial or boundary conditions.

Kd-trees (k-dimensional trees) are a method for organizing a set of k-dimensional points. In particular, I would like to find the nearest neighbors in three-dimensions. Many interpolation tools (such as csa or delaunay only work with 2D data.

The advantage of using kd-trees is twofold: First storing the tree is not very memory intensive, approximately the same size as the original point field itself, and building the tree scales roughly as the number of points (n log(n)). Second, searching for the closest points to a query point is quite fast (log(n)). The ann library can be used to return indicies of and distances to the M closest points to a query point. An example of how to use the ann library for interpolation is below the fold.

(more…)

The ROMS project movie (the only thing people seemed to miss during the black months when pong was down) is now back online. See it here.

I have been working on creating images and animations from google earth. I started out here for inspiration. Here are some samples of an image and an animation. The code is actually quite simple: (more…)

Finally, after a massive hard drive failure, my web log is back up. To make a long story short, I promise to make better backups and buy Steve a beer.. I feel a pent up pile of posts — so after waiting for so long, be sure to check often now.

Here is a sample script for generating a series of figures (that can be turned into a movie with ffmpeg, see below). The case used is the box_curvilinear test case. (more…)

People have asked me how I make my HOWTO movies, like this one. The good news is that it is ridiculously simple. (more…)

It seems that Mac OS X Leopard has quite a few excellent features installed by default. Including numpy!

Some interesting unix commands I recently discovered: xargs and seq. With these you can do this, for example:

Find all of the files named foo, and create a tar file:

find . -name foo | xargs tar cvfz tarfile.tar.gz

Iterate through your cluster nodes:

for i in `seq 0 7` ; do scp foolib c0-$i:/usr/local/lib ; done

Based on some advice I found here, I discovered how to make high quality movies from series of png files using ffmpeg. I use png files because they have a high color depth (unlike 256 for gif), and they are non-lossy (unlike jpg).

First you need to make a sequence of ordered files that have filenames like frame_001.png, frame_002.png, etc. For example, in python’s matplotlib, you would have a line like:

saveframe('frame_%03d.png' % framenumber)

Then, you need to run ffmpeg on all of these files. The command is quite simple:

ffmpeg -r 10 -sameq -i frame_%03d.png test.mp4

where the -r flag sets the frames per second. Otherwise, the only other key option is the -sameq flag, which keeps the same non-lossy quality of the png files. Pretty easy. Also, you don’t need to install xvid or h264 libraries for this to work. The codecs that come with ffmpeg seem to work just fine.

Also, the size of the resulting movie file is quite reasonable. I tested 121 frames at 1000×600 (with considerable non-changing space in the field). The resulting movie was 2.8M big. This is about seven times smaller than simply concatenating all of the images together. Not too bad.

This may not be new news, but it was a good find. Here is a very extensive archive of publicly available model output. This could be used, for example, for regional model boundary conditions:

http://apdrc.soest.hawaii.edu/dods/public_data/

An example python script for making a movie of ssh in the Indian ocean from the Navy Layer Ocean Model is found after the fold (more…)