halem Quick Start Guide

Step 1: Connect to halem from your workstation (ssh)

System access is allowed only through the secure shell (ssh) command and only from an authorized workstation. You may get an error if your workstation address is not enetered into our hosts.allow file. In that case, please contact the NCCS User Services Group with your workstation address.

If your workstation address is entered into the "hosts.allow" file, then enter the following command:

ssh your_userid@login.nccs.nasa.gov

At the PASSCODE: prompt, enter your 4-digit NCCS PIN + 6-digit Authentication Key Token tokencode, together as one number, with no spaces.

At the host: prompt, enter halem (all lowercase). Finally, enter your halem password at the password: prompt.

Note that once you are inside the halem environment, you may ssh to other NCCS systems.

| Top of Page |

Step 2: Check your user startup files (.cshrc, .login files)

All new users have the Posix shell (sh) by default. If you want to change to the C shell or Korn shell, you can issue the chsh command. Consult man chsh to learn more about the chsh command or read the halem documentation on shells for more information.

The startup files for C shell are .login and .cshrc, and the startup file for Posix shell is .profile.

After editing your .cshrc file, you must issue the command source .cshrc for your changes to take effect.

| Top of Page |

Step 3: Transfer your codes from your workstation (scp)

If your workstation allows inbound scp connections, you may issue the command scp (secure copy) to transfer your code to halem from your workstation. Consult man scp for more information.

For example, suppose you have the file file1 in the /tmp directory of your workstation. To scp that file to your current working directory on halem, issue the following command:

scp your_workstation_userid@your_workstation:/tmp/file1 ./file1

You will be prompted for your workstation password, and then the transfer will be made.

If your workstation does not allow inbound scp connections, refer to documentation on moving files to dirac to transfer your code to dirac and then to halem from dirac.

| Top of Page |

Step 4: Compile and run your code (f90, cc, prun)

To compile your code myprogram.f using a Fortran compiler, issue the following command:

f90 myprogram.f

Many different optimization options are available. The default optimization level is -O4. Consult man f90 for more information. Detailed information on different compilers (C, Fortran, Java) and options to read and save binary data is available in the halem compiler tools documentation.

To run the executable, use ./a.out. The command prun can be used to run the executable in batch processing.

| Top of Page |

Step 5: Submit the job to the batch queue (bsub, LSF)

If you have a job to run that requires more than one processor, you can submit it to the batch queue. The default batch queue is the "general" queue. A detailed description of other halem queues and resources is available.

You can run your job using an interactive batch environment or by submitting a batch script file. In either case you must provide your Computational Project (formerly "Sponsor Code Account"). More information on batch script files and commands to run batch jobs is available.

To a submit a batch script file, say, myscript, issue the following command:

bsub -P sponsor_code < myscript

| Top of Page |

Step 6: Use OpenMP (OMP)

OpenMP is an extension to standard Fortran, C and C++ that supports shared memory parallel execution. Directives must be added to your source code to parallelize loops and specify certain properties of variables. Fortran with OpenMP directives is compiled as:

% f90 -omp ompcode.f -o ompcode.exe

Compaq Fortran supplies OpenMP Fortran API Compiler Directives for True64 UNIX.

| Top of Page |

Step 7: Use Message Passing Interface Model (MPI, MPI-IO, SHMEM)

MPI

To build Fortran MPI programs, insert the MPI header file in your source files (include 'mpif.h'), compile the program with f90, and link with MPI and Elan libraries using the following command:

% f90 -o mpicode mpicode.f90 -lmpi

Because the MPI library contains the dynamic load of Elan libraries, it is not necessary to explicitly link Elan. The list and man pages of MPI routines available on halem can be found at /usr/opt/mpi/man. MPI-2 is fully supported on SC45, but documentation is scarce. For the index and some examples of MPI-2, refer to the MPI Forum Standard Index.

More information is also available on using MPI on halem.

MPI-IO

To write the MPI-IO Fortran program, insert the MPI header file in the source file:

include "mpif.h"

and then compile the code with MPI and MPI-IO libraries:

% f90 -o mpi_iocode mpi_iocode.f90 -lmpi -lmpio

The file mpif.h has definitions for Fortran MPI-IO programs. More information can be found in the halem MPI-IO documentation, including a short sample proram for testing.

SHMEM

To build Fortran SHMEM programs, insert the SHMEM header file in your source files (include 'shmemf.h'), compile the program with f90, and link with the SHMEM library:

% f90 -o shmemcode shmemcode.f90 -lshmem

More information on SHMEM and sample programs are available in the halem SHMEM documentation.

| Top of Page |

Step 8: Use Performance Analysis Tools (prof, pixie)

To analyze the performance of your code, compile, run, and use prof. To create the profile information, issue the following commands:

% f90 -p -g3 program.f
% prun -n2 ./a.out
% prof ./a.out mon.out>prog.prof

For more information, consult man prof. The instruction-counting profiler pixie can also be used with prof; consult man pixie for more information. More information on using prof is available in the halem documentation.

| Top of Page |

Step 9: Use Debugging Tools (ladebug, totalview, vampir)

Ladebug is a source-level, symbolic debugger that supports Compaq Fortran data types, syntax, and use. The following commands create (compile and link) the executable program and invoke the character-cell interface to the Ladebug debugger:

% f90 -g -ladebug -o squares squares.f90
% ladebug squares

The -g(n) options control the amount of information placed in the object file for debugging. To use Ladebug, you should specify the -ladebug option along with the -g (= -g2), or -g3 options. Descriptions on how to use Ladebug can be found in Ladebug Debugger Manual.

Totalview, a well known debugging tool, is also installed on halem for graphical debugging of serial and parallel codes.

% totalview a.out

VAMPIR is another powerful tool for analyzing MPI code performance.

% f90 vt.f -lVT -lmpi
% vampir

After issuing the above commands, you can then load the respective trace file from the "File" menu.

| Top of Page |

Step 10: Use software on halem

Different software products are available on halem. For detailed information on how to use these libraries on halem, refer to the corresponding documents.

| Top of Page |

Step 11: Use the data storage system to store files (sftp, SAM-QFS commands)

The scratch directory /scr on halem should not be used for permanent storage. If you have access to our mass storage system, then you can use sftp to transfer files to dirac/jimpf for long-term storage. See halem documentation on data storage for more information.

| Top of Page |