Lab 4 Getting Started on the Cray

Each of you should have access to the ISTEC Cray.
You should look to the ISTEC Cray Website to answer any questions that come up during this lab. If you can't find the answer there ask the TA or professor. Please do not email ISTEC for help unless you have already spoken to us.

Information Needed to Run Jobs on the Cray

  1. Log onto the cray using your account: ssh username@cray2.colostate.edu
  2. File Storage. The files in the home directory are backed up. Keep all your files there. To execute paralel program you need to be in the lustrefs directory. (You have one provided.) Also, Large files should be stored in lustrefs directory. Lustrefs is not backed up. It is best to store executables and their input output data in lustrefs, and copy anything valuable back to your home directory.
  3. Transferring files from your computer onto CRAY home directory
    Execute this command from your computer.
    scp "file-to-be-transferred" "personnel-id"@cray2.colostate.edu:"destination-directory"
  4. Compilers on the Cray.
    The Cray user guide version 3 was used as reference for the information below. The default compiler loaded is the cray compiler. There are only a few licenses for this compiler, so we use the gcc compiler suite. Use the following command to swap to gcc
      module list
      module swap PrgEnv-cray PrgEnv-gnu
    
    Use the Makefile to compile your codes.
  5. Executing your program on a login node.
    Move the code to your lustrefs directory, and execute it using the aprun command as follows:
      mv program lu*
      cd lu*
      aprun -n1 -d1 program
    
    Check "man aprun" for aprun options. The -n option specifies the number of times (spawns) of "program". The -d option (used for openMP specifies the number of cores executing the code in a node. (each node consists of 24 cores: 4 processors each with 6 cores, memory, and interconnect).
  6. Executing your program on a compute node.
    You will need to write a short bash script.
    #!/bin/bash
    
    #PBS -N PA1
    #PBS -j oe
    #PBS -l mppwidth=2
    #PBS -l walltime=00:01:00
    #PBS -q small
    cd /home/[username]/lustrefs/PA1
    export OMP_NUM_THREADS=2
    aprun -n1 -d2 ./jacobi_1D 4000 200000
    
    Save this to a file and then on the command line run:
    qsub filename
    
    qstat -a will show you the current state of the queue.

For Lab 4: Take your own PA1 and run it on a single compute node of the Cray.


Compare and contrast the results for 1 through 24 threads to what you observed on the CS machines.