for an account contact Balagopal.Pillai@Dal.Ca

To see what software is available on perun check out the page http://perun.biochem.dal.ca/doku.php?id=available_software (username is perun, password is Perun_) as well as other useful pages about using perun.

When text appears in bold this indicates that you should type it either on the command line or into a text editor as part of a script.

Open a terminal on the Mac. Login to perun using

ssh -l username perun.biochem.dal.ca

You will be asked for your password Then, always

qrsh

You may or may not be asked for your password. If you don't enter qrsh you will be in a limited part of the system, unable to access all the programs.

You will always start in your home directory. You will notice on the cursor line that the absolute pathname of your location is present and should look something like this /home/username@perun11 /home/username is the absolute pathname while @perun11 indicates that you have been assigned to node 11

Every person who has an account on perun has a home directory. To see the other home directories

cd ..

then

ls

You should see the names of the other perun account holders. Unless they have modified their home directory permissions you cannot enter or access the contents of their home directories (unless you are a superuser). To go back to your home directory simply type

cd

This will work regardless of where you are in the computer system.

Home Directory vs Scratch directories Perun is a shared resource. While there is a lot of storage space and computing power it has to be shared among the potential users. Because of the way the system is designed, if the home directory (/home/) fills up to capacity then all processes slow to a crawl and no one can do much. So your home directory (/home/username) should only be used as the place to do small jobs that won't produce large result files and you should only keep small and/or important result files there. The combined storage space for /home/ is 650 GB but this is shared among several users and it is usually between 50% and 80% full all the time. Therefore, if you want to work with large files (like the ones generated by NGS machines) and/or produce large result files you should do this in /scratch1/username or /scratch2/username. The storage capacity for scratch1 is 21 Tb and 64 Tb for scratch2. One thing to keep in mind is that the home directory is backed up three times a week while the scratch directories are not backed up. The cluster is stable and there is little chance that any data in the scratch areas will be lost but to be on the safe side you should keep or copy your important files in your home directory and/or on your local computer. To see the capacity and how much is currently being used on the system

df -h

Running Scripts and Processes - Command line vs the Queue Perun contains lots of scripts and programs for your use. There are two main ways to run scripts/programs on perun. You can simply run the process from the command line

./myscript.pl datafile1 datafile2

and the script/program will run right away. This is what you are doing when you issue a UNIX command like ls or cd. You are running a little script called ls or cd that does something very simple and very fast. The other way to run a script/program is by putting the appropriate commands and options in a little script and then submitting the script to a queue. The queue is a perun program that decides where and when to run your script/program depending on what computing resources are available. This means, depending on how many users are using the system and what jobs they are running, that your script/program might not run right away. Once there are CPUs available the queue will then allow your script/program to run. So, why would you submit your script/program to the queue rather than run it directly? 1. Perun has 17 nodes with multiple CPUs per node. When you run something on the command line directly it is only using the computing power of the head node which is used by everyone else to do simple tasks. Consequently you are not using the full potential of perun and you are slowing down everyone else's simple tasks like ls or cd. 2. You may not be getting all the appropriate environmental parameters necessary to run the script/program because it is expecting to be called through the queue. 3. The script/program will run in the background without you having to have a terminal open or even logged in to perun.

How to decide when to use the queue and when to run the script/program directly? As mentioned UNIX commands are simple scripts. These can and should be run from the command line. If the script/program is fairly short and you expect it to finish quickly (under 15 seconds) and not take up large amounts of computing power then you can probably run it from the command line. Everything else, especially scripts/programs that have been installed on perun by the system administrator, should be called from inside a queue submission script.

Creating a submission script Either open a text editor on perun (pico, vi, emacs) or create the script on your local computer (and then copy and paste it into a pico editor on perun or scp it to perun). Every submission script should start with the line

#!/bin/sh

at the very top. This tells the UNIX system what type of shell to use (this one is the bourne shell). You could use other UNIX shells like csh or tcsh if you wanted to. The next two lines should be

#$ -S /bin/sh

and

. /etc/profile

to ensure that the correct and current environmental variables are inherited

Then you can have options beneath these first lines. Some of the more common ones are

#$ -cwd

(this means run the process in the current working directory ie where you submitted the script)

#$ -o logfilename_of_your_choice

(this creates a log file that contains any output that your script/program might generate that is not directly related to the results, so things like error messages, updates of where the process is, etc)

#$ -pe threaded number

(how many CPUs you want to assign to the task - the default is 1 which is typically adequate for most jobs)

After the options you would then have the name of the program you are calling along with any program parameters and/or options. When you run the program from the command line you often just need the name of the program (and any options or parameters). However, when you call a program from a submission script it is generally a good idea to use the full path of the program like so /opt/perun/bin/blastn blah blah blah (See below for how to find the pathname of programs).

Submitting the script Once the submission script is finished save it. To submit the script

qsub name_of_script

There are special queues for large computationally heavy job like large assemblies or large trees. To use these special queues you use a modified qsub command

qsub -q 144G-batch name_of_script (nodes with 144 GB RAM, 24 slots per node (1)) qsub -q 16G-batch name_of_script (nodes with 16 GB RAM, 8 slots per node (12)) qsub -q 256G0batch name_of _script (nodes with 256 GB RAM, 48 slots per node (2))

Monitoring your run To see if you job is running

qstat

This will list all the jobs currently being run on perun as well as who is logged in. To just see your jobs

qstat | grep username

The qstat columns give you information about the processes that are being run. The first column is the job id. This can be used to delete/stop a job

qdel job_id

The third column is the name of the process or script being run. If it says QRLOGIN this is simply the terminal (and the qrsh command you issued at the beginning of the session). The fourth column is the username of who is running the script/process. The fifth column indicates whether the process is running or not. An r means it is actively running while qw means it is waiting in the queue for a slot to be free. If you see Eqw (this should not happen very often) it means your process is waiting in an error state and probably should be deleted. To see what the error is

qstat -j job_id | grep error

If you do ls in the directory where you submitted the script myscript.x to the queue you will see that there is a file called something like this - myscript.x.e798764. This is the error file associated with your submission and one is created every single time you submit something regardless of whether there are any errors. Usually this error file will be empty if the script/program runs successfully. However, sometimes it contains useful information generated by the script/program that has nothing to do with errors so once your process finishes you should check this file. The number after the e is the job_id of the process.

You can have multiple perun sessions/terminals open at any one time. You can also remain logged in indefinitely. If you wish to exit you can close the terminal or else enter

You can change the job's hard queue after submission (without killing it) by:

qalter -q [node you want, eg 256G-batch] [job ID number]

exit

The first exit will terminate the qrsh environment you created so you need to do another exit to log off completely from perun.

Finding the path of an installed program Most programs are installed in common, standard places on UNIX systems. On perun many of the programs/packages are installed in /opt/perun/ or /opt/perun/bin/

cd /opt/perun/

ls

cd bin

ls

to see the names of the programs

If you know the precise name of the program you can find its path by

which velvetg

or

whereis velvetg

New programs If you want to use a linux based piece of free software first check to see if it is already installed on perun. If it is not currently available you can email the system administrator, Balagopal, and ask him to install it. You should provide the URL so he knows where to obtain the software. You should also check to see if the software you are interested in requires ancillary programs that may or may not be installed on perun. Balagopal's email is Balagopal.Pillai@Dal.Ca

There is also the option of trying to install the software in your home directory and then running the program locally. With simple scripts this may work but more sophisticated software usually has dependencies (other software) that it may not be able to find.

The ins and outs of the program Information about the various options and parameters for a program can be obtained from several sources. 1. You can probably find a manual online for a particular program. Perun has some manuals installed at http://perun.biochem.dal.ca/manuals/ 2. Most of the bioinformatics software you will be using is free and was developed by other scientists (mainly grad students and postdocs) for their own purposes. They have generously made their programs available to the wider community. However, because these programs are not commercial the level of instruction about how to use them varies from practically non-existent to pretty good. Sometimes the best way to find out how to do something with a program or why you are getting a particular error is to google it because chances are someone else had a similar question or problem. 3. Almost all command line programs employ several common ways to give the basics about running a script

1. simply type the name of the script on the command line

eg velvetg

1. type the name of the script followed by -h

eg blastn -h

1. type the name of the script followed by -help

eg blastn -help 4. If all else fails you can try and read the program code (if it is in perl or python as opposed to C or C++) and try and figure out what the authors want.