University of Alaska Fairbanks
Geophysical Institute

Beyond the Mouse 2010 - The geoscientist's computational chest.

Lab 8: Unix / Shell Introduction

Lab slides

last slides from lecture

Running the VirtualBox (Read this before doing anything else!)

I installed VirtualBox on all the machines in the lab. VirtualBox allows to install many different guest operating systems into one host operating system. The host is, big surprise, Windows XP into which I installed a Linux distribution; namely Ubuntu. I decided for Ubuntu because it's rather user friedly and these days it's also what I run on my laptop (read: familar). The installation routine is rather hassle free and a lot of effort is put in supporting modern hardware. This, of course, comes with drawbacks I am not going to debate here. You can read through one of the comparisions of the most widely used Linux distributions if you're interested.

How to get your Ubuntu started? Double-click on the ``Oracle VM VirualBox'' icon on the desktop, select 'Ubuntu10.10' and click ``start''. The startup takes a while: an actual operating system is being loaded; Ubuntu doesn't know a thing about the Windows it's running in; poor thing.

At some point, after the system is initialized (which may take a while because everything being virtual makes slows things a bit) you will see a Desktop with task bar and all that stuff. Welcome to your working environment for the next labs (this window manager is called gdm (gnome desktop manager)). Here are a few useful shortcuts:

• (right) ctrl+f: Fullscreen mode on / off (a virtual box shortcut, must be right control!)
• ctrl+f2: start an application
• alt+tab: switch active window
• alt+shift+tab: switch active window backwards
• ctrl+alt+cursor: switch active desktop

I recommend working in fullscreen mode; but that's just me.

You will be automatically logged in as user btm_user. This is a local user which exists on all the computers in the lab. In your home directory (/home/btm_user/) you will find a directory N_DRIVE; this is your personal N:\ from the Windows world. I mounted this as a shared drive so that you have a place to save your data/scripts. Everything you save to /home/btm_user/N_DRIVE (case sensitive!) will be available on all machines. Everything else is local to the GEOS-XX machine you're working on. So save your stuff to N_DRIVE or your personal flash drive (given that usb works, untested). If there's any software you want to install you can do that locally in /home/btm_user only. If it's a reasonably useful application (read: for everybody) let me know and I'll sudo it on the machines.

This distribution comes with quite a few applications already; check out the applications menu; or go to /usr/bin. The standard text editor is gedit (press alt+F2 and type gedit and press enter). I added a few other things including MATLAB, GMT, LaTeX compiler, and several other standard development tools which we'll probably not use, though. Feel free to poke around; one objective of these labs is indeed getting you acquainted with a (maybe) unfamilar operating system (a crusade in disguise: one can do the virtualbox thing also the other way around -- Windows inside a Linux ;) ).

In the upper task bar, you'll find a shortcut to the gnome-terminal application (you could also press alt+F2 and type gnome-terminal and press enter to open such a window). Another popular terminal application is xterm. Whichever you choose; your shell will be a tcsh. Well, yeah, I want you to open that window now ...

You will be greeted by a prompt: GEOS-14:~> Let's look at what this means ('>' marks the prompt):

• GEOS-14: The hostname, obviously all the installations are a clone of GEOS-14 where I first installed Ubuntu
• Everything between the colon and the '>' tells you which directory you are in
• ~: is synonymous for a user's home directory (for us a shortcut to /home/btm_user

Exercise 0: Warm-up with some shell tricks.

Type these commands and follow the output on the screen. I want you to get to know the OS a bit better:

• > cd /usr/bin
  change directory to /usr/bin, look at the prompt; it should have changed
• > pwd
  print working directory
• > whoami
  get current user name (sometimes that's pretty helpful)
• > echo $USER
  same thing, the user name
• > ls
  list contents in /usr/bin ... wait what?
• > ls | more
  list contents in /usr/bin and pipe it into a pager
• > man more
  learn about more or other commands you find (exit with 'q')
• > man man
  man is a command, too
• > cd ~
  go home
• > pwd
  print working directory
• > cd N_DRIVE
  print working directory
• > matlab &
  starts Matlab; the & pushes the new process in the background which allows you to continue using this terminal window.

Sweet. A useful thing to know is that with the up and down arrows you can browse through the history of your commands. Quite handy if you just typed a very long command and you want to do something quite similar again (Well, it that case it might be time for some shell scripting).

Now for some tricks with the PATH and other variables. Be sure to do exactly what is listed. Answer questions where I ask.

• > echo $PATH
  What do these directories, separated by ':' tell you?
• > set mytmpPATH=$PATH
  What are we doing here?
• > setenv PATH && echo $PATH && whoami
  What's going on? (Hint: && is a logical AND which executes the later commands based on the success of the earlier commands, test this with: > setenv PATH=0 && echo $PATH && whoami, no whoami error)
• > echo $USER
  get current user name (sometimes that's pretty helpful)
• > matlab
  also not working ...
• > setenv PATH $mytmpPATH && echo $PATH && whoami
  
• > matlab
  Yeah - still questions about the usefulness of PATH?

You can learn more about other environment variables by using env | more.

Exercise 1: Writing a Shell script (Solution for a slightly modified version in Lab 09)

• go to your N_DRIVE in a Terminal
• retrieve a package with data files using wget:
  wget http://www.gps.alaska.edu/programming/lectures/lab08.tar.gz
• unpack a *.tar.gz archive using tar: tar xfz lab08.tar.gz
• check whether you have a directory lab08

In this directory you will find GPS data for a certain day. That's not essential. The key is that there are many, many files. Some of which are gzipped, others are duplicates: gzipped and unzipped. What I want you to do now is find all the duplicates and rename the unzipped files to all upper case:

• open a text editor (gedit)
• write a shell script which you will save to N_DRIVE/lab08 that will:
• loop over all qm files (use foreach, and backtick operator in which you do a ls)
• check whether the file exists in gzipped version (hint: if statement)
• echo the duplicate

Now the funky part: Rename the file such that all lower case letters are upper case. You find almost a full solution at this website. You will have to do the conversion from lower case to upper case since they convert from upper case to lower case. I find this task challenging and yet rewarding enough that giving the solution away is fine with me. However, you still have to find the correct line on the website, copy it correctly into your script, modify and explain to me what this line does (use man pages and Internet to find answers).

As a guideline: my neatly formatted, yet uncommented solution script is 8 lines long.

Once you're done try > ls *QM | wc -l
The result should be 944.

You've just changed the name of 944 files. Given the boredom caused by doing the actual conversion by hand and the number of files, writing the script, testing, failing, fixing, testing, succeeding was still a lot faster.

Now you could go ahead and remove all files that are upper case using rm ./*.QM. A safe way to mark files and hopefully avoid deleting precious data accidentally - good thing you have backups.

To be fair though, in real life you would simply call: gzip *qm and let gzip complain about existing files. But the point of this exercise was to introduce you to a few unix tools, get you to do some scripting and do a simple task on many, many files. I hope this objective was accomplished.