Research Projects
Classes GEOS 293/393 Hawaii Field Course GEOS 436/636 Beyond the Mouse GEOS 602 Geophysical Fields GEOS 620 Geodynamics GEOS 655 Tectonic Geodesy GEOS 692 Geophys. Seminar
Current & Past Students
Student Opportunities
Curriculum Vitae
Recent Publications
Info for Surveyors
Personal Info

Climb higher, beyond the mouse...

Beyond the Mouse LAB 7: Unix Tools 1

October 17,19

Instructor: Jeff Freymueller

x7286 Elvey 413B jfreymueller@alaska.edu

TA: Shanshan Li

Last Updated: October 10, 2017

Due: Tuesday Oct 24, before class

Lab slides

Refer to the last slides from the lecture.

We are going to run the exercises for the next few labs in a "Virtual Machine", or a computer within a computer. This allows us to have you use a machine with the Linux operating system even though we only have Windows machines in the lab. There are two parts to this solution: a software program that sets up a "virtual computer" in software, and a file on the Windows machine that contains the image of the hard disk(s) for the virtual machine. Because of the setup of our computer lab, you will need to bring a USB drive with you for the lab and save all your work to that -- you cannot assume that any files you create within the virtual machine will be there the next time you log in (and if they are there, they might be someone else's files).

Setting up the VirtualBox (Read this before doing anything else!)

The VirtualBox software is installed on all the machines in the lab. VirtualBox allows you to install many different guest operating systems into one host operating system. For us the guest operating system is a Linux distribution; namely Fedora. You can read through one of the comparisions of the most widely used Linux distributions if you're interested and wondering what you would choose if you install it on your own machine (personally, I use a different distribution, openSuSE).

Because of the setup of our computer lab, you will need to go through several setup steps before you can use it. This should only be needed one time:

  1. Open up a Windows Explorer window and go to the folder "C:\Program Files\Oracle\VirtualBox\HardDisks". Copy the file "fbtm22.vdi" to your Documents folder.
  2. Double-click on the "Oracle VM VirtualBox" icon on the desktop, or select the program from the Start Menu.
  3. Click on the "New" button in the upper left of the window, which will start the process of setting up a new virtual machine.
  4. Type "Fedora" or some other name you prefer (this is only for your use) in the "Name" box. Select "Fedora-64bit" from the menu for Version. Type should be set to "Linux". Then click Next to go to the next dialog box.
  5. Change the Memory size from 1024 MB to 2048 and click Next again.
  6. The next dialog box should say "Hard drive". Click on "Use an existing virtual hard drive file", and then click on the folder icon to broswe and select the file "C:\Users\username\My Documents\fbtm22.vdi", where username should be replaced with your username. Double-click this file, or click once and then click Open.
  7. Click Create, and your virtual machine is now configured.

All of the above steps should only have to be done one time. Now you need to start up your virtual machine. The remaining steps you will do each time you run the program and use the Virtual Machine.

  1. Click Start to start it up. It will take just a few seconds -- it is actually booting up the virtual computer and loading the operating system.
  2. You will get a couple of dialog boxes with information, which you can dismiss. Basically they are telling you that as long as the mouse is over the virtual computer's window and that window is in front, everything you type and every mouse movement will go to the virtual machine.
  3. You may get an error message with a "Press S to skip" message -- press S. This is actually a leftover from the configuration of the previous computer lab -- it is nothing to worry about.
  4. Some messages will flash up while Fedora boots up. You can ignore them. You will eventually get a window with a colorful background, which is the desktop of the virtual machine.
  5. You may see a dialog box telling you that this version of Fedora is no longer supported. Click close to dismiss the box. Also, something called "Update Manager" will start up and offer to install updates for you. Don't install any updates! Click the red X in the upper left corner of it's window to get rid of Update Manager.
  6. When you are ready to exit VirtualBox, you have the choice of shutting down the virtual machine, or suspending it. If you suspend it, then it will be in the same state you left it when you come back to the program later.

The Fedora installation doesn't know a thing about the Windows it's running in; poor thing. As far as it can tell, it is running on a real computer! The VirtualBox software has some settings and controls that determine how the virtual machine interacts with the real computer, but the only one you need to use here is the one that looks like a USB connector. You will use this to tell make a USB drive attached to the real computer usable within the virtual machine.

At some point, after the system is initialized you will see a Desktop with a task bar and all that stuff. Welcome to your working environment for the next labs (this window manager is called gdm (gnome desktop manager)).

You can resize the window and make it as big as you like (bigger is better here).

You will be automatically logged in as user geos436. This is a local user which exists on all the computers in the lab. Your home directory is /home/geos436/. Everything on the virtual machine is local to the geos436>> machine you're working on. So save your scripts (NOT THE DATA!) to your personal flash drive, or open a web browser (mozilla, an early version of Firefox) and upload files to your Google Drive, or email them to yourself. See below for instructions on using a USB drive.

This distribution comes with quite a few applications already; check out the applications menu; or go to /usr/bin. The text editor we will use is gedit (press alt+F2 and type gedit and press enter). I added a few other things including MATLAB, GMT, LaTeX compiler, and several other standard development tools which we'll probably not use, though. Feel free to poke around; one objective of these labs is indeed getting you acquainted with a (maybe) unfamiliar operating system (a crusade in disguise: one can do the virtualbox thing also the other way around -- Windows inside a Linux ;) ).

In the upper task bar, you'll find a shortcut to the gnome-terminal application (you could also press alt+F2 and type gnome-terminal and press enter to open such a window). Another popular terminal application is xterm. Whichever you choose; your shell will be a tcsh. Yes, I want you to open that window now ...

You will be greeted by a prompt: geos436>> Let's look at what this means ('>' marks the prompt):

Using a USB drive on the virtual machine. Here is how to mount and dismount a USB drive on the virtual machine. Note that once you have mounted the USB drive to the virtual machine, Windows can't see it any more. I don't know why they did it that way -- it is actually possible for the two OSes to share, but I suspect it is because Windows doesn't like sharing control.

  1. Insert your USB drive into the computer. Windows will go through the usual stuff, and then tell you that the device is ready to use.
  2. There is a little icon that looks like a USB conector in the window frame of the virtual machines's window, at the bottom. You need to right-click on this, and then select your USB device from the menu. The keyboard and mouse will also be listed. Don't select one of these by accident as it might turn them off.
  3. Now wait a minute or two. You will see Windows telling you again about device drivers, and then after that is done a window will appear on your VM desktop with icons for all the files on your USB drive. You can browse the files using that window -- it is more or less the same was Windows Explorer.
  4. You can also access the USB drive via the command line. It is called /media/drivename, where drivename is whatever name you gave the device. For example, if in Windows your device is called "USB8GB", then you would access it using /media/USB8GB.
  5. The safest way to unmount the drive when you are done is to click on the little eject icon in the file browser window. You can also do umount /media/drivename. As with windows, if any program is still using the drive, it won't unmount.

Exercise 0: Warm-up with some shell tricks.

Type these commands and follow the output on the screen. I want you to get to know this world a bit better:

A useful thing to know is that with the up and down arrows you can browse through the history of your commands. Quite handy if you just typed a very long command and you want to do something quite similar again (Well, it that case it might be time for some shell scripting).

You can learn more about other environment variables by using env | more.

You don't need to turn anything in from this exercise; just play around.

Exercise 1: Commands and Piping

Below we give a list of unix commands which we find useful and you'll get to know this week. Some of these commands work on files/directories. The home directory of your virtual machine may have a directory called lab07_ex1, which contains a lot of files. If for some reason it is not there, or if you accidentally mangle it, then look for the file lab07_ex1.tar.gz in your home directory (if it is not there, download it to your home directory). You can unpack it using tar xfz lab07_ex1.tar.gz. Now cd into the newly created directory lab07_ex1.

Each of the commands given in the tables below does something and creates output that that can be piped into the others (multiple pipes are perfectly fine). The general syntax of Unix commands given in the man-pages is generally something like command [options] file(s). This means you write the command name on the command line, you get to chose whether you want to use any of the options the command offers (square brackets usually indicate that things can be left out), and then you operate on one or more files. Here are the commands:

commanduseful optionsexplanation
ls -l, -s, -t, -rlist files in current directory
wc -l, -c, -w count lines, characters (bytes), words
head -#### (### represents number) output first part of a file
tail -#### (### represents number) output last part of a file
diffcompare files line by line
sort-n -r -ksort lines of text files
historynone.lists history of commands
df / du-k -hshow available disk-space / space used by files
catdisplay / concatenate files
top-n show process statistics, for piping use top -n 1 as it keeps running otherwise

Here are a few examples that can be run in lab07_ex1, which should illustrate how these commands work:

The first thing you will do is play with these commands and their command line options. If you need any files to work on, explore the example directory we gave you, maybe with the example commands listed above. You should go to the man pages and read up on the options we've highlighted in the table if you want to know what they actually mean (> man COMMAND). You will send to us 1) three commands that may include command line options of your choice (please tell us from which directory you invoke the command) 2) an explanation of what the command does, and 3) the output of each command.

Now it's time to step it up a bit: From the list of commands use as many as you like, pipe the output into other commands to create 5 new commands that do something you find useful, redirect the final output to a new file. Send in: (1) the command, (2) a description of the command, (3) a file with the redirected output for each command.

Exercise 2a: Permanently changing your Path and stuff

Since there is only one user geos436 for the VirtualBox, but many Windows users that will use this log-in, everybody will have to use the same directory name for the scripts. Create a directory btm_unix_scripts. Check here to see how to do this. You will want to save a copy of your scripts directory to your USB drive, because you might need to use another machine the next time, and the directory might or might not be there next time (this is a limitation of the Department's computers; it has nothing to do with the virtual machine or Linux).

Now that you all have this directory you will have to edit the .tcshrc file. This is the "Run Command" (rc) file for the tcsh-shell. It is executed every time you log into a shell or open a new terminal Window or subshell. All environment variables, aliases, etc. will therefore be available in any shell session you start on this system. Here is a brief description of things that happen during the login process (for a shell). You might see that you can easily configure your working environment using this file. If it does not exist, you will have to create it. (The leading dot is important; it's part of the filename and 'hides' the file in normal ls listings. This is generally used for configuration files and directories that have to be in your home directory; but you don't have much business messing around with them (or so the developer thought) To see all the stuff that's in your directory, try ls -lisa. The options l,i,s,a are explained in the man pages of ls.)

After all this talk, here's what to do (assuming you created /home/geos436/btm_unix_scripts):

Exercise 2b: Writing a Shell script

In this directory you will find GPS data for a certain day. The files are binary files and you can't read them directly, but knowing the contents of the files is not essential. The key point is that there are many, many files. Some of which are gzipped, others are duplicates: gzipped and unzipped. What I want you to do now is find all the duplicates and rename the unzipped files to all upper case:

While developing this script you might want to test it. To test it you will have to make it an executable and make sure that the first line of the file reads #!/bin/tcsh. To make a file executable, in a terminal window:

You will have to repeat this for any script you want to be executed on the command line. Otherwise you'll get a "could not find ..." response.

The testing happens in ~/lab07_ex2! Open a new terminal window (to refresh the path contents with your new executable), go to ~/lab07_ex2, and execute whatever you called your script. And yes, I will do exactly that and expect your script to work, no matter where it is stored.

Now the funky part: Rename the duplicate files such that all lower case letters are upper case. You can find a nearly complete solution at this website. You will have to do the conversion from lower case to upper case since they convert from upper case to lower case. I find this task challenging and yet rewarding enough that giving the solution away is fine with me. However, you still have to find the correct line on the website, copy it correctly into your script, modify and explain what this line does (use man pages and Internet to find answers).

As a guideline: my neatly formatted, yet uncommented solution script is 8 lines long. There are also some tools you can use to do this in a different way with fewer commands (see for example, rename), but those may require cryptic options, and this way will require you to put together a loop, tests, and some useful shell commands and tools.

Once you're done try ls *QM | wc -l
The result should be 944. I think. It has been a scramble to get the virtual machines up so I am not 100% sure that the one that was saved from last year had exactly that many duplicate files.

You've just changed the name of 944 files. Given the boredom caused by doing the actual conversion by hand and the number of files, writing the script, testing, failing, fixing, testing, succeeding was still a lot faster.

There should be no files matching *qm any more:
> ls *qm
ls: No match.

To be fair though, in real life you might simply call: gzip *qm and let gzip complain about existing files. But the point of this exercise was to introduce you to a few unix tools, get you to do some scripting and do a simple task on many, many files. I hope this objective was accomplished.

Dr. Jeffrey T. Freymueller
Professor of Geophysics
Geophysical Institute
University of Alaska, Fairbanks
Fairbanks, AK 99775-7320

jfreymueller -at- alaska.edu
Phone 907-474-7286
Fax 907-474-7290
Office 413B Elvey