Difference between revisions of "GPS analysis system"

Revision as of 22:10, 14 March 2008

We use a GPS data analysis system based on the GIPSY software developed at JPL. Most of the GIPSY programs are called by shell scripts written by Jeff Freymueller. Using these scripts, we can analyze a large amount of data either as part of network solutions or in Precise Point Positioning (PPP) mode.

Where do you put RINEX files?

RINEX files should be put into the hopper, $RAWDATA/hopper. What, you don't have RINEX files yet? See RINEXing. Once files are in the hopper, you can either let the first processing stages happen automatically overnight (see next section), or run the autofront and autoclean programs manually.

What happens automatically?

Quite a lot.

Autoftp runs every night beginning at 6pm local time and fetches data files. These are placed into the hopper ($RAWDATA/hopper), a directory where all data files are put for entry into the system and processing. Autofront then runs at midnight to process all files in the hopper, including any placed there manually (from campaigns, for example). Finally, autoclean runs at 4am local to carry out automated screening for cycle slips and other bad data.

Autoftp

Autoftp is an efficient data-fetching tool that uses wget to automatically get data from any of several internet GPS data archives. It reads a list of desired sites from a request file, which contains the date in the filename, and attempts to find and download data from as many sites as possible. It is intended to run automatically on a daily basis under cron, and when acccompanied by another simple program to generate a standard request file every day, it can easily fetch a standard set of sites on a daily basis for analysis. Because it keeps track in the request file of sites that it has found already, autoftp can be run multiple times with the same request file and it will not repeatedly fetch data. This is ideal for the real world, in which data from some sites are available rapidly while data from other sites may require many hours or days to become available.

Autofront

Autofront is a script intended to run under cron that carried out the initial "front end" processing on a set of GPS data files. When executed, it will process all files in the hopper directory, and will place each resulting qm file into the appropriate week directory.

Autofront runs the following steps 1. Checks on the validity of RINEX file and repair of some common problems. 2. Depending on receiver type, clockprep -fixtags 3. (optional, presently not default) PhasEdit 4. ninja

Autoclean

Autoclean carries out automated cleaning of cycle slips, based on point positioning solutions. It is quite effective and at present it rarely misses cycle slips unless they are smaller than its minimum tolerance (10 cm). Autoclean operates on an edit-request file, which contains the name of the directory (week directory) and a list of qm files that need to be cleaned. It will clean all files on the list as long as orbits and clocks are available, and it marks off files that have been cleaned so that it can safely be run multiple times.

Autoclean operates in an iterative mode. It's zeroth iteration is to do a pseudorange-only solution and identify and delete extremely bad pseudorange data. In this step it uses a tolerance that catches only grossly biased data. (Explain it). It then carries out 1 or more iterations of screening the phase data. In each iteration, it uses postbreak to identify discontinuities in the residuals of a point positioning solution. Postbreak is run with an adaptive tolerance (minimum 10 cm), and it is critical that my slightly modified version of postbreak be used. If any cycle slips are discovered, they are flagged and another iteration is run. Autoclean runs a maximum of 4 iterations on the phase data.

Where do the data files go?

Data files from each station are stored in the QM format that is native to GIPSY. QM files (and all other) files are stored in directories by GPS week. For each week directory there are several subdirectories; qm files are stored in $ANALYSIS/wwww/qm, where wwww is the 4 character GPS week number (with a leading zero if needed).

Running Solutions

In the flt directory for each week there will (hopefully) be a UNIX script called
make-*
This script runs another script called
standard_*_solution
which again runs another script called
solve
for each network of stations for each day. The solve script runs solutions for each of these networks based on the data from sites in the network that are available in the qm directory.

To run solutions, copy the make script to a file called make-flt (for example).

Check that the make-flt script contains all the days that you want to run.

Check which computer is free to run the script (to do so, type:)

  check-solves

log on to a free computer and type

  submit make-flt

As the script runs, files will appear in the flt directory for each network for each day. Usually this script will have been run once automatically, so there will often already be files in the flt directory ready to be cleaned and then re-run.

Solve, a very flexible script. (link to detailed help)

Philosophy of solve

Subnet files and campaign files

Standard solutions

(text of standard_Alaska_solution)

Running several days at once: make-make-flt and make-alaska

(text of a standard make-alaska file and variant)

Running several weeks at once

(text of sample rerun-* file)

Cleaning Solutions

Sometimes bad data (outliers and cycle slips) make it past the automatic editors. When this happens the bad data are removed from the qm files by either deleting points or by inserting new phase ambiguities to deal with cycle slips. The steps, commands and scrips to use are somewhat explained here. Once the data are cleaned, the files should be deleted from the flt directory and the solutions re-run (run make-flt again). Usually you will have to go through 2-3 iterations of the cleaning-rerunning cycle. To be clean, the .point files for a solution do not exist or are small (below 1000 bytes). Once a solution is clean the files should remain in the flt directory and the lines to rerun that solution should be deleted from the make-flt file.

Initial Explanation of terms

Expected residuals from a clean solution

Automated screening: postfit, the point file, postbreak

Checking for bad pseudorange data: badp, allbadp

Removing biased pseudorange data: del_pcode_arc

Automatically identified cycle slips: breaks, allbreaks

Quickly scanning through residuals: short_hand
The data problems can be identified and fixed using the program

  short_hand 

(follow the link to read about and ask for help to get started with this program).
Limitations of short_hand

Manually checking residuals and fixing problems

Procedure for Running Solutions for a week for the first time

A few special things need to be done the very first time solutions in a week are run. First, you need to make up a script to run all days of the week. This may need to be edited if the JPL final orbits are not available at the time. The standard_Alaska_solution uses the non-fiducial orbits and thus requires that the final JPL orbits be present. If they are not, you can run rapid_Alaska_solution instead.

Then, the log files from autoclean should be moved away to a subdirectory, and any problem stations identified by autoclean should be checked. Then, you are ready to run the solutions.

First, make a script to run solutions. For example, to make a script to run all days of the week for the Alaska solution:

cd $ANALYSIS
make-make-flt 1381
cd 1381/flt
vi make-alaska
#  Edit the file if needed so that you are ready to run the rapid solutions.

cat make-alaska
#!/bin/csh -f
#
setenv CAMP $ANALYSIS/1381
#
#standard_Alaska_solution 06jul01
#standard_Alaska_solution 06jun25
#standard_Alaska_solution 06jun26
#standard_Alaska_solution 06jun27
#standard_Alaska_solution 06jun28
#standard_Alaska_solution 06jun29
#standard_Alaska_solution 06jun30
#
rapid_Alaska_solution 06jul01
rapid_Alaska_solution 06jun25
rapid_Alaska_solution 06jun26
rapid_Alaska_solution 06jun27
rapid_Alaska_solution 06jun28
rapid_Alaska_solution 06jun29
rapid_Alaska_solution 06jun30

The script make-make-flt finds all unique dates for qm files in that week's directory, and uses that to generate the script, so if you run it before the end of the week you will get a partial script. If the final JPL orbits are not yet present, you will need to edit the script to change "standard" ro "rapid". Or better yet, copy all the lines and comment one set out, then modify the others to read "rapid_Alaska_solution <date>".

Next, make a subdirectory called logfiles, and move all of autoclean's logfiles into it. You will need to use two separate mv commands because the list of files is too big for one.

mkdir logfiles
mv *____*.i1* logfiles
mv *____*.i* logfiles

Now look for a file called make-problems, which lists all files that autoclean had a problem with. Sometimes these files are almost clean, but sometimes they are full of junk or horribly mangled by the automated editing. There should be PPP solutions already run for these files, so they are ready to be checked.

Set the CAMP variable (if not set):

setenv CAMP $ANALYSIS/wwww

Again, wwww is the 4-digit GPS-week.

Now run the solutions. The first time you run the solutions, look at the residuals very carefully before trying short_hand. Uncompress the postlog, postfit and postbreak files, and then use allbadp to check the pseudorange and allbreaks to check for major cycle slips.

A very common problem is that for several stations per week, there will be one satellite arc of pseudorange data that all have residuals of roughly 2000 cm. If you see these, don't delete the data, but instead run del_pcode_arc to remove only the pseudorange data. I am not sure why these show up, but it could be either a hardware channel bias or a pre-processing glitch. They happen much more often with Ashtechs than any others, and are particuarly common with the US Coast Guard CORS sites. In the qm directory,

del_pcode_arc *02gus2* GUS2 GPS41

If you just run short_hand without looking, it will probably either throw out all the pseudorange for one site, or delete a lot of data (phase and pseudorange) where only the pseudorange needs to be deleted. So don't do that. Then I delete a batch of bad pseudorange data to get the number of pseudorange outliers under control for the next run.

cd $ANALYSIS/1381/flt
gunzip *alaska*post*
allbadp
allbreaks
# Based on this, run del_pcode_arc as above, and add ambiguities manually if needed.
#
delete_allbadp 50
#  This creates a file called delete.
vi delete
#    Remove lines for any points for which you have already run del_pocde_arc.
cd ../qm
sh ../flt/delete

At this point, don't worry too much about phase outliers. Basically we are trying to get the number of pseudorange outliers down into the range where short_hand will do the right thing when we run it later. Now may be a good time to run Alaska_cleaning_solution $date, which runs a smaller and much faster solution including the sites that most often need some cleaning.

Data Backup

RINEX file backups. There are either 1 or 2 separate backups of the raw rinex files. For data we collected ourselves, a copy of the original rinex files can be found in either the campaign directory (/gps/akda/Campaigns/Data.2007/<project>, where <project> is the project name, and Data.2007 will change with the year), or in the continuous site ftp area (/gps/akda/Permanent/2007/260/, where 2007 is the year and 160 is the day of year). Also, every rinex file put through the hopper is moved to a directory like $RAWDATA/2007/260/ (again, year and day of year may change). However, the $RAWDATA/2007/260/ directories are not really archived and eventually they will be deleted. But in practice we have most of the last few years of rinex files online in case something goes wrong.

QM file backups. Before autoclean makes any changes to a qm file, it copies the file to a subdirectory called "original" in the qm directory. so if you completely destroy a qm file by accident, you can still go back the original version. Of course, that loses all editing done to the file, but at least the original data can be recovered easily. In general, it is not a good idea to go back to the version in the original subdirectory unless you know what you are doing, because doing that can make a lot more work for everyone. Mostly we do that when files have been mangled by autoclean. It is actually hard to mangle data files using our usual editing procedures.

Customized Solutions

Sometimes customized solutions are required for various reasons. This link will provide some strategies that may improve your situation.
Kinematic Processing
Ambiguity Resolution

Products / file contents

Where to find certain information and what are the structure / content of output files? It is started to be summarized on the files page.