GeodesyLab - User contributions [en]

WHOS

2007-09-19T17:44:06Z

Nekram: /* Week Schedule as of 9/7/07 */

The WHOS list tracks assigned GPS weeks for processing.

More details about analysis at [[GPS analysis system]].

===Recent weeks===
<pre>
week # start date last processed who status
****************************************************************
1375 05/14/2006 Trivikram/trivikram 1/ CLEAN EXCEPT 2 DAYS
1376 05/21/2006 Trivikram/1-2/ CLEAN
1377 05/28/2006 07/20/2006 Trivikram 1
1378 06/04/2006 07/20/2006 Trivikram 1
1379 06/11/2006 07/20/2006 Trivikram 1
1380 06/18/2006 07/20/2006 Trivikram 1
1381 06/25/2006 07/12/2006 Ryan
1382 07/02/2006 Julie
1383 07/09/2006 Julie
1384 07/16/2006 Julie
1385 07/23/2006 Julie

1413 02/04/2007 05/01/2007 Tom
</pre>
===Week Schedule as of 9/7/07===
<pre>
week # start date last processed who status
*************************************************************
1430 Brandon one more iteration
1431 Lissy
1432 Yuning
1433 09/14/2007 09/14/2007 Ronni figuring stuff out
1434 Julie
1435 Julie
1436 Julie
1437 Julie
1438 Julie
1439 Brandon one more iteration
1440 Lissy
1441 Yuning
1442 Ronni
1443 Brandon
1444 Lissy
1445 Yuning
1446 Ronni
1447 Brandon
1448 Lissy
1449 Yuning
1450 Ronni
1451 Brandon
1452 Lissy
1453 Yuning
1454 Ronni
1455 Brandon
1456 Lissy
1457 Yuning
1458 Ronni
1459 Julie
</pre>

===2006===
<pre>
week # start date last processed who status
****************************************************************
1357 01/08/2006 Sandeep CLEANING Venkat
1358 01/15/2006 Sandeep CLEANING Venkat
1359 01/22/2006 Sandeep CLEAN Venkat (AK Point File Size < 300)
1360 01/29/2006 Sandeep CLEANING Venkat
1361 02/05/2006 Trivikram
1362 02/12/2006 Trivikram CLEAN (AK Point file size < 400)
1363 02/19/2006 Venkat CLEAN (AK Point file size < 400)
1364 02/26/2006 Trivikram
1365 03/05/2006 Venkat CLEANING
1366 03/12/2006 Trivikram
1367 03/19/2006 Trivikram
1368 03/26/2006 Trivikram problem with AK CLEANING SOLN
1369 04/02/2006 Trivikram CLEAN
1370 04/09/2006 Tom
1371 04/16/2006 Venkat CLEANING
1372 04/23/2006 Venkat CLEANING
1373 04/30/2006 Venkat CLEANING
1374 05/07/2006 Venkat CLEANING
</pre>

===2005===
<pre>
week # start date last processed who status
****************************************************************
1304 01/02/2005 09/16/2005 Ryan CLEAN
1305 01/09/2005 08/30/2005 Pravee CLEAN
1306 01/16/2005 07/15/2005 Jill WORKING Trivikram(tibet) 07/26/2006 CLEAN
1307 01/23/2005 08/31/2005 Pravee CLEAN Trivikram(tibet) 07/26/2006 CLEAN
1308 01/30/2005 04/05/2005 Jill CLEAN Trivikram(tibet) 07/26/2006 CLEAN
1309 02/06/2005 08/31/2005 Pravee CLEAN
1310 02/13/2005 03/12/2005 Julie CLEAN
1311 02/20/2005 03/12/2005 Pravee CLEANING
1312 02/27/2005 03/31/2005 Tom CLEAN
1313 03/06/2005 03/31/2005 Pravee CLEANING
1314 03/13/2005 03/31/2005 Tom CLEAN
1315 03/20/2005 06/13/2005 Samik CLEAN (Point File Size < 500)
1316 03/27/2005 06/13/2005 Samik CLEAN (Point File Size < 400)
1317 04/03/2005 06/16/2005 Samik CLEAN (Point File Size < 400)
1318 04/10/2005 9/15/2005 Ryan not sure what's going on?
1319 04/17/2005 10/4/2005 Ryan ak SUBMITTED
1320 04/24/2005 Sandeep Done 1 2 Trivikram(tibet) 07/27/2006 1 2 CLEAN except 2 days
1321 05/01/2005 Sandeep Done 1 2 Trivikram(tibet) 07/28/2006 1
1322 05/08/2005 Sandeep Done 1 2 Trivikram(tibet) 07/28/2006 1
1323 05/15/2005 Sandeep Done 1 2 Trivikram(tibet) 08/04/2006 1
1324 05/22/2005 Sandeep Done 1 2 Trivikram(tibet) 08/04/2006 1
1325 05/29/2005 Sigrun DONE Trivikram(tibet) 08/04/2006 1
1326 06/05/2005 Sigrun DONE Trivikram(tibet) 08/06/2006 1
1327 06/12/2005 Sandeep Done 1 2 NO TIBET
1328 06/19/2005 Sandeep Done 1 2 No TIBET
1329 06/26/2005 Sandeep Done 1 2 Trivikram(tibet) 08/18/2006 1
1330 07/03/2005 9/14/2005 Ryan DONE
1331 07/10/2005 11/10/2005 Tom Submitted
1332 07/17/2005 Sandeep CLEAN Venkat (AK Point File Size < 500)
1333 07/24/2005 Sandeep CLEAN Trivikram
1334 07/31/2005 Sandeep CLEAN Trivikram
1335 08/07/2005 9/12/2005 Tom CLEAN Venkat (AK Point File Size < 400)
1336 08/14/2005 Sandeep CLEAN Venkat (AK Point File Size < 300)
1337 08/21/2005 10/17/2005 Ryan ak CLEAN Venkat (AK Point File Size < 300)
1338 08/28/2005 10/17/2005 Ryan ak CLEAN Venkat (AK Point File Size < 300)
1339 09/04/2005 11/08/2005 Ryan CLEAN Trivikram (AK Point File Size < 300)
1340 09/11/2005 Sandeep CLEAN Venkat (AK Point File Size < 300)
1341 09/18/2005 Sandeep CLEAN Venkat (AK Point File Size < 400)
1342 09/25/2005 Sandeep CLEAN Venkat (AK Point File Size < 300)
1343 10/02/2005 Sandeep CLEAN Venkat (AK Point File Size < 300)
1344 10/09/2005 Sandeep CLEAN Venkat (AK Point File Size < 400)
1345 10/16/2005 Sandeep CLEAN Venkat (AK Point File Size < 400)
1346 10/23/2005 Sandeep CLEAN Venkat (AK Point File Size < 400)
1347 10/30/2005 Sandeep CLEAN Venkat (AK Point File Size < 400)
1348 11/06/2005 Sandeep CLEAN Venkat (AK Point File Size < 400)
1349 11/13/2005 Sandeep CLEAN Venkat (AK Point File Size < 300)
1350 11/20/2005 Sandeep CLEAN Venkat (AK Point File Size < 400)
1351 11/27/2005 Sandeep CLEAN Venkat (AK Point File Size < 400)
1352 12/04/2005 Sandeep CLEAN Venkat (AK Point File Size < 300)
1353 12/11/2005 Sandeep CLEAN Venkat (AK Point File Size < 400)
1354 12/18/2005 Sandeep CLEANING Venkat
1356 12/25/2005 Sandeep CLEANING Venkat
</pre>

===Older Stuff===
<pre>
week # start date last processed who status
****************************************************************
****************************************************************
***************************2002*********************************
**************************EARTHQUAKE****************************
1189 10/20/02 02/26/05 Josh PENDING
1190 10/27/02 Josh PENDING
1191 11/03/02 03/04/05 Josh DONE
1192 11/10/02 03/11/05 Josh DONE
1193 11/17/02 03/11/05 Josh DONE
1194 11/24/02 03/11/05 Josh DONE
1195 12/01/02 03/11/05 Josh RUNNING
1196 12/08/02 03/11/05 Josh DONE
1197 12/15/02 03/11/05 Josh DONE
1198 12/22/02 03/12/05 Josh DONE
1199 12/29/02 03/12/05 Josh DONE

*************************2003***********************************

1200 01/05/03 03/12/05 Josh DONE
1201 01/12/03 03/12/05 Josh DONE
1202 01/19/03 03/15/05 Josh RUNNING
1203 01/26/03 03/14/05 Josh DONE
1204 02/02/03 03/14/05 Josh DONE
1205 02/09/03 Josh PENDING
1206 02/16/03 Josh PENDING
1207 02/23/03 Josh PENDING
1208 03/02/03 02/26/05 Josh ALMOST CLEAN
1209 03/09/03 02/26/05 Josh ALMOST CLEAN
1210 03/16/03 02/26/05 Josh NEEDS CLEAN
1211 03/23/03 02/27/05 Josh DONE
1212 03/30/03 02/26/05 Josh DONE
1213 04/06/03 03/12/05 Josh RUNNING
1214 04/13/03 02/26/05 Josh CLEANED
1215 04/20/03 02/27/05 Josh CLEANED
1216 04/27/03 02/26/05 Josh PENDING
1217 05/04/03 02/26/05 Josh NEEDS CLEAN
1218 05/11/03 02/27/05 Josh NEEDS CLEAN
1219 05/18/03 02/27/05 Josh WORKING
1220 05/25/03 02/27/05 Josh PENDING
1221 06/01/03 Josh PENDING
1222 06/08/03 Josh PENDING
1223 06/15/03 Josh PENDING
1224 06/22/03 Josh PENDING
1225 06/29/03 02/26/05 Josh FAILED
1226 07/06/03 02/26/05 Josh ALMOST CLEAN
1227 07/13/03 02/26/05 Josh FAILED
1228 07/20/03 Josh FAILED
1229 07/27/03 11/10/05 Tom Submitted
1230 08/03/03 Josh FAILED
1231 08/10/03 Josh FAILED
1232 08/17/03 11/10/05 Tom Submitted
1233 08/24/03 02/26/05 Josh ALMOST CLEAN
1234 08/31/03 02/26/05 Josh ALMOST CLEAN
1235 09/07/03 02/26/05 Josh ALMOST CLEAN
1236 09/14/03 02/26/05 Josh ALMOST CLEAN
1237 09/21/03 02/26/05 Josh NEEDS CLEAN
1238 09/28/03 02/26/05 Josh NEEDS CLEAN
1239 10/05/03 02/26/05 Josh WORKING
1240 10/12/03 02/26/05 Josh PENDING
1241 10/19/03 02/26/05 Josh NEEDS CLEAN
1242 10/26/03 02/26/05 Josh ALMOST CLEAN
1243 11/02/03 02/26/05 Josh NEEDS CLEAN
1244 11/09/03 02/26/05 Josh NEEDS CLEAN
1245 11/16/03 02/26/05 Josh ALMOST CLEAN
1246 11/23/03 02/26/05 Josh ALMOST CLEAN
1247 11/30/03 02/26/05 Josh ALMOST CLEAN
1248 12/07/03 02/26/05 Josh NEEDS CLEAN
1249 12/14/03 02/26/05 Josh NEEDS CLEAN
1250 12/21/03 02/26/05 Josh NEEDS CLEAN
1251 12/28/03 02/26/05 Josh WORKING

***************************2004*********************************

1253 01/11/04 02/26/05 Josh PENDING
1254 01/18/04 Josh PENDING
1255 01/25/04 Josh NEEDS
1256 02/01/04 Josh PENDING
1257 02/08/04 Josh PENDING
1258 02/15/04 Josh PENDING
1259 02/22/04 Josh PENDING
1260 02/29/04 Josh PENDING
1261 03/07/04 02/26/05 Josh PENDING
1262 03/14/04 02/26/05 Josh NEEDS CLEAN
1263 03/21/04 02/26/05 Josh NEEDS CLEAN
1264 03/28/04 02/26/05 Josh NEEDS CLEAN
1265 04/04/04 02/26/05 Josh FAILED No files found, needs re
run
1266 04/11/04 02/26/05 Josh FAILED No files found, needs re
run
1267 04/18/04 02/26/05 Josh NEEDS CLEAN
1268 04/25/04 02/26/05 Josh NEEDS CLEAN
1269 05/02/04 02/26/05 Josh NEEDS CLEAN
1270 05/09/04 02/26/05 Josh NEEDS CLEAN
1271 05/16/04 02/26/05 Josh NEEDS CLEAN
1272 05/23/04 02/26/05 Josh NEEDS CLEAN
1273 05/30/04 02/26/05 Josh NEEDS CLEAN
1274 06/06/04 02/26/05 Josh NEEDS CLEAN
1275 06/13/04 02/26/05 Josh NEEDS CLEAN
1276 06/20/04 02/27/05 Samik DONE
1277 06/27/04 03/19/05 Jeff NEEDS CLEAN
1278 07/04/04 03/19/05 Jeff NEEDS CLEAN
1279 07/11/04 03/19/05 Jeff NEEDS CLEAN
1280 07/18/04 03/19/05 Jeff NEEDS CLEAN
1281 07/25/04 03/19/05 Jeff NEEDS CLEAN
1282 08/01/04 Josh PENDING
1283 08/08/04 Josh PENDING
1284 08/15/04 Josh PENDING
1285 08/22/04 Josh PENDING
1286 08/29/04 Josh PENDING
1287 09/05/04 Josh PENDING
1288 09/12/04 Josh PENDING
1289 09/19/04 03/19/05 Jeff ??
1290 09/26/04 Josh PENDING
1291 10/03/04 Josh PENDING
1292 10/10/04 03/19/05 Jeff ??
1293 10/17/04 Josh PENDING
1294 10/24/04 03/19/05 Jeff ??
1295 10/31/04 03/25/05 Tom CLEAN
1296 11/07/04 04/07/05 Tom Cleaning
1297 11/14/04 03/19/05 Jeff ??
1298 11/21/04 NEEDS CLEAN
1299 11/28/04 NEEDS CLEAN
1300 12/05/04 08/16/05 Pravee CLEAN
1301 12/12/04 08/19/05 Pravee CLEAN
1302 12/19/04 08/19/05 Pravee CLEAN trivikram(tibet) CLEAN 07/26/2006
1303 12/26/04 08/26/05 Pravee CLEAN
</pre>

===Archaic (pre-1996)===
<pre>
week # start date last processed who status
****************************************************************
At the moment, Jeff is dealing with these.
</pre>

GPS analysis system

2007-09-15T21:00:21Z

Nekram: /* Cleaning Solutions */

We use a GPS data analysis system based on the [http://sideshow.jpl.nasa.gov GIPSY] software developed at JPL. Most of the GIPSY programs are called by shell scripts written by Jeff Freymueller. Using these scripts, we can analyze a large amount of data either as part of network solutions or in Precise Point Positioning (PPP) mode.

===Where do you put RINEX files?===
RINEX files should be put into the hopper, <code>$RAWDATA/hopper</code>. What, you don't have RINEX files yet? See [[RINEXing]]. Once files are in the hopper, you can either let the first processing stages happen automatically overnight (see next section), or run the <code>autofront</code> and <code>autoclean</code> programs manually.

===What happens automatically?===

Quite a lot.

Autoftp runs every night beginning at 6pm local time and fetches data files. These are placed into the '''hopper''' ($RAWDATA/hopper), a directory where all data files are put for entry into the system and processing. Autofront then runs at midnight to process all files in the hopper, including any placed there manually (from campaigns, for example). Finally, autoclean runs at 4am local to carry out automated screening for cycle slips and other bad data.

====Autoftp====

Autoftp is an efficient data-fetching tool that uses wget to automatically get data from any of several internet GPS data archives. It reads a list of desired sites from a ''request file'', which contains the date in the filename, and attempts to find and download data from as many sites as possible. It is intended to run automatically on a daily basis under cron, and when acccompanied by another simple program to generate a standard request file every day, it can easily fetch a standard set of sites on a daily basis for analysis. Because it keeps track in the request file of sites that it has found already, autoftp can be run multiple times with the same request file and it will not repeatedly fetch data. This is ideal for the real world, in which data from some sites are available rapidly while data from other sites may require many hours or days to become available.

====Autofront====

Autofront is a script intended to run under cron that carried out the initial "front end" processing on a set of GPS data files. When executed, it will process all files in the '''hopper''' directory, and will place each resulting qm file into the appropriate week directory.

Autofront runs the following steps
1. Checks on the validity of RINEX file and repair of some common problems.
2. Depending on receiver type, clockprep -fixtags
3. (optional, presently not default) PhasEdit
4. ninja

====Autoclean====

Autoclean carries out automated cleaning of cycle slips, based on point positioning solutions. It is quite effective and at present it rarely misses cycle slips unless they are smaller than its minimum tolerance (10 cm). Autoclean operates on an ''edit-request'' file, which contains the name of the directory (week directory) and a list of qm files that need to be cleaned. It will clean all files on the list as long as orbits and clocks are available, and it marks off files that have been cleaned so that it can safely be run multiple times.

Autoclean operates in an iterative mode. It's zeroth iteration is to do a pseudorange-only solution and identify and delete extremely bad pseudorange data. In this step it uses a tolerance that catches only grossly biased data. (Explain it). It then carries out 1 or more iterations of screening the phase data. In each iteration, it uses postbreak to identify discontinuities in the residuals of a point positioning solution. Postbreak is run with an adaptive tolerance (minimum 10 cm), and it is critical that my slightly modified version of postbreak be used. If any cycle slips are discovered, they are flagged and another iteration is run. Autoclean runs a maximum of 4 iterations on the phase data.

===Where do the data files go?===
Data files from each station are stored in the QM format that is native to GIPSY. QM files (and all other) files are stored in directories by GPS week. For each [[week_directory|week directory]] there are several [[subdirectories]]; qm files are stored in <code>$ANALYSIS/wwww/qm</code>, where <code>wwww</code> is the 4 character GPS week number (with a leading zero if needed).

===Running Solutions===
In the [[subdirectories|flt]] directory for each [[week directory|week]] there will (hopefully) be a UNIX script called 
''make-*'' 
This script runs another script called 
''standard_*_solution'' 
which again runs another script called 
''solve'' 
for each [[subnets | network of stations]] for each day. The solve script runs solutions for each of these networks based on the data from sites in the network that are available in the qm directory.

To run solutions, copy the ''make'' script to a file called ''make-flt'' (for example).

Check that the ''make-flt'' script contains all the days that you want to run.

Check which [[Linux Computer System | computer]] is free to run the script (to do so, type:)
check-solves
log on to a free computer and type
submit make-flt

As the script runs, files will appear in the [[subdirectories|flt]] directory for each [[subnets|network]] for each day. Usually
this script will have been run once [[#What happens automatically? | automatically]], so there will often already be files in the [[subdirectories|flt]]
directory ready to be cleaned and then re-run.

Solve, a very flexible script. (link to detailed help)

Philosophy of solve

Subnet files and campaign files

Standard solutions

(text of standard_Alaska_solution)

Running several days at once: make-make-flt and make-alaska

(text of a standard make-alaska file and variant)

Running several weeks at once

(text of sample rerun-* file)

===Cleaning Solutions===

Sometimes bad data (outliers and cycle slips) make it past the automatic editors. When this
happens the bad data are removed from the qm files by either deleting points or by inserting
new phase ambiguities to deal with cycle slips. The steps, commands and scrips to use are explained [[How to clean a solution? | here]]. Once the data are cleaned, the files should be deleted from the [[subdirectories|flt]] directory and the solutions re-run (run <code>make-flt</code> again). Usually you will have to go through 2-3
iterations of the cleaning-rerunning cycle. To be clean, the .point files for a solution do not
exist or are small (below 1000 bytes). Once a solution is clean the files should remain in
the [[subdirectories|flt]] directory and the lines to rerun that solution should be deleted from the <code>make-flt</code> file.

Initial Explanation of terms

Expected residuals from a clean solution

Automated screening: postfit, the point file, postbreak

Checking for bad pseudorange data: badp, allbadp

Removing biased pseudorange data: del_pcode_arc

Automatically identified cycle slips: breaks

Quickly scanning through residuals: short_hand 
The data problems can be identified and
fixed using the program
[[short-hand]]
(follow the link to read about and ask for help to get started with this program). 
Limitations of short_hand

Manually checking residuals and fixing problems (link)

===Procedure for Running Solutions for a week for the first time===

A few special things need to be done the very first time solutions in a week are run. First, you need to make up a script to run all days of the week. This may need to be edited if the JPL final orbits are not available at the time. The <code>standard_Alaska_solution</code> uses the non-fiducial orbits and thus requires that the final JPL orbits be present. If they are not, you can run <code>rapid_Alaska_solution</code> instead.

Then, the log files from autoclean should be moved away to a subdirectory, and any problem stations identified by autoclean should be checked. Then, you are ready to run the solutions.

First, make a script to run solutions. For example, to make a script to run all days of the week for the Alaska solution:
<pre>
cd $ANALYSIS
make-make-flt 1381
cd 1381/flt
vi make-alaska
# Edit the file if needed so that you are ready to run the rapid solutions.

cat make-alaska
#!/bin/csh -f
#
setenv CAMP $ANALYSIS/1381
#
#standard_Alaska_solution 06jul01
#standard_Alaska_solution 06jun25
#standard_Alaska_solution 06jun26
#standard_Alaska_solution 06jun27
#standard_Alaska_solution 06jun28
#standard_Alaska_solution 06jun29
#standard_Alaska_solution 06jun30
#
rapid_Alaska_solution 06jul01
rapid_Alaska_solution 06jun25
rapid_Alaska_solution 06jun26
rapid_Alaska_solution 06jun27
rapid_Alaska_solution 06jun28
rapid_Alaska_solution 06jun29
rapid_Alaska_solution 06jun30

</pre>

The script <code>make-make-flt</code> finds all unique dates for qm files in that week's directory, and uses that to generate the script, so if you run it before the end of the week you will get a partial script. If the final JPL orbits are not yet present, you will need to edit the script to change "standard" ro "rapid". Or better yet, copy all the lines and comment one set out, then modify the others to read "rapid_Alaska_solution <date>".

Next, make a subdirectory called <code>logfiles</code>, and move all of autoclean's logfiles into it. You will need to use two separate <code>mv</code> commands because the list of files is too big for one.

<pre>
mkdir logfiles
mv *____*.i1* logfiles
mv *____*.i* logfiles
</pre>

Now look for a file called <code>make-problems</code>, which lists all files that autoclean had a problem with. Sometimes these files are almost clean, but sometimes they are full of junk or horribly mangled by the automated editing. There should be PPP solutions already run for these files, so they are ready to be checked.

Set the <code>CAMP</code> variable (if not set):
<pre>
setenv CAMP $ANALYSIS/wwww
</pre>
Again, <code>wwww</code> is the 4-digit GPS-week.

Now run the solutions. The first time you run the solutions, look at the residuals very carefully before trying <code>short_hand</code>. Uncompress the postlog, postfit and postbreak files, and then use <code>allbadp</code> to check the pseudorange and <code>allbreaks</code> to check for major cycle slips.

A very common problem is that for several stations per week, there will be one satellite arc of pseudorange data that all have residuals of roughly 2000 cm. If you see these, don't delete the data, but instead run del_pcode_arc to remove only the pseudorange data. I am not sure why these show up, but it could be either a hardware channel bias or a pre-processing glitch. They happen much more often with Ashtechs than any others, and are particuarly common with the US Coast Guard CORS sites. In the <code>qm</code> directory,

<pre>
del_pcode_arc *02gus2* GUS2 GPS41
</pre>

If you just run <code>short_hand</code> without looking, it will probably either throw out all the pseudorange for one site, or delete a lot of data (phase and pseudorange) where only the pseudorange needs to be deleted. So don't do that. Then I delete a batch of bad pseudorange data to get the number of pseudorange outliers under control for the next run.

<pre>
cd $ANALYSIS/1381/flt
gunzip *alaska*post*
allbadp
allbreaks
# Based on this, run del_pcode_arc as above, and add ambiguities manually if needed.
#
delete_allbadp 50
# This creates a file called delete.
vi delete
# Remove lines for any points for which you have already run del_pocde_arc.
cd ../qm
sh ../flt/delete
</pre>

At this point, don't worry too much about phase outliers. Basically we are trying to get the number of pseudorange outliers down into the range where <code>short_hand</code> will do the right thing when we run it later. Now may be a good time to run <code>Alaska_cleaning_solution $date</code>, which runs a smaller and much faster solution including the sites that most often need some cleaning.