Difference between revisions of "RINEXing"

From GeodesyLab
Jump to: navigation, search
 
 
(2 intermediate revisions by the same user not shown)
Line 41: Line 41:
  
 
The final step is to compress the files using gzip.
 
The final step is to compress the files using gzip.
 +
 +
===Splitting mulit-day dat files===
 +
Normally we set our receivers to start a new file every day at UTC midnight and track for 24 hours. Every so often we get a data file that has several days all in one file, because the receiver was not programmed correctly. Most often this happens with borrowed receivers that we forget to reprogram.
 +
 +
If the file contains just a few days of data you can just create a RINEX file as above, then use <pre>rinexwin</pre> to split it into daily files. For example,
 +
 +
<pre>
 +
rinexwin warr2061.05o -ge "25-jul-2005 00:00" -lt "26-jul-2005 00:00" > ../warr2060.05o
 +
rinexwin warr2061.05o -ge "26-jul-2005 00:00" -lt "27-jul-2005 00:00" > ../warr2070.05o
 +
</pre>
 +
 +
However, once a file spans a long enough time it requires some special handling. For files spanning several days you can generally add the +smtt flag to the teqc command (at the moment you have to edit scripte or run it manually to do this). For longer files you need to split the dat file up into daily dat files. The reason for this has to do with the convention typically used in RINEX for handling the 1 millisecond clock offsets in the Trimble and other receivers. These files require some manual fiddling.
 +
 +
There is a 4 step process:
 +
1. Use teqc to create a diagnostic file that contains information about the time and byte position of each data record in the file.
 +
2. Run a perl program that processes this file to make a list of epochs around UTC midnight along with their byte locations.
 +
3. Edit this file, or a copy, to keep only those lines for the epochs at midnight.
 +
4. Use a simple awk command to convert this file into a script to run the program chop, which chops bytes out of a file.
 +
5. Edit this script to make a final simple change, and run it. OK, that is 5 steps.
 +
 +
The reason for all of these steps is because of the millisecond offsets in the data, so that it becomes difficult for a long file to predict exactly which observation is the one at UTC midnight. To do it would require figuring out which way the clock is drifting. It would not be too hard to do that, but is too much programming hassle.
 +
 +
This was developed before the teqc +smtt option, and I believe that by using teqc +smtt it should be possible to create a single perl program that would do all of the above steps in one.
 +
 +
For example, suppose the file tk5e1440.dat contains a couple of months of data. The sequence of commands to split the file is given below.
 +
 +
<pre>
 +
teqc +diag -O.obs - tk5e1440.dat >& tk5e1440.dat.diag
 +
~jeff/bin/split_big_dat tk5e1440.dat.diag > rectimes
 +
</pre>
 +
 +
 +
The file tk5e1440.dat.diag has the RINEX file lines with the observations times, plus a bunch of lines that refer to Trimble download frames. There is one of these with type=17 for each data record.
 +
 +
<pre>
 +
05  5 24  2 41 15.0000000  0  4G 4G13G16G23
 +
Trimble download frame t @ 0o00004373 = 0x000008fb = 00002299  type= 0x15 = 21
 +
Trimble download frame t @ 0o00004671 = 0x000009b9 = 00002489  type= 0x15 = 21
 +
Trimble download frame t @ 0o00005167 = 0x00000a77 = 00002679  type= 0x15 = 21
 +
Trimble download frame t @ 0o00005465 = 0x00000b35 = 00002869  type= 0x11 = 17
 +
05  5 24  2 41 30.0000000  0  4G 4G13G16G23
 +
Trimble download frame t @ 0o00006152 = 0x00000c6a = 00003178  type= 0x15 = 21
 +
Trimble download frame t @ 0o00006450 = 0x00000d28 = 00003368  type= 0x15 = 21
 +
Trimble download frame t @ 0o00006746 = 0x00000de6 = 00003558  type= 0x15 = 21
 +
Trimble download frame t @ 0o00007244 = 0x00000ea4 = 00003748  type= 0x15 = 21
 +
Trimble download frame t @ 0o00007542 = 0x00000f62 = 00003938  type= 0x11 = 17
 +
05  5 24  2 41 45.0000000  0  8G 2G 4G 8G10G13G16G23G20
 +
Trimble download frame t @ 0o00010667 = 0x000011b7 = 00004535  type= 0x11 = 17
 +
05  5 24  2 42  0.0000000  0  8G 2G 4G 8G10G13G16G23G20
 +
Trimble download frame t @ 0o00011764 = 0x000013f4 = 00005108  type= 0x15 = 21
 +
Trimble download frame t @ 0o00012262 = 0x000014b2 = 00005298  type= 0x11 = 17
 +
05  5 24  2 42 15.0000000  0  8G 2G 4G 8G10G13G16G23G20
 +
</pre>
 +
 +
The split_big_dat program is written in perl, and it processes this file and boils it down to an essential one line per data record, giving the byte number and the observation time:
 +
 +
<pre>
 +
5097 at byte number 03354160 : 05 5 24 23 55 14.9710000
 +
5098 at byte number 03354829 : 05 5 24 23 55 29.9710000
 +
5099 at byte number 03355498 : 05 5 24 23 55 44.9710000
 +
5100 at byte number 03356167 : 05 5 24 23 55 59.9710000
 +
5101 at byte number 03356836 : 05 5 24 23 56 14.9710000
 +
5102 at byte number 03357505 : 05 5 24 23 56 29.9710000
 +
5103 at byte number 03358174 : 05 5 24 23 56 44.9710000
 +
</pre>
 +
 +
Now the trick is to edit this file and keep only the lines that correspond to the day boundaries. Be careful about the millisecond offsets. The split_big_dat program only prints out the lines that are near to midnight, so it will be pretty easy to do this. A trick is that once you have found the first one, you can delete 43 lines and the line after that will be the next one. The first few lines of the edited file look like this:
 +
 +
<pre>
 +
5116 at byte number 03366871 : 05 5 24 23 59 59.9710000
 +
10876 at byte number 07145513 : 05 5 25 23 59 59.9390000
 +
16636 at byte number 10925649 : 05 5 26 23 59 59.9060000
 +
22396 at byte number 14704101 : 05 5 27 23 59 59.8720000
 +
</pre>
 +
 +
In this file, every few days there was some problem with the data point at 0000 UTC, so I picked the one at 0015 for those days. This example file had data from 43 different days (42 lines in the file).
 +
 +
Next we make up a script that will run a program called chop over and over again to chop out pieces of the file. Chop needs to know the byte number of the start and end of the file.
 +
 +
<pre>
 +
mkdir split
 +
awk 'BEGIN {last = 0} {print "~jeff/bin/chop +" last " -" $5 " tk5e1440.dat > split/tk5exxx0.dat" ; last = $5}' rectimes.edited > splits
 +
 +
more splits
 +
~jeff/bin/chop +0 -03366871  tk5e1440.dat > split/tk5exxx0.dat
 +
~jeff/bin/chop +03366871 -07145513  tk5e1440.dat > split/tk5exxx0.dat
 +
~jeff/bin/chop +07145513 -10925649  tk5e1440.dat > split/tk5exxx0.dat
 +
~jeff/bin/chop +10925649 -14704101  tk5e1440.dat > split/tk5exxx0.dat
 +
~jeff/bin/chop +14704101 -18486658  tk5e1440.dat > split/tk5exxx0.dat
 +
~jeff/bin/chop +18486658 -22266015  tk5e1440.dat > split/tk5exxx0.dat
 +
~jeff/bin/chop +22266015 -26042019  tk5e1440.dat > split/tk5exxx0.dat
 +
~jeff/bin/chop +26042019 -29821755  tk5e1440.dat > split/tk5exxx0.dat
 +
~jeff/bin/chop +29821755 -33716237  tk5e1440.dat > split/tk5exxx0.dat
 +
</pre>
 +
 +
The last step is to edit the splits file and put in the correct day number for each of the output RINEX files. You have to add a final line for the last day by hand. It should look like this:
 +
 +
<pre>
 +
~jeff/bin/chop +157859599 -161760773  tk5e1440.dat > split/tk5e1840.dat
 +
~jeff/bin/chop +161760773  tk5e1440.dat > split/tk5e1850.dat
 +
</pre>
 +
 +
Then run the script, and you have daily .dat files! One catch is that some of the header meta-data may be missing from the files, unless it is repeated every day. So you may need to provide more meta-data when running teqc to have a complete RINEX header.

Latest revision as of 04:58, 21 October 2005

The program we now use to convert most files to rinex is called teqc, which is written and maintained by Lou Estey at UNAVCO. Teqc can handle data from many different receiver types. This page describes how to convert the data from Trimble receivers into RINEX format.

Converting Trimble .t00/.t01 files to .dat

Normally you don't have to do this. The exception is the data from the Net-RS receiver, which under some circumstances we will only get in .t01 format. Or if you copy .t00 files straight off a flash card.

.t00 files

There are two ways to convert these files to .dat format. You can do it on a Windows PC by right-clicking on the file and selecting "Convert to DAT format" from the contextual menu. Or you can convert the files on our Linux system using: 

/usr/local/UNAVCO/bin/runpkr00 -d file.t00

This will create the file file.dat.

.t01 files

At the moment, these files can only be converted on a Windows PC that has the latest version of Trimble Data Transfer installed. Right-click on the file and choose "Convert to DAT format" from the contextual menu.

Converting dat files to RINEX

You can create a single file to RINEX using teqc. We have a script that will create RINEX files and also add appropriate information into the RINEX headers. The easiest way to convert multiple files is to make up a script and then execute it. 

/bin/ls *.dat | awk '{print "newrinex_teqc -5700 tr_d " $0}' > rinexem

There are some options you can add to this:

  • The -5700 flag really means to use the Zephyr Geodetic antenna, you can use it for the Net-RS as well
  • For data from a Trimble 4000 receiver, use -4000 instead.
  • You can add the operator name using -oper Name (no spaces)
  • If the data are from a non-UAF group, add -agency Agency
  • For 4000 receivers, add -slant 0.2334 -0.0591 to convert from slant to vertical height

Then execute the script: 

sh rinexem

There should be one RINEX observation file (*.YYo, YY = last 2 digits of the year) and one RINEX navigation file (*.YYn) for each .dat file.

To make a fully complete RINEX file, you would have to edit the file in a text editor and add any missing items that are blank. That is important for files that we might send to someone else, but it doesn't matter for our own processing.

The final step is to compress the files using gzip.

Splitting mulit-day dat files

Normally we set our receivers to start a new file every day at UTC midnight and track for 24 hours. Every so often we get a data file that has several days all in one file, because the receiver was not programmed correctly. Most often this happens with borrowed receivers that we forget to reprogram.

If the file contains just a few days of data you can just create a RINEX file as above, then use 
rinexwin
to split it into daily files. For example, 
rinexwin warr2061.05o -ge "25-jul-2005 00:00" -lt "26-jul-2005 00:00" > ../warr2060.05o
rinexwin warr2061.05o -ge "26-jul-2005 00:00" -lt "27-jul-2005 00:00" > ../warr2070.05o

However, once a file spans a long enough time it requires some special handling. For files spanning several days you can generally add the +smtt flag to the teqc command (at the moment you have to edit scripte or run it manually to do this). For longer files you need to split the dat file up into daily dat files. The reason for this has to do with the convention typically used in RINEX for handling the 1 millisecond clock offsets in the Trimble and other receivers. These files require some manual fiddling.

There is a 4 step process: 1. Use teqc to create a diagnostic file that contains information about the time and byte position of each data record in the file. 2. Run a perl program that processes this file to make a list of epochs around UTC midnight along with their byte locations. 3. Edit this file, or a copy, to keep only those lines for the epochs at midnight. 4. Use a simple awk command to convert this file into a script to run the program chop, which chops bytes out of a file. 5. Edit this script to make a final simple change, and run it. OK, that is 5 steps.

The reason for all of these steps is because of the millisecond offsets in the data, so that it becomes difficult for a long file to predict exactly which observation is the one at UTC midnight. To do it would require figuring out which way the clock is drifting. It would not be too hard to do that, but is too much programming hassle.

This was developed before the teqc +smtt option, and I believe that by using teqc +smtt it should be possible to create a single perl program that would do all of the above steps in one.

For example, suppose the file tk5e1440.dat contains a couple of months of data. The sequence of commands to split the file is given below. 

teqc +diag -O.obs - tk5e1440.dat >& tk5e1440.dat.diag
~jeff/bin/split_big_dat tk5e1440.dat.diag > rectimes


The file tk5e1440.dat.diag has the RINEX file lines with the observations times, plus a bunch of lines that refer to Trimble download frames. There is one of these with type=17 for each data record. 

 05  5 24  2 41 15.0000000  0  4G 4G13G16G23
Trimble download frame t @ 0o00004373 = 0x000008fb = 00002299   type= 0x15 = 21
Trimble download frame t @ 0o00004671 = 0x000009b9 = 00002489   type= 0x15 = 21
Trimble download frame t @ 0o00005167 = 0x00000a77 = 00002679   type= 0x15 = 21
Trimble download frame t @ 0o00005465 = 0x00000b35 = 00002869   type= 0x11 = 17
 05  5 24  2 41 30.0000000  0  4G 4G13G16G23
Trimble download frame t @ 0o00006152 = 0x00000c6a = 00003178   type= 0x15 = 21
Trimble download frame t @ 0o00006450 = 0x00000d28 = 00003368   type= 0x15 = 21
Trimble download frame t @ 0o00006746 = 0x00000de6 = 00003558   type= 0x15 = 21
Trimble download frame t @ 0o00007244 = 0x00000ea4 = 00003748   type= 0x15 = 21
Trimble download frame t @ 0o00007542 = 0x00000f62 = 00003938   type= 0x11 = 17
 05  5 24  2 41 45.0000000  0  8G 2G 4G 8G10G13G16G23G20
Trimble download frame t @ 0o00010667 = 0x000011b7 = 00004535   type= 0x11 = 17
 05  5 24  2 42  0.0000000  0  8G 2G 4G 8G10G13G16G23G20
Trimble download frame t @ 0o00011764 = 0x000013f4 = 00005108   type= 0x15 = 21
Trimble download frame t @ 0o00012262 = 0x000014b2 = 00005298   type= 0x11 = 17
 05  5 24  2 42 15.0000000  0  8G 2G 4G 8G10G13G16G23G20

The split_big_dat program is written in perl, and it processes this file and boils it down to an essential one line per data record, giving the byte number and the observation time: 

5097 at byte number 03354160 : 05 5 24 23 55 14.9710000
5098 at byte number 03354829 : 05 5 24 23 55 29.9710000
5099 at byte number 03355498 : 05 5 24 23 55 44.9710000
5100 at byte number 03356167 : 05 5 24 23 55 59.9710000
5101 at byte number 03356836 : 05 5 24 23 56 14.9710000
5102 at byte number 03357505 : 05 5 24 23 56 29.9710000
5103 at byte number 03358174 : 05 5 24 23 56 44.9710000

Now the trick is to edit this file and keep only the lines that correspond to the day boundaries. Be careful about the millisecond offsets. The split_big_dat program only prints out the lines that are near to midnight, so it will be pretty easy to do this. A trick is that once you have found the first one, you can delete 43 lines and the line after that will be the next one. The first few lines of the edited file look like this: 

5116 at byte number 03366871 : 05 5 24 23 59 59.9710000
10876 at byte number 07145513 : 05 5 25 23 59 59.9390000
16636 at byte number 10925649 : 05 5 26 23 59 59.9060000
22396 at byte number 14704101 : 05 5 27 23 59 59.8720000

In this file, every few days there was some problem with the data point at 0000 UTC, so I picked the one at 0015 for those days. This example file had data from 43 different days (42 lines in the file).

Next we make up a script that will run a program called chop over and over again to chop out pieces of the file. Chop needs to know the byte number of the start and end of the file. 

mkdir split
awk 'BEGIN {last = 0} {print "~jeff/bin/chop +" last " -" $5 " tk5e1440.dat > split/tk5exxx0.dat" ; last = $5}' rectimes.edited > splits

more splits
~jeff/bin/chop +0 -03366871  tk5e1440.dat > split/tk5exxx0.dat
~jeff/bin/chop +03366871 -07145513  tk5e1440.dat > split/tk5exxx0.dat
~jeff/bin/chop +07145513 -10925649  tk5e1440.dat > split/tk5exxx0.dat
~jeff/bin/chop +10925649 -14704101  tk5e1440.dat > split/tk5exxx0.dat
~jeff/bin/chop +14704101 -18486658  tk5e1440.dat > split/tk5exxx0.dat
~jeff/bin/chop +18486658 -22266015  tk5e1440.dat > split/tk5exxx0.dat
~jeff/bin/chop +22266015 -26042019  tk5e1440.dat > split/tk5exxx0.dat
~jeff/bin/chop +26042019 -29821755  tk5e1440.dat > split/tk5exxx0.dat
~jeff/bin/chop +29821755 -33716237  tk5e1440.dat > split/tk5exxx0.dat

The last step is to edit the splits file and put in the correct day number for each of the output RINEX files. You have to add a final line for the last day by hand. It should look like this: 

~jeff/bin/chop +157859599 -161760773  tk5e1440.dat > split/tk5e1840.dat
~jeff/bin/chop +161760773  tk5e1440.dat > split/tk5e1850.dat

Then run the script, and you have daily .dat files! One catch is that some of the header meta-data may be missing from the files, unless it is repeated every day. So you may need to provide more meta-data when running teqc to have a complete RINEX header.

Retrieved from "http://gps.alaska.edu/internal/index.php?title=RINEXing&oldid=1566"