RINEXing - Revision history

Jeff at 04:58, 21 October 2005

2005-10-21T04:58:05Z

Jeff at 00:56, 21 October 2005

2005-10-21T00:56:23Z

Jeff at 01:13, 20 October 2005

2005-10-20T01:13:54Z

Jeff at 07:01, 4 October 2005

2005-10-04T07:01:22Z

New page

The program we now use to convert most files to rinex is called teqc, which is written and maintained by Lou Estey at UNAVCO. Teqc can handle data from many different receiver types. This page describes how to convert the data from Trimble receivers into RINEX format.

===Converting Trimble .t00/.t01 files to .dat===
Normally you don't have to do this. The exception is the data from the Net-RS receiver, which under some circumstances we will only get in .t01 format. Or if you copy .t00 files straight off a flash card.

====.t00 files====
There are two ways to convert these files to .dat format. You can do it on a Windows PC by right-clicking on the file and selecting "Convert to DAT format" from the contextual menu. Or you can convert the files on our Linux system using:

<pre>
/usr/local/UNAVCO/bin/runpkr00 -d file.t00
</pre>

This will create the file <code>file.dat</code>.

====.t01 files====
At the moment, these files can only be converted on a Windows PC that has the latest version of Trimble Data Transfer installed. Right-click on the file and choose "Convert to DAT format" from the contextual menu.

===Converting dat files to RINEX===
You can create a single file to RINEX using teqc. We have a script that will create RINEX files and also add appropriate information into the RINEX headers. The easiest way to convert multiple files is to make up a script and then execute it.

<pre>
/bin/ls *.dat | awk '{print "newrinex_teqc -5700 tr_d " $0}' > rinexem
</pre>

There are some options you can add to this:
* The -5700 flag really means to use the Zephyr Geodetic antenna, you can use it for the Net-RS as well
* For data from a Trimble 4000 receiver, use -4000 instead.
* You can add the operator name using -oper Name (no spaces)
* If the data are from a non-UAF group, add -agency Agency
* For 4000 receivers, add -slant 0.2334 -0.0591 to convert from slant to vertical height

Then execute the script:

<pre>
sh rinexem
</pre>

There should be one RINEX observation file (<code>*.YYo</code>, YY = last 2 digits of the year) and one RINEX navigation file (<code>*.YYn</code>) for each .dat file.

To make a fully complete RINEX file, you would have to edit the file in a text editor and add any missing items that are blank. That is important for files that we might send to someone else, but it doesn't matter for our own processing.

The final step is to compress the files using gzip.

@@ Line 56: / Line 56: @@
 There is a 4 step process:
 . Use teqc to create a diagnostic file that contains information about the time and byte position of each data record in the file.
-. Run a perl program that processes this file to make a list of
+. Run a perl program that processes this file to make a list of epochs around UTC midnight along with their byte locations.
 For example, suppose the file tk5e1440.dat contains a couple of months of data. The sequence of commands to split the file is given below.

@@ Line 116: / Line 116: @@
 <pre>
 mkdir split
-awk 'BEGIN {last = 0} {print "~jeff/bin/chop +" last " -" $5 "  tk5e1440.dat > split/tk5exxx0.dat"; last = $5}' rectimes.edited > splits
+awk 'BEGIN {last = 0} {print "~jeff/bin/chop +" last " -" $5 " tk5e1440.dat > split/tk5exxx0.dat" ; last = $5}' rectimes.edited > splits
 more splits

← Older revision		Revision as of 01:13, 20 October 2005
Line 41:		Line 41:

	The final step is to compress the files using gzip.		The final step is to compress the files using gzip.
		+
		+	===Splitting mulit-day dat files===
		+	Normally we set our receivers to start a new file every day at UTC midnight and track for 24 hours. Every so often we get a data file that has several days all in one file, because the receiver was not programmed correctly. Most often this happens with borrowed receivers that we forget to reprogram.
		+
		+	If the file contains just a few days of data you can just create a RINEX file as above, then use <pre>rinexwin</pre> to split it into daily files. For example,
		+
		+	<pre>
		+	rinexwin warr2061.05o -ge "25-jul-2005 00:00" -lt "26-jul-2005 00:00" > ../warr2060.05o
		+	rinexwin warr2061.05o -ge "26-jul-2005 00:00" -lt "27-jul-2005 00:00" > ../warr2070.05o
		+	</pre>
		+
		+	However, once a file spans a long enough time it requires some special handling. For files spanning several days you can generally add the +smtt flag to the teqc command (at the moment you have to edit scripte or run it manually to do this). For longer files you need to split the dat file up into daily dat files. The reason for this has to do with the convention typically used in RINEX for handling the 1 millisecond clock offsets in the Trimble and other receivers. These files require some manual fiddling.
		+
		+	There is a 4 step process:
		+	1. Use teqc to create a diagnostic file that contains information about the time and byte position of each data record in the file.
		+	2. Run a perl program that processes this file to make a list of
		+
		+
		+	For example, suppose the file tk5e1440.dat contains a couple of months of data. The sequence of commands to split the file is given below.
		+
		+	<pre>
		+	teqc +diag -O.obs - tk5e1440.dat >& tk5e1440.dat.diag
		+	~jeff/bin/split_big_dat tk5e1440.dat.diag > rectimes
		+	</pre>
		+
		+
		+	The file tk5e1440.dat.diag has the RINEX file lines with the observations times, plus a bunch of lines that refer to Trimble download frames. There is one of these with type=17 for each data record.
		+
		+	<pre>
		+	05 5 24 2 41 15.0000000 0 4G 4G13G16G23
		+	Trimble download frame t @ 0o00004373 = 0x000008fb = 00002299 type= 0x15 = 21
		+	Trimble download frame t @ 0o00004671 = 0x000009b9 = 00002489 type= 0x15 = 21
		+	Trimble download frame t @ 0o00005167 = 0x00000a77 = 00002679 type= 0x15 = 21
		+	Trimble download frame t @ 0o00005465 = 0x00000b35 = 00002869 type= 0x11 = 17
		+	05 5 24 2 41 30.0000000 0 4G 4G13G16G23
		+	Trimble download frame t @ 0o00006152 = 0x00000c6a = 00003178 type= 0x15 = 21
		+	Trimble download frame t @ 0o00006450 = 0x00000d28 = 00003368 type= 0x15 = 21
		+	Trimble download frame t @ 0o00006746 = 0x00000de6 = 00003558 type= 0x15 = 21
		+	Trimble download frame t @ 0o00007244 = 0x00000ea4 = 00003748 type= 0x15 = 21
		+	Trimble download frame t @ 0o00007542 = 0x00000f62 = 00003938 type= 0x11 = 17
		+	05 5 24 2 41 45.0000000 0 8G 2G 4G 8G10G13G16G23G20
		+	Trimble download frame t @ 0o00010667 = 0x000011b7 = 00004535 type= 0x11 = 17
		+	05 5 24 2 42 0.0000000 0 8G 2G 4G 8G10G13G16G23G20
		+	Trimble download frame t @ 0o00011764 = 0x000013f4 = 00005108 type= 0x15 = 21
		+	Trimble download frame t @ 0o00012262 = 0x000014b2 = 00005298 type= 0x11 = 17
		+	05 5 24 2 42 15.0000000 0 8G 2G 4G 8G10G13G16G23G20
		+	</pre>
		+
		+	The split_big_dat program is written in perl, and it processes this file and boils it down to an essential one line per data record, giving the byte number and the observation time:
		+
		+	<pre>
		+	5097 at byte number 03354160 : 05 5 24 23 55 14.9710000
		+	5098 at byte number 03354829 : 05 5 24 23 55 29.9710000
		+	5099 at byte number 03355498 : 05 5 24 23 55 44.9710000
		+	5100 at byte number 03356167 : 05 5 24 23 55 59.9710000
		+	5101 at byte number 03356836 : 05 5 24 23 56 14.9710000
		+	5102 at byte number 03357505 : 05 5 24 23 56 29.9710000
		+	5103 at byte number 03358174 : 05 5 24 23 56 44.9710000
		+	</pre>
		+
		+	Now the trick is to edit this file and keep only the lines that correspond to the day boundaries. Be careful about the millisecond offsets. The split_big_dat program only prints out the lines that are near to midnight, so it will be pretty easy to do this. A trick is that once you have found the first one, you can delete 43 lines and the line after that will be the next one. The first few lines of the edited file look like this:
		+
		+	<pre>
		+	5116 at byte number 03366871 : 05 5 24 23 59 59.9710000
		+	10876 at byte number 07145513 : 05 5 25 23 59 59.9390000
		+	16636 at byte number 10925649 : 05 5 26 23 59 59.9060000
		+	22396 at byte number 14704101 : 05 5 27 23 59 59.8720000
		+	</pre>
		+
		+	In this file, every few days there was some problem with the data point at 0000 UTC, so I picked the one at 0015 for those days. This example file had data from 43 different days (42 lines in the file).
		+
		+	Next we make up a script that will run a program called chop over and over again to chop out pieces of the file. Chop needs to know the byte number of the start and end of the file.
		+
		+	<pre>
		+	mkdir split
		+	awk 'BEGIN {last = 0} {print "~jeff/bin/chop +" last " -" $5 " tk5e1440.dat > split/tk5exxx0.dat"; last = $5}' rectimes.edited > splits
		+
		+	more splits
		+	~jeff/bin/chop +0 -03366871 tk5e1440.dat > split/tk5exxx0.dat
		+	~jeff/bin/chop +03366871 -07145513 tk5e1440.dat > split/tk5exxx0.dat
		+	~jeff/bin/chop +07145513 -10925649 tk5e1440.dat > split/tk5exxx0.dat
		+	~jeff/bin/chop +10925649 -14704101 tk5e1440.dat > split/tk5exxx0.dat
		+	~jeff/bin/chop +14704101 -18486658 tk5e1440.dat > split/tk5exxx0.dat
		+	~jeff/bin/chop +18486658 -22266015 tk5e1440.dat > split/tk5exxx0.dat
		+	~jeff/bin/chop +22266015 -26042019 tk5e1440.dat > split/tk5exxx0.dat
		+	~jeff/bin/chop +26042019 -29821755 tk5e1440.dat > split/tk5exxx0.dat
		+	~jeff/bin/chop +29821755 -33716237 tk5e1440.dat > split/tk5exxx0.dat
		+	</pre>
		+
		+	The last step is to edit the splits file and put in the correct day number for each of the output RINEX files. You have to add a final line for the last day by hand. It should look like this:
		+
		+	<pre>
		+	~jeff/bin/chop +157859599 -161760773 tk5e1440.dat > split/tk5e1840.dat
		+	~jeff/bin/chop +161760773 tk5e1440.dat > split/tk5e1850.dat
		+	</pre>
		+
		+	Then run the script, and you have daily .dat files! One catch is that some of the header meta-data may be missing from the files, unless it is repeated every day. So you may need to provide more meta-data when running teqc to have a complete RINEX header.