Processing LTA data with quick_adcp.py

This document was originally written for a unix-savvy data-literate person going on a cruise with both a NB150 (DAS2.48) and an OS75 (VmDAS). References are made to that cruise as an example.

Batch processing

The first step is to get through one complete pass through the processing. Sufficient documentation exists (online or contained withing a CODAS installation) to train a person on processing LTA data. There is far more documentation for processing of pingdata (from the nb150), so you may still need to consult that for some details.

The approach of this document is to take a stab at a black box, and hope it works. I will describe the black box, the inputs and outputs, and some rough sanity checks on the data, liberally referring to the existing documentation (only linking to it in the html version).

CODAS (Common Ocean Data Access System) is fundamentally a database. We load the averaged data into it and then in various steps, access the database and write files to the disk, manipulate those files to create different ones, and put information from those new files back into the database.

Extractions are done with C programs, called as “action action.cnt” (where “action.cnt” is an ascii file with a specified format and contains configurable options for “action”). Manipulations are done with a combination of C and Matlab code. Usually the matlab code is a small script or wrapper which contains variables to change, and when run, calls outside scripts or functions to perform its duty. Changes to the database are implimented with more C code. “LTA” refers to the original “Long Term Averages” (5-minute averages, for instance) assembled by VmDAS and written to disk as “*.LTA” files.

The acquisition PC is probably networked, and may be writing to a networked disk. You need to get the data onto your computer. The best way to do this is to make sure that the data directory is exported (shared) for all to read, and then using your PC, just read the LTA files from the share (or with linux use smbclient or smbmount) to access the files.

Because the steps in processing are quite regular we have written a Python script to automate these steps. Quick_adcp.py runs through the steps, changing to the right directory, writing the control file (or the matlab file), running it, and changing back to the original directory, asking the user at each step whether to run the next step. Al the control files are suffixed with “.tmp”, all the matlab files end in “_tmp.m”, and a record of steps is written to a file ending in “.runlog”.

Complete “processing” of ADCP data actually requires several iterations:

1. get the velocity data and the navigation into the database, clean up (smooth) the navigation using an oceanic reference layer,
2. apply any phase or amplitude necessary (** see below) clean up (smooth) the navigation using an oceanic reference layer, calculate residual phase and amplitude calibration values
3. edit bad bins, profiles, and bottom interference (for historical reasons this is a two-stage process: (a) write ascii files to the disk which contain information about the bad data; (b) apply these flags to the database (we turn on bits indicating bad data, we do not actually delete any of the data) clean up (smooth) the navigation using an oceanic reference layer, calculate residual phase and amplitude calibration values
4. make some matlab files suitable for vector or contour plotting

Calibration values: You should determine the heading source used for your data. See these links (heading correction setup and example for more details. After the first pass of processing, there may still be an offset in heading. Look for “phase” under the “mean” and “median” columns cal/watertrk/adccal.out and cal/botmtrk/btcaluv.out. If there are sufficient points (at least 10 watertrack and 40 bottomtrack is reasonable) in “edited”, then a value between the bottomtrack and watertrack values for the rotation. Amplitude for the Melville looks OK, so you shouldn’t have to make that adjustment. Ultimately, with enough points, we expect the final calibration amplitude mean to be around 0.997-1.003 and phase to be around -.01 to 0.01 (stddev around 0.3-0.4 is good). In this example, we are starting with an unknown calibration phase and amplitude, but once we know what they are we can build that in on the first step (see part II).

Data processing takes place in a directory created by “adcptree.py”. Do this once to start a new processing directory. Processing takes place in that directory. Manually, each step would take place within a subdirectory devoted to a particular steps (scanning, loading, navigation-related, calibration-related, editing, etc). For each step, quick_adcp.py changes directory to the appropraite location, runs the step, and changes directory back. Quick_adcp.py needs to know various things about the data, the most basic if which are:

variable name    :    what it is
-------------    :    ----------
yearbase         :    current year
datadir          :    where the data are
datafile_glob    :    wildcard expansion for files
                 :       !! QUOTE IT if it is on the command line
                 :       (do not quote if in a control file)
dbname           :    5-character basename for database
instname         :    os75, os150, os38 (so output matlab files
                 :       have good depth defaults
use_refsm        :    reference layer calculation (choose one of):
                 :       use_refsm
                 :       use_smoothr

Note

quick_adcp.py must be run in one line of text. HTML may be wrapping the line: you may be mislead

For data residing in /home/data/mv0407/adcp, named *.LTA, the quick_adcp.py steps corresponding to the numbers above are:

1. load the data

quick_adcp.py --yearbase 2004 --dbname a0407 --datadir /home/data/mv0407/adcp --datafile_glob "\*.LTA" --instclass os --datatype lta --use_refsm

2. rotate the data, if you already know the angle

quick_adcp.py --yearbase 2004 --use_refsm --instname os75  --steps2rerun apply_edit:navsteps:calib:matfiles

3. Go to the edit/ subdirectory, run gautoedit and edit the data (i.e. create the ascii files which will then become flags in the database. Apply the prfile flags to the database

quick_adcp.py –yearbase 2004 –use_refsm –instname os75 –steps2rerun apply_edit:navsteps:calib:matfiles

4. (automatically generated when adding “matfiles” to the “steps2rerun” switch. see the following files, (do “help load_adcp” for info on reading):

   • vector/*.mat (for 1-hour 50m averages)
   • contour/*.mat (for 15minute, 10m averages)
   • contour/albins_*.mat (all bins, all profiles). load with load_getmat.m

Note

quick_adcp.py can also be run with the variables in a control file. If you look at the output of the following two commands you will see an example of (1) LTA processing commands and (2) LTA processing steps (summarized), respectively:

quick_adcp.py --commands LTA
quick_adcp.py --howto_lta

Note

If you are looking for a tutorial about LTA processing, you don’t need the rest of this. Go look at the “LTA demo” in the “Practice Datasets” section of this quick_adcp.py overview

If you are at sea and you are trying to “update” CODAS processing of an LTA dataset during the cruise, keep reading