Navigation

This Page

Reading binary ADCP data files

Matlab requirements

You must have UH Currents group matlab code installed and the matlab path configured. Read the relevant parts of this page to find the and download the matlab code (“matlab.zip”), and set up the path.

Reading raw data

Programs exist to read raw data from data files collected by various logging programs, sometimes more than one program can log data for a given instrument. The binary data format differs between instruments; the filenames vary between logging programs. The “class” (below) indicates the class of instrument (specifically the kind of binary data stored). There is one read program for each class of instruments: specifying the instrument class is necessary during the read process so matlab will know which “read” program to run. These ‘read’ programs only work with instruments that have 4 beams (regardless of how many are working):

class   instrument   self-contained    underway
          name         (lowered,         (requires a  data
                        moored)         acquisition program)
-----   ----------   ----------     ------------------------------
  bb     bb adcp         yes         yes: VmDAS  (ENR, ENS, ENX, STA, LTA
                                     yes: Transect (*.r files only)
  bb     workhorse       yes         yes: VmDAS  (ENR, ENS, ENX, STA, LTA
  os   ocean surveyor    NA          yes: VmDAS  (ENR, ENS, ENX, STA, LTA
                                          UHDAS    (*.raw)
  hr     HDSS            NA          yes: HDSS     (ugly file name)
 son    Sontek           yes         NA

The “read” function returns a data structure with fields present for each beam in a three-dimensional array. Type “help nb/read”, “help bb/read”, (etc) to see the options for the program. In addition to required options (instrument class, file name, and yearbase) you can specify the data variables to output (eg. ‘vel’ is velocity, ‘ts’ is time series (heading, temperature,...)) and the indices to extract. Note that not all fields are available in all data sets.

There are two types of single-ping data:

  • from the instrument: (eg. moored, lowered) or for shipboard ADCP, the ENR files (from VmDAS) or “raw” files (from UHDAS). These data have no positions, the instrument clock sets the time stamp, and if heading is present it probably came from an internal flux gate compass and/or tilt sensors.
  • “fully navigated”: These have passed through a data acqusition system and additional information has been added (position, attitude, and corrected time stamp)

NOTE that the data collected may be in earth or beam coordinates, depending on the way the instrument was set up. Fully navigated shiboard data are in earth coordinatees.

Data present in the table above have the following fields:

"RAW" (from the
 instrument)        lon  coord      time        pings per   heading
                    lat  system     base        ensemble    (pitch,roll)
------------------+-----+---------+-----------+-----------+-------------
moored or lowered    no   note(1)  instrument   note(1)     internal sensor

shipboard:
'raw'
    VmDAS ENR        no   beam     instrument      1          see note(2)
    UHDAS raw        no   beam     instrument      1          see note(2)

shipboard:
partially
navigated
    VmDAS ENS       yes   beam       UTC           1          see note(2)

shipboard:
fully
navigated
    VmDAS ENX       yes   earth      UTC           1          as specified
    VmDAS STA       yes   earth      UTC        note(2)       as specified
    VmDAS LTA       yes   earth      UTC        note(2)       as specified

    UHDAS raw       yes   earth      UTC           1          as specified
                                                              (see note(3))
    HDSS  uglyname  yes   earth      UTC           1          as specified
                                                             (see note(4))

NOTES

  1. Moored or lowered are usually collected in earthcoordinates, but the transformation matrices are available in this suite of programs to switch from beam to earth coordinates given heading.
  2. Raw RDI (un-navigated) shipboard ADCP data will only have a heading if it is put into the deck unit using the synchro input. You cannot assume this is being used: heading usually comes in a serial input and is not available until the “fully navigated” stage.
  3. UHDAS raw data files are un-navigated, and are stored along with the raw serial inputs (gzipped files with GGA messages, etc). Two steps are needed to create ancillary navigation files, at which point a simple tool (in this sofware suite) can be used to attach the navigation UTC time, and attitude to the UHDAS files.
  4. HDSS data routines are experimental: newly written, and unsupported. HDSS files are fully navigated, but badly documented. If you must deal with HDSS files, check the “HDSS blog” (currents.soest.hawaii.edu) for more information, caveats, warnings, pitfalls, and experience. It will probably be painful.

Working with binary ADCP data files:

You must run “adcppath” and then “radcppath” to have access to the programs that read raw data from RDI instruments

  • DO NOT add any subdirectories under ‘rawadcp” other than those provided by this mechanism, or you will be running obsolete code.
  • NEVER name a variable ‘nb’, ‘bb’, or ‘os’, since this will override the class definition and you will be unable to use that read function until you clear the variable.
  • Hint: start by reading a subsample of the data so you know how much memory you will be using, and how long it will take to read the file.
  • Hint: see also ‘cutadcp’ and ‘catadcp’ for crude tools to subsample and concatenate data structures. These work best with data structures obtained using ‘read’ before using restruct. See below for details and use matlab command-line ‘help’ for these commands

Examples:

  1. UHDAS Ocean surveyor
  1. Read all variables, first 300 values

    • pingdata type = ‘first’
    • data are RAW (straight from the instrument)
-------------------------
>> osfile = 'lmg0501/raw/os38/lg2004_365_14400.raw';
>> osdata=read(os, osfile,'vars','all','ilist',[1:300])
osdata =
           vel: [80x4x300 double]              beam velocities
           cor: [80x4x300 double]              correlation
           amp: [80x4x300 double]              signal/noise
            pg: [80x4x300 double]              0 or 1 flag
       heading: [1x300 double]                  NA, present for consistency
         pitch: [1x300 double]                  NA, present for consistency
          roll: [1x300 double]                  NA, present for consistency
    soundspeed: [1x300 double]                 sound speed from instrument
   temperature: [1x300 double]                 temperature from instrument
   tr_pressure: [1x300 double]                  NA, present for consistency
            bt: [1x1 struct]                   bottom track data
           nav: [1x1 struct]                   NA, present for consistency
          dday: [1x300 double]                 instrument time
       ens_num: [1x300 double]                 profile number
     num_pings: [1x300 double]                 number of pings averaged (1)
        config: [1x1 struct]                   full of useful info
         depth: [80x1 double]                  nominal depth
          info: [1x1 struct]                   (added by 'read'; machinery)
         error: []                             (added by 'read'; machinery)
-------------------------
  1. Read all variables, every 40th point

    • FULLY NAVIGATED data (requires presence of gbin files)
-------------------------
>> osfile = 'lmg0501/raw/os38/lg2004_365_14400.raw';
>> osdata = get_xfraw(os,filelist(40).name,'step',40);
-------------------------
  1. get the data in earth coordinates

    • if you know the transducer alignment, you can get the data in earth coordinates. See this link for more detail
---------------------------
>> os_halign = 42.3; %set this to what you found in #2
>> data=get_xfraw(os, filelist(12).name,'h_align', os_halign);
>> [osdd, cfg] = restruct(os, data);
---------------------------
  1. Read all variables, every 40th point
    • FULLY NAVIGATED data (requires the three config files used during data acquisition and processing)
-------------------------
>>  cfg = get_uhdascfg('cfgbase','lmg0501', ...
                      'cfgpath', 'lmg0501/raw/config',...
                      'instname', 'os38',...
                      'pingtype', 'bb',...
                      'logfpath', '.');
>> osdata = getsome_adcp(cfg, 'ddrange', [364 364.1])
-------------------------
  1. VmDAS data
  1. read some data (this scheme works for all VmDAS data files)
-------------------------
>> filename = 'KMADCP093_000000.LTA';  % this is fully navigated, averaged data
>> osdata = read(os, filename, 'vars', 'all');
-------------------------
  1. concatenating files:
-------------------------
>> filename1 = 'KMADCP093_000000.LTA';
>> filename2 = 'KMADCP093_000001.LTA';
>> filename3 = 'KMADCP093_000002.LTA';
>> osdata1 = read(os, filename1, 'vars', 'all');
>> osdata2 = read(os, filename2, 'vars', 'all');
>> osdata = catadcp(osdata1, osdata2);
>> osdata = catadcp(osdata, read(os, filename3, 'vars', 'all');
>> dd = restruct(os, osdata); % make the variables 2-dimensional; add some
-------------------------
  1. HDSS data (see HDSS blog for caveats).
  • There are two HDSS instruments, and you have to specify which one (to get the right transformation matrices) and which beams to use for the transformation (you never know which ones might work for your dataset: defaults are :
beams [1 2 3 4] for HR140
beams [1 2 3 4]   for HR50
  • This example is from a time when only two beams of the 50kHz instrument were working
-------------------------
>> filename = '50K_ZHNG02RR[12_01_05.185500]';
>> hrdata = read(hr, filename','yearbase',2005,...
             'log_yearbase', 2003,...
             'lonsign', -1,...
             'beams2do', [3 4],...
             'instname', 'hr50);
-------------------------

transforming the data into earth coordinates

To get raw data with U, V, W, and error velocity you need to transform the velocities from beam coordinates to earth coordinates. For this, you must know the angle of the transducer relative to the hull. This usually comes as a rough guess of the angle, which is used during aquisition, and a calibration value, which is determined after processing. In this example, the heading offset used during aquisition was 45deg, and the “phase” from earlier codas calibrations was 0.55deg. The ‘geometry’ varies with the instrument, but is usually ‘convex’ for the OS and ‘concave’ for NB and BB instruments.

-------------------------
OH = 45.0 + .55;  %Mounting Angle +  phase
hdg = data.heading;
[data.enu data.xyze] = beam_enu(data.vel, OH + hdg, 0*hdg, 0*hdg', ...
               'geometry','convex');
-------------------------

Using raw data

Sometimes if is nice to have all four beams available without having to type ‘squeeze’. The function “restruct” returns variables starting as [NBINS x NBEAMS x NPROFS] in four separate variables, each [NBINS x NPROFS]. Restruct requires the instrument class as the first argument. (The optional output structure ‘config’ holds title information used for plotting)

-------------------------
[dd, config] = restruct(bb, data);  % "config" is used for plotting; see below

eg:        data.vel:           [50x4x50 double]    (from original read)
turns into
           dd.vel1:          [50x50 double]        (after restruct)
           dd.vel2:          [50x50 double]
           dd.vel3:          [50x50 double]
           dd.vel4:          [50x50 double]
-------------------------

“restruct” is most useful with fully navigated data.

From Example (1b) above:

------------------------------------------
dd=                               (same as original "data", plus some)
          dday: [1x1200 double]    original instrument decimal day
     ens_num: [1x1200 double]      profile number

        cor1: [80x1200 double]  \  correlation, beam 1 (for a NB, the
        cor2: [80x1200 double]   \ correlation, beam 2     field would be
        cor3: [80x1200 double]   / correlation, beam 3     "spectral width"
        cor4: [80x1200 double]  /  correlation, beam 4     not correlation

        amp1: [80x1200 double]  \  signal/noise, beam 1
        amp2: [80x1200 double]   \ signal/noise, beam 2
        amp3: [80x1200 double]   / signal/noise, beam 3
        amp4: [80x1200 double]  /  signal/noise, beam 4

         pg1: [80x1200 double]  \  flag: 0 or 1, beam 1
         pg2: [80x1200 double]   \ flag: 0 or 1, beam 2
         pg3: [80x1200 double]   / flag: 0 or 1, beam 3
         pg4: [80x1200 double]  /  flag: 0 or 1, beam 4

        vel1: [80x1200 double]   \ if ORIGINALLY beam coords, these are
        vel2: [80x1200 double]       measured velocities for the 4 beams
        vel3: [80x1200 double]     if ORIGINALLY in earth coords, (eg. LTA)
        vel4: [80x1200 double]   /   meas vel in [east, north , up, errvel]

       xyze1: [80x1200 double]  \  measured velocities in ship coords:
       xyze2: [80x1200 double]   \     starboard, forward, up, errvel
       xyze3: [80x1200 double]   /
       xyze4: [80x1200 double]  /

        enu1: [80x1200 double]  \  measured velocoties in earth coordinates
        enu2: [80x1200 double]   \    east,north,up,errvel
        enu3: [80x1200 double]   /
        enu4: [80x1200 double]  /

       uship: [1x1200 double]      ship's speed from navigation
       vship: [1x1200 double]             uses 'll2mps'

       umeas: [80x1200 double]  \  same as enu1:   m/s
       vmeas: [80x1200 double]   \ same as enu2:   m/s
       wmeas: [80x1200 double]   / same as enu3:   m/s
       emeas: [80x1200 double]  /  same as enu4:   m/s
       fmeas: [80x1200 double]     measured velocity aligned with ship heading
       pmeas: [80x1200 double]     measured velocity aligned with port dir


   corr_dday: [1x1200 double]  --> UTC time
     heading: [1x1200 double]      heading from some source (depends on   )
       pitch: [1x1200 double]      pitch from some source   (data type and)
        roll: [1x1200 double]      roll from some source    (acquisition  )
 temperature: [1x1200 double]      temperature from thermistor
  soundspeed: [1x1200 double]      soundspeed used (instrument-dependent)

        nav: [1x1 struct]         navigation structure, UTC time, lon, lat
        mps: [1x1200 double]      ship speed (from ll2mps)
        cog: [1x1200 double]      course over ground
        lon: [1x1200 double]      lon from source
        lat: [1x1200 double]      lat from source
       uabs: [80x1200 double]     ocean u: umeas + uship (no corrections!!)
       vabs: [80x1200 double]     ocean v: vmeas + vship (no corrections!!)
       fabs: [80x1200 double]     ocean vel in direction of ship's heading
         bt: [1x1 struct]         bottom track data
       bins: [80x1 double]        bin count (positive away from instrument)
      depth: [80x1 double]        nominal depth (meters) away from instr
     config: [1x1 struct]         FULL OF USEFUL INFO, eg: bin, pulse...
------------------------------------------

Plotting raw data

After you load the data and restructure it, you can plot any 2-dimensional parameter with “aplotit”, for example:

----------------------

aplotit(data,config,'fn','amp1');

or

aplotit(data,config,'fn','amp1','cur_ax', subplot(211));
aplotit(data,config,'fn','cor1','cur_ax', subplot(212));
----------------------