|You are here: NODC Home > Satellite Oceanography Group > Satellite Data Formats|
Satellite Data Formats and Compression
The Satellite Oceanography Group strives to provide its data in formats useful
to the widest array of users. For this reason, we attempt to provide data
in standard formats whenever possible. Below are brief descriptions of
some of the standard formats we use.
The GeoTIFF* format was chosen as one of our standards, primarily for its ability to bring image-oriented data into GIS systems. GeoTIFF is merely a variant of standard TIFF, with some additional tags to include georeferencing information. This information can be utilized by some programs, notably GIS software, to properly place the data with respect to other georeferenced objects or "layers". Other applications will simply ignore the GeoTIFF tags and display the images as if they were regular TIFF. Additionally, unlike some image formats, the original data remain intact within the TIFF image and can be loaded and analyzed in various data analysis environments like Matlab and IDL.
HDF4-Scientific Data Set
In addition to GeoTIFF, the Satellite Oceanography Group employs the Hierarchical Data Format Version 4 (HDF4) Scientific Data Set (SDS) model. This HDF4-SDS format is extremely useful for large datasets, as it permits internal metadata tags, inclusion of multiple data layers, and perhaps most importantly an internal compression scheme known as tiling, which breaks the large data set into individually compressed pieces. These pieces can be individually decompressed, resulting in far greater access speeds when selecting subsetted regions from the global fields. Many software tools exist for working with HDF data. Please see a very nice list at the HDF Group web site: http://www.hdfgroup.org/tools.html*. Currently, our AVHRR Pathfinder Version SST data are provided in HDF4-SDS. Also provided in HDF4-SDS is CoRTAD Version 1.
The newer HDF-5 format is used for CoRTAD Versions 2 and 3. HDF-5 is an i extremely powerful and flexible file format, but can be challenging to use. However, it is needed in situations where data files exceed 2 GB in size. See http://www.hdfgroup.org/HDF5/ for more information on HDF-5.
netCDF-3 & netCDF-4
Similar to HDF, the network Common Data Form (netCDF) file format allows for multiple data layers with internal metadata tags. Maintenance of the format and a full description of netCDF are provided by Unidata*. All of our GHRSST data are currently provided in netCDF-3 or netCDF-4. CoRTAD Version 4 is in netCDF-4.
Gzip and Bzip2 Compression
Because of the large size of many high resolution and global datasets, they are often compressed using gzip or bzip2. Compressing with gzip adds the ".gz" extension to each of the file names while bzip2 adds ".bz2". These are free programs to efficiently compress files.
Gzip is developed by GNU*. Executables to compress and decompress "gzipped" files are available for a wide variety of operating systems, including UNIX and Linux platforms, Windows machines, and Macs. Please see the gzip home page* for more information, including access to the source code, executables, user manuals, etc. Other programs are also capable of compressing and decompressing gzipped files. PowerArchiver 6.1* (freeware), PKZIP*, and WinZip* are three such programs. Decompressing a gzipped file is very straightforward. On UNIX/Linux systems, simply "gunzip filename.gz". On other platforms, once you have installed the software, simply double-clicking the filename should launch the proper application.The bzip2 compression is very similar to gzip and more information is available at the Bzip2 web site*. Bzip2 takes about twice as long to compress and decompress files when compared to gzip, but is able to compress the data about twice as well as gzip.
AIP SIP DIP GOSTA NPOESS VIIRS OPeNDAP DODS LAS HRPT LAC GAC HDF-SDS DMAC PO.DAAC LTSRF CoRTAD
|Last modified: Wed, 2-May-2012 18:58 UTC||NODC.Webmaster@noaa.gov|
Freedom of Information Act (FOIA)
USA.gov - The U.S. Government's Web Portal