---------- Forwarded message ----------
From: Signell, Richard <rsignell@usgs.gov>
Date: Tue, Aug 23, 2016 at 3:47 PM
Subject: Re: USGS time series data collection Archive Appraisal Committee Questions
To: Mathew Biddle - NOAA Affiliate <mathew.biddle@noaa.gov>
Cc: "Montgomery, Ellyn" <emontgomery@usgs.gov>, Kyle Wilcox <kyle@axiomdatascience.com>


Matt,
With this handy script from Kyle Wilcox:

#!/bin/bash

export CMG_OBS=/usgs/data2/emontgomery/stellwagen/CF-1.6
export THREDDS_URL=http://geoport.whoi.edu/thredds/fileServer

sha256sum $CMG_OBS/**/*.nc | \
grep -E '([0-9]+\..*|.*(-(a|A)|(ls|[0-9]s|isus|[aAB]s|sgs|aqds|vs|pcs|d|tide)-ca
l){1}(?!lp)(?!1(h|H))\.*.*)' | \
sed "s#$CMG_OBS#$THREDDS_URL$CMG_OBS#" \
> /var/www/waf/data/ts/manifest.txt

we were able to create the manifest.txt you can see in this WAF:

http://geoport.whoi.edu/waf/data/ts/manifest.txt

@Ellyn, when we get more data, I guess you can run this again and diff it to provide a new list, or add some logic to only search data added since a specified date.   This script is living at
emontgomery@gam:/usgs/data2/emontgomery/stellwagen/CF-1.6/do_manifest