Dapper in-situ conventions spec available

[Joe Sirott]

Hi Jennifer,

I believe that there are several reasons for always using 2-level sequences:

- A client only has to parse one kind of OPeNDAP data structure
- The (x,y,z,t) position and units of the observation(s) are always 
known. I suppose one could imply that a missing 'z' value is 0, but I 
believe that it is better to specify this explicitly.
- It's flexible. For example, suppose you want to aggregate a set of 
time series ocean moorings with sensors at different depths (like the 
TAO data array). How you would represent this with a 1-level sequence?
- Data access is faster because there is less data redundancy (lat and 
lon values only have to be sent once per time series).
- It's been tested and scales -- our DChart web application uses OPeNDAP 
to access all of our datasets, including the World Ocean Database (4 
million profiles) and the GTSPP (2.5 million profiles), NDBC buoys, the 
NCDC Global Summary of the Day, etc.

I don't think a netCDF standard is going to solve this problem (at least 
netcdf-3). It's not difficult to come up with a standard for 
representing a single profile or time series as a netCDF file, but I 
don't think that anyone has come up with an efficient way  for 
representing multiple profiles or time series in a single netCDF file -- 
especially if you want to query the data (e.g. return all the profiles 
for 2004 from the Bering Sea).

Jennifer M. Adams wrote:
> Dear All,
> As a GrADS developer, I have multiple issues/concerns/ideas regarding 
> serving and accessing in situ data via OPeNDAP.  GrADS is a natural 
> client for this data type, as it already has an internal data model 
> for in situ data, with many built-in functions for the display and 
> analysis of the data. For two years, GDS has been serving in situ data 
> -- surface obs in real time -- with 4000-9000 hits per month over the 
> past year. Not overwhelming, but the usage has been consistent. 
>
> Now that Dapper, the THREDDS Data Server, and OPeNDAP Server 4 are 
> also out there, I think it would be wonderful if there were  some 
> consistency in the way these data are presented to the world. I think 
> a spec or set of conventions is a great idea -- it certainly seems 
> like it would make my life easier when trying to enable GrADS to be a 
> client for all the possible servers that users will want to access. 
>
> To get GrADS to read Dapper, I had to make a few changes that reflect 
> the differences between GDS and Dapper conventions: 
> 1. The variable "Int32 _id"  in Dapper is the analog to the  "String 
> stid" variable in GDS 
> 2. Time axis information is presented a little differently
> 3. Constraint protocol and syntax (the reason why GrADS 1.9 can't 
> access the new Dapper)
>
> The more significant difference between GDS and Dapper is that Dapper 
> presents a time series from a single station as a 2D nested sequence, 
> in the same way you would present a vertical profile at a single time. 
> Why complicate things by have two meanings for the inner sequence? 
> Swapping Z and T as variables for the inner and outer sequences seems 
> like it's defeating the purpose of having a convention. I prefer the 
> simpler notion that it's a 2D sequence if the data at the same 
> location vary in height/depth, if not, it's a 1D sequence. 
>
> I suspect the real problem is that there is no NetCDF standard for 
> this data type. As soon as one is developed, and an API is available 
> for subsetting on variables as well as coordinates, then I reckon 
> everyone will jump to that standard and this whole discussion will 
> become an artifact of history. Anything on this brewing at UCAR? 
>
> Jennifer 
>
> --
> Jennifer M. Adams
> IGES/COLA
> 4041 Powder Mill Road, Suite 302
> Beltsville, MD 20705
> jma at cola.iges.org <mailto:jma at cola.iges.org>
>
>
>