Fwd: attributes on sequence fields

[James Gallagher]

Joe, here's a thread from an earlier conversation about the DAS,  
hierarchy and the DAP2 spec. The summary is that the DAP2 spec says  
that the hierarchy of attribute containers should match the hierarchy  
of variables, but (as you point out) practice differs. DAP4 addresses  
this by binding attributes with variables, so the association is  
never ambiguous.

I was just suggesting that a footnote or brief explanation saying  
'This differs from ... and here's why: ...'

James

Begin forwarded message:

> From: Joe Sirott <Joe.Sirott at noaa.gov>
> Date: January 24, 2006 3:35:33 PM MST
> To: Tech DODS <dods-tech at unidata.ucar.edu>
> Subject: Re: attributes on sequence fields
>
> At this point, I'm leaning towards not changing the DAS in Dapper:
>
> - The current DAS works with pyDap, ncBrowse, Java Ocean Atlas,  
> GrADS, and DChart. I'll have to test all five of these clients if I  
> change the DAS.
> - The current DAS doesn't work with John Caron's software, which is  
> based on Java-DODS. But this isn't due to any incompatibility with  
> Java-DODS because Java-DODS doesn't bind DAS attributes to DDS  
> variables. It's up to the software that incorporates Java-DODS to  
> make the connection. I'm not sure what's going on here.
> - While I would ordinarily agree with the "Be strict in what you  
> write" rule, what should one do when part of a specification is  
> (almost) universally ignored and the "common" implementation works  
> as well as the specified  implementation? I'd argue that in this  
> case the spec has to acknowledge the "reality on the ground."
> - Although I might be misreading the spec, it  appears that the  
> spec is also ignored for the Grid type. Shouldn't a Grid like this:
> Dataset { Grid { ARRAY: Float32 cprate[time = 636][lat = 94][lon =  
> 192]; MAPS: Float64 time[time = 636]; ... } cprate;  have a DAS  
> like this:
> Attributes { cprate { time { } rather than the commonly implemented:
> Attributes { time { } - Although DAP 4 will replace DAP 2 at some  
> point, the nature of protocol changes is such that DAP 2 will be  
> around for many years. So I think that it's still important to  
> resolve this problem (of binding attributes to variables) in DAP 2.
>
> James Gallagher wrote:
>>
>> On Jan 22, 2006, at 5:51 PM, Peter Cornillon wrote:
>>
>>> This is an important discussion. Hopefully James (as well as  
>>> others of dods-tech) will chime in after the weekend (for you in  
>>> Europe and the Americas).
>>>
>>> First, whatever we decide to do with the relationship between the  
>>> dds and the das, I like John Caron's rule: Be lenient in what you  
>>> read, be strict in what you write. Which leads to: Clients should  
>>> deal with common variants, but servers should be changed to  
>>> follow the spec. I believe that that is the intent in the  
>>> migration from DAP2 to DAP4.
>>
>> I agree, esp. with the part about the servers.
>>
>>>
>>> More in-line below.
>>>
>>> On Jan 21, 2006, at 10:14 PM, Joe Sirott wrote:
>>>
>>>> Nathan,
>>>>
>>>> This leads me to a some questions:
>>>>
>>>> - Given that there are apparently no servers serving Sequence  
>>>> data that follow the spec and that DAP 2 spec will soon (?) be  
>>>> replaced by DAP 4, does it make sense to now change the way the  
>>>> DAS is constructed for sequences?  The implication is that  
>>>> parsers will have to accept both methods since old servers will  
>>>> not be updated.
>>>> - Have nested attributes for sequences been tested in any of the  
>>>> publicly available DAP parsers (C++ or Java) ?
>>>> - I don't understand the need to duplicate the DDS hierarchy in  
>>>> the DAS. It seems simpler to have a reference to the variable  
>>>> (either via a fully qualified name or some XPath like construct).
>>>>
>>>> And a (partially related) comment:
>>>>
>>>> If the DAP specification is truly going to be a community  
>>>> standard, it should be opened up for public comment and proposed  
>>>> changes should be announced on this list.
>>>
>>> I agree completely here. My recollection is that there has been  
>>> announcements of and requests for comments re both DAP2 and DAP4,  
>>> but that does not mean that we should not consider more input,  
>>> especially since this discussion is probably more relevant to  
>>> sequences than to arrays/grids and serious clients addressing the  
>>> former are now beginning to see significant use. (This was not  
>>> the case when a lot of the DAP4 discussion took place.) My sense  
>>> is that DAP2 is pretty much frozen in place, so the comments  
>>> should focus on DAP4.
>>
>> True. I had thought that the DAP2 process was pretty open, but I  
>> guess unless you were plugged into the NASA realm, maybe it was  
>> not. Hmmm. I'll have to ensure we don't make that mistake with DAP4.
>>>
>>> James and Nathan, I think that it would be useful to provide the  
>>> rationale for linking the das with the dds and second to reopen  
>>> the discussion, or better, to integrate this discussion in with  
>>> previous comments.
>>>
>>> For those who are interested the developer's discussion for DAP4  
>>> can be found by clicking on the DAP3/4 link in: http:// 
>>> scm.opendap.org:8090/trac/
>>>
>>> As an aside, I note that this web site "traks" a number of on- 
>>> going discussions related to OPeNDAP, such as Server4 issues.
>>>
>>> I want to emphasize the point the Joe brings up here. OPeNDAP is  
>>> an open source project and we welcome all input relative to  
>>> OPeNDAP related issues, be they issues related to specific  
>>> OPeNDAP elements or issues related to how we can better integrate  
>>> the community into these discussions.
>>
>> Briefly, I have 'linked' the DAS and DDS to form one response/ 
>> document because that seemed best based on two types of comments  
>> which I have received from many users/developers. First, it was  
>> hard for many to use attribute information because servers  
>> returned DAS objects that were 'all over the place' when compared  
>> to the corresponding DDS objects. This was OK with most of the  
>> datasets, and that's part of the reason it went on for so long.  
>> But once more and more datasets with structures came into the  
>> picture, it stopped being so benign. It turns out that it's fairly  
>> hard to match attribute containers to variables not because the  
>> design is incorrect, but because the design is easy to implement  
>> incorrectly. So the first reason I made (or _suggested we make_,  
>> because it's certainly not cast in stone) the change was to have a  
>> design that would be easier to implement correctly. That is, I  
>> wanted an unambiguous design, even to a casual reader of the  
>> specification.
>>
>> The second reason is that there was a significant group of  
>> developers who knew they were going to want both the DAS and DDS  
>> and saw no reason not to get all that information in one request.  
>> Purely an optimization, but one I thought was important. There are  
>> many ways to do this; the DDX design was one.
>>
>> As I have worked more with the DDX object/response, I've come to  
>> like it more and more. First, it's really nice to have the  
>> attributes right there with the variable. Even if the DAS and DDS  
>> are 100% to spec, it's still more work to iterate through both  
>> synchronously switching back and forth between the 'variable  
>> space' and the 'attribute space.' One suggestion I got (not  
>> implemented, but worth thinking about) was to do away with  
>> attributes entirely and just move all that info into a structure.  
>> But I digress.
>>
>> Another nice thing I've noticed is that the XML notation, combined  
>> with the single document representation, fits nicely with work on  
>> RDF/OWL that other groups are doing. Several of Benno's  
>> suggestions about using namespaces for really fit nicely.
>>
>> James
>>>
>>> Peter
>>>
>>>> I don't recall seeing any announcements about the DAP 2 spec  
>>>> here and only one, maybe two, request for comments about the DAP  
>>>> 4 spec. I would like Dapper to be compliant with the DAP spec,  
>>>> but I can only maintain compliance if I'm aware of changes to  
>>>> the spec.
>>>>
>>>> - Joe
>>>>
>>>>
>>>> Nathan Potter wrote:
>>>>
>>>>> There are a bunch of important issues here, and I need to chime  
>>>>> in.
>>>>>
>>>>> Background:
>>>>>
>>>>> James and I have been working (intermittently) on the DAP 4  
>>>>> spec, and  prior to that the formalization of the DAP 2 spec.  
>>>>> The idea that a  correctly formatted DAS has "nested"  
>>>>> attributes that conform to the  hierarchy of the DDS was  
>>>>> documented from the beginning of the DODS  project.
>>>>> Unfortunately the lax rules regarding the semantic metadata  
>>>>> (i.e. the  DAS) allowed people to do what ever creative thing  
>>>>> they wished. This  combined with code that attempted  
>>>>> (mistakenly IMHO) to associate top  level DAS attributes with  
>>>>> variables deep in the hierarchy created the  current state of  
>>>>> affairs.
>>>>>
>>>>> The direction that we are moving is towards strengthening the   
>>>>> structural relationships between the DAS and the DDS, not   
>>>>> perpetuating the weak ones.
>>>>>
>>>>> I am basically imploring you NOT to flatten your DAS. Structure  
>>>>> is  important, and really is a requirement.
>>>>>
>>>>>
>>>
>>> --
>>>  Peter Cornillon
>>>    Graduate School of Oceanography     -  Telephone: (401) 874-6283
>>>       University of Rhode Island                 -                
>>> Fax: (401) 874-6728
>>>         Narragansett, RI 02882                    -           E- 
>>> mail: pcornillon at gso.uri.edu
>>>
>>
>> --
>> James Gallagher                jgallagher at opendap.org
>> OPeNDAP, Inc                   406.723.8663
>>
>

--
James Gallagher                jgallagher at opendap.org
OPeNDAP, Inc                   406.723.8663