Sequences, Lists, and Relational Constraint Expressions

[James Gallagher]

On May 8, 2006, at 6:27 AM, Roy Mendelssohn wrote:

> I think to be useful you must be able to constrain more than lat- 
> lon-depth-time.  You should be able to constrain things like  
> "station", "line", and some other properties of the variables.  I  
> understand that allowing constraints on everything can be  
> difficult. If there were a series of well-formed conventions, than  
> some limit could be agreed to based on the structure of the  
> convention.
>
> Certainly formalizing the DAPPER convention, and also  firming up  
> and comparing the conventions John Caron has recommended woul dbe a  
> good start.

I agree. Originally the DAP had a data type called 'Function' which  
was not a function like f(x) but a Sequence where the columns were  
divided into two groups: Independent and dependent. It was never used  
and I dropped it (just like I dropped List). There's no silver bullet  
here, but I thought the history might be useful in some way. Maybe  
now we're a point where it makes sense to divide things in a sequence  
along these lines?

James
>
> -Roy
>
>
> At 2:45 PM -0600 5/5/06, John Caron wrote:
>> Steve Hankin wrote:
>>> Hi John,
>>>
>>> This dilemma is the result of a mismatch between the "discipline  
>>> neutral" generality of the OPeNDAP DAP and the relatively  
>>> focussed data models that Unidata is working with.  The obvious  
>>> way to address this is through conventions -- just as the CF  
>>> conventions are already needed in order to map the generality of  
>>> netCDF onto the Common Data Model.
>>>
>>> It seems to me that the DAPPER conventions, if fleshed out, could  
>>> address the problem that you pose.  Fully fleshed out DAPPER  
>>> conventions would clarify the process for identifying the  
>>> independent (coordinate) variables, georeferencing, the  
>>> requirement (or not) for a 2-level Sequence, the scope of  
>>> structures that would be permitted inside the sequences, the  
>>> adoption (or not) of CF standard names, etc.   Through the  
>>> conventions one could also impose restrictions (or convey  
>>> information to clients) on the types of relational operations  
>>> that are allowed and/or the variables to which they can be applied.
>>
>> Hmmm, I dont think you could use conventions to alter the  
>> semantics of the data access protocol. That is, operations will  
>> still be allowed, but they may fail because they exceed resource  
>> limits.
>>
>> The problem is we want to model our data with variable length  
>> arrays, and right now we only have Sequences to do that, which  
>> require supporting RCE on all fields. We could move to supporting  
>> a List data type, but that has no RCE on any fields. I want to  
>> support RCE on some fields of a variable length array, because I  
>> can only do that efficiently on some of the fields. (This is true  
>> even with an RDB, namely the fields that you have indexed).
>>
>> More abstractly, this is an example where I (ideally) want to  
>> express what opendap calls are practical. The client should be  
>> able to exmine the DDS and decide how to best get the info it  
>> wants. Because of the generality of opendap (i assume following  
>> relational theory) i cant do that.
>>
>> So you are suggesting that instead of indicating "what is  
>> practical" in the DDS, we should do it at some layer above that,  
>> eg the "Dapper convention layer". Thats probably what we have to  
>> do if we cant do it in the DDS. I wanted to see if anyone else had  
>> other ideas before we follow that path, though.
>>
>>>
>>>    - steve
>>>
>>> =================================================
>>>
>>> John Caron wrote:
>>>
>>>> Hi James, et al:
>>>>
>>>> One problem I have in implementing opendap sequences is that you  
>>>> have to support relational constraint expressions (RCE) on all  
>>>> of the fields of a sequence. This only scales if you use a DBMS.  
>>>> I would prefer to support RCE on some of the fields, eg the  
>>>> space and time coordinates and station ids.
>>>> Im wondering if anyone has this problem, and if there is  
>>>> anything to do about it?
>
>
> -- 
> **********************
> "The contents of this message do not reflect any position of the  
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> 1352 Lighthouse Avenue
> Pacific Grove, CA 93950-2097
>
> e-mail: Roy.Mendelssohn at noaa.gov (Note new e-mail address)
> voice: (831)-648-9029
> fax: (831)-648-8440
> www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
>

--
James Gallagher                jgallagher at opendap.org
OPeNDAP, Inc                   406.723.8663