Sequences, Lists, and Relational Constraint Expressions

[Steve Hankin]

Joe Sirott wrote:

> Hi John,
>
> I don't know if there are /any /OpenDap server implementations that 
> fully implement the selection constraints as described in the OpenDap 
> spec. For instance, if I try and constrain a string variable with a 
> netCDF server I get a parse error (although I could be doing something 
> wrong). In practice, it's difficult to see how a server /could/ 
> practically implement all of the mandated constraints -- imagine 
> having to search a database of several million profiles with several 
> constraints on variable values. Even though such a server would be 
> obeying the "letter of the law," a timeout after five minutes is 
> probably worse behavior for a client then an immediate error message.
>
> The practical alternative is to use some convention. Dapper already 
> uses an ad-hoc convention that might be useful to you (as Steve 
> mentioned) since it does identify the coordinate variables and allows 
> constraints on those variables. 
>
>
> John Caron wrote:
>
>>
>>
>> Steve Hankin wrote:
>>
>>> Hi John,
>>>
>>> This dilemma is the result of a mismatch between the "discipline 
>>> neutral" generality of the OPeNDAP DAP and the relatively focussed 
>>> data models that Unidata is working with.  The obvious way to 
>>> address this is through conventions -- just as the CF conventions 
>>> are already needed in order to map the generality of netCDF onto the 
>>> Common Data Model.
>>>
>>> It seems to me that the DAPPER conventions, if fleshed out, could 
>>> address the problem that you pose.  Fully fleshed out DAPPER 
>>> conventions would clarify the process for identifying the 
>>> independent (coordinate) variables, georeferencing, the requirement 
>>> (or not) for a 2-level Sequence, the scope of structures that would 
>>> be permitted inside the sequences, the adoption (or not) of CF 
>>> standard names, etc.   Through the conventions one could also impose 
>>> restrictions (or convey information to clients) on the types of 
>>> relational operations that are allowed and/or the variables to which 
>>> they can be applied.
>>
>>
>> Hmmm, I dont think you could use conventions to alter the semantics 
>> of the data access protocol. That is, operations will still be 
>> allowed, but they may fail because they exceed resource limits.
>
Hi Jon,

Joe pointed out that there is already a pretty substantial mis-match 
between what real OPeNDAP servers can do and what the  URL syntax 
permits.  You're already in good company.

I'd add to this that when the server advertises the conventions it is 
using (CF or DAPPER or whatever) it is indicating a "contract" that it 
is willing to enter into with clients. If the clients do not honor that 
contract (if they ask for things that the conventions forbid them from 
requesting) then isn't it perfectly valid for the server to refuse?  
That outlook is why I would see the DAPPER conventions as a potential 
solution to your situation.  (Doesn't your Java netCDF code behave 
similarly if (say) CF conventions are violated and the user makes a 
georeferenced API request?)

>> The problem is we want to model our data with variable length arrays, 
>> and right now we only have Sequences to do that, which require 
>> supporting RCE on all fields.
>
>> We could move to supporting a List data type, but that has no RCE on 
>> any fields. I want to support RCE on some fields of a variable length 
>> array, because I can only do that efficiently on some of the fields. 
>> (This is true even with an RDB, namely the fields that you have 
>> indexed).
>>
>> More abstractly, this is an example where I (ideally) want to express 
>> what opendap calls are practical. The client should be able to exmine 
>> the DDS and decide how to best get the info it wants. Because of the 
>> generality of opendap (i assume following relational theory) i cant 
>> do that.
>>
>> So you are suggesting that instead of indicating "what is practical" 
>> in the DDS, we should do it at some layer above that, eg the "Dapper 
>> convention layer". Thats probably what we have to do if we cant do it 
>> in the DDS. I wanted to see if anyone else had other ideas before we 
>> follow that path, though.
>>
Yes (assuming I am understanding you, that is)
 
The lofty goal of OPeNDAP has always been interoperability by hiding the 
differences between on-disk data management strategies.  And optimizing 
interoperability through this approach almost always involves making 
compromizes in functionality.  In the case of in-situ data that means we 
want (e.g.) collections of netCDF time series or profiles, relational 
databases, and your proposed netCDF conventions for observations all to 
look the same through OPeNDAP.

Instead of asking "what is practical" in the DDS, this approach asks 
"what is the common denominator" that all of the on-disk data management 
strategies can  work with.  If the common denominator is good enough to 
meet the requirements of the user community, and if you document it well 
in the form of conventions that get widely used, then you have produced 
a potentially very useful standard.  In my opinion that's the best and 
most meaningful usage of the over-used word "standard".  And 
DAPPER/OPeNDAP is only a few, small steps away from being able to 
achieve that.

Enjoy your weekend!

    - Steve

>>
>>>
>>>    - steve
>>>
>>> =================================================
>>>
>>> John Caron wrote:
>>>
>>>> Hi James, et al:
>>>>
>>>> One problem I have in implementing opendap sequences is that you 
>>>> have to support relational constraint expressions (RCE) on all of 
>>>> the fields of a sequence. This only scales if you use a DBMS. I 
>>>> would prefer to support RCE on some of the fields, eg the space and 
>>>> time coordinates and station ids.
>>>> Im wondering if anyone has this problem, and if there is anything 
>>>> to do about it?
>>>>
>>>
>

-- 
º¤ø,¸¸,ø¤º°`°º¤ø,¸¸,ø¤º°`°º¤ø,¸¸,ø¤º°`°º¤º¤ø,¸¸,ø¤º

Steve Hankin, NOAA/PMEL -- Steven.C.Hankin at noaa.gov 
7600 Sand Point Way NE, Seattle, WA 98115-6349
ph. (206) 526-6080, FAX (206) 526-6744