BOUNCE dods-tech at unidata.ucar.edu: Non-member submission from [Daniel Holloway <d.holloway at webhost.opendap.org>]
owner-dods-tech at unidata.ucar.edu
owner-dods-tech at unidata.ucar.edu
Thu May 4 09:34:14 PDT 2006
_From support at unidata.ucar.edu Thu May 4 10:34:07 2006
Return-Path: <support at unidata.ucar.edu>
Received: from coral.gso.uri.edu (coral.gso.uri.edu [131.128.101.71])
by unidata.ucar.edu (UCAR/Unidata) with ESMTP id k44GY6V9015066
for <dods-tech at unidata.ucar.edu>; Thu, 4 May 2006 10:34:06 -0600 (MDT)
Organization: UCAR/Unidata
Keywords: 200605041634.k44GY6V9015066
Received: from [192.168.10.37] (host134.gso.uri.edu [131.128.110.134])
by coral.gso.uri.edu (8.12.11.20060308/8.12.11) with ESMTP id k44GXuZQ015482;
Thu, 4 May 2006 12:33:56 -0400
In-Reply-To: <445A0B03.10506 at ifremer.fr>
References: <445A0B03.10506 at ifremer.fr>
Mime-Version: 1.0 (Apple Message framework v749.3)
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <CDEC8E4E-F67E-4801-8E64-0B3B39D2539A at opendap.org>
Cc: thomas LOUBRIEU <Thomas.Loubrieu at ifremer.fr>, dods-tech at unidata.ucar.edu
Content-Transfer-Encoding: 7bit
From: Daniel Holloway <d.holloway at webhost.opendap.org>
Subject: Re: Structures in Dapper dds & Loaddap
Date: Thu, 4 May 2006 12:34:52 -0400
To: Arnaud FOREST <Arnaud.Forest at ifremer.fr>
X-Mailer: Apple Mail (2.749.3)
X-Virus-Scanned: by amavisd-new at gso.uri.edu
X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on
laraine.unidata.ucar.edu
X-Spam-Level:
X-Spam-Status: No, score=-2.3 required=5.0 tests=AWL,BAYES_00,SPF_SOFTFAIL
autolearn=no version=3.0.1
Hi Arnaud,
There are several issues here, the primary culprit is handling
complex sequences not structures. I'll provide some input inline below.
On May 4, 2006, at 10:09 AM, Arnaud FOREST wrote:
> Hi all,
>
> I work on an opendap server for in-situ vertical profiles stored in
> a Oracle DBMS.
> Our dds structure is copied from the 'argo' dapper interface
> (http://dapper.pmel.noaa.gov/dapper/argo/argo_all.cdp.dds) but I
> notice that matlab struct tools(Libdap : 3.6.2, loaddap : 3.5.2,
> Matlab : 7 ) doesn't completly manage the in-situ profile dapper
> interface.
>
> The 'structures' are especially not very well managed by the matlab
> client :
The problem is with handling nested sequences for this
particular data source. The following data source is a complex
structure that loads fine with loaddap(v-3.5.2)
http://test.opendap.org:8080/dods/dts/complex_structs.03.dds
---------
>> loaddap('http://test.opendap.org:8080/dods/dts/complex_structs.03')
>> whos
Name Size Bytes Class
Outermost 1x1 1472 struct array
Grand total is 68 elements using 1472 bytes
>> Outermost
Outermost
SimpleStructure: [1x1 struct]
>> Outermost.SimpleStructure
ans
Innermost: [1x1 struct]
>> Outermost.SimpleStructure.Innermost
ans
i32: [10x1 double]
ui32: [10x1 double]
i16: [10x1 double]
ui16: [10x1 double]
f32: [10x1 double]
f64: [10x1 double]
>> Outermost.SimpleStructure.Innermost.i32
ans
0
2048
4096
6144
8192
10240
12288
14336
16384
18432
>>
---------
Place those structs inside a Sequence and baboom.... So yes,
there is a bug in loaddap with respect to these nested sequences. I
didn't realize it was there as I used loaddap recently to load
another Dapper data source and it worked fine.
---------
>> clear
>> loaddap('http://las.pfeg.noaa.gov/dods/ndbcMet/
all_noaa_time_series.cdp?
LAT,LON,WSPD1&LAT>39.58&LAT<43&TIME<=1105315200000&TIME>=1104537600000')
>> whos
Name Size Bytes Class
location 1x1 11348 struct array
Grand total is 1328 elements using 11348 bytes
>> location
location
profile: [6x1 struct]
LON: [6x1 double]
LAT: [6x1 double]
>> location.LAT(2)
ans
42.7500
>> location.LON(2)
ans
235.1500
>> location.profile(2).WSPD1(1:5)
ans
5.1000
5.8000
5.9000
7.2000
4.6000
>>
---------------
So, I was not aware that Dapper output would cause loaddap to
fail, so I'll look into what the problem might be that this
particular data source is causing the client.
Nested Sequences can be a bear to deal with...
>
> 1) "Loaddap" doesn't support the structure of structure, a
> fatal error occures. (the matlab's crash dump is at the end
> of the
> message)
>
>
>
> Is there a known bug or updates foreseen about the structure
> management in the loaddap matalb toolbox ?
There is now, at least for this particular data source.
>
> Why the dapper interface has been defined with so many structures
> in it?
I shouldn't try to explain the rationale behind the Dapper use
of Sequences but IMO it's a pretty good representation of the
underlying relationships.
Basically Dapper provides the following form for all of its
data sources:
Dataset {
Sequence { ... } location;
Structure { independent dims } constrained_ranges;
} data source;
Each data source has a series of location data, and a single
structure listing the extent of the independent dimensions (x,y,z,t)
for the data source itself (as constrained by the request)
------------
To represent the 'location' series Dapper uses a nested
sequence to represent the relationships between the parts of the
series, such that values which do not change for a particular series
which in this case are the lat/lon/juld variables are stored in the
outer sequence element. The variables that change most frequently,
whether that is a time-series, or vertical profile, etc., are stored
in the inner sequence, in this case the variable 'profile'.
-----------
Dataset {
Sequence {
Float64 JULD;
Float32 LONGITUDE;
Float32 LATITUDE;
Int32 _id;
Sequence {
Float32 PSAL_QC;
Float32 CNDC_ADJUSTED_QC;
Float32 TEMP_ADJUSTED_ERROR;
...
} profile;
...
} location;
}
------------
Logically, you have a set of locations, which have a JULD, LAT,
LON, and foreach of these you have a series of observations (profile).
The potentially confusing part is that for this data source
Dapper includes two additional structure variables, but it's
important to note that these structure variables reside in the
outermost sequence element of 'location'. That means that the
values in the 'attribute' structure and the 'variable_attributes'
structure apply to the outermost relationship. Long story short the
values in 'attributes' like 'PLATFORM_NUMBER' don't change as a
function of the 'profile' series. The nested Structure
'variable_attributes' list the extent or range of the independent
dimension implicit within the nested sequence 'profile', which in
this example is PRES (pressure).
Extending the above, you have a set of locations, which have a
JULD, LAT, LON and attributes like PLATFORM_NUMBER, etc., and the
range on the independent dim (PRES) for the series 'profile' is
contained in the structure 'variable_attributes.PRES.valid_ranges
[0:1], and all the observations recorded at that location are stored
in the profile series which happens to use PRES as the independent
dimension between observations.
------------
OK, I've probably butchered that explanation... There are a couple
of issues at work here that you should be aware of:
1: Not sure why they use a nested structure for
'variable_attributes', maybe they envision supporting more than 2
levels of nesting to represent a complicated relationship but I think
you don't have to have nested structures for this particular variable.
2: If you write a server that uses nested sequences the server
should serialize any constructor variables (.e.g., structures) before
the inner nested sequence itself. I believe this is documented in a
recent RFC on the DAP but I'll have to double check. Logically from
a client's perspective there's not difference between the ordering
but from the existing implementations standpoint there is a big
difference. I'm not sure if that's the reason why the client is
failing for this particular data source or not but will look into it.
>
> Does anyone know what opendap clients are fully compliant with the
> dapper output (structures of structures, sequences of sequences,
> sequences of structures...) and what is planned for the improvement
> of the compliance between dapper interface and opendap clients
> (matlab, ferret, nco, Opendap Data connector, pyDAP, GrADS... I
> guess C++ and JAVA API are right).
>
I can't speak for other client developers, we will make every
effort to insure that our supported clients can read any valid DAP
response. We support the Matlab, IDL and ODC, as well as the API
implementations we distribute. I doubt if every client application
will be able to support reading Dapper responses, they won't easily
map into some of the underlying client APIs.
Dan
>
> Thanks a lot,
>
> Arnaud and Thomas
>
> --------------------- Matlab Crash Dump ---------------------
>
>
> >>loaddap('http://dapper.pmel.noaa.gov/dapper/argo/argo_all.cdp?
> &location.JULD>1143929418000&location.JULD<1144447818000&location.LATI
> TUDE>30
> &location.LATITUDE<50&location.LONGITUDE>-51&location.LONGITUDE<-5')
>
>> ---------------------------------------------------------------------
>> ---
>> Segmentation violation detected at Thu May 4 15:06:08 2006
>> ---------------------------------------------------------------------
>> ---
>> Configuration:
>> MATLAB Version: 7.0.1.24704 (R14) Service Pack 1
>> MATLAB License: 122551
>> Operating System: Linux 2.6.9-11.EL #1 Wed Jun 8 16:59:52 CDT
>> 2005 i686
>> Window System: Hummingbird Communications Ltd. (7000),
>> display br144-122:0.0
>> Current Visual: 0x23 (class 4, depth 24)
>> Processor ID: x86 Family 15 Model 0 Stepping 10, GenuineIntel
>> Virtual Machine: Java is not enabled
>> Default Charset: UTF-8
>> Register State:
>> eax = 00000000 ebx = 00cbc148
>> ecx = 02d09560 edx = 0896bd40
>> esi = 025250d0 edi = 00000003
>> ebp = bfff8bc8 esp = bfff8bc8
>> eip = 00cb737b flg = 00210286
>> Stack Trace:
>> [0] loaddap.mexglx:vfprintf~(0x0896bd40, 0x025250d0, 0xbfffafb0
>> "location", 0x00cb8d64) + 13179 bytes
>> [1] loaddap.mexglx:0x00cb8e6c(0x08642338, 0xbfffb20c,
>> 0x00cba7c6, 1)
>> [2] loaddap.mexglx:0x00cb97ba(0x08642338, 0x00cba429, 1,
>> 0xbfffbe30)
>> [3] loaddap.mexglx:mexFunction~(0, 0xbfffbdd0, 1, 0xbfffbe30) +
>> 304 bytes
>> [4] libmex.so:mexRunMexFile(0, 0xbfffbdd0, 1, 0xbfffbe30) + 93
>> bytes
>> [5] libmex.so:Mfh_mex::dispatch_file(int, mxArray_tag**, int,
>> mxArray_tag**)(0x02b40fb0, 0, 0xbfffbdd0, 1) + 537 bytes
>> [6] libmwm_dispatcher.so:Mfh_file::dispatch_fh(int,
>> mxArray_tag**, int, mxArray_tag**)(0x02b40fb0, 0, 0xbfffbdd0, 1) +
>> 262 bytes
>> [7] libmwm_interpreter.so:inDispatchFromStack(455, 0x0869ad20
>> "loaddap", 0, 1) + 1240 bytes
>> [8] libmwm_interpreter.so:inDispatchCall(char const*, int, int,
>> int, int*, int*)(0x0869ad20 "loaddap", 455, 0, 1) + 112 bytes
>> [9] libmwm_interpreter.so:.L924(2, 0, 0, 0) + 165 bytes
>> [10] libmwm_interpreter.so:inInterPcodeSJ(inDebugCheck, int,
>> int, opcodes, inPcodeNest_tag*)(2, 0, 0, 0) + 315 bytes
>> [11] libmwm_interpreter.so:inInterPcode(2, 0, 0xbfffc3f8,
>> 0x0095e39b) + 93 bytes
>> [12] libmwm_interpreter.so:in_local_call_eval_function(int*,
>> _pcodeheader*, int*, mxArray_tag**, inDebugCheck)(0, 0xbfffcdf0,
>> 0xbfffce7c, 0xbfffcea8) + 163 bytes
>> [13] libmwm_interpreter.so:inEvalStringWithIsVarFcn
>> (_memory_context*, char const*, EvalType, int, mxArray_tag**,
>> inDebugCheck, _pcodeheader*, int*, bool (*)(void*, char const*),
>> void*)(0x008dd468, 0x08744bd0 "loaddap('http://
>> dapper.pmel.noaa..", 0, 0) + 2358 bytes
>> [14] libmwm_interpreter.so:inEvalCmdNoEnd(0x08744bd0 "loaddap
>> ('http://dapper.pmel.noaa..", 0x08744bd0 "loaddap('http://
>> dapper.pmel.noaa..", 0xbfffd048 ", 0x00de8c27) + 85 bytes
>> [15] libmwbridge.so:mnParser(0x00cd21e8 "@@@", 0x00cd22d8
>> "mnParser", 1, 0xbfffd0a4) + 471 bytes
>> [16] libmwmcr.so:mcrInstance::mnParser()(0x080a01c0, 0,
>> 0xbffff398, 0x0804a902) + 96 bytes
>> [17] MATLAB:mcrMain(int, char**)(2, 0xbffff444, 0x0804ad1c,
>> 0xbffff3b8) + 308 bytes
>> [18] MATLAB:main(2, 0xbffff444, 0xbffff450, 0x0047ebe6) + 23 bytes
>> [19] libc.so.6:__libc_start_main~(0x0804a7c4, 2, 0xbffff444,
>> 0x0804a3d8) + 211 bytes
>> This error was detected while a MEX-file was running. If the MEX-
>> file
>> is not an official MathWorks function, please examine its source code
>> for errors. Please consult the External Interfaces Guide for
>> information
>> on debugging MEX-files.
>
> FOREST Arnaud
More information about the Opendap-tech
mailing list