telecon today

[Jose H. Garcia]

Hi James,

I just spoke with Patrick about possible reasons why the BES can  
handle about 1 million of request before getting frisky with memory.  
In my experience, the default 'new' and 'delete' operators in C++, as  
well as the standard allocator in the STL, fragment memory. These  
operators are implemented in the same fashion as the C language pair  
'malloc/free', that is, they do maintain an internal list of areas  
where memory has been allocated in the heap. Given a pointer to one  
of those areas, they know how big such space is.

In my knowledge, this mechanism is imperfect and leads to memory  
fragmentation. The problem is well illustrated in "C++ Pointers and  
Dynamic Memory Management" by Michael Daconta, pages 360-366 and  
there are some suggestions on how to deal with this in Effective C++  
by Scott Meyers, items 5 through 10.

My point is that we don't  use arenas, object pools or any other  
approach for that matter in libdap nor any of the handlers . In other  
words, it is expected that after many cycles, the heap becomes  
unorganized and a simple call to 'new' will fail.

I want to mention that another element often overlooked is heap  
control.  In the BES, heap control is built in upon initialization.  
For this matter, we pre-expand the heap in each BES to a  
predetermined size (defined in the configuration file) and we stop  
the heap from growing any larger than that. This takes care of  
avoiding process starvation, as each BES will not be allowed to grow  
the heap beyond its limit, yet eventually due to fragmentation, a BES  
worker will hit a wall.

I think it is important to mention that most operative system *do  
not* decrease the heap as a respond to calls to either 'free' or  
'delete'. Programmers in general simple rely on the facilities  
provided by C and C++ to mark buffers as free without calling the OS  
to do any heap reduction.

Nathan, a good test sometime for you to try to let the heap in your  
BES grow very large. You can do this by changing the configuration  
file entries:

#----------------------------------------------------------------------- 
#
# Defines size of system global memory  
pool                             #
#----------------------------------------------------------------------- 
#
BES.Memory.GlobalArea.EmergencyPoolSize=1
BES.Memory.GlobalArea.MaximunHeapSize=2
BES.Memory.GlobalArea.Verbose=no
BES.Memory.GlobalArea.ControlHeap=no


Please set BES.Memory.GlobalArea.MaximunHeapSize to a larger number  
(the units are in megabytes) and see if the number of request you can  
hit before failing reaches more than a million. Patrick, will you do  
the same please using the bescmdln?

In summary, if we want to support long term memory control, we must  
overload new, delete and the standard allocator in C++. In my opinion  
this is not that important of a feature, lets just keeps things  
simple like the OLFS is currently doing, that is, recycle a pool of  
workers.
The BES listener never goes away, and paying for a new fork after 1  
million of orders is not too much.

Thanks guys,

Jose



On Apr 30, 2007, at 2:19 PM, Gallagher James wrote:

> All: Sorry, in my haste I sent out the notice of our development  
> group's teleconference to the opendap-tech list. In fact, anyone  
> _is_ welcome to join the teleconference, but you might find it  
> fairly boring and I don't generally announce it for that reason.  
> These are also intended to be very informal with goal of clearing  
> away things are presenting stumbling blocks for us. We don't decide  
> on things so much as report on what's happened. To cut down on the  
> admin overhead, we don't take notes, except that I often jot down a  
> personal to-do list lest I forget something.
>
> You can guess what figured big at today's meeting... With that in  
> mind, what follows is a short summary of the security issues.
>
> On Apr 30, 2007, at 11:27 AM, Roy Mendelssohn wrote:
>
>> Oops, just got this.  Can you put out a summary when completed.
>
>
> We will be coordinating with US-CERT on the security breach. It  
> seems mostly in hand, but some sites still may need to be notified.  
> We're working on an interim security plan until a working group can  
> get a more formal one in place. US-CERT will work with agency CIRTs  
> and has a guide to help people whose machines were affected. We  
> (opendap) had a machine that was broken into - some people were  
> concerned that it was the machine that holds our development code,  
> it wasn't - and that break-in provided a good baseline for the  
> cleanup people will need to do. It appears to be very limited and  
> easy although I want to communicate more with US-CERT before I make  
> that a formal statement.
>
> Other things:
>
> We have tested the new patch to Hyrax and it fixes the issues with  
> concurrency. The server has been tested to over a million accesses.
>
> Progress on a WCS gateway is good.
>
> The PyDAP code seems to fit the bill for getting data off of an  
> opendap server and into a RDB - a project one attendee is working on.
>
> We are going to move forward with a semantics working group.
>
> I have been working on the Working Group Terms of Reference.
>
> James
> --
> James Gallagher                jgallagher at opendap.org
> OPeNDAP, Inc                   406.723.8663
>
>
>
>