Re: [Omaha.pm] Self monitoring a perl scripts memory usage...

[Dan Linder <dan@linder.org>]

On Wed, Oct 21, 2009 at 15:40, Jay Hannah <jay@jays.net> wrote:
> Wow. That's a big process.  :)  You have to have all that in memory all at
> once? Can you cache out some of it or something? (Cache::FileCache?)

The script is building up a large hash table in RAM which is then
dumped to a text file when it finishes.  In this exact instance, the
system is expanding user/group entries that were dumped from a Windows
domain controller.  Since the domain is very dynamic, the actual RAM
requirements change day-to-day so it doesn't fail every day.

The quick and dirty solution is to bump up the ulimit on the account
that the process is running under.  But, that's not the Perl solution.

My first inclination was to capture the system events and exit the
subroutine cleanly.  Unfortunately the OOM signal is not allowed to be
captured by the program...

I did some digging around and there is a special Perl variable $^M -
the "Emergency Memory pool" variable.  In short, the variable is
_documented_ to allow a Perl script to cleanly handle getting low on
memory, but apparently it is not handled within the Perl interperter
itself.

See "Is $^M a leftover April Fool?" :
http://www.perlmonks.org/index.pl?node_id=287850

The only work around I have right now is that the Perl interpreter
seems (or the OS?) prints "out of memory" to the screen as it is being
killed.  When I run the script from within a wrapper script, the
wrapper can capture the "out of memory" in the output and I can search
for that and report back.  Not a great answer, but it's something.

I'm still pursuing the $^M variable.  I'm hoping I can find the key
code that makes it work.

Dan

-- 
******************* ***************** ************* ***********
******* ***** *** **
"Quis custodiet ipsos custodes?" (Who can watch the watchmen?) -- from
the Satires of Juvenal
"I do not fear computers, I fear the lack of them." -- Isaac Asimov (Author)
** *** ***** ******* *********** ************* *****************
*******************