armi.bookkeeping.memoryProfiler module

Interface to help diagnose memory issues during debugging/development.

There are many approaches to memory profiling.

1. You can ask psutil for the memory used by the process from an OS perspective. This is great for top-down analysis. This module provides printouts that show info from every process running. This is very fast.

2. You can use asizeof (part of pympler) to measure the size of various individual objects. This will help you pin-point your issue. But it’s slow.

3. You can use gc.get_objects() to list all objects that the garbage collector is tracking. If you want, you can filter it down and get the counts and sizes of objects of interest (e.g. all armi objects).

This module has tools to do all of this. It should help you out.

Note that if psutil is reporting way more memory usage than the asizeof reports, then you probably are dealing with a garbage collection queue. If you free a large number of objects, call gc.collect() to force a garbage collection and the psutil process memory should fall by a lot. Also, it seems that even if your garbage is collected, Windows does not de-allocate all the memory. So if you are a worker and you just got a 1.6GB reactor but then deleted it, Windows will keep you at 1.6GB for a while.

See Also:

http://packages.python.org/Pympler/index.html https://pythonhosted.org/psutil/ https://docs.python.org/2/library/gc.html#gc.garbage

armi.bookkeeping.memoryProfiler.describeInterfaces(cs)

Function for exposing interface(s) to other code

class armi.bookkeeping.memoryProfiler.MemoryProfiler(r, cs)

Bases: armi.interfaces.Interface

Construct an interface.

The r and cs arguments are required, but may be None, where appropriate for the specific Interface implementation.

Parameters

• r (Reactor) – A reactor to attach to

• cs (Settings) – Settings object to use

Raises

RuntimeError – Interfaces derived from Interface must define their name

name = 'memoryProfiler'

interactBOL()

Called at the Beginning-of-Life of a run, before any cycles start.

interactEveryNode(cycle, node)

Called at each time node/subcycle of every cycle.

interactEOL()

End of life hook. Good place to wrap up or print out summary outputs

displayMemoryUsage(timeDescription)

Print out some information to stdout about the memory usage of ARMI.

Makes use of the asizeof utility.

Useful when the debugMem setting is set to True.

Turn these on as appropriate to find all your problems.

class armi.bookkeeping.memoryProfiler.KlassCounter(reportSize)

Bases: object

countObjects(ao)

Recursively find non-list,dict, tuple objects in containers.

Essential for traversing the garbage collector

class armi.bookkeeping.memoryProfiler.InstanceCounter(classType, reportSize)

Bases: object

add(item)

class armi.bookkeeping.memoryProfiler.ObjectSizeBreakdown(name, minMBToShowAttrBreakdown=30.0, excludedAttributes=None, initialZeroes=0)

Bases: object

property size

calcSize(obj)

class armi.bookkeeping.memoryProfiler.ProfileMemoryUsageAction(timeDescription)

Bases: armi.mpiActions.MpiAction

invokeHook()

This method must be overridden in sub-clases.

This method is called by worker nodes, and has access to the worker node’s operator, reactor, and settings (through self.o, self.r, and self.cs). It must return a boolean value of True or False, otherwise the worker node will raise an exception and terminate execution.

Returns

result – Dependent on implementation

Return type

object

class armi.bookkeeping.memoryProfiler.SystemAndProcessMemoryUsage

Bases: object

class armi.bookkeeping.memoryProfiler.PrintSystemMemoryUsageAction

Bases: armi.mpiActions.MpiAction

property minProcessMemoryInMB

property maxProcessMemoryInMB

invokeHook()

This method must be overridden in sub-clases.

This method is called by worker nodes, and has access to the worker node’s operator, reactor, and settings (through self.o, self.r, and self.cs). It must return a boolean value of True or False, otherwise the worker node will raise an exception and terminate execution.

Returns

result – Dependent on implementation

Return type

object

printUsage(description=None)

This method prints the usage of all MPI nodes.

The printout looks something like:

SYS_MEM EDWARD102.hpcc.tp.int 14.4% RAM. Proc mem (MB): 491 472 471 471 471 470 SYS_MEM EDWARD103.hpcc.tp.int 13.9% RAM. Proc mem (MB): 474 473 472 471 460 461 SYS_MEM EDWARD104.hpcc.tp.int … SYS_MEM EDWARD107.hpcc.tp.int …