Share @ LinkedIn Facebook  memory-profiling
Pympler - Monitor Memory Usage By Python Objects

Pympler - Monitor Memory Usage By Python Objects

Python has different sets of libraries for performing memory profiling like memory_profiler, guppy/heapy, scalene, etc. All these libraries provide usage of memory by python code in different ways. But there is no provision of monitoring memory usage of object created using user-defined classes in any of them. There are situations where we need to monitor memory usage by a particular kind of object and a Python library named pympler can be very useful for those kinds of requirements. The pympler has a list of modules that lets us monitor memory usage by python code in various ways. As a part of this tutorial, we'll be explaining various modules available in pympler with examples.

Below is a list of modules that we'll discuss as a part of this tutorial.

  • asizeof - This module provides us with various methods to measure the size of objects.
  • classtracker - This module provides us with methods to monitor the memory usage of objects created by user define classes.
  • classtracker_stats - This module lets us format data captured using the classtracker module in different ways.
  • tracker - This module allows us to track changes in overall memory over time.
  • muppy - This module lets us monitor memory usage by a list of objects over time.
  • garbagegraph - This module lets us analyze objects which are creating a reference cycle hence becomes hard to collect by the garbage collector.
  • refbrowser - This module lets us perform tree-like exploration of object referrers.
  • refgraph - This module provides ways to illustrate objects and their references as directed graphs. It can even generate graphviz directed graphs.
  • summary - This module provides functions to summarize information for a list of objects.

We'll start by importing pympler.

In [1]:
import pympler

asizeof

As a part of this section, we'll explore various methods available with asizeof module to find out memory usage by various objects.

  • asizeof.asizeof() This method takes as input single or multiple objects and returns the size of each object in bytes.

Below we have created two lists where the first is an actual list with all elements and the second is the python generator. We have then measured the size of both of them using the asizeof() method.

In [2]:
from pympler import asizeof
In [3]:
l1 = [i for i in range(10)]
l2 = range(10)

print("Size of List l1              : %d bytes"%asizeof.asizeof(l1))
print("Size of List l1              : %d bytes"%asizeof.asizeof(l2))
print("Size of List l1,l2 Combined  : %d bytes"%asizeof.asizeof(l1,l2))
print("Size of List l1 & l2         : %d bytes, %d bytes"%asizeof.asizesof(l1, l2))
Size of List l1              : 504 bytes
Size of List l1              : 48 bytes
Size of List l1,l2 Combined  : 552 bytes
Size of List l1 & l2         : 504 bytes, 48 bytes
  • asizeof.asized() - This method takes as input objects as input and returns a list of objects of type pympler.asizeof.Asized which has information about the memory usage of objects passed.
  • Asized.format() - This method lets us format memory information present for particular object. This gives us detailed insight into complex objects.

Below we have explained the usage of both methods with simple examples.

In [4]:
asized = asizeof.asized(l1)
print("Object Type : ", type(asized))
print(asized)
print("Flat Size : %d, Total Size : %d\n"%(asized.flat, asized.size))

asized_objs = asizeof.asized(l1,l2)
print(asized_objs[0].format())
print(asized_objs[1].format())

print("\nObject Stats : ")
print(asizeof.asized(l1, detail=0, stats=1).format())
print("\nObject Details : ")
print(asizeof.asized(l1, detail=1).format())
Object Type :  <class 'pympler.asizeof.Asized'>
size 504, flat 192, refs[0], name '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]'
Flat Size : 192, Total Size : 504

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] size=504 flat=192
range(0, 10) size=48 flat=48

Object Stats :

asized(detail=0, stats=1): size 504, flat 192, refs[0], name '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]'
 504 bytes
   8 byte aligned
   8 byte sizeof(void*)
   1 object given
  11 objects sized
  11 objects seen
   0 objects missed
   0 duplicates
   1 deepest recursion
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] size=504 flat=192

Object Details :
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] size=504 flat=192
    1 size=32 flat=32
    2 size=32 flat=32
    3 size=32 flat=32
    4 size=32 flat=32
    5 size=32 flat=32
    6 size=32 flat=32
    7 size=32 flat=32
    8 size=32 flat=32
    9 size=32 flat=32
    0 size=24 flat=24
  • itemsize() - This method returns size of single object as bytes from a list of objects.
  • basicsize() - This method returns basic size of object as bytes.
  • flatsize() - This method returns flat size of object as bytes.
  • refs() - This method return list of objects referred by object passed to it.

We have explained how we can use the above methods.

In [10]:
print("Size of Single Item in a List : %d bytes"%asizeof.itemsize(l1))
print("Basic Size of Object          : %d bytes"%asizeof.basicsize(l1))
print("Flat Size of Object           : %d bytes"%asizeof.flatsize(l1))
print("List of Objects Referred by an Object : ", asizeof.refs(l1))
Size of Single Item in a List : 8 bytes
Basic Size of Object          : 64 bytes
Flat Size of Object           : 192 bytes
List of Objects Referred by an Object :  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  • isclass() - This method takes object as input and returns True if object is class else False.
  • isbuiltin() - This method takes object as input and returns True if object is builtin python keyword else False.
  • iscode() - This method takes object as input and returns True if object is code else False.
  • isframe() - This method takes object as input and returns True if object is frame else False.
  • isfunction() - This method takes object as input and returns True if object is function else False.
  • ismethod() - This method takes object as input and returns True if object is method else False.
  • ismodule() - This method takes object as input and returns True if object is module else False.

We have explained the usage of the above methods with simple examples below.

In [6]:
print("Is Object Class    : ",asizeof.isclass(l1))
print("Is Object Builtin  : ",asizeof.isbuiltin(l1))
print("Is Object Code     : ",asizeof.iscode(l1))
print("Is Object Frame    : ",asizeof.isframe(l1))
print("Is Object Function : ",asizeof.isfunction(l1))
print("Is Object method   : ",asizeof.ismethod(l1))
print("Is Object Module   : ",asizeof.ismodule(l1))

print()
import random

print("Is Object Module   : ",asizeof.ismodule(random))

print()
def method_test():
    return random.randint(1,100)

print()
print("Is Object Function : ",asizeof.ismethod(method_test))
print("Is Object method   : ",asizeof.isfunction(method_test))

class Testing:
    def __init__(self):
        pass

    def test_print(self):
        print("Testing Class")

testing  = Testing()

print()
print("Is Object Function : ",asizeof.ismethod(testing.test_print))
print("Is Object method   : ",asizeof.isfunction(testing.test_print))
print("Is Object Class    : ",asizeof.isclass(Testing))

print()
print("Is Object Builtin  : ",asizeof.isbuiltin(min))
print("Is Object Builtin  : ",asizeof.isbuiltin(sum))
print("Is Object Builtin  : ",asizeof.isbuiltin(max))
Is Object Class    :  False
Is Object Builtin  :  False
Is Object Code     :  False
Is Object Frame    :  False
Is Object Function :  False
Is Object method   :  False
Is Object Module   :  False

Is Object Module   :  True


Is Object Function :  False
Is Object method   :  True

Is Object Function :  True
Is Object method   :  False
Is Object Class    :  True

Is Object Builtin  :  True
Is Object Builtin  :  True
Is Object Builtin  :  True

classtracker & classtracker_stats

As a part of this section, we'll explain through various examples how we can trace memory usage by particular kind of user-defined objects and then format stats of monitoring in different ways using classtracker and classtracker_stats modules.

Example 1

Our first example is a simple example that monitors memory usage by objects created using the class RandomNumbersGenerator. We have created a simple class named RandomNumbersGenerator which when instantiated with size generates a numpy array of that size with a random number in the range 1-100. We'll be monitoring all memory usage by all objects created using this class.

In order to trace memory usage by a particular class, we first need to create an instance of ClassTracker(). We then need to class which we want to monitor by calling the track_class() method of ClassTracker by giving it class reference and name as input. We can then call create_snapshot() on the ClassTracker instance and it'll record memory usage by objects of classes monitored by it at that time. We can call create_snapshot() as many times as we want and it'll record memory usage at all times. We can then call the print_summary() method from the stats object of ClassTracker and it'll print all memory snapshots taken for registered classes.

  • classtracker.ClassTracker() - It creates an object which will be used to monitor the memory usage of an object of a particular type.
  • ClassTracker.track_class() - It takes as input reference to class which we want to monitor.
  • ClassTracker.create_snapshot() - It'll record memory usage by registered classes at the point when its called.
  • ClassTracker.stats.print_summary() - It'll print all snapshots taken since beginning.
  • ClassTracker.close() - It'll inform class tracker to stop monitoring everything.

Below we have explained the usage of methods by creating 3 lists and then deleting one list. We have taken snapshots at the beginning, after lists creation, and after one list deletion to notice changes.

In [7]:
import numpy as np

class RandomNumbersGenerator:
    def __init__(self, size):
        self.random_ints = np.random.randint(1,100, size)
In [8]:
from pympler import classtracker

ct = classtracker.ClassTracker()
ct.track_class(RandomNumbersGenerator, name=RandomNumbersGenerator.__name__)

ct.create_snapshot(description="Start")

rand1 = RandomNumbersGenerator((1000,1000))
rand2 = RandomNumbersGenerator((1000,1000))
rand3 = RandomNumbersGenerator((1000,1000))

ct.create_snapshot(description="Intermediate")

#ct.stats.print_stats()

del rand3

ct.create_snapshot(description="End")

ct.stats.print_summary()
ct.close()
---- SUMMARY ------------------------------------------------------------------
Start                                    active      0     B      average   pct
  RandomNumbersGenerator                      0      0     B      0     B    0%
Intermediate                             active      0     B      average   pct
  RandomNumbersGenerator                      3     22.89 MB      7.63 MB    3%
End                                      active      0     B      average   pct
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    2%
-------------------------------------------------------------------------------

Example 2

As a part of our second example, we'll be demonstrating how we can monitor more than one class using a class tracker and how to clear stats collected.

  • ClassTracker.clear() - It clears all stats collected till now.

Below we have created another class named RandomDistribution which generates a random uniform distribution for size passed to it. We have explained how we can monitor the usage of both RandomDistribution and RandomNumbersGenerator classes.

In [9]:
class RandomDistribution:
    def __init__(self, size):
        self.uniform_dist = np.random.uniform(size=size)
In [10]:
from pympler import classtracker

ct = classtracker.ClassTracker()
ct.track_class(RandomNumbersGenerator, name=RandomNumbersGenerator.__name__)
ct.track_class(RandomDistribution, name=RandomDistribution.__name__)

ct.create_snapshot(description="Start")

rand1 = RandomNumbersGenerator((1000,1000))
rand2 = RandomNumbersGenerator((1000,1000))

unif_dist1 = RandomDistribution((100,100))
unif_dist2 = RandomDistribution((100,100))

ct.create_snapshot(description="Intermediate")

#ct.stats.print_stats()
del rand2, unif_dist2

ct.create_snapshot(description="End")

ct.stats.print_summary()

ct.clear()
ct.stats.print_summary()

ct.close()
---- SUMMARY ------------------------------------------------------------------
Start                                    active      0     B      average   pct
  RandomDistribution                          0      0     B      0     B    0%
  RandomNumbersGenerator                      0      0     B      0     B    0%
Intermediate                             active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    2%
End                                      active      0     B      average   pct
  RandomDistribution                          1     78.46 KB     78.46 KB    0%
  RandomNumbersGenerator                      1      7.63 MB      7.63 MB    1%
-------------------------------------------------------------------------------
---- SUMMARY ------------------------------------------------------------------
-------------------------------------------------------------------------------

Example 3

As a part of this example, we'll explain how we can inform the class tracker to monitor only a particular object of a class rather than all objects.

  • ClassTracker.track_object() - This method will monitor memory usage by object passed to it.
  • ClassTracker.track_change() - This method will monitor memory usage by object passed to it as well as changes to it.

Below we have created two RandomNumbersGenerator and two RandomDistribution objects. We are then monitoring them using the above described methods.

In [11]:
from pympler import classtracker

rand1 = RandomNumbersGenerator((1000,1000))
rand2 = RandomNumbersGenerator((1000,1000))

unif_dist1 = RandomDistribution((100,100))
unif_dist2 = RandomDistribution((100,100))

ct = classtracker.ClassTracker()
ct.track_object(rand1)
ct.track_change(rand1)
ct.track_object(rand2)
ct.track_change(rand2)
ct.track_object(unif_dist1)
ct.track_change(unif_dist1)
ct.track_object(unif_dist2)
ct.track_change(unif_dist2)

ct.create_snapshot(description="Start")

rand1.random_ints = np.random.randint(1,100, (100,100))

ct.create_snapshot(description="Intermediate")

del rand2

ct.create_snapshot(description="End")

ct.stats.print_summary()

#ct.stats.print_stats(clsname="RandomDistribution")

ct.close()
---- SUMMARY ------------------------------------------------------------------
Start                                    active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    2%
Intermediate                             active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      2      7.71 MB      3.85 MB    1%
End                                      active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      1     78.46 KB     78.46 KB    0%
-------------------------------------------------------------------------------

Example 4

As a part of this example, we'll explain how we can remove a class that we don't want to monitor as well as how we can dump monitoring stats to an output file that we can load later.

  • ClassTracker.detach_class() - This method takes a class reference as input and removes that class from the monitoring list of a class tracker.
  • ClassTracker.stats.sort_stats() - This method sorts stats of monitoring. We can give it sorting key like the name of the class, active, average, pct. etc.
  • ClassTracker.stats.dump_stats() - This method dumps monitoring stats to an output file.

Below we have first registered two classes which we created earlier for monitoring, then created instances of classes, deleted a few instances, and then unregistered one class from monitoring. We have also taken a memory snapshot between steps. We have then checked whether it’s still monitoring a new instance of that class getting created. At last, we have stored monitoring stats to an output file.

In [12]:
from pympler import classtracker

ct = classtracker.ClassTracker()
ct.track_class(RandomNumbersGenerator, name=RandomNumbersGenerator.__name__)
ct.track_class(RandomDistribution, name=RandomDistribution.__name__)
ct.create_snapshot(description="Start")

rand1 = RandomNumbersGenerator((1000,1000))
rand2 = RandomNumbersGenerator((1000,1000))

unif_dist1 = RandomDistribution((100,100))
unif_dist2 = RandomDistribution((100,100))

ct.create_snapshot(description="Intermediate")

#ct.stats.print_stats()
del rand2, unif_dist2

ct.create_snapshot(description="End")

ct.stats.print_summary()

## Detaches "RandomNumbersGenerator".
## Already monitored instance will be monitored but future instance won't be monitored
## detach_all() and detach_all_classes() detaches all objects and classes.
ct.detach_class(RandomNumbersGenerator)

del rand1
rand3  = RandomNumbersGenerator((1000,1000))

ct.create_snapshot(description="Last")
ct.stats.sort_stats().print_summary()

print("Tracked Classes : ", ct.stats.tracked_classes)
ct.stats.dump_stats("pympler_stats.out")

ct.close()
---- SUMMARY ------------------------------------------------------------------
Start                                    active      0     B      average   pct
  RandomDistribution                          0      0     B      0     B    0%
  RandomNumbersGenerator                      0      0     B      0     B    0%
Intermediate                             active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    2%
End                                      active      0     B      average   pct
  RandomDistribution                          1     78.46 KB     78.46 KB    0%
  RandomNumbersGenerator                      1      7.63 MB      7.63 MB    1%
-------------------------------------------------------------------------------
---- SUMMARY ------------------------------------------------------------------
Start                                    active      0     B      average   pct
  RandomDistribution                          0      0     B      0     B    0%
  RandomNumbersGenerator                      0      0     B      0     B    0%
Intermediate                             active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    2%
End                                      active      0     B      average   pct
  RandomDistribution                          1     78.46 KB     78.46 KB    0%
  RandomNumbersGenerator                      1      7.63 MB      7.63 MB    1%
Last                                     active      0     B      average   pct
  RandomDistribution                          1     78.46 KB     78.46 KB    0%
  RandomNumbersGenerator                      0     16     B      0     B    0%
-------------------------------------------------------------------------------
Tracked Classes :  ['RandomDistribution', 'RandomNumbersGenerator']

The classtracker module has a ConsoleStats class which lets us load monitoring stats from a file. Below we have reloaded again monitoring stats store to a file in a previous step and printed stats again to verify.

In [13]:
ct = classtracker.ClassTracker()
console_stats = classtracker.ConsoleStats(ct, "pympler_stats.out")

console_stats.print_summary()
print("Tracked Classes : ",console_stats.tracked_classes)
---- SUMMARY ------------------------------------------------------------------
Start                                    active      0     B      average   pct
  RandomDistribution                          0      0     B      0     B    0%
  RandomNumbersGenerator                      0      0     B      0     B    0%
Intermediate                             active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    2%
End                                      active      0     B      average   pct
  RandomDistribution                          1     78.46 KB     78.46 KB    0%
  RandomNumbersGenerator                      1      7.63 MB      7.63 MB    1%
Last                                     active      0     B      average   pct
  RandomDistribution                          1     78.46 KB     78.46 KB    0%
  RandomNumbersGenerator                      0     16     B      0     B    0%
-------------------------------------------------------------------------------
Tracked Classes :  ['RandomDistribution', 'RandomNumbersGenerator']

Example 5

As a part of this example, we have again explained the usage of dumping stats to a file and then reloading it.

In [15]:
from pympler import classtracker

ct = classtracker.ClassTracker()
ct.track_class(RandomNumbersGenerator, name=RandomNumbersGenerator.__name__)
console_stats = classtracker.ConsoleStats(ct)

ct.create_snapshot(description="Start")

rand1 = RandomNumbersGenerator((1000,1000))
rand2 = RandomNumbersGenerator((1000,1000))
rand3 = RandomNumbersGenerator((1000,1000))

ct.create_snapshot(description="Intermediate")

#ct.stats.print_stats()

del rand3

ct.create_snapshot(description="End")

console_stats.print_summary()

console_stats.dump_stats("pympler_stats.out")
print("Tracked Classes : ",console_stats.tracked_classes)

ct.close()
---- SUMMARY ------------------------------------------------------------------
Start                                    active      0     B      average   pct
  RandomNumbersGenerator                      0      0     B      0     B    0%
Intermediate                             active      0     B      average   pct
  RandomNumbersGenerator                      3     22.89 MB      7.63 MB    3%
End                                      active      0     B      average   pct
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    2%
-------------------------------------------------------------------------------
Tracked Classes :  ['RandomNumbersGenerator']
In [16]:
ct = classtracker.ClassTracker()
console_stats = classtracker.ConsoleStats(ct, "pympler_stats.out")

console_stats.print_summary()
print("Tracked Classes : ",console_stats.tracked_classes)
---- SUMMARY ------------------------------------------------------------------
Start                                    active      0     B      average   pct
  RandomNumbersGenerator                      0      0     B      0     B    0%
Intermediate                             active      0     B      average   pct
  RandomNumbersGenerator                      3     22.89 MB      7.63 MB    3%
End                                      active      0     B      average   pct
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    2%
-------------------------------------------------------------------------------
Tracked Classes :  ['RandomNumbersGenerator']

Example 6

As a part of this example, we have explained how we can format monitoring statistics as an HTML file which has charts explaining memory usage of a class.

  • classtracker_stats.HtmlStats() - This class takes as input class tracker instance and will be responsible for formatting monitoring stats as html.

Below we have again recreated an example almost the same as one of our previous examples but this time introduced HTML stats as well.

In [18]:
from pympler import classtracker, classtracker_stats

ct = classtracker.ClassTracker()
ct.track_class(RandomNumbersGenerator, name=RandomNumbersGenerator.__name__)
ct.track_class(RandomDistribution, name=RandomDistribution.__name__)

html_stats = classtracker_stats.HtmlStats(ct)

ct.create_snapshot(description="Start")

rand1 = RandomNumbersGenerator((1000,1000))
rand2 = RandomNumbersGenerator((1000,1000))

unif_dist1 = RandomDistribution((100,100))
unif_dist2 = RandomDistribution((100,100))

ct.create_snapshot(description="Intermediate")

#ct.stats.print_stats()
del rand2, unif_dist2

ct.create_snapshot(description="End")

ct.stats.print_summary()

html_stats.dump_stats("pympler_html_stats.out")
print("Tracked Classes : ",html_stats.tracked_classes)

ct.close()
---- SUMMARY ------------------------------------------------------------------
Start                                    active      0     B      average   pct
  RandomDistribution                          0      0     B      0     B    0%
  RandomNumbersGenerator                      0      0     B      0     B    0%
Intermediate                             active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    1%
End                                      active      0     B      average   pct
  RandomDistribution                          1     78.46 KB     78.46 KB    0%
  RandomNumbersGenerator                      1      7.63 MB      7.63 MB    0%
-------------------------------------------------------------------------------
Tracked Classes :  ['RandomDistribution', 'RandomNumbersGenerator']
  • HtmlStats.create_html() - This method will create HTML file of monitoring stats. It'll also create supporting folder of files with the same name as HTML file name.
In [19]:
html_stats.create_html("html_stats.html")

pympler

Other Files pympler pympler pympler

Example 7

As a part of our seventh example, we are explaining how we can inform class tracker to take snapshots periodically rather than we manually taking snapshot each time.

  • classtracker.PeriodicThread() - This class takes as input class tracker instance and interval in seconds. It'll then take snapshots as that interval.
  • PeriodicThread.tracker.stop_periodic_snapshots() - We can call this method to inform periodic thread to stop taking snapshots.
  • PeriodicThread.tracker.start_periodic_snapshots() - We can call this method to inform periodic thread to start taking snapshots.

Below we have explained the usage of the above methods with a simple example.

In [104]:
import time

ct = classtracker.ClassTracker()
ct.track_class(RandomNumbersGenerator, name=RandomNumbersGenerator.__name__)
ct.track_class(RandomDistribution, name=RandomDistribution.__name__)

periodic_thread = classtracker.PeriodicThread(ct, 0.1)
periodic_thread.start()

rand1 = RandomNumbersGenerator((1000,1000))
unif_dist1 = RandomDistribution((100,100))
In [106]:
ct.stats.print_summary()

periodic_thread.tracker.stop_periodic_snapshots()

rand2 = RandomNumbersGenerator((1000,1000))
unif_dist2 = RandomDistribution((100,100))

ct.stats.print_summary()

periodic_thread.tracker.start_periodic_snapshots()

ct.stats.print_summary()

ct.close()
---- SUMMARY ------------------------------------------------------------------
-------------------------------------------------------------------------------
---- SUMMARY ------------------------------------------------------------------
                                         active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    0%
                                         active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    0%
-------------------------------------------------------------------------------
---- SUMMARY ------------------------------------------------------------------
                                         active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    0%
                                         active      0     B      average   pct
  RandomDistribution                          2    156.86 KB     78.43 KB    0%
  RandomNumbersGenerator                      2     15.26 MB      7.63 MB    0%
-------------------------------------------------------------------------------

tracker

The tracker module lets us monitor overall memory usage over time. It can let us track the difference in memory usage between summaries.

In [1]:
from pympler import tracker
  • tracker.SummaryTracker() - This class lets us monitor memory usage between summaries.
  • SummaryTracker.create_summary() - This method creates a memory usage summary.
  • SummaryTracker.print_diff() - This method finds out difference between two summaries.
  • diff() - This method returns difference between two summaries as a list.

Below we have explained the usage of methods with a simple example.

In [115]:
summary_tracker = tracker.SummaryTracker()
summary_tracker.print_diff()
                                   types |   # objects |   total size
======================================== | =========== | ============
                   pympler.asizeof._Seen |           0 |      1.25 MB
                                    list |       12723 |      1.21 MB
                                     str |       14515 |   1005.61 KB
                                     int |       29009 |    890.34 KB
                                   tuple |        3556 |    222.26 KB
                  pympler.asizeof.Asized |        1950 |    137.11 KB
                                    dict |         326 |     48.56 KB
  pympler.process._ProcessMemoryInfoProc |         162 |      8.86 KB
           pympler.classtracker.Snapshot |         162 |      8.86 KB
                                   float |         162 |      3.80 KB
                                    code |           1 |    144     B
                   function (store_info) |           1 |    136     B
                                 weakref |           1 |     80     B
                        _ast.Interactive |           1 |     56     B
                             _ast.Module |          -1 |    -56     B
In [127]:
s0 = summary_tracker.create_summary()
l = [i*i for i in range(10000)] ## Creating Temporary List to Check its presence in difference
s1 = summary_tracker.create_summary()

summary_tracker.print_diff(summary1=s0, summary2=s1)
                                   types |   # objects |   total size
======================================== | =========== | ============
                   pympler.asizeof._Seen |           0 |      1.94 MB
                                     int |       53126 |      1.62 MB
                                   tuple |        3983 |    248.94 KB
                  pympler.asizeof.Asized |        2263 |    159.12 KB
                                     str |        2265 |    123.53 KB
                                    list |         165 |     67.95 KB
                                    dict |         348 |     51.54 KB
  pympler.process._ProcessMemoryInfoProc |         173 |      9.46 KB
           pympler.classtracker.Snapshot |         173 |      9.46 KB
                                   float |         173 |      4.05 KB
                       _io.TextIOWrapper |           2 |    432     B
                      _io.BufferedReader |           2 |    352     B
                              _io.FileIO |           2 |    144     B
              builtin_function_or_method |           2 |    144     B
      encodings.utf_8.IncrementalDecoder |           2 |    112     B
In [131]:
sorted(summary_tracker.diff(summary1=s0, summary2=s1), key=lambda x: x[2], reverse=True)
Out[131]:
[['pympler.asizeof._Seen', 0, 2031616],
 ['int', 53126, 1695136],
 ['tuple', 3983, 254912],
 ['pympler.asizeof.Asized', 2263, 162936],
 ['str', 2265, 126495],
 ['list', 165, 69576],
 ['dict', 348, 52776],
 ['pympler.process._ProcessMemoryInfoProc', 173, 9688],
 ['pympler.classtracker.Snapshot', 173, 9688],
 ['float', 173, 4152],
 ['_io.TextIOWrapper', 2, 432],
 ['_io.BufferedReader', 2, 352],
 ['_io.FileIO', 2, 144],
 ['builtin_function_or_method', 2, 144],
 ['encodings.utf_8.IncrementalDecoder', 2, 112],
 ['_io.IncrementalNewlineDecoder', 2, 80],
 ['weakref', 1, 80],
 ['list_iterator', -1, -56],
 ['tuple_iterator', -2, -112],
 ['dict_itemiterator', -3, -240],
 ['function (_N)', -2, -272],
 ['function (<lambda>)', -2, -272],
 ['cell', -7, -336],
 ['method', -6, -384],
 ['generator', -6, -720],
 ['bytes', -12, -1693]]

muppy

The muppy module lets the developer detect memory leaks. As a part of this section, we'll explain the usage of various methods of muppy module.

In [112]:
from pympler import muppy
  • muppy.get_objects() - It returns list of all objects in the memory.
In [139]:
objs = muppy.get_objects()
print("Number of Objects : ", len(objs))
Number of Objects :  5746097
  • muppy.filter() - It lets us filter objects passed to it to keep only objects of a particular type.

Below we have only kept objects of type list.

In [146]:
objs_by_type = muppy.filter(objs, Type=list)
objs_by_type[:5]
Out[146]:
[[],
 [(2293,
   139,
   'objs = muppy.get_objects()\nprint("Number of Objects : ", len(objs))',
   'objs = muppy.get_objects()\nprint("Number of Objects : ", len(objs))')],
 [],
 [],
 []]
  • muppy.get_referents() - It lets us find objects referring to object passed to it.
In [165]:
l = [i*i for i in range(4999)]

all_objs_referred_by_l = muppy.get_referents(l)

print("Number of Objects Referred by List l : ", len(all_objs_referred_by_l))
Number of Objects Referred by List l :  4999
  • muppy.print_summary() - It prints memory usage as table.
In [114]:
muppy.print_summary()
                         types |   # objects |   total size
============================== | =========== | ============
                          list |       68727 |     55.18 MB
         pympler.asizeof._Seen |           5 |     53.06 MB
                           int |     1388401 |     41.84 MB
                           str |      337139 |     28.01 MB
                          dict |       68441 |     14.16 MB
                         tuple |      169164 |     10.41 MB
        pympler.asizeof.Asized |       58192 |      4.00 MB
    parso.python.tree.Operator |       33195 |      3.04 MB
                          code |       20331 |      2.80 MB
                          type |        2883 |      2.69 MB
  parso.python.tree.PythonNode |       35220 |      2.15 MB
        parso.python.tree.Name |       26297 |      2.01 MB
     parso.python.tree.Keyword |        8891 |    833.53 KB
     parso.python.tree.Newline |       10613 |    829.14 KB
                           set |        1515 |    741.91 KB
  • muppy.getsizeof() - It returns size of object as bytes.
In [137]:
l = [i*i for i in range(10000)]
print("Size of List l : %d bytes"%muppy.getsizeof(l))
Size of List l : 87624 bytes

garbagegraph

This module lets us monitor cyclic objects. It has a class named GarbageGraph which takes an input list of objects and generates graphviz visualization showing the relation between them which can help us detect a cycle. The GarbageGraph is an extension of ReferenceGraph which will introduce in the refgraph section.

Below we have tried to create cyclic objects and then created a garbage graph of it.

In [12]:
from pympler import garbagegraph
In [15]:
a = "A"
b = True
c = {a:b}
d= [1,2,3]
e = {2:3}

ref_graph = garbagegraph.GarbageGraph([a,b,c, d, e])
  • GarbageGraph.write_graph() - It saves reference graph to an output file.
  • GarbageGraph.render() - It renders graph created using write_graph() method as graphviz graph with .ps extension.
In [186]:
ref_graph.write_graph("ref_graph.out")
In [187]:
!cat ref_graph.out
// Process this file with graphviz
digraph G {
    node [shape=box];
    "Xx7f39954b7340" [ label = "'A'\nstr"  ];
    "Xx55ca999273e0" [ label = "True\nbool"  ];
    "Xx7f38d1955cf0" [ label = "{'A': True}\ndict"  ];
    "Xx7f38d0ab2508" [ label = "[1, 2, 3]\nlist"  ];
    "Xx7f38d1404b88" [ label = "{2: 3}\ndict"  ];
    Xx7f38d1955cf0 -> Xx55ca999273e0 [label="A"];
}
In [181]:
ref_graph.render("ref_graph1.out")
Out[181]:
True

pympler

refbrowser

The refbrowser module lets us print tree-like illustration for object referrers. Below we have created a tree-like exploration of objects using ConsoleBrowser and FileBrowser classes of the refbrowser module.

In [1]:
from pympler import refbrowser
In [ ]:
a = "A"
b = "B"
c = [a, b]

browser = refbrowser.ConsoleBrowser(b)

browser.print_tree()
str-+-list--dict-+-dict
    |            +-module(__main__)
    |            +-dict
    |            +-dict
    |
    +-frozenset--dict-+-module(sre_parse)
    |                 +-function (_class_escape)
    |                 +-function (_escape)
    |                 +-function (_uniq)
    |                 +-function (_parse_sub)
    |                 +-function (_parse)
    |                 +-function (_parse_flag
In [6]:
file_browser = refbrowser.FileBrowser(b)

file_browser.print_tree("tree.out")
str-+-list--dict-+-dict
    |            +-module(__main__)
    |            +-dict
    |            +-dict
    |
    +-frozenset--dict-+-module(sre_parse)
    |                 +-function (_class_escape)
    |                 +-function (_escape)
    |                 +-function (_uniq)
    |                 +-function (_parse_sub)
    |                 +-function (_parse)
    |                 +-function (_parse_flag

refgraph

The refgraph module creates a graphviz graph of references.

  • refgraph.ReferenceGraph() - It takes as an input list of objects and then creates a reference graph showing the relation between them.
  • ReferenceGraph.write_graph() - It saves reference graph to an output file.
  • ReferenceGraph.render() - It renders graph created using write_graph() method as graphviz graph with .ps extension.

Below we have created reference graph of a few objects and then also created a graphviz directed graph of it.

In [7]:
from pympler import refgraph
In [8]:
a = "A"
b = True
c = {a:b}
d= [1,2,3]
e = {2:3}

ref_graph = refgraph.ReferenceGraph([a,b,c, d, e])
In [9]:
ref_graph.write_graph("ref_graph_graphviz")
In [10]:
!cat ref_graph.out
// Process this file with graphviz
digraph G {
    node [shape=box];
    "Xx7f2a599f6340" [ label = "'A'\nstr"  ];
    "Xx564cf9f753e0" [ label = "True\nbool"  ];
    "Xx7f2a2c37ac18" [ label = "{'A': True}\ndict"  ];
    "Xx7f2a2b657bc8" [ label = "[1, 2, 3]\nlist"  ];
    "Xx7f2a2c36dc18" [ label = "{2: 3}\ndict"  ];
    Xx7f2a2c37ac18 -> Xx564cf9f753e0 [label="A"];
}
In [16]:
ref_graph.render("ref_graph_graphviz")
Out[16]:
True

pympler

summary

The summary module lets us create a summary of the list of objects.

  • summary.summarize() - It creates summary for list of objects as a table.
  • summary.print_() - This method prints output of summarize() method in a table format.
  • summary.get_diff() - It lets us find out two different types of summary generated by summarize() method.

Below we have explained the usage of these methods with simple examples.

In [196]:
from pympler import summary
In [202]:
sum_by_type = summary.summarize(objs_by_type)

summary.print_(sum_by_type)
                   types |   # objects |   total size
======================== | =========== | ============
                    list |      130798 |    696.24 MB
  traceback.StackSummary |           4 |    544     B
In [205]:
sum_by_l = summary.summarize(all_objs_referred_by_l)

summary.print_(sum_by_l)
  types |   # objects |   total size
======= | =========== | ============
    int |        4999 |    136.69 KB
In [206]:
sum_full = summary.summarize(objs)

summary.print_(sum_full)
                                   types |   # objects |   total size
======================================== | =========== | ============
                                    list |      130798 |    696.30 MB
                                     int |     4181941 |    126.72 MB
                   pympler.asizeof._Seen |          13 |     74.31 MB
                                     set |        1519 |     66.75 MB
                                     str |      503288 |     37.75 MB
                                   tuple |      382887 |     23.45 MB
                                    dict |       88952 |     18.45 MB
                  pympler.asizeof.Asized |      172733 |     11.86 MB
              parso.python.tree.Operator |       33220 |      3.04 MB
                                    code |       20346 |      2.81 MB
                                    type |        2883 |      2.69 MB
            parso.python.tree.PythonNode |       35236 |      2.15 MB
                  parso.python.tree.Name |       26323 |      2.01 MB
           pympler.classtracker.Snapshot |       18684 |   1021.78 KB
  pympler.process._ProcessMemoryInfoProc |       18684 |   1021.78 KB
In [215]:
diff = summary.get_diff(sum_by_type, sum_full)
summary.print_(diff)
                                   types |   # objects |   total size
======================================== | =========== | ============
                                    list |      130798 |    696.30 MB
                                     int |     4176942 |    126.58 MB
                   pympler.asizeof._Seen |          13 |     74.31 MB
                                     set |        1519 |     66.75 MB
                                     str |      503288 |     37.75 MB
                                   tuple |      382887 |     23.45 MB
                                    dict |       88952 |     18.45 MB
                  pympler.asizeof.Asized |      172733 |     11.86 MB
              parso.python.tree.Operator |       33220 |      3.04 MB
                                    code |       20346 |      2.81 MB
                                    type |        2883 |      2.69 MB
            parso.python.tree.PythonNode |       35236 |      2.15 MB
                  parso.python.tree.Name |       26323 |      2.01 MB
  pympler.process._ProcessMemoryInfoProc |       18684 |   1021.78 KB
           pympler.classtracker.Snapshot |       18684 |   1021.78 KB


Sunny Solanki  Sunny Solanki