Share @ LinkedIn Facebook  memoery-profiling, guppy, heapy
guppy/heapy - Profile Memory Usage in Python

guppy/heapy - Profile Memory Usage in Python

The guppy is a python package that provides the user status of the current heap and objects present in the heap. This information can help the developer with memory profiling and analysis. The guppy has a module named heapy which provides a list of important methods and current heap status. The guppy when called for heap status gives us a special C-nodesets object which has heap status for each object present in the memory. It has information about all reachable and/or unreachable objects present in the heap as a special type of list. It also provides information about a number of objects, % of memory occupied by the object in memory, size in bytes, and type information as well. Apart from this guppy module provides a list of useful methods that can let us access an individual element of the heap status, find the difference between heap statuses, find the size of objects in bytes, and many other functionalities. We'll be explaining the usage of guppy to collect information about memory usage in python through various examples as a part of this tutorial.

Example 1

We'll first explain how to access heap status as a part of our first example and then will slowly build on it.

  • guppy.hpy() - The guppy has method named hpy() which gives us access to heapy object. This will be the first method called by all our examples in order to get the heapy object which will provide a common interface for accessing heap status and performing other operations as well.

  • heap() - This method is available through object created by calling hpy() method. This method provides a list of objects accessible from the root of the heap which are reachable and visible. The returned result is presented in a table format. Please make a note that it does not include objects used as a part of guppy.

  • setref() - This method is available through heapy object created by calling hpy(). It is used to set a reference point and all objects created after this reference point will be available in the next call to the heap() method for heap status. It'll not include all objects present in heap but only once created after the reference point.

Below we have explained the usage of all these 3 methods. We have first collected heap status at the beginning of the script. We have then set the reference point and retrieved the heap status again. Then we have created a few objects like list, string, and numpy array of random numbers. After the creation of these objects, we have again called the heap() method to get heap status which has information about these objects.

We can see from the output that it has information about a number of objects and the total size of the whole heap as well as object count, size, % of memory used by that object type, and type information. The second heap status does not much info as not much has happened after setting up a reference point. The third heap status object has information about objects created after setting a reference point.

CODE

import guppy
from guppy import hpy
import numpy as np


heap = hpy()

print("Heap Status At Starting : ")
heap_status1 = heap.heap()
print("Heap Size : ", heap_status1.size, " bytes\n")
print(heap_status1)


heap.setref()

print("\nHeap Status After Setting Reference Point : ")
heap_status2 = heap.heap()
print("Heap Size : ", heap_status2.size, " bytes\n")
print(heap_status2)

a = [i for i in range(1000)]
b = "A"
c = np.random.randint(1,100, (1000,))

print("\nHeap Status After Creating Few Objects : ")
heap_status3 = heap.heap()
print("Heap Size : ", heap_status3.size, " bytes\n")
print(heap_status3)

print("\nMemory Usage After Creation Of Objects : ", heap_status3.size - heap_status2.size, " bytes")

OUTPUT

Heap Status At Starting :
Heap Size :  12985791  bytes

Partition of a set of 98239 objects. Total size = 12985791 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  27441  28  4200501  32   4200501  32 str
     1  26394  27  2002360  15   6202861  48 tuple
     2  13607  14  1021370   8   7224231  56 bytes
     3   6772   7   978824   8   8203055  63 types.CodeType
     4   6235   6   847960   7   9051015  70 function
     5    940   1   801552   6   9852567  76 type
     6    273   0   541800   4  10394367  80 dict of module
     7    940   1   509240   4  10903607  84 dict of type
     8   1202   1   482112   4  11385719  88 dict (no owner)
     9    791   1   191880   1  11577599  89 dict of function
<248 more rows. Type e.g. '_.more' to view.>

Heap Status After Setting Reference Point :
Heap Size :  416  bytes

Partition of a set of 1 object. Total size = 416 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100      416 100       416 100 types.FrameType

Heap Status After Creating Few Objects :
Heap Size :  40968  bytes

Partition of a set of 838 objects. Total size = 40968 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0    834 100    23352  57     23352  57 int
     1      1   0     9024  22     32376  79 list
     2      1   0     8096  20     40472  99 numpy.ndarray
     3      1   0      416   1     40888 100 types.FrameType
     4      1   0       80   0     40968 100 builtins.weakref

Memory Usage After Creation Of Objects :  40552  bytes

Example 2

As a part of the second example, we'll explain how to access objects which are not reachable from the root of the heap. This is generally created using cycle references.

  • heapu() - This method is available through an object created by calling hpy() method of guppy module. This method provides us heap status about the list of objects which are not reachable from the root of the heap. The object returned by heapu() method is a status object (guppy.heapy.Part.Stat) which has stats about unreachable objects.

Below we have first retrieved a list of unreachable objects from the heap and then used various methods to access an individual row of status to retrieve information about the individual data type.

If you are interested in learning about memory management in python then please feel free to check our tutorial on the same.

CODE

import guppy
from guppy import hpy


heap = hpy()

print("GC Collectable Objects Which Are not Reachable from Root of Heap")

stats = heap.heapu()

print("Total Objects : ", stats.count)
print("Total Size : ", stats.size, " Bytes")
print("Number of Entries : ", stats.numrows)

print("Entries : ")

print("Index Count  Size  Cumulative Size         Object Name")
for row in stats.get_rows():
    print("%5d%5d%8d%8d%30s"% (row.index, row.count, row.size, row.cumulsize, row.name))


print("\nFirst 5 Entries : ")
print("Index Count  Size  Cumulative Size         Object Name")
for row in stats.rows[:5]:
    print("%5d%5d%8d%8d%30s"%(row.index, row.count, row.size, row.cumulsize, row.name))

print("\nDirectly Printing Results Without Iteration")
print(heap.heapu(stat=0))

print("\nMeasuring Unreachable Objects From This Reference Point Onwards")
heap.setref()
print(heap.heapu(stat=0))

OUTPUT

GC Collectable Objects Which Are not Reachable from Root of Heap
Total Objects :  681
Total Size :  119273  Bytes
Number of Entries :  16
Entries :
Index Count  Size  Cumulative Size         Object Name
    0   65   50136   50136                          dict
    1   44   17600   67736                          type
    2   44   10880   78616                           set
    3  125   10000   88616   types.WrapperDescriptorType
    4  102    8160   96776              builtins.weakref
    5   90    6480  103256       types.BuiltinMethodType
    6   88    5368  108624                         tuple
    7   50    3600  112224    types.MemberDescriptorType
    8   46    3312  115536    types.MethodDescriptorType
    9   13    1737  117273                           str
   10    3    1176  118449                          list
   11    7     504  118953    types.GetSetDescriptorType
   12    1     168  119121                     sys.flags
   13    1      80  119201                        module
   14    1      56  119257                  staticmethod
   15    1      16  119273                 Token.MISSING

First 5 Entries :
Index Count  Size  Cumulative Size         Object Name
    0   65   50136   50136                          dict
    1   44   17600   67736                          type
    2   44   10880   78616                           set
    3  125   10000   88616   types.WrapperDescriptorType
    4  102    8160   96776              builtins.weakref

Directly Printing Results Without Iteration
Partition of a set of 790 objects. Total size = 137320 bytes.
 Index  Count   %     Size   % Cumulative  % Type
     0     77  10    58552  43     58552  43 dict
     1     52   7    20800  15     79352  58 type
     2    143  18    11440   8     90792  66 types.WrapperDescriptorType
     3     44   6    10880   8    101672  74 set
     4    102  13     8160   6    109832  80 builtins.weakref
     5    108  14     7776   6    117608  86 types.BuiltinMethodType
     6    109  14     6712   5    124320  91 tuple
     7     69   9     4968   4    129288  94 types.MethodDescriptorType
     8     51   6     3672   3    132960  97 types.MemberDescriptorType
     9     19   2     2216   2    135176  98 str
<6 more rows. Type e.g. '_.more' to view.>

Measuring Unreachable Objects From This Reference Point Onwards
Partition of a set of 790 objects. Total size = 137320 bytes.
 Index  Count   %     Size   % Cumulative  % Type
     0     77  10    58552  43     58552  43 dict
     1     52   7    20800  15     79352  58 type
     2    143  18    11440   8     90792  66 types.WrapperDescriptorType
     3     44   6    10880   8    101672  74 set
     4    102  13     8160   6    109832  80 builtins.weakref
     5    108  14     7776   6    117608  86 types.BuiltinMethodType
     6    109  14     6712   5    124320  91 tuple
     7     69   9     4968   4    129288  94 types.MethodDescriptorType
     8     51   6     3672   3    132960  97 types.MemberDescriptorType
     9     19   2     2216   2    135176  98 str
<6 more rows. Type e.g. '_.more' to view.>

Example 3

As a part of this example, we'll explain how we can retrieve an individual entry from the whole heap status object.

Below we have retrieved heap status after creating few lists. The individual object can also be accessed from the heap status object by simply using list indexing. Below we have printed various individual entry of heap status. We have also created a simple method that takes as input size in bytes and returns size in KB/MB/GB.

CODE

import guppy
from guppy import hpy
import numpy as np

heap = hpy()

print("Heap Status At Starting : ")

l1 = [i*i for i in range(1000)]
l2 = np.random.randint(1,100, (1000,1000))

heap_status1 = heap.heap()

print(heap_status1)

print("\nAccessing Individual Element of Heap")

print("\nFirst Element : ")
print(heap_status1[0])

print("\nSecond Element : ")
print(heap_status1[1])

print("\nThird Element : ")
print(heap_status1.parts[2])

def convert_size(size):
    if size <1024:
        return size
    elif (size >= 1024) and (size < (1024 * 1024)):
        return "%.2f KB"%(size/1024)
    elif (size >= (1024*1024)) and (size < (1024*1024*1024)):
        return "%.2f MB"%(size/(1024*1024))
    else:
        return "%.2f GB"%(size/(1024*1024*1024))

print("\nTotal Heap Size : ", convert_size(heap_status1.size), "\n")
for i in range(10):
    print("Size Of Object : %d - "%i, convert_size(heap_status1[i].size))

OUTPUT

Heap Status At Starting :
Partition of a set of 99240 objects. Total size = 21023362 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0     69   0  8006950  38   8006950  38 numpy.ndarray
     1  27445  28  4200686  20  12207636  58 str
     2  26396  27  2002552  10  14210188  68 tuple
     3  13609  14  1021552   5  15231740  72 bytes
     4   6773   7   978968   5  16210708  77 types.CodeType
     5   6235   6   847960   4  17058668  81 function
     6    940   1   801552   4  17860220  85 type
     7    273   0   541800   3  18402020  88 dict of module
     8    940   1   509240   2  18911260  90 dict of type
     9   1202   1   482112   2  19393372  92 dict (no owner)
<248 more rows. Type e.g. '_.more' to view.>

Accessing Individual Element of Heap

First Element :
Partition of a set of 69 objects. Total size = 8006950 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0     69 100  8006950 100   8006950 100 numpy.ndarray

Second Element :
Partition of a set of 27445 objects. Total size = 4200686 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     1  27445 100  4200686 100  12207636 291 str

Third Element :
Partition of a set of 26396 objects. Total size = 2002552 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     2  26396 100  2002552 100  14210188 710 tuple

Total Heap Size :  20.05 MB

Size Of Object : 0 -  7.64 MB
Size Of Object : 1 -  4.01 MB
Size Of Object : 2 -  1.91 MB
Size Of Object : 3 -  997.61 KB
Size Of Object : 4 -  956.02 KB
Size Of Object : 5 -  828.09 KB
Size Of Object : 6 -  783.16 KB
Size Of Object : 7 -  529.10 KB
Size Of Object : 8 -  497.30 KB
Size Of Object : 9 -  470.81 KB

Example 4

As a part of our fourth example, we'll explain how we can find out the difference between two heap status to check how many objects were created between two calls of heap status.

  • diff() - This method is available as a part of the heap status object which takes as input another heap status to find out objects difference between two statuses. It returns an object of type guppy.heapy.Part.Stat which we have explored as a part of our second example.

  • disjoint() - This method is available through a heap status object that takes input another heap status and returns True/False based on whether two heap status is disjoint or not.

Below we have first taken heap status at the beginning. We have then created a few lists of objects and string object. After the creation of these objects, we have again taken another heap status. We have then called the diff() method on the second heap status object passing it first heap status object to get the difference between two screenshots of the heap. We have then looped through stats object and printed the difference of objects.

CODE

import guppy
from guppy import hpy
import numpy as np


heap = hpy()

print("Heap Status At Starting : ")
heap_status1 = heap.heap()
print("Heap Size : ", heap_status1.size, " bytes\n")
print(heap_status1)


a = [i for i in range(1000)]
b = "A"
c = np.random.randint(1,100, (1000,))

print("\nHeap Status After Creating Few Objects : ")
heap_status2 = heap.heap()
print("Heap Size : ", heap_status2.size, " bytes\n")
print(heap_status2)

print("\nMemory Usage After Creation Of Objects : ", heap_status2.size - heap_status1.size, " bytes")

print("\nFinding Out Difference Between Two Heap Status : ")
stats = heap_status2.diff(heap_status1)

print("Whether Two Heap Status Are Disjoint : ", heap_status1.disjoint(heap_status2))
print("Total Objects : ", stats.count)
print("Total Size : ", stats.size, " Bytes")
print("Number of Entries : ", stats.numrows)

print("Entries : ")

print("Index Count  Size  Cumulative Size         Object Name")
for row in stats.get_rows():
    print("%5d%5d%8d%8d%30s"% (row.index, row.count, row.size, row.cumulsize, row.name))

OUTPUT

Heap Status At Starting :
Heap Size :  12986611  bytes

Partition of a set of 98247 objects. Total size = 12986611 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  27449  28  4201065  32   4201065  32 str
     1  26394  27  2002488  15   6203553  48 tuple
     2  13607  14  1021474   8   7225027  56 bytes
     3   6772   7   978824   8   8203851  63 types.CodeType
     4   6235   6   847960   7   9051811  70 function
     5    940   1   801552   6   9853363  76 type
     6    273   0   541800   4  10395163  80 dict of module
     7    940   1   509240   4  10904403  84 dict of type
     8   1202   1   482112   4  11386515  88 dict (no owner)
     9    791   1   191880   1  11578395  89 dict of function
<248 more rows. Type e.g. '_.more' to view.>

Heap Status After Creating Few Objects :
Heap Size :  13027083  bytes

Partition of a set of 99083 objects. Total size = 13027083 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  27449  28  4201065  32   4201065  32 str
     1  26394  27  2002488  15   6203553  48 tuple
     2  13607  14  1021474   8   7225027  55 bytes
     3   6772   7   978824   8   8203851  63 types.CodeType
     4   6235   6   847960   7   9051811  69 function
     5    940   1   801552   6   9853363  76 type
     6    273   0   541800   4  10395163  80 dict of module
     7    940   1   509240   4  10904403  84 dict of type
     8   1202   1   482112   4  11386515  87 dict (no owner)
     9    791   1   191880   1  11578395  89 dict of function
<248 more rows. Type e.g. '_.more' to view.>

Memory Usage After Creation Of Objects :  40472  bytes

Finding Out Difference Between Two Heap Status :
Whether Two Heap Status Are Disjoint :  False
Total Objects :  836
Total Size :  40472  Bytes
Number of Entries :  5
Entries :
Index Count  Size  Cumulative Size         Object Name
    0  834   23352   23352                           int
    1    1    9024   32376                          list
    2    1    8096   40472                 numpy.ndarray
    3    0       0   40472          ctypes.CFunctionType
    4    0       0   40472                   numpy.int64

Example 5

As a part of this example, we'll explore a few methods available through heap status methods and attributes.

  • count - This attribute of heap status object returns an integer specifying a total number of objects present when the status was taken.
  • size - This attribute of heap status object returns an integer specifying the total size of the heap when the status was taken.
  • referents - This attribute returns another heap status object with only entry for objects which are referred to by other objects.
  • referrers - This attribute returns another heap status object with only entry for objects which are referring to other objects.
  • stat - It returns guppy.heapy.Part.Stat which can be looped to get information about the individual entry of the heap status as we had explained in our previous examples.
  • dump() - It takes as input filename to which heap status will be dumped.
  • load() - It takes as input filename from which heap status will be loaded. This method can be called over the object created by calling the hpy() method. It returns guppy.heapy.Part.Stat object of heap status present in the file.

Below we have explained the usage of the above properties and methods.

CODE

import guppy
from guppy import hpy
import numpy as np


heap = hpy()

heap_status1 = heap.heap()
print("\nFew Important Properties/Methods of Heap Status Object(%s) : "%type(heap_status1))

print("\nCount of Objects in Heap : ", heap_status1.count)
print("\nHeap Size : ", heap_status1.size, " bytes")
print("\nHeap 1st Entry : ")
print(heap_status1.parts[0])

print("\nSet of Objects Referred to By Any Objects : \n")
print(heap_status1.referents)

print("\nSet of Objects Directly Refer to Any Objects : \n")
print(heap_status1.referrers)

print("\nStat Object : ")
print(heap_status1.stat)
print("First 3 Entries : ")
print("Index Count  Size  Cumulative Size         Object Name")
for row in list(heap_status1.stat.get_rows())[:3]:
    print("%5d%5d%8d%8d%30s"% (row.index, row.count, row.size, row.cumulsize, row.name))


print("\nDumping Heap Status to a File : ")
heap_status1.dump("guppy_heap_status.out")
print(open("guppy_heap_status.out").readlines()[:5])

loaded_heap_stat = heap.load("guppy_heap_status.out")

print("First 3 Entries From Loaded File : ")
print("Index Count  Size  Cumulative Size         Object Name")
for row in list(loaded_heap_stat.get_rows())[:3]:
    print("%5d%5d%8d%8d%30s"% (row.index, row.count, row.size, row.cumulsize, row.name))

OUTPUT

Few Important Properties/Methods of Heap Status Object(<class 'guppy.heapy.UniSet.IdentitySetMulti'>) : 

Count of Objects in Heap :  98237

Heap Size :  12986124  bytes

Heap 1st Entry :
Partition of a set of 27446 objects. Total size = 4200962 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  27446 100  4200962 100   4200962 100 str

Set of Objects Referred to By Any Objects :

Partition of a set of 98230 objects. Total size = 12980396 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  27446  28  4200962  32   4200962  32 str
     1  26392  27  2002320  15   6203282  48 tuple
     2  13605  14  1021458   8   7224740  56 bytes
     3   6771   7   978680   8   8203420  63 types.CodeType
     4   6236   6   848096   7   9051516  70 function
     5    940   1   801960   6   9853476  76 type
     6    273   0   541800   4  10395276  80 dict of module
     7    940   1   509240   4  10904516  84 dict of type
     8   1198   1   476560   4  11381076  88 dict (no owner)
     9    791   1   191880   1  11572956  89 dict of function
<248 more rows. Type e.g. '_.more' to view.>

Set of Objects Directly Refer to Any Objects :

Partition of a set of 52393 objects. Total size = 7348954 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  26396  50  2002832  27   2002832  27 tuple
     1   6772  13   978824  13   2981656  41 types.CodeType
     2   6236  12   848096  12   3829752  52 function
     3    941   2   803016  11   4632768  63 type
     4    273   1   541800   7   5174568  70 dict of module
     5    940   2   509240   7   5683808  77 dict of type
     6    810   2   388032   5   6071840  83 dict (no owner)
     7   2113   4   169040   2   6240880  85 types.WrapperDescriptorType
     8    454   1   111000   2   6351880  86 dict of function
     9     49   0    98528   1   6450408  88 set
<219 more rows. Type e.g. '_.more' to view.>

Stat Object :
<guppy.heapy.Part.Stat object at 0x7fb7db895ba8>
First 3 Entries :
Index Count  Size  Cumulative Size         Object Name
    027446 4200962 4200962                           str
    126392 2002320 6203282                         tuple
    213605 1021458 7224740                         bytes

Dumping Heap Status to a File :
['.loader: _load_stat\n', '.format: SetFormat\n', '.timemade: 1603777688.456192\n', '.count: 98230\n', '.size: 12985525\n']
First 3 Entries From Loaded File :
Index Count  Size  Cumulative Size         Object Name
    027439 4200493 4200493                           str
    126392 2002296 6202789                         tuple
    213605 1021352 7224141                         bytes

Example 6

As a part of the sixth example, we have explained various attributes available through heap status object which lets us group heap status entries based on different attributes like, type, size, referrers, memory address, etc.

  • bytype - This attribute of heap status groups heap status entries by type of object. All dict entries will be combined into one entry.
  • byrcs - This attribute of heap status groups heap status entries by type of referrers.
  • bymodule - This attribute of heap status groups heap status entries by the module.
  • bysize - This attribute of heap status groups heap status entries by the individual size of object.
  • byunity - This attribute of heap status groups heap status entries by total size.
  • byvia -This attribute of heap status groups heap status entries by objects via which they are referred.
  • byidset - This attribute of heap status groups heap status entries by idset.
  • byid - This attribute of heap status groups heap status entries by memory address.

Below we have explained all attributes mentioned above one by one. We can easily see results to see how objects are grouped by.

CODE

from guppy import hpy

heap = hpy()
heap_status1 = heap.heap()

print("=========== Heap Status At Starting : ============")
print(heap_status1)

print("\n============ Heap Status Grouped By Type : ==========")
print(heap_status1.bytype)

print("\n==== Heap Status Grouped By Referrers of kind(class/dict of class) : ===")
print(heap_status1.byrcs)

print("\n========== Heap Status Grouped By Module : ============")
print(heap_status1.bymodule)

print("\n========== Heap Status Grouped By Individual Size : ==============")
print(heap_status1.bysize)

print("\n========= Heap Status Grouped By Total Size : ============")
print(heap_status1.byunity)

print("\n======= Heap Status Grouped By Referred Via : =============")
print(heap_status1.byvia)

print("\n========= Heap Status Grouped By IDset : ============")
print(heap_status1.byidset)

print("\n======== Heap Status Grouped By Address : ==========")
print(heap_status1.byid)

OUTPUT

=========== Heap Status At Starting : ============
Partition of a set of 38336 objects. Total size = 4307591 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  11218  29   988030  23    988030  23 str
     1   9449  25   688544  16   1676574  39 tuple
     2   2472   6   357512   8   2034086  47 types.CodeType
     3   4922  13   343021   8   2377107  55 bytes
     4    444   1   336552   8   2713659  63 type
     5   2282   6   310352   7   3024011  70 function
     6    444   1   244808   6   3268819  76 dict of type
     7    101   0   174656   4   3443475  80 dict of module
     8    253   1   117584   3   3561059  83 dict (no owner)
     9   1090   3    87200   2   3648259  85 types.WrapperDescriptorType
<116 more rows. Type e.g. '_.more' to view.>

============ Heap Status Grouped By Type : ==========
Partition of a set of 38336 objects. Total size = 4307591 bytes.
 Index  Count   %     Size   % Cumulative  % Type
     0  11218  29   988030  23    988030  23 str
     1   9449  25   688544  16   1676574  39 tuple
     2   1301   3   639080  15   2315654  54 dict
     3   2472   6   357512   8   2673166  62 types.CodeType
     4   4922  13   343021   8   3016187  70 bytes
     5    444   1   336552   8   3352739  78 type
     6   2282   6   310352   7   3663091  85 function
     7   1090   3    87200   2   3750291  87 types.WrapperDescriptorType
     8     41   0    73184   2   3823475  89 set
     9     64   0    63344   1   3886819  90 abc.ABCMeta
<78 more rows. Type e.g. '_.more' to view.>

==== Heap Status Grouped By Referrers of kind(class/dict of class) : ===
Partition of a set of 38336 objects. Total size = 4307847 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  12186  32   891421  21    891421  21 types.CodeType
     1   4454  12   491986  11   1383407  32 function
     2   4396  11   480762  11   1864169  43 dict of type
     3   1445   4   337424   8   2201593  51 type
     4   4573  12   312848   7   2514441  58 tuple
     5   1422   4   240269   6   2754710  64 dict of module
     6    651   2   160561   4   2915271  68 function, tuple
     7   1645   4   107779   3   3023050  70 set
     8     53   0   101968   2   3125018  73 function, module
     9     91   0    98608   2   3223626  75 dict of module, tuple, types.GetSetDescriptorType
<420 more rows. Type e.g. '_.more' to view.>

========== Heap Status Grouped By Module : ============
Partition of a set of 38336 objects. Total size = 4308103 bytes.
 Index  Count   %     Size   % Cumulative  % Module
     0  38235 100  4300023 100   4300023 100 ~module
     1      1   0       80   0   4300103 100 __main__
     2      1   0       80   0   4300183 100 _abc
     3      1   0       80   0   4300263 100 _bootlocale
     4      1   0       80   0   4300343 100 _codecs
     5      1   0       80   0   4300423 100 _collections
     6      1   0       80   0   4300503 100 _ctypes
     7      1   0       80   0   4300583 100 _functools
     8      1   0       80   0   4300663 100 _heapq
     9      1   0       80   0   4300743 100 _imp
<92 more rows. Type e.g. '_.more' to view.>

========== Heap Status Grouped By Individual Size : ==============
Partition of a set of 38336 objects. Total size = 4308103 bytes.
 Index  Count   %     Size   % Cumulative  % Individual Size
     0   2427   6   349488   8    349488   8       144
     1   2357   6   320552   7    670040  16       136
     2   3735  10   268920   6    938960  22        72
     3    248   1   261888   6   1200848  28      1056
     4   4281  11   239736   6   1440584  33        56
     5   2735   7   218800   5   1659384  39        80
     6   2738   7   175232   4   1834616  43        64
     7    108   0   127872   3   1962488  46      1184
     8    415   1    99600   2   2062088  48       240
     9    268   1    98624   2   2160712  50       368
<577 more rows. Type e.g. '_.more' to view.>

========= Heap Status Grouped By Total Size : ============
Partition of a set of 38336 objects. Total size = 4308103 bytes.
 Index  Count   %     Size   % Cumulative  % <unclassified>
     0  38336 100  4308103 100   4308103 100 <Anything>

======= Heap Status Grouped By Referred Via : =============
Partition of a set of 38336 objects. Total size = 4308231 bytes.
 Index  Count   %     Size   % Cumulative  % Referred Via:
     0   1021   3   423524  10    423524  10 '.__dict__'
     1   2169   6   313800   7    737324  17 '.__code__'
     2   2472   6   230886   5    968210  22 '.co_code'
     3   2219   6   187992   4   1156202  27 '.co_names'
     4   2436   6   179552   4   1335754  31 '.co_varnames'
     5   2389   6   166768   4   1502522  35 '.co_consts'
     6    628   2   157587   4   1660109  39 '.__doc__', '[0]'
     7    402   1   142935   3   1803044  42 "['__doc__']"
     8   1725   4   121048   3   1924092  45 '.__qualname__'
     9   1661   4   108579   3   2032671  47 'list(_)[0]'
<6455 more rows. Type e.g. '_.more' to view.>

========= Heap Status Grouped By IDset : ============
Partition of a set of 38336 objects. Total size = 4308359 bytes.
 Index  Count   %     Size   % Cumulative  % by identity set
     0      1   0    32992   1     32992   1 <1 set: 0x7fe8a7f38e48>
     1      1   0    18528   0     51520   1 <1 dict of module: os>
     2      1   0     9320   0     60840   1 <1 dict (no owner): 0x7fe8a93721f8*323>
     3      1   0     9320   0     70160   2 <1 dict (no owner): 0x7fe8a941dca8*288>
     4      1   0     9320   0     79480   2 <1 dict of module: posix>
     5      1   0     8416   0     87896   2 <1 set: 0x7fe8a93e09e8>
     6      1   0     8416   0     96312   2 <1 set: 0x7fe8a93e0c88>
     7      1   0     7423   0    103735   2 <1 str: 'The class Bi... copy of S.\n'>
     8      1   0     6036   0    109771   3 <1 str: "Support for ... 'error'.\n\n">
     9      1   0     4791   0    114562   3 <1 str: 'Heap queues\...at Art! :-)\n'>
<38326 more rows. Type e.g. '_.more' to view.>

======== Heap Status Grouped By Address : ==========
Set of 38336 <mixed> objects. Total size = 4308359 bytes.
 Index     Size   %   Cumulative  %   Brief
     0    32992   0.8     32992   0.8 set: 0x7fe8a7f38e48
     1    18528   0.4     51520   1.2 dict of module: os
     2     9320   0.2     60840   1.4 dict (no owner): 0x7fe8a93721f8*323
     3     9320   0.2     70160   1.6 dict (no owner): 0x7fe8a941dca8*295
     4     9320   0.2     79480   1.8 dict of module: posix
     5     8416   0.2     87896   2.0 set: 0x7fe8a93e09e8
     6     8416   0.2     96312   2.2 set: 0x7fe8a93e0c88
     7     7423   0.2    103735   2.4 str: 'The class Bi... copy of S.\n'
     8     6036   0.1    109771   2.5 str: "Support for ... 'error'.\n\n"
     9     4791   0.1    114562   2.7 str: 'Heap queues\...at Art! :-)\n'
<38326 more rows. Type e.g. '_.more' to view.>

Example 7

As a part of this example, we'll explain two useful methods available through the heap object.

  • iso() - It takes as input single object or multiple objects and returns status specifying object size. It considers complex objects like list, dict as one object.
  • idset() - It takes as input single object or multiple objects and returns status specifying object size. It considers complex objects like list, dict as a list of the individual object.

Please feel free to go through the below example to see the difference between the results of the two methods.

CODE

import guppy
from guppy import hpy
import numpy as np


heap = hpy()

a = [i for i in range(1000000)]
b = "A"
c = "WORLD"


print("============== ISO Method Examples ====================")
print()
print(heap.iso(a))
print()
print(heap.iso(b))
print()
print(heap.iso(c))
print()
print(heap.iso([]))
print()
print(heap.iso(""))

print("\n============== IDSET Method Examples ===================")
print()
print(heap.idset(a))
print()
print(heap.idset(b))
print()
print(heap.idset(c))
print()
print(heap.idset([]))
print()
print(heap.idset(""))

OUTPUT

============== ISO Method Examples ====================

Partition of a set of 1 object. Total size = 8697464 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100  8697464 100   8697464 100 list

Partition of a set of 1 object. Total size = 50 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100       50 100        50 100 str

Partition of a set of 1 object. Total size = 54 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100       54 100        54 100 str

Partition of a set of 1 object. Total size = 64 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100       64 100        64 100 list

Partition of a set of 1 object. Total size = 49 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100       49 100        49 100 str

============== IDSET Method Examples ===================

Partition of a set of 1000000 objects. Total size = 27999996 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0 1000000 100 27999996 100  27999996 100 int

Partition of a set of 1 object. Total size = 50 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100       50 100        50 100 str

Partition of a set of 5 objects. Total size = 250 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      5 100      250 100       250 100 str

<Nothing>

<Nothing>

Example 8

As a part of our eighth and last example, we have demonstrated how we can check doc of particular method/attributes of objects available through guppy.

Below we have accessed the document of the heap by calling the doc attribute of a heap object. It'll show documentation for the whole heap object listing all methods/attributes available through that object. We have also accessed the document of individual method/attribute by calling that method attributes name on the doc attribute as explained below.

We have also accessed documentation of the heap status object. Please feel free to go through the below example to check how we can access docs of guppy objects.

CODE

from guppy import hpy

heap = hpy()

heap_status1 = heap.heap()

print("============== Heap Documents ====================")
print(heap.doc)

print("Doc On Individual Method/Function/Class : ")
print("\nISO Method Doc : \n")
print(heap.doc.iso)

print("\nRoot Class Doc : \n")
print(heap.doc.Root)


print("============= Heap Status Documents ================")
print(heap_status1.doc)

print("Doc On Individual Method/Function/Class : ")

print("\nDoc on parts Property of Heap Status : \n")
print(heap_status1.doc.parts)
print("\nDoc on byrcs Property of Heap Status : \n")
print(heap_status1.doc.byrcs)

OUTPUT

============== Heap Documents ====================
Top level interface to Heapy. Available attributes:
Anything            Prod                Via                 iso
Clodo               Rcs                 doc                 load
Id                  Root                findex              monitor
Idset               Size                heap                pb
Module              Type                heapu               setref
Nothing             Unity               idset               test
Use eg: heap.doc.<attribute> for info on <attribute>.
Doc On Individual Method/Function/Class :

ISO Method Doc :

iso( 0..*:node: Any+) -> IdentitySet[1]

Create a set of objects based on identity.

Argument
    node: Any+
        Any kind of objects are valid arguments.
Note
    This method is the same as idset[2] except for the argument.
References
    [0] heapy_Use.html#heapykinds.Use.iso
    [1] heapy_UniSet.html#heapykinds.IdentitySet
    [2] heapy_Use.html#heapykinds.Use.idset

Root Class Doc :

Root: RootStateType

This attribute is a symbolic root containing attributes from which all
reachable objects in the heap can be reached. It is the only value (a
singleton) of its kind; see [1] for a description of its attributes.

References
        [0] heapy_Use.html#heapykinds.Use.Root
        [1] heapy_RootState.html#heapykinds.RootStateType
============= Heap Status Documents ================
all                 byvia               get_render          pathsout
biper               count               get_rp              prod
brief               dictof              get_shpaths         referents
by                  diff                imdom               referrers
byclodo             disjoint            indisize            rp
byid                doc                 kind                shpaths
byidset             dominos             maprox              size
bymodule            domisize            more                sp
byprod              dump                nodes               stat
byrcs               er                  owners              test_contains
bysize              fam                 partition           theone
bytype              get_ckc             parts
byunity             get_examples        pathsin
Doc On Individual Method/Function/Class :

Doc on parts Property of Heap Status :

An iterable object, that can be used to iterate over the 'parts' of
self. The iteration order is determined by the sorting order the set
has, in the table printed when partitioned.

Doc on byrcs Property of Heap Status :

A copy of self, but with 'Rcs' as the equivalence relation.


Sunny Solanki  Sunny Solanki