Updated On : Oct-28,2020 Tags tracemalloc, memory-tracing
tracemalloc - How to Trace Memory Usage in Python Code

tracemalloc - How to Trace Memory Usage in Python Code

The tracemalloc module which became available from Python version 3.4. The tracemalloc module allows us to monitor memory allocations in the python code. It lets us take memory snapshots at a particular point, perform various statistics on snapshots as well as perform difference between two snapshots to check object allocation between two snapshots.

As a part of this tutorial, we'll explain how we can use tracemalloc API to trace memory usage in python code and perform various operations. We'll be explaining the usage of various classes, methods, and attributes available through the module with the help of various examples.

If you are interested in learning about other python profilers then please feel free to check our references section which has a list of tutorials on python profilers.

Example 1

As a part of the first example, we'll simply explain how to start tracing memory using tracemalloc, take snapshots, and print traced output.

Below is a list of methods which we'll explain as a part of this example.

  • tracemalloc.start() - This method is available from tracemalloc module calling which will start tracing of memory. It accepts an integer argument named nframe which mentions a number of frames to allocate per call. The default value is 1.
  • tracemalloc.take_snapshot() - This method is available from the tracemalloc module which takes memory snapshot when it’s called and returns theSnapshot object.
  • Snapshot.statistics() - This method is available through the Snapshot object and takes as input key using which to sort records of tracing. It returns an iterator that has a list of Statistic objects. The single Statistic object has information about a single traceback (single line of code generally) which includes a number of objects and size of objects recorded in that traceback. Below is a list of keys that this method takes as input.
    • filename - Sort traces according to file names
    • lineno - sort traces according to line no in the file.
    • traceback - Sort traces according to trace taken order.

Below we have introduced the above-mentioned method with a simple example. We are starting tracing at the beginning and then creating three lists of integers. We have then taken a memory snapshot and printed a list of tracebacks collected by that snapshot from the starting of the traceback.

CODE

import tracemalloc

tracemalloc.start()

l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]

snapshot = tracemalloc.take_snapshot()

for stat in snapshot.statistics("lineno"):
    print(stat)

OUTPUT

tracemalloc_ex1.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex1.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex1.py:5: size=352 KiB, count=9745, average=37 B

Example 2

As a part of our second example, we'll explain a few attributes and methods of tracemalloc and statistic objects.

  • Statistic.traceback - The traceback attribute of statistic object returns an object of class Traceback. The Traceback has information about individual trace which can have more than one frame (Frame).
  • Traceback.format() - This method formats traceback as a list of lines that were traced for this traceback.
  • tracemalloc.get_traceback_limit() - It returns an integer specifying maximum number of frames stored in single traceback. We can specify how many maximum frames to record per traceback in tracemalloc.start() method.
  • tracemalloc.get_object_traceback() - This method takes as input individual object and returns traceback object specifying where traceback for this object was taken.
  • tracemalloc.get_traced_memory() - It returns tuple of two integer specifying current size and peak size of block traced in bytes.
  • tracemalloc.get_tracemalloc_memory() - It returns integer specifying memory used by tracemalloc module in bytes.
  • tracemalloc.is_tracing() - It returns boolean specifying whether tracing is happening or stopped.

Below we have explained the usage of the above-mentioned method through example.

CODE

import tracemalloc
import numpy as np

tracemalloc.start(25)

l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]
l4 = np.random.randint(1,100, (1000,))

snapshot = tracemalloc.take_snapshot()

print("========== SNAPSHOT =============")
for stat in snapshot.statistics("lineno"):
    print(stat)
    print(stat.traceback.format())

print("\n=========== USEFUL METHODS ===========")
print("\nTraceback Limit : ", tracemalloc.get_traceback_limit(), " Frames")

print("\nAllocation Location for List l4 : ",  tracemalloc.get_object_traceback(l4))

print("\nTraced Memory (Current, Peak): ", tracemalloc.get_traced_memory())

#tracemalloc.reset_peak()

#print("\nTraced Memory : ", tracemalloc.get_traced_memory())

print("\nMemory Usage by tracemalloc Module : ", tracemalloc.get_tracemalloc_memory(), " bytes")

print("\nTracing Status : ",tracemalloc.is_tracing())

OUTPUT

========== SNAPSHOT =============
tracemalloc_ex2.py:8: size=394 KiB, count=9994, average=40 B
['  File "tracemalloc_ex2.py", line 8', '    l3 = [i*i*i for i in range(10000)]']
tracemalloc_ex2.py:7: size=358 KiB, count=9984, average=37 B
['  File "tracemalloc_ex2.py", line 7', '    l2 = [i*i for i in range(10000)]']
tracemalloc_ex2.py:6: size=352 KiB, count=9744, average=37 B
['  File "tracemalloc_ex2.py", line 6', '    l1 = [i for i in range(10000)]']
tracemalloc_ex2.py:9: size=8138 B, count=3, average=2713 B
['  File "tracemalloc_ex2.py", line 9', '    l4 = np.random.randint(1,100, (1000,))']
<__array_function__ internals>:6: size=536 B, count=1, average=536 B
['  File "<__array_function__ internals>", line 6']
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:2911: size=528 B, count=1, average=528 B
['  File "/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2911', '    keepdims=keepdims, initial=initial, where=where)']
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/_dtype.py:334: size=488 B, count=1, average=488 B
['  File "/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/_dtype.py", line 334', '    if np.issubdtype(dtype, np.flexible) and not _isunsized(dtype):']
<__array_function__ internals>:4: size=464 B, count=1, average=464 B
['  File "<__array_function__ internals>", line 4']
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:74: size=448 B, count=1, average=448 B
['  File "/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 74', '    passkwargs = {k: v for k, v in kwargs.items()']
tracemalloc_ex2.py:11: size=432 B, count=1, average=432 B
['  File "tracemalloc_ex2.py", line 11', '    snapshot = tracemalloc.take_snapshot()']
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:90: size=145 B, count=2, average=72 B
['  File "/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 90', '    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)']

=========== USEFUL METHODS ===========

Traceback Limit :  25  Frames

Allocation Location for List l4 :  tracemalloc_ex2.py:9

Traced Memory (Current, Peak):  (1472281, 1487679)

Memory Usage by tracemalloc Module :  2504012  bytes

Tracing Status :  True

Example 3

As a part of this example, we have simply explained how we can start/stop tracemalloc and clear traces.

  • tracemalloc.stop() - This method stops tracemalloc from taking traces.
  • tracemalloc.clear_traces() - This method clears all traces taken from the beginning of the start() method. Please make a note that traces info would still be available for snapshot objects which were taken but it won't be available if you call the take_snapshot() method after this method.

Below we have first taken a snapshot after creating three lists of integers. We have then created another list and taken snapshots again. We have then cleared all traces which are traces of 4 lists created since the beginning. We have then created the fifth list and taken snapshot again. Please make a note from the output that the 3rd snapshot has information only about the fifth list created and all traces before it is cleared. We have also explained the usage of stop() at the end.

CODE

import tracemalloc

tracemalloc.start()

l1 = [i for i in range(1000000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]

snapshot1 = tracemalloc.take_snapshot()

print("================ SNAPSHOT 1 =================")
for stat in snapshot1.statistics("lineno"):
    print(stat)

l4 = [i*i*i*i for i in range(10000)]



snapshot2 = tracemalloc.take_snapshot()

print("\n================ SNAPSHOT 2 =================")
for stat in snapshot2.statistics("lineno"):
    print(stat)

tracemalloc.clear_traces()

l5 = [i*2 for i in range(10000)]

snapshot3 = tracemalloc.take_snapshot()

print("\n================ SNAPSHOT 3 =================")
for stat in snapshot3.statistics("lineno"):
    print(stat)


print("\nTracing Status : ", tracemalloc.is_tracing())

tracemalloc.stop()

print("\nTracing Status : ", tracemalloc.is_tracing())

try:
    print("\nTrying to Take Snapshot After Tracing is Stopped.")
    snap = tracemalloc.take_snapshot()
except Exception as e:
    print("Exception : ", e)

OUTPUT

================ SNAPSHOT 1 =================
tracemalloc_ex3.py:5: size=35.0 MiB, count=999745, average=37 B
tracemalloc_ex3.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex3.py:6: size=358 KiB, count=9984, average=37 B

================ SNAPSHOT 2 =================
tracemalloc_ex3.py:5: size=35.0 MiB, count=999745, average=37 B
tracemalloc_ex3.py:15: size=433 KiB, count=9997, average=44 B
tracemalloc_ex3.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex3.py:6: size=358 KiB, count=9984, average=37 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:209: size=864 B, count=2, average=432 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:165: size=832 B, count=2, average=416 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:397: size=656 B, count=5, average=131 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:479: size=576 B, count=3, average=192 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:507: size=576 B, count=1, average=576 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:534: size=552 B, count=3, average=184 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:472: size=480 B, count=2, average=240 B
tracemalloc_ex3.py:12: size=464 B, count=1, average=464 B
tracemalloc_ex3.py:13: size=456 B, count=1, average=456 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:53: size=448 B, count=1, average=448 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:509: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:194: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:54: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:475: size=416 B, count=1, average=416 B
tracemalloc_ex3.py:11: size=314 B, count=5, average=63 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:459: size=240 B, count=2, average=120 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:458: size=240 B, count=2, average=120 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:185: size=168 B, count=3, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:65: size=144 B, count=2, average=72 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:291: size=112 B, count=2, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:508: size=64 B, count=1, average=64 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:289: size=56 B, count=1, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:180: size=56 B, count=1, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:477: size=28 B, count=1, average=28 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:476: size=28 B, count=1, average=28 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:27: size=24 B, count=1, average=24 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:15: size=24 B, count=1, average=24 B

================ SNAPSHOT 3 =================
tracemalloc_ex3.py:27: size=356 KiB, count=9873, average=37 B

Tracing Status :  True

Tracing Status :  False

Trying to Take Snapshot After Tracing is Stopped.
Exception :  the tracemalloc module must be tracing memory allocations to take a snapshot

Example 4

As a part of our fourth example, we'll explain how we can store a snapshot of traces into a file on disk and then load it again.

  • Snapshot.dump() - This method takes as input filename to which store traces information.
  • Snapshot.load() - This method takes as input filename from which load traces information.

Below we have explained the usage of both methods.

CODE

import tracemalloc

tracemalloc.start()

l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]

snapshot = tracemalloc.take_snapshot()

for stat in snapshot.statistics("lineno"):
    print(stat)

snapshot.dump("snap.out")

snapshot_loaded = tracemalloc.Snapshot.load("snap.out")

print("\nLoaded Snapshot From File :")
for stat in snapshot_loaded.statistics("lineno"):
    print(stat)

OUTPUT

tracemalloc_ex4.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex4.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex4.py:5: size=352 KiB, count=9745, average=37 B

Loaded Snapshot From File :
tracemalloc_ex4.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex4.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex4.py:5: size=352 KiB, count=9745, average=37 B

Example 5

As a part of this example, we'll explain how we can filter out traces from a list of all traces recorded by the tracemalloc. There are two main classes provided by tracemalloc for filtering traces.

  • DomainFilter - This class lets us filter traces by address space of objects. It takes two parameters as input.

    • domain - It accepts integer specifying address space of memory block. The integer 0 is generally used by python memory traces but the number will be different for C.
    • inclusive - It accepts boolean specifying whether to include or exclude traces specified by the domain attribute.

  • Filter - This class lets filter traces based on combinations of the filename, line number, and domain. It has the below-mentioned parameters.

    • domain - It accepts integer specifying address space of memory block. The integer 0 is generally used by python memory traces but the number will be different for C.
    • filename_pattern - It accepts string specifying pattern for which filenames will be matched in traces and based on the inclusive flag they will be included or excluded.
    • lineno - It accepts integer specifying traces with that line numbers to include.
      • inclusive - It accepts boolean specifying whether to include or exclude traces specified by domain, filename_pattern, and lineno attributes.
    • all_frames - It accepts a boolean flag specifying whether to include all frames per traces or not.

We have explained the usage of filters through the examples below. The first snapshot includes all traces for memory blocks used by python. The second snapshot includes traces of memory blocks that are not created by python. It can be due to C as numpy is built on it. The third snapshot includes entries where the filename is tracemalloc_ex5.py and the fourth snapshot excludes entries with that file names.

CODE

import tracemalloc
import numpy as np

tracemalloc.start(10)

l2 = [i*i for i in range(10000)]
l1 = [i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]
l4 = [i*2 for i in range(10000)]
l5 = np.random.randint(1,100, (1000,))

snapshot = tracemalloc.take_snapshot()

print("========== Original Snapshot ===========")
for stat in snapshot.statistics("lineno"):
    print(stat)

print("\n========= Filtered Snapshot 1 =============")
filtr = tracemalloc.DomainFilter(inclusive=True, domain=0)
snap_shot = snapshot.filter_traces(filters=[filtr])

for stat in snap_shot.statistics("lineno"):
    print(stat)


filtr = tracemalloc.DomainFilter(inclusive=False, domain=0)
snap_shot = snapshot.filter_traces(filters=[filtr])

print("\n========= Filtered Snapshot 2 =============")
for stat in snap_shot.statistics("lineno"):
    print(stat)

filtr = tracemalloc.Filter(inclusive=True, filename_pattern="tracemalloc_ex5.py")
snap_shot = snapshot.filter_traces([filtr])

print("\n========= Filtered Snapshot 3 =============")
for stat in snap_shot.statistics("lineno"):
    print(stat)


filtr = tracemalloc.Filter(inclusive=False, filename_pattern="tracemalloc_ex5.py")
snap_shot = snapshot.filter_traces([filtr])

print("\n========= Filtered Snapshot 4 =============")
for stat in snap_shot.statistics("lineno"):
    print(stat)

OUTPUT

========== Original Snapshot ===========
tracemalloc_ex5.py:8: size=394 KiB, count=9994, average=40 B
tracemalloc_ex5.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex5.py:9: size=355 KiB, count=9872, average=37 B
tracemalloc_ex5.py:7: size=352 KiB, count=9744, average=37 B
tracemalloc_ex5.py:10: size=9050 B, count=5, average=1810 B
<__array_function__ internals>:6: size=536 B, count=1, average=536 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:2911: size=528 B, count=1, average=528 B
<__array_function__ internals>:4: size=464 B, count=1, average=464 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:74: size=448 B, count=1, average=448 B
tracemalloc_ex5.py:12: size=432 B, count=1, average=432 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:90: size=145 B, count=2, average=72 B

========= Filtered Snapshot 1 =============
tracemalloc_ex5.py:8: size=394 KiB, count=9994, average=40 B
tracemalloc_ex5.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex5.py:9: size=355 KiB, count=9872, average=37 B
tracemalloc_ex5.py:7: size=352 KiB, count=9744, average=37 B
tracemalloc_ex5.py:10: size=1050 B, count=4, average=262 B
<__array_function__ internals>:6: size=536 B, count=1, average=536 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:2911: size=528 B, count=1, average=528 B
<__array_function__ internals>:4: size=464 B, count=1, average=464 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:74: size=448 B, count=1, average=448 B
tracemalloc_ex5.py:12: size=432 B, count=1, average=432 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:90: size=145 B, count=2, average=72 B

========= Filtered Snapshot 2 =============
tracemalloc_ex5.py:10: size=8000 B, count=1, average=8000 B

========= Filtered Snapshot 3 =============
tracemalloc_ex5.py:8: size=394 KiB, count=9994, average=40 B
tracemalloc_ex5.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex5.py:9: size=355 KiB, count=9872, average=37 B
tracemalloc_ex5.py:7: size=352 KiB, count=9744, average=37 B
tracemalloc_ex5.py:10: size=9050 B, count=5, average=1810 B
tracemalloc_ex5.py:12: size=432 B, count=1, average=432 B

========= Filtered Snapshot 4 =============
<__array_function__ internals>:6: size=536 B, count=1, average=536 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:2911: size=528 B, count=1, average=528 B
<__array_function__ internals>:4: size=464 B, count=1, average=464 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:74: size=448 B, count=1, average=448 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:90: size=145 B, count=2, average=72 B

Example 6

As a part of this example, we'll explain how we can compare two snapshots and find out the difference in traces between them.

  • Snapshot.compare_to() - This method can be called on any snapshot object passing it another snapshot object to get the difference between two snapshots. It also takes as input the key_type parameter which will be used as the primary key to find out the difference between two snapshots. It'll return an object of class StatisticDiff. This object has information about differences.

Below we have explained through example, how we can get the difference between two snapshots. Please make a note that different traces taken before the first snapshot are not present.

CODE

import tracemalloc

tracemalloc.start()

l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]

snapshot1 = tracemalloc.take_snapshot()

print("SNAPSHOT - 1")
for stat in snapshot1.statistics("lineno"):
    print(stat)


l4 = [i*i*i*i for i in range(10000)]
l5 = [i*i*i*i*i for i in range(10000)]

print("\nSNAPSHOT - 2")
snapshot2 = tracemalloc.take_snapshot()

for stat in snapshot2.statistics("lineno"):
    #if "tracemalloc_ex6.py" in str(stat):
    print(stat)

print("\nDIFFERENCE")
for diff in snapshot2.compare_to(snapshot1,"lineno"):
    #if "tracemalloc_ex6.py" in str(diff):
    print(diff)

OUTPUT

SNAPSHOT - 1
tracemalloc_ex6.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex6.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex6.py:5: size=352 KiB, count=9745, average=37 B

SNAPSHOT - 2
tracemalloc_ex6.py:17: size=436 KiB, count=9998, average=45 B
tracemalloc_ex6.py:16: size=433 KiB, count=9997, average=44 B
tracemalloc_ex6.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex6.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex6.py:5: size=352 KiB, count=9745, average=37 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:209: size=864 B, count=2, average=432 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:165: size=832 B, count=2, average=416 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:479: size=576 B, count=3, average=192 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:507: size=576 B, count=1, average=576 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:397: size=528 B, count=3, average=176 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:472: size=480 B, count=2, average=240 B
tracemalloc_ex6.py:12: size=464 B, count=1, average=464 B
tracemalloc_ex6.py:13: size=456 B, count=1, average=456 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:53: size=448 B, count=1, average=448 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:509: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:194: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:54: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:475: size=416 B, count=1, average=416 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:65: size=144 B, count=2, average=72 B
tracemalloc_ex6.py:11: size=106 B, count=2, average=53 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:508: size=64 B, count=1, average=64 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:534: size=56 B, count=1, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:185: size=56 B, count=1, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:291: size=40 B, count=1, average=40 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:477: size=28 B, count=1, average=28 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:476: size=28 B, count=1, average=28 B

DIFFERENCE
tracemalloc_ex6.py:17: size=436 KiB (+436 KiB), count=9998 (+9998), average=45 B
tracemalloc_ex6.py:16: size=433 KiB (+433 KiB), count=9997 (+9997), average=44 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:209: size=864 B (+864 B), count=2 (+2), average=432 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:165: size=832 B (+832 B), count=2 (+2), average=416 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:479: size=576 B (+576 B), count=3 (+3), average=192 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:507: size=576 B (+576 B), count=1 (+1), average=576 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:397: size=528 B (+528 B), count=3 (+3), average=176 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:472: size=480 B (+480 B), count=2 (+2), average=240 B
tracemalloc_ex6.py:12: size=464 B (+464 B), count=1 (+1), average=464 B
tracemalloc_ex6.py:13: size=456 B (+456 B), count=1 (+1), average=456 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:53: size=448 B (+448 B), count=1 (+1), average=448 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:509: size=424 B (+424 B), count=1 (+1), average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:194: size=424 B (+424 B), count=1 (+1), average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:54: size=424 B (+424 B), count=1 (+1), average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:475: size=416 B (+416 B), count=1 (+1), average=416 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:65: size=144 B (+144 B), count=2 (+2), average=72 B
tracemalloc_ex6.py:11: size=106 B (+106 B), count=2 (+2), average=53 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:508: size=64 B (+64 B), count=1 (+1), average=64 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:534: size=56 B (+56 B), count=1 (+1), average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:185: size=56 B (+56 B), count=1 (+1), average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:291: size=40 B (+40 B), count=1 (+1), average=40 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:477: size=28 B (+28 B), count=1 (+1), average=28 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:476: size=28 B (+28 B), count=1 (+1), average=28 B
tracemalloc_ex6.py:7: size=394 KiB (+0 B), count=9994 (+0), average=40 B
tracemalloc_ex6.py:6: size=358 KiB (+0 B), count=9984 (+0), average=37 B
tracemalloc_ex6.py:5: size=352 KiB (+0 B), count=9745 (+0), average=37 B

Example 7

Our seventh example is exactly the same as our sixth example with the only difference that we have used a filter to remove entry related to the tracemalloc module itself.

We have specified tracemalloc filename (tracemalloc.__file__) to filename_pattern of Filter class with the inclusive attribute as False to exclude entries with tracemalloc module.

CODE

import tracemalloc

tracemalloc.start()

l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]

snapshot1 = tracemalloc.take_snapshot()

print("SNAPSHOT - 1")
for stat in snapshot1.statistics("lineno"):
    print(stat)


l4 = [i*i*i*i for i in range(10000)]
l5 = [i*i*i*i*i for i in range(10000)]

print("\nSNAPSHOT - 2")
snapshot2 = tracemalloc.take_snapshot()
filtr = tracemalloc.Filter(inclusive=False, filename_pattern=tracemalloc.__file__)
snapshot2 = snapshot2.filter_traces([filtr])

for stat in snapshot2.statistics("lineno"):
        print(stat)

print("\nDIFFERENCE")
for diff in snapshot2.compare_to(snapshot1,"lineno"):
        print(diff)

OUTPUT

SNAPSHOT - 1
tracemalloc_ex7.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex7.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex7.py:5: size=352 KiB, count=9745, average=37 B

SNAPSHOT - 2
tracemalloc_ex7.py:17: size=436 KiB, count=9998, average=45 B
tracemalloc_ex7.py:16: size=433 KiB, count=9997, average=44 B
tracemalloc_ex7.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex7.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex7.py:5: size=352 KiB, count=9745, average=37 B
tracemalloc_ex7.py:12: size=464 B, count=1, average=464 B
tracemalloc_ex7.py:13: size=456 B, count=1, average=456 B
tracemalloc_ex7.py:11: size=106 B, count=2, average=53 B

DIFFERENCE
tracemalloc_ex7.py:17: size=436 KiB (+436 KiB), count=9998 (+9998), average=45 B
tracemalloc_ex7.py:16: size=433 KiB (+433 KiB), count=9997 (+9997), average=44 B
tracemalloc_ex7.py:12: size=464 B (+464 B), count=1 (+1), average=464 B
tracemalloc_ex7.py:13: size=456 B (+456 B), count=1 (+1), average=456 B
tracemalloc_ex7.py:11: size=106 B (+106 B), count=2 (+2), average=53 B
tracemalloc_ex7.py:7: size=394 KiB (+0 B), count=9994 (+0), average=40 B
tracemalloc_ex7.py:6: size=358 KiB (+0 B), count=9984 (+0), average=37 B
tracemalloc_ex7.py:5: size=352 KiB (+0 B), count=9745 (+0), average=37 B

Example 8

Our eighth example is also exactly the same as the seventh example but we have specified in a different way how we can include only entries pertaining to the file which we are tracing. We have specified tracemalloc_ex8.py as a filename pattern to the Filter object. This way it'll eliminate entries of tracemalloc module and only keep the entry of this file.

CODE

import tracemalloc

tracemalloc.start()

l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]

snapshot1 = tracemalloc.take_snapshot()

print("SNAPSHOT - 1")
for stat in snapshot1.statistics("lineno"):
    print(stat)


l4 = [i*i*i*i for i in range(10000)]
l5 = [i*i*i*i*i for i in range(10000)]

print("\nSNAPSHOT - 2")
snapshot2 = tracemalloc.take_snapshot()
filtr = tracemalloc.Filter(inclusive=True, filename_pattern="tracemalloc_ex8.py")
snapshot2 = snapshot2.filter_traces([filtr])

for stat in snapshot2.statistics("lineno"):
        print(stat)

print("\nDIFFERENCE")
for diff in snapshot2.compare_to(snapshot1,"lineno"):
        print(diff)

OUTPUT

SNAPSHOT - 1
tracemalloc_ex8.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex8.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex8.py:5: size=352 KiB, count=9745, average=37 B

SNAPSHOT - 2
tracemalloc_ex8.py:17: size=436 KiB, count=9998, average=45 B
tracemalloc_ex8.py:16: size=433 KiB, count=9997, average=44 B
tracemalloc_ex8.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex8.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex8.py:5: size=352 KiB, count=9745, average=37 B
tracemalloc_ex8.py:12: size=464 B, count=1, average=464 B
tracemalloc_ex8.py:13: size=456 B, count=1, average=456 B
tracemalloc_ex8.py:11: size=106 B, count=2, average=53 B

DIFFERENCE
tracemalloc_ex8.py:17: size=436 KiB (+436 KiB), count=9998 (+9998), average=45 B
tracemalloc_ex8.py:16: size=433 KiB (+433 KiB), count=9997 (+9997), average=44 B
tracemalloc_ex8.py:12: size=464 B (+464 B), count=1 (+1), average=464 B
tracemalloc_ex8.py:13: size=456 B (+456 B), count=1 (+1), average=456 B
tracemalloc_ex8.py:11: size=106 B (+106 B), count=2 (+2), average=53 B
tracemalloc_ex8.py:7: size=394 KiB (+0 B), count=9994 (+0), average=40 B
tracemalloc_ex8.py:6: size=358 KiB (+0 B), count=9984 (+0), average=37 B
tracemalloc_ex8.py:5: size=352 KiB (+0 B), count=9745 (+0), average=37 B

This ends our small tutorial explaining how to use tracemalloc in a different way to monitor memory usage in python. Please feel free to let us know your views in the comments section.

References



Sunny Solanki  Sunny Solanki
Published On : Oct-28,2020 Time Investment : ~30 mins

Other Python Tutorials

signal - Simple Guide to Send, Receive and Handle System Signals in Python
asyncio - Concurrent Programming using Async/Await Syntax in Python
difflib - Simple Way to Find Out Differences Between Sequences/File Contents using Python
filecmp - Compare Files and Directories using Python
hmac - Hash-based Message Authentication Code using Python

Other Categories

Artificial Intelligence (2)
Data Science (62)
Digital Marketing (7)
Machine Learning (33)
Newsletter Subscription