tracemalloc - How to Profile Memory Usage By Python Code?¶
> What is Memory Profiling?¶
Memory Profiling is the process of recording memory usage (space complexity) by various parts of python code. We can record memory usage taken by individual parts or whole Python script/program as well.
It is one type of profiling whereas another one is time profiling (time complexity) where we record time taken by Python code/script/program. We have listed some useful time profilers in References section at the end of tutorial.
> Which Python Libraries to Use for Profiling Memory Usage?¶
Till 'v3.4', Python default installation did not have any memory profilers available in it. The 'v3.4' introduced a new module named 'tracemalloc' offering functionality to record memory usage traces.
The tracemalloc module allows us to monitor memory allocations in the python code. It lets us take memory snapshots at a particular point, perform various statistics on snapshots as well as perform differences between two snapshots to check object allocation between two snapshots.
Apart from 'tracemalloc', Python has other memory profilers (memory_profiler, pympler, guppy/heapy, Scalene, etc) which needs to be installed separately using 'pip' or 'conda'.
> What You Can Learn From This Article?¶
As a part of this tutorial, we have explained how to use Python Module tracemalloc to profile memory usage by python code and perform various operations. We have thoroughly covered the usage of various classes, methods, and attributes available through the module with the help of various examples. We have covered whole API of tracemalloc module. Tracemalloc can be used to profile memory usage by individual parts of Python program as well as whole Python program/script.
Below, we have listed important sections of tutorial to give an overview of the material covered.
Important Sections Of Tutorial¶
- Steps to Use "tracemalloc" for Memory Profiling
- Simple Example Explaining How to Use "tracemalloc"
- Useful Methods and Attributes of "tracemalloc" Library
- How to Clear Memory Traces?
- Store Memory Snapshots to a File and Load it Again
- Filter Unwanted Traces
- Compare Memory Snapshots to Find Out Differences
- Remove "tracemalloc" Module Entries from Traces
- Filter Snapshots to Keep Entries of Specified File
- How are Classes of "tracemalloc" Module Organized?
1. Steps to Use "tracemalloc" for Memory Profiling ¶
- Start Memory Tracer ('start()' Method)
- Take snapshots using 'take_snapshot()'.
- Once tracer has started, we can take Snapshots wherever we want using 'take_snapshot()'.
- The snapshot will have memory usage from tracer start till the line where 'take_snapshot()' was called.
- Print Statistics by calling 'statistics()' Method on Snapshot object.
- Stop memory tracer ('stop()' Method).
2. Simple Example Explaining How to Use "tracemalloc" ¶
As a part of the first example, we'll simply explain how to start tracing memory using tracemalloc, take snapshots, and print traced output.
Below is a list of methods that we have explained as a part of this example.
- tracemalloc.start() - This method is available from tracemalloc module calling which will start tracing of memory. It accepts an integer argument named nframe which mentions a number of frames to allocate per call. The default value is 1.
- tracemalloc.take_snapshot() - This method is available from the tracemalloc module which takes a memory snapshot when it’s called and returns the Snapshot object.
- Snapshot.statistics() - This method is available through the Snapshot object and takes as an input key using which to sort records of tracing. It returns an iterator that has a list of Statistic objects. The single Statistic object has information about a single traceback (single line of code generally) which includes a number of objects and size of objects recorded in that traceback. Below is a list of keys that this method takes as input.
- filename - Sort traces according to file names
- lineno - sort traces according to line no in the file.
- traceback - Sort traces according to trace taken order.
Below we have introduced the above-mentioned method with a simple example.
We are starting tracing at the beginning and then creating three lists of integers. We have then taken a memory snapshot and printed a list of tracebacks collected by that snapshot from the start of the traceback.
CODE: tracemalloc_ex1.py
import tracemalloc
tracemalloc.start()
l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]
snapshot = tracemalloc.take_snapshot()
for stat in snapshot.statistics("lineno"):
print(stat)
tracemalloc.stop()
OUTPUT
tracemalloc_ex1.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex1.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex1.py:5: size=352 KiB, count=9745, average=37 B
3. Useful Methods and Attributes of "tracemalloc" Library ¶
As a part of our second example, we'll explain a few attributes and methods of tracemalloc and statistic objects.
Below is a list of methods that we have explained as a part of this example.
- Statistic.traceback - The traceback attribute of statistic object returns an object of class Traceback. The Traceback has information about individual traces which can have more than one frame (Frame).
- Traceback.format() - This method formats traceback as a list of lines that were traced for this traceback.
- tracemalloc.get_traceback_limit() - It returns an integer specifying maximum number of frames stored in single traceback. We can specify how many maximum frames to record per traceback in tracemalloc.start() method.
- tracemalloc.get_object_traceback() - This method takes as input individual object and returns traceback object specifying where traceback for this object was taken.
- tracemalloc.get_traced_memory() - It returns tuple of two integer specifying current size and peak size of block traced in bytes.
- tracemalloc.get_tracemalloc_memory() - It returns integer specifying memory used by tracemalloc module in bytes.
- tracemalloc.is_tracing() - It returns boolean specifying whether tracing is happening or stopped.
Below we have explained the usage of the above-mentioned method through example.
CODE: tracemalloc_ex2.py
import tracemalloc
import numpy as np
tracemalloc.start(25)
l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]
l4 = np.random.randint(1,100, (1000,))
snapshot = tracemalloc.take_snapshot()
print("========== SNAPSHOT =============")
for stat in snapshot.statistics("lineno"):
print(stat)
print(stat.traceback.format())
print("\n=========== USEFUL METHODS ===========")
print("\nTraceback Limit : ", tracemalloc.get_traceback_limit(), " Frames")
print("\nAllocation Location for List l4 : ", tracemalloc.get_object_traceback(l4))
print("\nTraced Memory (Current, Peak): ", tracemalloc.get_traced_memory())
#tracemalloc.reset_peak()
#print("\nTraced Memory : ", tracemalloc.get_traced_memory())
print("\nMemory Usage by tracemalloc Module : ", tracemalloc.get_tracemalloc_memory(), " bytes")
print("\nTracing Status : ",tracemalloc.is_tracing())
tracemalloc.stop()
OUTPUT
========== SNAPSHOT =============
tracemalloc_ex2.py:8: size=394 KiB, count=9994, average=40 B
[' File "tracemalloc_ex2.py", line 8', ' l3 = [i*i*i for i in range(10000)]']
tracemalloc_ex2.py:7: size=358 KiB, count=9984, average=37 B
[' File "tracemalloc_ex2.py", line 7', ' l2 = [i*i for i in range(10000)]']
tracemalloc_ex2.py:6: size=352 KiB, count=9744, average=37 B
[' File "tracemalloc_ex2.py", line 6', ' l1 = [i for i in range(10000)]']
tracemalloc_ex2.py:9: size=8138 B, count=3, average=2713 B
[' File "tracemalloc_ex2.py", line 9', ' l4 = np.random.randint(1,100, (1000,))']
<__array_function__ internals>:6: size=536 B, count=1, average=536 B
[' File "<__array_function__ internals>", line 6']
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:2911: size=528 B, count=1, average=528 B
[' File "/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2911', ' keepdims=keepdims, initial=initial, where=where)']
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/_dtype.py:334: size=488 B, count=1, average=488 B
[' File "/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/_dtype.py", line 334', ' if np.issubdtype(dtype, np.flexible) and not _isunsized(dtype):']
<__array_function__ internals>:4: size=464 B, count=1, average=464 B
[' File "<__array_function__ internals>", line 4']
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:74: size=448 B, count=1, average=448 B
[' File "/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 74', ' passkwargs = {k: v for k, v in kwargs.items()']
tracemalloc_ex2.py:11: size=432 B, count=1, average=432 B
[' File "tracemalloc_ex2.py", line 11', ' snapshot = tracemalloc.take_snapshot()']
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:90: size=145 B, count=2, average=72 B
[' File "/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 90', ' return ufunc.reduce(obj, axis, dtype, out, **passkwargs)']
=========== USEFUL METHODS ===========
Traceback Limit : 25 Frames
Allocation Location for List l4 : tracemalloc_ex2.py:9
Traced Memory (Current, Peak): (1472281, 1487679)
Memory Usage by tracemalloc Module : 2504012 bytes
Tracing Status : True
4. How to Clear Memory Traces? ¶
As a part of this example, we have simply explained how we can start/stop tracemalloc and clear traces.
Below is a list of methods that we have explained as a part of this example.
- tracemalloc.stop() - This method stops tracemalloc from taking traces.
- tracemalloc.clear_traces() - This method clears all traces taken from the beginning of the start() method.
Please make a NOTE that traces info would still be available for snapshot objects which were taken but it won't be available if you call the take_snapshot() method after this method.
Below we have first taken a snapshot after creating three lists of integers.
We have then created another list and taken snapshots again. We have then cleared all traces which are traces of 4 lists created since the beginning.
We have then created the fifth list and taken a snapshot again.
Please make a note from the output that the 3rd snapshot has information only about the fifth list created and all traces before it is cleared.
We have also explained the usage of stop() at the end.
CODE: tracemalloc_ex3.py
import tracemalloc
tracemalloc.start()
l1 = [i for i in range(1000000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]
snapshot1 = tracemalloc.take_snapshot()
print("================ SNAPSHOT 1 =================")
for stat in snapshot1.statistics("lineno"):
print(stat)
l4 = [i*i*i*i for i in range(10000)]
snapshot2 = tracemalloc.take_snapshot()
print("\n================ SNAPSHOT 2 =================")
for stat in snapshot2.statistics("lineno"):
print(stat)
tracemalloc.clear_traces()
l5 = [i*2 for i in range(10000)]
snapshot3 = tracemalloc.take_snapshot()
print("\n================ SNAPSHOT 3 =================")
for stat in snapshot3.statistics("lineno"):
print(stat)
print("\nTracing Status : ", tracemalloc.is_tracing())
tracemalloc.stop()
print("\nTracing Status : ", tracemalloc.is_tracing())
try:
print("\nTrying to Take Snapshot After Tracing is Stopped.")
snap = tracemalloc.take_snapshot()
except Exception as e:
print("Exception : ", e)
OUTPUT
================ SNAPSHOT 1 =================
tracemalloc_ex3.py:5: size=35.0 MiB, count=999745, average=37 B
tracemalloc_ex3.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex3.py:6: size=358 KiB, count=9984, average=37 B
================ SNAPSHOT 2 =================
tracemalloc_ex3.py:5: size=35.0 MiB, count=999745, average=37 B
tracemalloc_ex3.py:15: size=433 KiB, count=9997, average=44 B
tracemalloc_ex3.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex3.py:6: size=358 KiB, count=9984, average=37 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:209: size=864 B, count=2, average=432 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:165: size=832 B, count=2, average=416 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:397: size=656 B, count=5, average=131 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:479: size=576 B, count=3, average=192 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:507: size=576 B, count=1, average=576 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:534: size=552 B, count=3, average=184 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:472: size=480 B, count=2, average=240 B
tracemalloc_ex3.py:12: size=464 B, count=1, average=464 B
tracemalloc_ex3.py:13: size=456 B, count=1, average=456 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:53: size=448 B, count=1, average=448 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:509: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:194: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:54: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:475: size=416 B, count=1, average=416 B
tracemalloc_ex3.py:11: size=314 B, count=5, average=63 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:459: size=240 B, count=2, average=120 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:458: size=240 B, count=2, average=120 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:185: size=168 B, count=3, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:65: size=144 B, count=2, average=72 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:291: size=112 B, count=2, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:508: size=64 B, count=1, average=64 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:289: size=56 B, count=1, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:180: size=56 B, count=1, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:477: size=28 B, count=1, average=28 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:476: size=28 B, count=1, average=28 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:27: size=24 B, count=1, average=24 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:15: size=24 B, count=1, average=24 B
================ SNAPSHOT 3 =================
tracemalloc_ex3.py:27: size=356 KiB, count=9873, average=37 B
Tracing Status : True
Tracing Status : False
Trying to Take Snapshot After Tracing is Stopped.
Exception : the tracemalloc module must be tracing memory allocations to take a snapshot
5. Store Memory Snapshots to a File and Load it Again ¶
As a part of our fourth example, we'll explain how we can store a snapshot of traces into a file on disk and then load it again.
Below is a list of methods that we have explained as a part of this example.
- Snapshot.dump() - This method takes as input filename to which store traces information.
- Snapshot.load() - This method takes as input filename from which load traces information.
Below we have explained the usage of both methods.
CODE: tracemalloc_ex4.py
import tracemalloc
tracemalloc.start()
l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]
snapshot = tracemalloc.take_snapshot()
for stat in snapshot.statistics("lineno"):
print(stat)
snapshot.dump("snap.out")
snapshot_loaded = tracemalloc.Snapshot.load("snap.out")
print("\nLoaded Snapshot From File :")
for stat in snapshot_loaded.statistics("lineno"):
print(stat)
tracemalloc.stop()
OUTPUT
tracemalloc_ex4.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex4.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex4.py:5: size=352 KiB, count=9745, average=37 B
Loaded Snapshot From File :
tracemalloc_ex4.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex4.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex4.py:5: size=352 KiB, count=9745, average=37 B
6. Filter Unwanted Traces ¶
As a part of this example, we'll explain how we can filter out traces from a list of all traces recorded by the tracemalloc. There are two main classes provided by tracemalloc for filtering traces.
DomainFilter - This class lets us filter traces by address space of objects. It takes two parameters as input.
- domain - It accepts integers specifying address space of memory block. The integer 0 is generally used by python memory traces but the number will be different for C.
- inclusive - It accepts boolean specifying whether to include or exclude traces specified by the domain attribute.
Filter - This class lets filter traces based on combinations of the filename, line number, and domain. It has the below-mentioned parameters.
- domain - It accepts integers specifying address space of memory block. The integer 0 is generally used by python memory traces but the number will be different for C.
- filename_pattern - It accepts string specifying pattern for which filenames will be matched in traces and based on the inclusive flag they will be included or excluded.
- lineno - It accepts integer specifying traces with that line numbers to include.
- inclusive - It accepts boolean specifying whether to include or exclude traces specified by domain, filename_pattern, and lineno attributes.
- all_frames - It accepts a boolean flag specifying whether to include all frames per trace or not.
We have explained the usage of filters through the examples below.
The first snapshot includes all traces for memory blocks used by python.
The second snapshot includes traces of memory blocks that are not created by python. It can be due to C as numpy is built on it.
The third snapshot includes entries where the filename is tracemalloc_ex5.py and the fourth snapshot excludes entries with that file name.
CODE: tracemalloc_ex5.py
import tracemalloc
import numpy as np
tracemalloc.start(10)
l2 = [i*i for i in range(10000)]
l1 = [i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]
l4 = [i*2 for i in range(10000)]
l5 = np.random.randint(1,100, (1000,))
snapshot = tracemalloc.take_snapshot()
print("========== Original Snapshot ===========")
for stat in snapshot.statistics("lineno"):
print(stat)
print("\n========= Filtered Snapshot 1 =============")
filtr = tracemalloc.DomainFilter(inclusive=True, domain=0)
snap_shot = snapshot.filter_traces(filters=[filtr])
for stat in snap_shot.statistics("lineno"):
print(stat)
filtr = tracemalloc.DomainFilter(inclusive=False, domain=0)
snap_shot = snapshot.filter_traces(filters=[filtr])
print("\n========= Filtered Snapshot 2 =============")
for stat in snap_shot.statistics("lineno"):
print(stat)
filtr = tracemalloc.Filter(inclusive=True, filename_pattern="tracemalloc_ex5.py")
snap_shot = snapshot.filter_traces([filtr])
print("\n========= Filtered Snapshot 3 =============")
for stat in snap_shot.statistics("lineno"):
print(stat)
filtr = tracemalloc.Filter(inclusive=False, filename_pattern="tracemalloc_ex5.py")
snap_shot = snapshot.filter_traces([filtr])
print("\n========= Filtered Snapshot 4 =============")
for stat in snap_shot.statistics("lineno"):
print(stat)
tracemalloc.stop()
OUTPUT
========== Original Snapshot ===========
tracemalloc_ex5.py:8: size=394 KiB, count=9994, average=40 B
tracemalloc_ex5.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex5.py:9: size=355 KiB, count=9872, average=37 B
tracemalloc_ex5.py:7: size=352 KiB, count=9744, average=37 B
tracemalloc_ex5.py:10: size=9050 B, count=5, average=1810 B
<__array_function__ internals>:6: size=536 B, count=1, average=536 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:2911: size=528 B, count=1, average=528 B
<__array_function__ internals>:4: size=464 B, count=1, average=464 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:74: size=448 B, count=1, average=448 B
tracemalloc_ex5.py:12: size=432 B, count=1, average=432 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:90: size=145 B, count=2, average=72 B
========= Filtered Snapshot 1 =============
tracemalloc_ex5.py:8: size=394 KiB, count=9994, average=40 B
tracemalloc_ex5.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex5.py:9: size=355 KiB, count=9872, average=37 B
tracemalloc_ex5.py:7: size=352 KiB, count=9744, average=37 B
tracemalloc_ex5.py:10: size=1050 B, count=4, average=262 B
<__array_function__ internals>:6: size=536 B, count=1, average=536 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:2911: size=528 B, count=1, average=528 B
<__array_function__ internals>:4: size=464 B, count=1, average=464 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:74: size=448 B, count=1, average=448 B
tracemalloc_ex5.py:12: size=432 B, count=1, average=432 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:90: size=145 B, count=2, average=72 B
========= Filtered Snapshot 2 =============
tracemalloc_ex5.py:10: size=8000 B, count=1, average=8000 B
========= Filtered Snapshot 3 =============
tracemalloc_ex5.py:8: size=394 KiB, count=9994, average=40 B
tracemalloc_ex5.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex5.py:9: size=355 KiB, count=9872, average=37 B
tracemalloc_ex5.py:7: size=352 KiB, count=9744, average=37 B
tracemalloc_ex5.py:10: size=9050 B, count=5, average=1810 B
tracemalloc_ex5.py:12: size=432 B, count=1, average=432 B
========= Filtered Snapshot 4 =============
<__array_function__ internals>:6: size=536 B, count=1, average=536 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:2911: size=528 B, count=1, average=528 B
<__array_function__ internals>:4: size=464 B, count=1, average=464 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:74: size=448 B, count=1, average=448 B
/home/sunny/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:90: size=145 B, count=2, average=72 B
7. Compare Memory Snapshots to Find Out Differences ¶
As a part of this example, we'll explain how we can compare two snapshots and find out the difference in traces between them.
Below is a list of methods that we have explained as a part of this example.
- Snapshot.compare_to() - This method can be called on any snapshot object passing it another snapshot object to get the difference between two snapshots. It also takes as input the key_type parameter which will be used as the primary key to find out the difference between two snapshots. It'll return an object of class StatisticDiff. This object has information about differences.
Below we have explained through example, how we can get the difference between two snapshots. Please make a note that different traces taken before the first snapshot are not present.
CODE: tracemalloc_ex6.py
import tracemalloc
tracemalloc.start()
l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]
snapshot1 = tracemalloc.take_snapshot()
print("SNAPSHOT - 1")
for stat in snapshot1.statistics("lineno"):
print(stat)
l4 = [i*i*i*i for i in range(10000)]
l5 = [i*i*i*i*i for i in range(10000)]
print("\nSNAPSHOT - 2")
snapshot2 = tracemalloc.take_snapshot()
for stat in snapshot2.statistics("lineno"):
#if "tracemalloc_ex6.py" in str(stat):
print(stat)
print("\nDIFFERENCE")
for diff in snapshot2.compare_to(snapshot1,"lineno"):
#if "tracemalloc_ex6.py" in str(diff):
print(diff)
tracemalloc.stop()
OUTPUT
SNAPSHOT - 1
tracemalloc_ex6.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex6.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex6.py:5: size=352 KiB, count=9745, average=37 B
SNAPSHOT - 2
tracemalloc_ex6.py:17: size=436 KiB, count=9998, average=45 B
tracemalloc_ex6.py:16: size=433 KiB, count=9997, average=44 B
tracemalloc_ex6.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex6.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex6.py:5: size=352 KiB, count=9745, average=37 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:209: size=864 B, count=2, average=432 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:165: size=832 B, count=2, average=416 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:479: size=576 B, count=3, average=192 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:507: size=576 B, count=1, average=576 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:397: size=528 B, count=3, average=176 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:472: size=480 B, count=2, average=240 B
tracemalloc_ex6.py:12: size=464 B, count=1, average=464 B
tracemalloc_ex6.py:13: size=456 B, count=1, average=456 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:53: size=448 B, count=1, average=448 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:509: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:194: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:54: size=424 B, count=1, average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:475: size=416 B, count=1, average=416 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:65: size=144 B, count=2, average=72 B
tracemalloc_ex6.py:11: size=106 B, count=2, average=53 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:508: size=64 B, count=1, average=64 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:534: size=56 B, count=1, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:185: size=56 B, count=1, average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:291: size=40 B, count=1, average=40 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:477: size=28 B, count=1, average=28 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:476: size=28 B, count=1, average=28 B
DIFFERENCE
tracemalloc_ex6.py:17: size=436 KiB (+436 KiB), count=9998 (+9998), average=45 B
tracemalloc_ex6.py:16: size=433 KiB (+433 KiB), count=9997 (+9997), average=44 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:209: size=864 B (+864 B), count=2 (+2), average=432 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:165: size=832 B (+832 B), count=2 (+2), average=416 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:479: size=576 B (+576 B), count=3 (+3), average=192 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:507: size=576 B (+576 B), count=1 (+1), average=576 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:397: size=528 B (+528 B), count=3 (+3), average=176 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:472: size=480 B (+480 B), count=2 (+2), average=240 B
tracemalloc_ex6.py:12: size=464 B (+464 B), count=1 (+1), average=464 B
tracemalloc_ex6.py:13: size=456 B (+456 B), count=1 (+1), average=456 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:53: size=448 B (+448 B), count=1 (+1), average=448 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:509: size=424 B (+424 B), count=1 (+1), average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:194: size=424 B (+424 B), count=1 (+1), average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:54: size=424 B (+424 B), count=1 (+1), average=424 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:475: size=416 B (+416 B), count=1 (+1), average=416 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:65: size=144 B (+144 B), count=2 (+2), average=72 B
tracemalloc_ex6.py:11: size=106 B (+106 B), count=2 (+2), average=53 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:508: size=64 B (+64 B), count=1 (+1), average=64 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:534: size=56 B (+56 B), count=1 (+1), average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:185: size=56 B (+56 B), count=1 (+1), average=56 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:291: size=40 B (+40 B), count=1 (+1), average=40 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:477: size=28 B (+28 B), count=1 (+1), average=28 B
/home/sunny/anaconda3/lib/python3.7/tracemalloc.py:476: size=28 B (+28 B), count=1 (+1), average=28 B
tracemalloc_ex6.py:7: size=394 KiB (+0 B), count=9994 (+0), average=40 B
tracemalloc_ex6.py:6: size=358 KiB (+0 B), count=9984 (+0), average=37 B
tracemalloc_ex6.py:5: size=352 KiB (+0 B), count=9745 (+0), average=37 B
8. Remove "tracemalloc" Module Entries from Traces ¶
Our seventh example is exactly the same as our sixth example with the only difference that we have used a filter to remove entries related to the tracemalloc module itself.
We have specified tracemalloc filename (tracemalloc.file) to filename_pattern of Filter class with the inclusive attribute as False to exclude entries with tracemalloc module.
CODE: tracemalloc_ex7.py
import tracemalloc
tracemalloc.start()
l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]
snapshot1 = tracemalloc.take_snapshot()
print("SNAPSHOT - 1")
for stat in snapshot1.statistics("lineno"):
print(stat)
l4 = [i*i*i*i for i in range(10000)]
l5 = [i*i*i*i*i for i in range(10000)]
print("\nSNAPSHOT - 2")
snapshot2 = tracemalloc.take_snapshot()
filtr = tracemalloc.Filter(inclusive=False, filename_pattern=tracemalloc.__file__)
snapshot2 = snapshot2.filter_traces([filtr])
for stat in snapshot2.statistics("lineno"):
print(stat)
print("\nDIFFERENCE")
for diff in snapshot2.compare_to(snapshot1,"lineno"):
print(diff)
tracemalloc.stop()
OUTPUT
SNAPSHOT - 1
tracemalloc_ex7.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex7.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex7.py:5: size=352 KiB, count=9745, average=37 B
SNAPSHOT - 2
tracemalloc_ex7.py:17: size=436 KiB, count=9998, average=45 B
tracemalloc_ex7.py:16: size=433 KiB, count=9997, average=44 B
tracemalloc_ex7.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex7.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex7.py:5: size=352 KiB, count=9745, average=37 B
tracemalloc_ex7.py:12: size=464 B, count=1, average=464 B
tracemalloc_ex7.py:13: size=456 B, count=1, average=456 B
tracemalloc_ex7.py:11: size=106 B, count=2, average=53 B
DIFFERENCE
tracemalloc_ex7.py:17: size=436 KiB (+436 KiB), count=9998 (+9998), average=45 B
tracemalloc_ex7.py:16: size=433 KiB (+433 KiB), count=9997 (+9997), average=44 B
tracemalloc_ex7.py:12: size=464 B (+464 B), count=1 (+1), average=464 B
tracemalloc_ex7.py:13: size=456 B (+456 B), count=1 (+1), average=456 B
tracemalloc_ex7.py:11: size=106 B (+106 B), count=2 (+2), average=53 B
tracemalloc_ex7.py:7: size=394 KiB (+0 B), count=9994 (+0), average=40 B
tracemalloc_ex7.py:6: size=358 KiB (+0 B), count=9984 (+0), average=37 B
tracemalloc_ex7.py:5: size=352 KiB (+0 B), count=9745 (+0), average=37 B
9. Filter Snapshots to Keep Entries of Specified File ¶
Our eighth example is also exactly the same as the seventh example but we have specified in a different way how we can include only entries pertaining to the file which we are tracing.
We have specified tracemalloc_ex8.py as a filename pattern to the Filter object. This way it'll eliminate entries of tracemalloc module and only keep the entry of this file.
CODE: tracemalloc_ex8.py
import tracemalloc
tracemalloc.start()
l1 = [i for i in range(10000)]
l2 = [i*i for i in range(10000)]
l3 = [i*i*i for i in range(10000)]
snapshot1 = tracemalloc.take_snapshot()
print("SNAPSHOT - 1")
for stat in snapshot1.statistics("lineno"):
print(stat)
l4 = [i*i*i*i for i in range(10000)]
l5 = [i*i*i*i*i for i in range(10000)]
print("\nSNAPSHOT - 2")
snapshot2 = tracemalloc.take_snapshot()
filtr = tracemalloc.Filter(inclusive=True, filename_pattern="tracemalloc_ex8.py")
snapshot2 = snapshot2.filter_traces([filtr])
for stat in snapshot2.statistics("lineno"):
print(stat)
print("\nDIFFERENCE")
for diff in snapshot2.compare_to(snapshot1,"lineno"):
print(diff)
tracemalloc.stop()
OUTPUT
SNAPSHOT - 1
tracemalloc_ex8.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex8.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex8.py:5: size=352 KiB, count=9745, average=37 B
SNAPSHOT - 2
tracemalloc_ex8.py:17: size=436 KiB, count=9998, average=45 B
tracemalloc_ex8.py:16: size=433 KiB, count=9997, average=44 B
tracemalloc_ex8.py:7: size=394 KiB, count=9994, average=40 B
tracemalloc_ex8.py:6: size=358 KiB, count=9984, average=37 B
tracemalloc_ex8.py:5: size=352 KiB, count=9745, average=37 B
tracemalloc_ex8.py:12: size=464 B, count=1, average=464 B
tracemalloc_ex8.py:13: size=456 B, count=1, average=456 B
tracemalloc_ex8.py:11: size=106 B, count=2, average=53 B
DIFFERENCE
tracemalloc_ex8.py:17: size=436 KiB (+436 KiB), count=9998 (+9998), average=45 B
tracemalloc_ex8.py:16: size=433 KiB (+433 KiB), count=9997 (+9997), average=44 B
tracemalloc_ex8.py:12: size=464 B (+464 B), count=1 (+1), average=464 B
tracemalloc_ex8.py:13: size=456 B (+456 B), count=1 (+1), average=456 B
tracemalloc_ex8.py:11: size=106 B (+106 B), count=2 (+2), average=53 B
tracemalloc_ex8.py:7: size=394 KiB (+0 B), count=9994 (+0), average=40 B
tracemalloc_ex8.py:6: size=358 KiB (+0 B), count=9984 (+0), average=37 B
tracemalloc_ex8.py:5: size=352 KiB (+0 B), count=9745 (+0), average=37 B
10. How are Classes of "tracemalloc" Module Organized? ¶
- Snapshot Object has list of Trace objects recording each trace of memory.
- Trace object has Traceback object with tracing information.
- Traceback object has recorded trace as a list of Frame objects.
- Statistic object has stats of one Trace.
- Filter objects let us filter traces from Snapshot object based on various conditions.
- StatisticDiff object has a difference in memory traces between two Snapshot objects.
This ends our small tutorial explaining how to use 'tracemalloc' in a different way to monitor memory usage by python.
References¶
Python Memory Profilers¶
- How to Profile Memory Usage in Python using memory_profiler?
- pympler - Monitor Memory Usage by Python Objects
- guppy/heapy - Profile Memory Usage in Python
Python Time Profilers¶
- line_profiler: Line by Line Profiling of Python Code
- Snakeviz - Visualize Profiling Results in Python
- How to Profile Python Code using cProfile and Profile
- Scalene - CPU and Memory Profiler for Python Code
- pprofile - Deterministic & Statistical Profiler For Python Code
- py-spy - Sampling Profiler for Python Code
- pyinstrument - Statistical Profiler for Python Code
- yappi - Yet Another Python Profiler
Garbage Collection in Python¶
Sunny Solanki
Sunny Solanki
Intro: Software Developer | Bonsai Enthusiast
About: Sunny Solanki holds a bachelor's degree in Information Technology (2006-2010) from L.D. College of Engineering. Post completion of his graduation, he has 8.5+ years of experience (2011-2019) in the IT Industry (TCS). His IT experience involves working on Python & Java Projects with US/Canada banking clients. Since 2019, he’s primarily concentrating on growing CoderzColumn.
His main areas of interest are AI, Machine Learning, Data Visualization, and Concurrent Programming. He has good hands-on with Python and its ecosystem libraries.
Apart from his tech life, he prefers reading biographies and autobiographies. And yes, he spends his leisure time taking care of his plants and a few pre-Bonsai trees.
Contact: sunny.2309@yahoo.in
![Share Views]()
Stuck Somewhere? Need Help with Coding? Have Doubts About the Topic/Code?
Stuck Somewhere? Need Help with Coding? Have Doubts About the Topic/Code?When going through coding examples, it's quite common to have doubts and errors.
If you have doubts about some code examples or are stuck somewhere when trying our code, send us an email at coderzcolumn07@gmail.com. We'll help you or point you in the direction where you can find a solution to your problem.
You can even send us a mail if you are trying something new and need guidance regarding coding. We'll try to respond as soon as possible.
![Share Views]()
Want to Share Your Views? Have Any Suggestions?
Want to Share Your Views? Have Any Suggestions?If you want to
- provide some suggestions on topic
- share your views
- include some details in tutorial
- suggest some new topics on which we should create tutorials/blogs
tracemalloc, memory-tracing