Updated On : Dec-12,2020 Tags magic-commands, jupyter-notebook
List of Useful Magic Commands in Jupyter Notebook/Lab

List of Useful Magic Commands in Jupyter Notebook/Lab

Jupyter Notebook/Lab is the go-to tool used by data scientists and developers worldwide to perform data analysis nowadays. It provides a very easy-to-use interface and lots of other functionalities like markdown, latex, inline plots, etc. Apart from these, it even provides a list of useful magic commands which let us perform a bunch of tasks from the jupyter notebook itself which developers need to do in the command prompt/shell. As a part of this tutorial, we'll cover some of the very commonly used magic commands.

There are two types of magic commands available with Jupyter Notebook/Lab:

  • Line Magic Commands: It applies the command to one line of the Jupyter cell as its name suggests.
  • Cell Magic Commands: It applies the command to the whole cell of the notebook and needs to be kept at the beginning of the cell.

We'll now explain the usage of magic commands one by one with simple examples.

Line Magic Commands

In this section, we'll explain the commonly used line magic command which can make the life of the developer easy by providing some of the useful functionalities in the notebook itself.

%lsmagic

The %lsmagic commands list all the available magic commands with a notebook.

In [1]:
%lsmagic
Out[1]:
Available line magics:
%alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %conda  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

If we run this command in Jupyter lab then it'll return an expandable tree-like structure for a list of commands as shown below.

List of Useful Magic Commands in Jupyter Notebook/Lab

%magic

The %magic commands print information about the magic commands system in the jupyter notebook. It kind of gives an overview of the magic commands system available in the notebook.

In [96]:
%magic

%quickref

The %quickref line command gives us a cheat-sheet covering an overview of each magic command available.

In [4]:
%quickref
The following magic functions are currently available:

%alias:
    Define an alias for a system command.
%alias_magic:
    ::
%autoawait:

%autocall:
    Make functions callable without having to type parentheses.
%automagic:
    Make magic functions callable without having to type the initial %.
%autosave:
    Set the autosave interval in the notebook (in seconds).
%bookmark:
    Manage IPython's bookmark system.

%alias_magic

The %alias_magic line command as its name suggests creates an alias for any existing magic command. We can then call the command by alias and it'll perform the same functionality as the original command. Below we have renamed the %pwd command to the %currdir command which displays the current working directory. We need to give a new name for the command followed by a command name to create an alias.

In [6]:
%alias_magic currdir pwd
Created `%currdir` as an alias for `%pwd`.
In [7]:
%currdir
Out[7]:
'/home/sunny'

%autocall

The %autocall line command lets us call functions in a notebook without typing parenthesis. We can type function name followed by a list of argument values separated by a comma. Below we have created a simple function that adds two numbers. We have then turned on autocall by calling the magic command. After turning on autocall, we are able to execute the function without parenthesis. We have then turned off autocall and calling the function without parenthesis fails.

In [8]:
def addition(a,b):
    return a+b
In [9]:
%autocall
Automatic calling is: Smart
In [10]:
addition 5, 5
------> addition(5, 5)
------> addition(5, 5)
Out[10]:
10
In [11]:
%autocall
Automatic calling is: OFF
In [12]:
addition 5, 5
  File "<ipython-input-12-4d245131862c>", line 1
    addition 5, 5
             ^
SyntaxError: invalid syntax

%automagic

The %automagic line command let us call magic command in jupyter notebook without typing % sign at the beginning. We can turn automagic on and off by executing the %automagic line command. Below we have explained the usage of the same.

In [13]:
%automagic
Automagic is OFF, % prefix IS needed for line magics.
In [14]:
pwd
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-14-86938b1e80ee> in <module>
----> 1 pwd

NameError: name 'pwd' is not defined
In [15]:
%automagic
Automagic is ON, % prefix IS NOT needed for line magics.
In [16]:
pwd
Out[16]:
'/home/sunny'

%pwd

The %pwd line command as its name suggests returns the present working directory.

In [17]:
%pwd
Out[17]:
'/home/sunny'

%cd

The %cd line command lets us change our working directory as explained below.

In [18]:
%cd Desktop/
/home/sunny/Desktop
In [19]:
%pwd
Out[19]:
'/home/sunny/Desktop'

%system

The %system command lets us execute Unix shell commands in the jupyter notebook. We can execute any single line Unix shell command from the notebook. We have explained below the usage of the command with two simple examples.

In [20]:
%system echo 'Hello World'
Out[20]:
['Hello World']
In [97]:
%system ls -lrt | grep python
Out[97]:
['-rw-r--r--  1 sunny sunny   3393723 Mar  4  2020 How to build dashboard using Python (Dash & Plotly) and deploy online (pythonanywhere.com).html~']

%sx

The %sx command works exactly like the %system command.

In [21]:
%sx echo 'Hello World'
Out[21]:
['Hello World']
In [98]:
%sx ls -lrt | grep python
Out[98]:
['-rw-r--r--  1 sunny sunny   3393723 Mar  4  2020 How to build dashboard using Python (Dash & Plotly) and deploy online (pythonanywhere.com).html~']

%time

The %time line command measures the execution time of the line which follows it using the time python module. It returns both, the CPU and wall time of execution. Below we have explained with a simple example of how to use command. It even returns the execution value of the command which we have kept in a variable. It's available as a cell command as well.

In [22]:
%time out = [i*i for i in range(1000000)]
CPU times: user 44.9 ms, sys: 11.3 ms, total: 56.3 ms
Wall time: 55.2 ms

%timeit

The %timeit line command measures the execution time of the function using the timeit python module. It provides a few other functionalities as well. It executes the command given as input for 7 rounds where each round executes code 10 times totaling 70 times by default. It takes the best of each iteration in each round and gives time measurement with standard deviation. Below are some useful arguments of the command.

  • -n <loops> - It accepts integer value specifying number of iteration per round.
  • -r <runs> - It accepts integer value specifying number of rounds to test timer.
  • -t - This option forces %timeit to use time.time to measure time which returns wall time.
  • -c - This option forces %timeit to use time.clock to measure time which returns CPU time.
  • -q - This option instructs %timeit to not print results to the output.
  • -o - This option returns TimeitResult object.

Below we have explained with simple example usage of the command.

In [99]:
%timeit out = [i*i for i in range(1000000)]
53.1 ms ± 1.27 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [100]:
%timeit -n 5 -r 5 out = [i*i for i in range(1000000)]
52.2 ms ± 1.87 ms per loop (mean ± std. dev. of 5 runs, 5 loops each)

%who

The %who line command returns all variables of a particular type. We can give variable type followed the command and it'll return a list of all variables with that type. We can even give more than one type if we want to see variables of different type which are currently active in jupyter notebook and not collected by the garbage collector.

Below we have explained with few simple examples of how we can use %who.

In [102]:
a = 100
b = 5.5
c = [11,12,13]
d = {'key1':'val1', 'key2':'val2'}
e = "Hello World"

%who str
%who dict
%who float
%who list
e
d
b
c
In [103]:
def addition(a,b):
    return a + b

def division(a,b):
    return a / b if b!=0 else 0

%who function
addition	 division
In [104]:
%who function list
addition	 c	 division

%who_ls

The %who_ls commands work exactly like %who but it returns a list of variable names as a list of strings which is sorted as well.

In [27]:
%who_ls function
Out[27]:
['addition', 'division']

%whos

The %whos command also works like %who but it gives a little more information about variables that match the given type.

In [28]:
a = 100
b = 5.5
c = [11,12,13]
d = {'key1':'val1', 'key2':'val2'}
e = "Hello World"

%whos str
%whos dict
%whos float
%whos list
Variable   Type    Data/Info
----------------------------
e          str     Hello World
Variable   Type    Data/Info
----------------------------
d          dict    n=2
Variable   Type     Data/Info
-----------------------------
b          float    5.5
Variable   Type    Data/Info
----------------------------
c          list    n=3
out        list    n=1000000
In [29]:
%whos function
Variable   Type        Data/Info
--------------------------------
addition   function    <function addition at 0x7fdedc575620>
division   function    <function division at 0x7fdedc63ba60>

%load

The %load command accepts the filename followed by it and loads the code present in that file in the current cell. It also comments execution of itself once the cell is executed. It can even accept URL where code is kept and loads it from there.

We can use the below-mentioned options along with the command if we want to load only a particular part of the file and not the whole file.

  • -r - It accepts integer range which only loads code that falls into that range in the file.
  • -s - It accepts class or function name and loads code of that class or function into the cell rather than the whole file.

Below we have explained the usage of the command with simple examples.

In [ ]:
# %load profiling_example.py

from memory_profiler import profile

@profile(precision=4)
def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
    print(tot)

if __name__ == "__main__":
    main_func()
In [ ]:
# %load -r 5-10 profiling_example.py
def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
In [ ]:
# %load -s main_func profiling_example.py
@profile(precision=4)
def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
    print(tot)

%load_ext

The %load_ext commands load any external module library which can then be used as a magic command in a notebook. Please make a note that only a few libraries that have implemented support for jupyter notebook can be loaded. The snakeviz, line_profiler and memory_profiler are examples of it.

Below we have loaded snakeviz as an extension in a notebook. We can then use %snakeviz to profile a line of code and visualize it.

If you are interested in learning about how to use snakeviz, line_profiler and memory_profiler with jupyter notebook then please feel free to check out tutorials on the same.

In [34]:
%load_ext snakeviz
In [ ]:
def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
    print(tot)

%snakeviz main_func()

List of Useful Magic Commands in Jupyter Notebook/Lab

%unload_ext

The %unload_ext line command unload any external loaded extension.

In [105]:
%unload_ext snakeviz
The snakeviz extension doesn't define how to unload it.

%reload_ext

The %reload_ext line command reloads externally loaded extension. We can reload it if it misbehaves.

In [37]:
%reload_ext snakeviz

%tb

The %tb command stack trace of the last failure which had happened in the notebook.

In [38]:
%tb
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
~/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py in find_user_code(self, target, raw, py_only, skip_encoding_cookie, search_ns)
   3736         try:                                              # User namespace
-> 3737             codeobj = eval(target, self.user_ns)
   3738         except Exception:

<string> in <module>

NameError: name 'profiling_example' is not defined

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-31-f6e2b7c9668b> in <module>
----> 1 get_ipython().run_line_magic('load', 'profiling_example.py')

~/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2325                 kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2326             with self.builtin_trap:
-> 2327                 result = fn(*args, **kwargs)
   2328             return result
   2329

<decorator-gen-46> in load(self, arg_s)

~/anaconda3/lib/python3.7/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188
    189         if callable(arg):

~/anaconda3/lib/python3.7/site-packages/IPython/core/magics/code.py in load(self, arg_s)
    331         search_ns = 'n' in opts
    332
--> 333         contents = self.shell.find_user_code(args, search_ns=search_ns)
    334
    335         if 's' in opts:

~/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py in find_user_code(self, target, raw, py_only, skip_encoding_cookie, search_ns)
   3738         except Exception:
   3739             raise ValueError(("'%s' was not found in history, as a file, url, "
-> 3740                                 "nor in the user namespace.") % target)
   3741
   3742         if isinstance(codeobj, str):

ValueError: 'profiling_example.py' was not found in history, as a file, url, nor in the user namespace.

%env

The %env line command can be used to get, set, and list environment variables. If we call the command without any argument then it'll list all environment variables. We can give the environment variable name followed by the command and it'll return the value of that environment variable. We can also set the value of the environment variable using it which we have explained with an example below.

In [39]:
%env DESKTOP_SESSION
Out[39]:
'ubuntu'
In [40]:
%env HOME
Out[40]:
'/home/sunny'
In [41]:
%env HOME=/home/sunny/Desktop
env: HOME=/home/sunny/Desktop
In [42]:
%env HOME
Out[42]:
'/home/sunny/Desktop'

%set_env

The %set_env command lets us set the value of environment variables.

In [43]:
%set_env HOME=/home/sunny
env: HOME=/home/sunny
In [44]:
%env HOME
Out[44]:
'/home/sunny'
In [106]:
home = "/home/sunny/Desktop/"

%set_env HOME=$home
env: HOME=/home/sunny/Desktop/
In [107]:
%env HOME
Out[107]:
'/home/sunny/Desktop/'

%conda

The %conda line command lets us execute the conda package manager command in the jupyter notebook. Below we are listing down a list of available conda environments on the system.

In [45]:
%conda env list
# conda environments:
#
base                  *  /home/sunny/anaconda3
py27                     /home/sunny/anaconda3/envs/py27
py37                     /home/sunny/anaconda3/envs/py37
scalene_env              /home/sunny/anaconda3/envs/scalene_env


Note: you may need to restart the kernel to use updated packages.

%pip

The %pip line command lets us install the python module using the pip package manager in the jupyter notebook.

In [46]:
%pip install sklearn
Requirement already satisfied: sklearn in ./anaconda3/lib/python3.7/site-packages (0.0)
Requirement already satisfied: scikit-learn in ./anaconda3/lib/python3.7/site-packages (from sklearn) (0.21.2)
Requirement already satisfied: numpy>=1.11.0 in ./anaconda3/lib/python3.7/site-packages (from scikit-learn->sklearn) (1.17.1)
Requirement already satisfied: scipy>=0.17.0 in ./anaconda3/lib/python3.7/site-packages (from scikit-learn->sklearn) (1.4.1)
Requirement already satisfied: joblib>=0.11 in ./anaconda3/lib/python3.7/site-packages (from scikit-learn->sklearn) (0.13.2)
Note: you may need to restart the kernel to use updated packages.

%dhist

The %dhist command lists down all directory which was visited in the notebook. It shows the history of directories visited.

In [47]:
%dhist
Directory history (kept in _dh)
0: /home/sunny
1: /home/sunny/Desktop
2: /home/sunny

%history

The %history line command list down the history of commands which were executed in a notebook. We can use the -n option to show commands which fall in a particular range in history. We can even store a history of commands executed to an output file using the -f option followed by the file name.

In [48]:
%history -n 6-9
   6: %alias_magic currdir pwd
   7: %currdir
   8:
def addition(a,b):
    return a+b
   9: %autocall
In [49]:
%history -n 6-9 -p
   6: >>> %alias_magic currdir pwd
   7: >>> %currdir
   8:
>>> def addition(a,b):
...     return a+b
...
   9: >>> %autocall
In [50]:
%history -n 6-9 -p -f magic.out
File 'magic.out' exists. Overwrite? y
Overwriting file.
In [51]:
!cat magic.out
   6: >>> %alias_magic currdir pwd
   7: >>> %currdir
   8:
>>> def addition(a,b):
...     return a+b
...
   9: >>> %autocall

%doctest_mode

The %doctest_mode line command informs the IPython kernel to behave as much as a normal python shell which will influence how it asks for values and prints output.

In [52]:
%doctest_mode
Exception reporting mode: Plain
Doctest mode is: ON
In [53]:
a=10

a/0
Traceback (most recent call last):

  File "<ipython-input-53-f9c81372a7d7>", line 3, in <module>
    a/0

ZeroDivisionError: division by zero
In [54]:
%doctest_mode
Exception reporting mode: Context
Doctest mode is: OFF

%prun

The %prun command lets us profile python code in jupyter notebook using the profile module. It lists down the time taken by various functions. Please feel free to check our tutorial on profile to learn about profiling.

It has a list of the below options which can be useful for different tasks.

  • -l - This option accepts an integer argument followed by it which will restrict a number of lines of profiling output printed to standard output.
  • -s - This option accepts string argument followed by it and will sort profiling argument based on that string. The string is one of the column names of the profiling output. Below is a list of possible values.
    • calls
    • tottime
    • cumulative
    • file
    • module
    • pcalls
    • name
    • line
  • -T - This option saves profiling results to a file. We need to give the file name after this option to save the output to it.
  • -q - This option prevents printing of profiling results to standard output.

Below we have explained the usage of %prun with simple examples. It’s also available as a cell command.

In [56]:
import random

%prun arr1 = [random.randint(1,10) for i in range(100000)]
 
559668 function calls in 0.140 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   100000    0.042    0.000    0.102    0.000 random.py:174(randrange)
   100000    0.042    0.000    0.060    0.000 random.py:224(_randbelow)
   100000    0.019    0.000    0.122    0.000 random.py:218(randint)
        1    0.018    0.018    0.140    0.140 <string>:1(<listcomp>)
   159664    0.014    0.000    0.014    0.000 {method 'getrandbits' of '_random.Random' objects}
   100000    0.005    0.000    0.005    0.000 {method 'bit_length' of 'int' objects}
        1    0.000    0.000    0.140    0.140 <string>:1(<module>)
        1    0.000    0.000    0.140    0.140 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
In [57]:
%prun -l 10 -s tottime -T prof_res.out arr1 = [random.randint(1,10) for i in range(100000)]
*** Profile printout saved to text file 'prof_res.out'.
In [58]:
!cat prof_res.out
         560268 function calls in 0.159 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   100000    0.048    0.000    0.116    0.000 random.py:174(randrange)
   100000    0.047    0.000    0.068    0.000 random.py:224(_randbelow)
   100000    0.023    0.000    0.139    0.000 random.py:218(randint)
        1    0.020    0.020    0.159    0.159 <string>:1(<listcomp>)
   160264    0.016    0.000    0.016    0.000 {method 'getrandbits' of '_random.Random' objects}
   100000    0.006    0.000    0.006    0.000 {method 'bit_length' of 'int' objects}
        1    0.000    0.000    0.159    0.159 {built-in method builtins.exec}
        1    0.000    0.000    0.159    0.159 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

%matplotlib

The %matplotlib line command sets up which backend to used to plot matplotlib plots. We can execute a command with the --list option and it'll return a list of available backend strings. If we call command without any argument then it'll set TkAgg as backend.

In [59]:
%matplotlib --list
Available matplotlib backends: ['tk', 'gtk', 'gtk3', 'wx', 'qt4', 'qt5', 'qt', 'osx', 'nbagg', 'notebook', 'agg', 'svg', 'pdf', 'ps', 'inline', 'ipympl', 'widget']
In [108]:
%matplotlib
Using matplotlib backend: TkAgg
In [109]:
import matplotlib.pyplot as plt

plt.plot(range(10), range(10))
plt.show()

List of Useful Magic Commands in Jupyter Notebook/Lab

In [62]:
%matplotlib inline
In [ ]:
import matplotlib.pyplot as plt

plt.plot(range(10), range(10))
plt.show()

List of Useful Magic Commands in Jupyter Notebook/Lab

In [1]:
%matplotlib widget
#%matplotlib notebook
In [ ]:
import matplotlib.pyplot as plt

plt.plot(range(10), range(10))
plt.show()

List of Useful Magic Commands in Jupyter Notebook/Lab

%pdef

The %pdef command prints the signature of any callable object. We can inspect the signature of functions using this line command which can be useful if a signature is quite long.

In [67]:
def division(a, b):
    return a / b if b!=0 else 0

%pdef division
 division(a, b)
 

%pdoc

The %pdoc line command prints docstring of callable objects. We can print a docstring of the function which has a general description of arguments and inner working of the function.

In [68]:
def division(a, b):
    '''
    This function divides first argument by second. 
    It return 0 if second argument is 0 to avoid divide by zero error
    '''
    return a / b if b!=0 else 0

%pdoc division
Class docstring:
    This function divides first argument by second.
    It return 0 if second argument is 0 to avoid divide by zero error
Call docstring:
    Call self as a function.

%precision

The %precision line command sets the precision of printing floating-point numbers. We can specify how many numbers to print after the decimal point. It'll round the number.

In [69]:
%precision 3
Out[69]:
'%.3f'
In [70]:
a = 1.23678

a
Out[70]:
1.237
In [71]:
%precision 0
Out[71]:
'%.0f'
In [72]:
a
Out[72]:
1

%psearch

The %psearch line command lets us search namespace to find a list of objects which match the wildcard argument given to it. We can search for variable names that have some string present in them using this command. We have explained the usage of the command below.

In [73]:
val1 = 10
val2 = 20
val3 = 50
top_val = 10000

%psearch val*
val1
val2
val3
In [74]:
%psearch *val*
eval
top_val
val1
val2
val3

%psource

The %psource command takes any object as input and prints the source code of it. Below we are using it to print the source code of the division function we had created earlier.

In [75]:
%psource division
def division(a, b):
    '''
    This function divides first argument by second. 
    It return 0 if second argument is 0 to avoid divide by zero error
    '''
    return a / b if b!=0 else 0

%pycat

The %pycat line command shows us a syntax-highlighted file which is given as input to it.

In [5]:
%pycat profiling_example.py
from memory_profiler import profile

@profile(precision=4)
def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
    print(tot)

if __name__ == "__main__":
    main_func()

%pylab

The %pylab command loads numpy and matplotlib to work into the namespace. After executing this command, we can directly call the numpy and matplotlib function without needing to import these libraries. We have explained the usage below.

In [1]:
%pylab
Using matplotlib backend: TkAgg
Populating the interactive namespace from numpy and matplotlib
In [2]:
arr = array([1,2,3,4])

type(arr)
Out[2]:
numpy.ndarray
In [3]:
rng = arange(10)
print(rng)
type(rng)
[0 1 2 3 4 5 6 7 8 9]
Out[3]:
numpy.ndarray
In [4]:
plot(range(10), range(10))
Out[4]:
[<matplotlib.lines.Line2D at 0x7f9ee61ec588>]

List of Useful Magic Commands in Jupyter Notebook/Lab

%recall

The %recall command puts a history of the command executed in the next cell. We can give it input integer of range of integer and it'll put that many commands from history in next cell.

In [5]:
%recall 4
In [ ]:
%quickref
In [11]:
%recall 1-4
In [ ]:
%lsmagic
%magic prun
%magic prun
%quickref

%rerun

The %rerun command reruns the previously executed cell.

In [12]:
%rerun
=== Executing: ===
%recall 1-4
=== Output: ===
In [ ]:
%lsmagic
%magic prun
%magic prun
%quickref

%reset

The %reset command resets namespace by removing all user-defined names.

In [85]:
%who_ls
Out[85]:
['a',
 'addition',
 'arr',
 'arr1',
 'b',
 'c',
 'd',
 'main_func',
 'out',
 'profile',
 'rng',
 'top_val',
 'val1',
 'val2',
 'val3']
In [86]:
%reset
Once deleted, variables cannot be recovered. Proceed (y/[n])? y
In [87]:
val1
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-87-9b02d54fbb1e> in <module>
----> 1 val1

NameError: name 'val1' is not defined

%reset_selective

The %reset_selective works like %reset but let us specify a pattern to remove only names that match that pattern. Below we are only removing variables that have the string val in their name.

In [88]:
%who_ls
Out[88]:
[]
In [92]:
val1 = 10
val2 = 20
val3 = 50
top_val = 10000

a = 10
rng = range(10)

%who_ls
Out[92]:
['a', 'rng', 'top_val', 'val1', 'val2', 'val3']
In [93]:
%reset_selective -f  val
In [94]:
%who_ls
Out[94]:
['a', 'rng']

%run

The %run command lets us run the python file in the jupyter notebook. We can also pass arguments to it followed by a file name as we do from shell/command prompt. We have created a simple profiling example mentioned below and run it for explanation purposes. It also accepts the -t option which measures the running time of the file.

profiling_example.py

from memory_profiler import profile

@profile(precision=4)
def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
    print(tot)

if __name__ == "__main__":
    main_func()
In [66]:
%run profiling_example.py
1102300
Filename: /home/sunny/profiling_example.py

Line #    Mem usage    Increment   Line Contents
================================================
     4 105.6602 MiB 105.6602 MiB   @profile(precision=4)
     5                             def main_func():
     6 105.6602 MiB   0.0000 MiB       import random
     7 105.6602 MiB   0.0000 MiB       arr1 = [random.randint(1,10) for i in range(100000)]
     8 105.6602 MiB   0.0000 MiB       arr2 = [random.randint(1,10) for i in range(100000)]
     9 105.6602 MiB   0.0000 MiB       arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    10 105.6602 MiB   0.0000 MiB       tot = sum(arr3)
    11 105.6602 MiB   0.0000 MiB       print(tot)


In [67]:
%run -t profiling_example.py
1100017
Filename: /home/sunny/profiling_example.py

Line #    Mem usage    Increment   Line Contents
================================================
     4 105.6602 MiB 105.6602 MiB   @profile(precision=4)
     5                             def main_func():
     6 105.6602 MiB   0.0000 MiB       import random
     7 105.6602 MiB   0.0000 MiB       arr1 = [random.randint(1,10) for i in range(100000)]
     8 105.6602 MiB   0.0000 MiB       arr2 = [random.randint(1,10) for i in range(100000)]
     9 105.6602 MiB   0.0000 MiB       arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    10 105.6602 MiB   0.0000 MiB       tot = sum(arr3)
    11 105.6602 MiB   0.0000 MiB       print(tot)



IPython CPU timings (estimated):
  User   :      14.68 s.
  System :       3.12 s.
Wall time:      17.86 s.

Cell Magic Commands

Cell magic commands are given at the starting of the cell and apply to the whole cell. It can be very useful when we want to perform some functionality at the cell level like measuring the running time of cells or profiling cell code. We'll now explain useful cell commands available in the jupyter notebook.

%%bash

The %%bash cell command lets us execute shell commands from the jupyter notebook. We can include the whole shell script into the cell and it'll execute it like it was executed in a shell.

In [24]:
%%bash

ls -lrt | grep "python"
-rw-r--r--  1 sunny sunny   3393723 Mar  4  2020 How to build dashboard using Python (Dash & Plotly) and deploy online (pythonanywhere.com).html~

%%script

The %%script cell command lets us execute scripts designed in different languages like Perl, pypy, python, ruby, and Linux shell scripting. We need to give the language name followed by the command and it'll execute shell contents using the interpreter of that language.

In [25]:
%%script bash

ls -lrt | grep "python"
-rw-r--r--  1 sunny sunny   3393723 Mar  4  2020 How to build dashboard using Python (Dash & Plotly) and deploy online (pythonanywhere.com).html~
In [26]:
%%script sh

ls -lrt | grep "python"
-rw-r--r--  1 sunny sunny   3393723 Mar  4  2020 How to build dashboard using Python (Dash & Plotly) and deploy online (pythonanywhere.com).html~

%%sh

The %%sh cell command let us execute UNIX shell commands into the jupyter notebook.

In [27]:
%%sh

ls -lrt | grep "python"
-rw-r--r--  1 sunny sunny   3393723 Mar  4  2020 How to build dashboard using Python (Dash & Plotly) and deploy online (pythonanywhere.com).html~

%%html

The %%html cell command renders the contents of the cell as HTML. We can keep HTML tags as input and it'll render them as HTML.

In [28]:
%%html

<h1>Heading Big</h1>
<h2>Heading Medium</h2>
<h3>Heading Small</h3>

# Comment

Heading Big

Heading Medium

Heading Small

# Comment

%%javascript

The %%javascript cell command will execute the contents of the cell as javascript. Below we have explained how we can use this cell command with a simple example. The output cell is available as element and we can modify it to append HTML. Please make a note that this command currently works only with Jupyter Lab, not with Jupyter Notebook.

In [ ]:
%%javascript

// program to find the largest among three numbers

// take input from the user
const num1 = 12
const num2 = 10
const num3 = 35
let largest;

// check the condition
if(num1 >= num2 && num1 >= num3) {
    largest = num1;
}
else if (num2 >= num1 && num2 >= num3) {
    largest = num2;
}
else {
    largest = num3;
}

// display the result
element.innerHTML =  '<h1>The Largest Number is : ' + largest + '</h1>'

List of Useful Magic Commands in Jupyter Notebook/Lab

%%js

The %%js cell command works exactly like %%javascript.

In [ ]:
%%js

// program to find the largest among three numbers

// take input from the user
const num1 = 12
const num2 = 10
const num3 = 35
let largest;

// check the condition
if(num1 >= num2 && num1 >= num3) {
    largest = num1;
}
else if (num2 >= num1 && num2 >= num3) {
    largest = num2;
}
else {
    largest = num3;
}

// display the result
element.innerHTML  = '<h1>The Largest Number is : ' + largest + '</h1>';

List of Useful Magic Commands in Jupyter Notebook/Lab

%%perl

The %%perl cell command executes cell content using Perl interpreter. We can use this command to execute Perl script in jupyter notebook.

In [53]:
%%perl

#!/usr/bin/perl
use strict;
use warnings;

print "Hello Bill\n";
Hello Bill
In [4]:
%%perl

$size=15;             # give $size value of 15
$y = -7.78;           # give $y value of -7.78
$z = 6 + 12;
print $y, "\n";
print $z, "\n";
print $size, "\n";
$num = 7;
$txt = "It is $num";
print $txt;
-7.78
18
15
It is 7

%%ruby

The %%ruby cell command executes cell content using a ruby interpreter. We can use this command to execute the ruby script in the jupyter notebook.

In [5]:
%%ruby

print "Hello, World!"
Hello, World!
In [6]:
%%ruby

tigers = 50
lions = 45

puts "There are #{tigers} Tigers and #{lions} Lions in the Zoo."
There are 50 Tigers and 45 Lions in the Zoo.

%%latex

The %%latex cell command lets us execute cell content as latex code. We can write latex code and it'll create formulas out of it. We have explained the usage of the same below with simple examples.

In [ ]:
%%latex

\begin{equation*}
   e^{\pi i} + 1 = 0
\end{equation*}

List of Useful Magic Commands in Jupyter Notebook/Lab

In [ ]:
%%latex

$
idf(t)  = {\log_{} \dfrac {n_d} {df(d,t)}} + 1
$

List of Useful Magic Commands in Jupyter Notebook/Lab

%%markdown

The %%markdown cell command lets us execute cell contents as markdown.

In [ ]:
%%markdown

# H1 Heading
# H2 Heading

* List 1
* List 2

**Bold Text**

List of Useful Magic Commands in Jupyter Notebook/Lab

%%writefile

The %%writefile cell command lets us save the contents of the cell to an output file.

In [30]:
%%writefile profiling_example.py

from memory_profiler import profile

@profile(precision=4)
def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
    print(tot)

if __name__ == "__main__":
    main_func()
Writing profiling_example.py
In [31]:
!cat profiling_example.py
from memory_profiler import profile

@profile(precision=4)
def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
    print(tot)

if __name__ == "__main__":
    main_func()

%%time

The %%time cell command works exactly like the %time line command but measures the time taken by code in the cell.

In [14]:
%%time

def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
    print(tot)

main_func()
1101325
CPU times: user 162 ms, sys: 6.25 ms, total: 169 ms
Wall time: 171 ms

%%timeit

The %%timeit cell command works exactly like the %timeit line command but measures the time taken by code in the cell.

In [16]:
%%timeit

def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)

main_func()
153 ms ± 3.7 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [47]:
%%timeit -n 5 -r 5

def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)

main_func()
153 ms ± 1.82 ms per loop (mean ± std. dev. of 5 runs, 5 loops each)
In [50]:
%%timeit -t -n 5 -r 5

def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)

main_func()
157 ms ± 2.99 ms per loop (mean ± std. dev. of 5 runs, 5 loops each)
In [51]:
%%timeit -c -n 5 -r 5

def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)

main_func()
152 ms ± 2.97 ms per loop (mean ± std. dev. of 5 runs, 5 loops each)

%%prun

The %%prun cell command profiles code of the cell exactly like the %prun profiles one line of code.

In [17]:
%%prun

def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
    print(tot)

main_func()
1098986
 
         1119655 function calls in 0.271 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   200000    0.082    0.000    0.196    0.000 random.py:174(randrange)
   200000    0.079    0.000    0.115    0.000 random.py:224(_randbelow)
   200000    0.037    0.000    0.233    0.000 random.py:218(randint)
   319612    0.027    0.000    0.027    0.000 {method 'getrandbits' of '_random.Random' objects}
        1    0.017    0.017    0.142    0.142 <string>:3(<listcomp>)
        1    0.014    0.014    0.122    0.122 <string>:4(<listcomp>)
   200000    0.009    0.000    0.009    0.000 {method 'bit_length' of 'int' objects}
        1    0.005    0.005    0.005    0.005 <string>:5(<listcomp>)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}
        1    0.000    0.000    0.270    0.270 <string>:1(<module>)
        1    0.000    0.000    0.271    0.271 {built-in method builtins.exec}
        3    0.000    0.000    0.000    0.000 socket.py:337(send)
        1    0.000    0.000    0.270    0.270 <string>:1(main_func)
        2    0.000    0.000    0.000    0.000 iostream.py:323(_schedule_flush)
        3    0.000    0.000    0.000    0.000 iostream.py:197(schedule)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.print}
        2    0.000    0.000    0.000    0.000 iostream.py:310(_is_master_process)
        2    0.000    0.000    0.000    0.000 iostream.py:386(write)
        3    0.000    0.000    0.000    0.000 threading.py:1080(is_alive)
        3    0.000    0.000    0.000    0.000 threading.py:1038(_wait_for_tstate_lock)
        3    0.000    0.000    0.000    0.000 {method 'acquire' of '_thread.lock' objects}
        3    0.000    0.000    0.000    0.000 iostream.py:93(_event_pipe)
        2    0.000    0.000    0.000    0.000 {built-in method posix.getpid}
        2    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        3    0.000    0.000    0.000    0.000 threading.py:507(is_set)
        3    0.000    0.000    0.000    0.000 {method 'append' of 'collections.deque' objects}
In [39]:
%%prun

def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
    print(tot)

main_func()
1102553
 
         1120032 function calls in 0.277 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   200000    0.082    0.000    0.199    0.000 random.py:174(randrange)
   200000    0.081    0.000    0.117    0.000 random.py:224(_randbelow)
   200000    0.037    0.000    0.235    0.000 random.py:218(randint)
   319989    0.027    0.000    0.027    0.000 {method 'getrandbits' of '_random.Random' objects}
        1    0.018    0.018    0.146    0.146 <string>:3(<listcomp>)
        1    0.014    0.014    0.122    0.122 <string>:4(<listcomp>)
   200000    0.009    0.000    0.009    0.000 {method 'bit_length' of 'int' objects}
        1    0.008    0.008    0.008    0.008 <string>:5(<listcomp>)
        1    0.001    0.001    0.001    0.001 {built-in method builtins.sum}
        1    0.000    0.000    0.277    0.277 <string>:1(<module>)
        1    0.000    0.000    0.277    0.277 {built-in method builtins.exec}
        1    0.000    0.000    0.276    0.276 <string>:1(main_func)
        3    0.000    0.000    0.000    0.000 socket.py:337(send)
        3    0.000    0.000    0.000    0.000 iostream.py:197(schedule)
        2    0.000    0.000    0.000    0.000 iostream.py:386(write)
        3    0.000    0.000    0.000    0.000 threading.py:1038(_wait_for_tstate_lock)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.print}
        2    0.000    0.000    0.000    0.000 iostream.py:310(_is_master_process)
        3    0.000    0.000    0.000    0.000 threading.py:1080(is_alive)
        2    0.000    0.000    0.000    0.000 {built-in method posix.getpid}
        3    0.000    0.000    0.000    0.000 iostream.py:93(_event_pipe)
        3    0.000    0.000    0.000    0.000 {method 'acquire' of '_thread.lock' objects}
        2    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
        2    0.000    0.000    0.000    0.000 iostream.py:323(_schedule_flush)
        3    0.000    0.000    0.000    0.000 {method 'append' of 'collections.deque' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        3    0.000    0.000    0.000    0.000 threading.py:507(is_set)
In [37]:
%%prun -l 10 -s tottime -T prof_res.out

def main_func():
    import random
    arr1 = [random.randint(1,10) for i in range(100000)]
    arr2 = [random.randint(1,10) for i in range(100000)]
    arr3 = [arr1[i]+arr2[i] for i in range(100000)]
    tot = sum(arr3)
    print(tot)

main_func()
1100623

*** Profile printout saved to text file 'prof_res.out'.
         1120156 function calls in 0.276 seconds

   Ordered by: internal time
   List reduced from 27 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   200000    0.085    0.000    0.201    0.000 random.py:174(randrange)
   200000    0.079    0.000    0.116    0.000 random.py:224(_randbelow)
   200000    0.037    0.000    0.238    0.000 random.py:218(randint)
   320113    0.028    0.000    0.028    0.000 {method 'getrandbits' of '_random.Random' objects}
        1    0.017    0.017    0.146    0.146 <string>:3(<listcomp>)
        1    0.014    0.014    0.123    0.123 <string>:4(<listcomp>)
   200000    0.009    0.000    0.009    0.000 {method 'bit_length' of 'int' objects}
        1    0.006    0.006    0.006    0.006 <string>:5(<listcomp>)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}
        1    0.000    0.000    0.276    0.276 <string>:1(<module>)
In [38]:
!cat prof_res.out
         1120156 function calls in 0.276 seconds

   Ordered by: internal time
   List reduced from 27 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   200000    0.085    0.000    0.201    0.000 random.py:174(randrange)
   200000    0.079    0.000    0.116    0.000 random.py:224(_randbelow)
   200000    0.037    0.000    0.238    0.000 random.py:218(randint)
   320113    0.028    0.000    0.028    0.000 {method 'getrandbits' of '_random.Random' objects}
        1    0.017    0.017    0.146    0.146 <string>:3(<listcomp>)
        1    0.014    0.014    0.123    0.123 <string>:4(<listcomp>)
   200000    0.009    0.000    0.009    0.000 {method 'bit_length' of 'int' objects}
        1    0.006    0.006    0.006    0.006 <string>:5(<listcomp>)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}
        1    0.000    0.000    0.276    0.276 <string>:1(<module>)

This ends our small tutorial explaining how we can use different line and cell magic commands available in the jupyter notebook. Please feel free to let us know your views in the comments section.

References



Sunny Solanki  Sunny Solanki