Updated On : Sep-01,2021 Tags os, useful-functions
Simple and Useful Functions of "OS" Module of Python

Simple and Useful Functions of "OS" Module of Python

Python provides a module named os which lets us access some of the functionalities of the underlying operating system and let us perform those functions from python. It let us performs functions like checking directories, creating directories, retrieve process details, modify environment variables, and many more. As a part of this tutorial, we'll be exploring some of the very commonly used and very often needed functions of os module. We have divided the tutorial into three sections where each section covers a bunch of os functions.

We'll start by importing os module.

In [2]:
import os

Very Commonly Used Functions

As a part of this section, we'll explore functions that will be useful very often like getting the current directory, changing the working directory, getting CPU cores, accessing/modifying environment variables, etc.

Get Current Working Directory (getcwd())

The getcwd() function returns current working directory. We can use it any time to get the current working path.

In [16]:
os.getcwd()
Out[16]:
'/home/sunny'

Change Current Working Directory (chdir())

The chdir() function accepts the string path and changes the current working directory to that path. Below we have changed the working directory to explain the usage of the function.

In [17]:
print(os.getcwd())

os.chdir("/home/sunny/Desktop")

print(os.getcwd())
/home/sunny
/home/sunny/Desktop

Get Environment Variables of OS (environ & getenv())

The os module provides us with environ attribute which lets us access and modifies environment variables. It also provides getenv() method which accepts string environment variable and returns the value of it.

Below we have retrieved few environment variables using environ attribute and getenv() method for explanation purposes.

In [22]:
print("Home Directory  : {}".format(os.getenv("HOME")))
print("User Name       : {}".format(os.getenv("USER")))
print("Desktop Session : {}".format(os.getenv("DESKTOP_SESSION")))
print("Language        : {}".format(os.getenv("LANG")))
Home Directory  : /home/sunny
User Name       : sunny
Desktop Session : ubuntu
Language        : en_IN
In [87]:
print("Home Directory  : {}".format(os.environ["HOME"]))
print("User Name       : {}".format(os.environ["USER"]))
print("Desktop Session : {}".format(os.environ["DESKTOP_SESSION"]))
print("Language        : {}".format(os.environ["LANG"]))
Home Directory  : /home/sunny
User Name       : sunny
Desktop Session : ubuntu
Language        : en_IN

Change/Add Environment Variables

We can modify existing environment variables as well as add new environment variables using environ attribute of os module. Below we have created a new environment variable and modify its value.

In [88]:
os.environ["New_ENV_VAR"] = "CoderzColumn"
In [89]:
print("User Environment Variable    : {}".format(os.getenv("New_ENV_VAR")))
print("User Environment Variable    : {}".format(os.environ["New_ENV_VAR"]))
User Environment Variable    : CoderzColumn
User Environment Variable    : CoderzColumn

Retrieve Underlying Operating System Details (uname())

The os module provides with a function named uname() which returns information about the underlying operating system as a tuple of 5 strings.

  • System Name
  • Machine Name on Network
  • OS Release
  • OS Version
  • Hardware

We can also access the underlying operating system name using name attribute of os module. It returns posix for unix and related systems, nt for windows, and java for java based systems.

In [33]:
sys_name, node_name, release, version, machine = os.uname()

print("System Name : {}".format(sys_name))
print("Node   Name : {}".format(node_name))
print("Release     : {}".format(release))
print("Version     : {}".format(version))
print("Machine     : {}".format(machine))
print("OS Name     : {}".format(os.name))
System Name : Linux
Node   Name : sunny-Vostro-3470
Release     : 5.4.0-67-generic
Version     : #75~18.04.1-Ubuntu SMP Tue Feb 23 19:17:50 UTC 2021
Machine     : x86_64
OS Name     : posix

List Down Contents of Given Directory (listdir())

We can list down all members in any given path using listdir() function. It returns a list of strings where each member is a member of the directory passed as a parameter to it. The member can be a file, directory, symbolic links, etc.

If we don't provide any path to the method then it retrieves all members of the current working directory.

In [40]:
os.listdir("/home/sunny/blogs")
Out[40]:
['terms-and-conditions.ipynb',
 '.git',
 'LICENSE',
 'Python',
 'privacy-policy.ipynb',
 '.ipynb_checkpoints',
 'Android',
 'Artificial Intelligence',
 'In Progress',
 'Digital Marketing',
 'Data Science',
 'README.md',
 'Machine Learning']

Scan Directory (scandir())

The os module provides us with method named scandir() which lets us scan any directory to get information about its members. It takes as the input directory name and returns an iterator of DirEntry instances. The DirEntry instance holds detail about members like file name, full path, file type, file stat, etc.

If we don't provide any path to the method then it'll return an iterator for the current directory. Below we have listed down the contents of the blogs directory.

In [13]:
for dr in os.scandir("/home/sunny/blogs"):
    print("Name             : {}".format(dr.name))
    print("Is File          : {}".format(dr.is_file()))
    print("Is Directory     : {}".format(dr.is_dir()))
    print("Is Symbolic Link : {}".format(dr.is_symlink()))
    print("Full Path        : {}\n".format(dr.path))
Name             : terms-and-conditions.ipynb
Is File          : True
Is Directory     : False
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/terms-and-conditions.ipynb

Name             : .git
Is File          : False
Is Directory     : True
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/.git

Name             : LICENSE
Is File          : True
Is Directory     : False
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/LICENSE

Name             : Python
Is File          : False
Is Directory     : True
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/Python

Name             : privacy-policy.ipynb
Is File          : True
Is Directory     : False
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/privacy-policy.ipynb

Name             : .ipynb_checkpoints
Is File          : False
Is Directory     : True
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/.ipynb_checkpoints

Name             : Android
Is File          : False
Is Directory     : True
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/Android

Name             : Artificial Intelligence
Is File          : False
Is Directory     : True
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/Artificial Intelligence

Name             : In Progress
Is File          : False
Is Directory     : True
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/In Progress

Name             : Digital Marketing
Is File          : False
Is Directory     : True
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/Digital Marketing

Name             : Data Science
Is File          : False
Is Directory     : True
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/Data Science

Name             : README.md
Is File          : True
Is Directory     : False
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/README.md

Name             : Machine Learning
Is File          : False
Is Directory     : True
Is Symbolic Link : False
Full Path        : /home/sunny/blogs/Machine Learning

Recursively List Down Contents of Directory (walk())

One of the most useful methods provided by os module for examining the contents of the path is walk() method. It takes as input any path and returns a list of tuples where each tuple has 3 entries.

  • Directory Name
  • Sub Directories (if any)
  • File in that Directory

If goes down recursively through all subdirectories of given input path.

In [71]:
for directory, sub_directories, files in os.walk("/home/sunny/blogs/Data Science"):
    print("Directory Name     : {}".format(directory))
    print("Sub-directories    :")
    for sub_dir in sub_directories:
        print("\t", sub_dir)
    print("Files              : ")
    for file in files:
        print("\t",file)
    print()
Directory Name     : /home/sunny/blogs/Data Science
Sub-directories    :
	 Data Visualization
Files              :

Directory Name     : /home/sunny/blogs/Data Science/Data Visualization
Sub-directories    :
	 .ipynb_checkpoints
Files              :
	 basic-principles-of-information-visualization.ipynb

Directory Name     : /home/sunny/blogs/Data Science/Data Visualization/.ipynb_checkpoints
Sub-directories    :
Files              :
	 basic-principles-of-information-visualization-checkpoint.ipynb

In [72]:
for directory, sub_directories, files in os.walk("/home/sunny/blogs/Data Science", topdown=False):
    print("Directory Name     : {}".format(directory))
    print("Sub-directories    :")
    for sub_dir in sub_directories:
        print("\t", sub_dir)
    print("Files              : ")
    for file in files:
        print("\t",file)
    print()
Directory Name     : /home/sunny/blogs/Data Science/Data Visualization/.ipynb_checkpoints
Sub-directories    :
Files              :
	 basic-principles-of-information-visualization-checkpoint.ipynb

Directory Name     : /home/sunny/blogs/Data Science/Data Visualization
Sub-directories    :
	 .ipynb_checkpoints
Files              :
	 basic-principles-of-information-visualization.ipynb

Directory Name     : /home/sunny/blogs/Data Science
Sub-directories    :
	 Data Visualization
Files              :

Execute Shell Commands (system())

The os module let us execute the underlying operating system command using system() method. It takes as input string containing the command and executes it.

Below we have executed Linux ls commands which list down the contents of a given directory.

In [76]:
os.system("ls /home/sunny/blogs")
Out[76]:
0
 Android           'In Progress'       Python
'Artificial Intelligence'   LICENSE        README.md
'Data Science'         'Machine Learning'      terms-and-conditions.ipynb
'Digital Marketing'     privacy-policy.ipynb
In [77]:
os.system("ls -l /home/sunny/blogs")
Out[77]:
0
drwxr-xr-x 2 sunny sunny 4096 Feb 17  2020  Android
drwxr-xr-x 3 sunny sunny 4096 Dec  8  2020 'Artificial Intelligence'
drwxr-xr-x 3 sunny sunny 4096 Feb  6  2020 'Data Science'
drwxr-xr-x 3 sunny sunny 4096 Aug  4  2020 'Digital Marketing'
drwxr-xr-x 2 sunny sunny 4096 Apr 23  2020 'In Progress'
-rw-r--r-- 1 sunny sunny 1070 Dec 30  2019  LICENSE
drwxr-xr-x 3 sunny sunny 4096 Jan  1  2020 'Machine Learning'
-rw-r--r-- 1 sunny sunny 4215 Feb 14  2020  privacy-policy.ipynb
drwxr-xr-x 3 sunny sunny 4096 Feb 25  2020  Python
-rw-r--r-- 1 sunny sunny   31 Dec 30  2019  README.md
-rw-r--r-- 1 sunny sunny 2685 Jan  1  2020  terms-and-conditions.ipynb

Below we have executed ls command and given the output of it to grep command which only keeps entries with string .ipynb present in them. The grep command is used to capture lines with a particular string in them.

In [78]:
os.system("ls -l /home/sunny/blogs | grep .ipynb")
Out[78]:
0
-rw-r--r-- 1 sunny sunny 4215 Feb 14  2020 privacy-policy.ipynb
-rw-r--r-- 1 sunny sunny 2685 Jan  1  2020 terms-and-conditions.ipynb

Retrieve Underlying Cores/CPU Count (cpu_count())

We can retrieve a number of CPU/cores on any computer by using cpu_count() command of os module.

In [80]:
os.cpu_count()
Out[80]:
4

Retrieve Random Bytes of Particular Size (urandom())

If you want random bytes of a particular size then urandom() command can be very useful. It takes as input integer specifying the number of bytes and returns that many random bytes. These can be useful in many cryptographic algorithms.

In [85]:
os.urandom(10)
Out[85]:
b'\x03b-\x95\x87\x9d\xc8\x85\xc8V'

Retrieve Terminal Size (get_terminal_size())

We can retrieve terminal size by using get_terminal_size() command. It returns us the terminal size of a number of columns and number of lines.

In [90]:
os.get_terminal_size()
Out[90]:
os.terminal_size(columns=80, lines=24)

Retrieve Login User Name (getlogin())

We can retrieve current logged in user name by using getlogin() function of os module.

In [92]:
os.getlogin()
Out[92]:
'sunny'

Retrieve List of Directories That Will Be Searched for Executable

If we want to know which directories will be searched when we try to execute any executable in the current operating system then we can use get_exec_path() function. It can be useful when we are facing an issue with some executables. It's better to check whether the path where executable is installed is getting checked or not.

In [161]:
os.get_exec_path()
Out[161]:
['/home/sunny/.local/psql/usr/lib/postgresql/12/bin',
 '/home/sunny/anaconda3/bin',
 '/home/sunny/anaconda3/condabin',
 '/usr/local/sbin',
 '/usr/local/bin',
 '/usr/sbin',
 '/usr/bin',
 '/sbin',
 '/bin',
 '/usr/games',
 '/usr/local/games',
 '/snap/bin']

Functions That Helps With Directories and Files

As a part of this section, we'll work with functions that can be helpful in directories and files related functionalities like creating directories, removing directories, retrieving directory stats, etc.

Create Directory

We can create a directory using mkdir() function. We need to give is an absolute or relative path. It accepts a string and creates a directory from it. It requires that in order to create a directory in a particular path, all parent directories are already created else it'll fail.

In [9]:
print("Current Directory       : {}".format(os.getcwd()))

os.mkdir("temp")
#os.mkdir("/home/sunny/Desktop/temp")

print("Is temp folder created? : {}".format("temp" in os.listdir(os.getcwd())))
Current Directory       : /home/sunny
Is temp folder created? : True

Recursively Create Directories

Sometimes we need to recursively create directories. In this kind of situation mkdir() function will fail. We can use makedirs() function for these purposes. It recursively creates all directories until the last directory.

Below we can notice that when we try to create directory dir3 using mkdir() function, it fails because the parent directories dir1 and dir2 are not created yet. The makedirs() method solves this problem.

In [10]:
print("Current Directory : {}".format(os.getcwd()))

os.mkdir("temp/dir1/dir2/dir3")
Current Directory : /home/sunny
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-10-cb9093440a59> in <module>
      1 print("Current Directory : {}".format(os.getcwd()))
      2
----> 3 os.mkdir("temp/dir1/dir2/dir3")

FileNotFoundError: [Errno 2] No such file or directory: 'temp/dir1/dir2/dir3'
In [11]:
print("Current Directory : {}".format(os.getcwd()))

os.makedirs("temp/dir1/dir2/dir3")

print("Contents of temp Folder : {}".format(os.listdir("temp")))
print("Contents of dir1 Folder : {}".format(os.listdir("temp/dir1")))
print("Contents of dir2 Folder : {}".format(os.listdir("temp/dir1/dir2")))
Current Directory : /home/sunny
Contents of temp Folder : ['dir1']
Contents of dir1 Folder : ['dir2']
Contents of dir2 Folder : ['dir3']

Remove Particular Directory

We can remove any particular directory by using rmdir() method. It takes as the input directory name and removes it. Below we have explained the example.

In [12]:
os.rmdir("temp/dir1/dir2/dir3")

print("Contents of temp Folder : {}".format(os.listdir("temp")))
print("Contents of dir1 Folder : {}".format(os.listdir("temp/dir1")))
print("Contents of dir2 Folder : {}".format(os.listdir("temp/dir1/dir2")))
Contents of temp Folder : ['dir1']
Contents of dir1 Folder : ['dir2']
Contents of dir2 Folder : []

Remove Directory and Sub-directories

The removedirs() function can be used to remove the directory and all its parent directories given in a path. Above, we deleted dir3 using rmdir() function which only deletes the given directory and not its parents.

Below we can notice that removedirs() deletes temp, dir1 and dir2 directories.

In [124]:
os.removedirs("temp/dir1/dir2")

print("Does temp folder exist? : {}".format("temp" in os.listdir(os.getcwd())))
Does temp folder exist? : False

Rename Directory/File Name

We can rename directory by using rename() function. It takes as input two strings where the first string is the current name of the directory and the second input is the new name of the directory. We can also rename file names using this function.

In [132]:
os.rename("temp/dir1", "temp/dir11")

print("Contents of temp Folder : {}".format(os.listdir("temp")))
Contents of temp Folder : ['dir11']
In [136]:
print("Contents of temp Folder Before Modification : {}".format(os.listdir("temp")))

os.rename("temp/temp.txt", "temp/temp_modified.txt")

print("Contents of temp Folder After Modification  : {}".format(os.listdir("temp")))
Contents of temp Folder Before Modification : ['dir11', 'temp.txt']
Contents of temp Folder After Modification  : ['dir11', 'temp_modified.txt']

File/Directory Stat Data

We can retrieve the stats of any file by using stat() function. It returns stat_result object which has a list of details.

  • st_mode - File Mode Bits
  • st_ino - Inode number of Unix systems and file index on windows
  • st_dev - Device identifier
  • st_nlink - Number of hard links to file/directory
  • st_uid - User id of the file owner
  • st_gid - Group id of the file owner
  • st_size - Size of file in bytes
  • st_atime - Time of most recent access in seconds.
  • st_mtime - Time of last modification in seconds.
  • st_ctime - Time of creation in seconds.
In [152]:
stat_result = os.stat("temp/temp_modified.txt")

print("File Mode Bits         : {}".format(stat_result.st_mode))
print("Inode Number           : {}".format(stat_result.st_ino))
print("Device Identifier      : {}".format(stat_result.st_dev))
print("Number of Hard Links   : {}".format(stat_result.st_nlink))
print("User Identifier        : {}".format(stat_result.st_uid))
print("Group Identifier       : {}".format(stat_result.st_gid))
print("File Size in Bytes     : {}".format(stat_result.st_size))

import datetime

print("\nRecent Access Time               : {}".format(datetime.datetime.fromtimestamp(stat_result.st_atime)))
print("Recent Content Modification Time : {}".format(datetime.datetime.fromtimestamp(stat_result.st_mtime)))
print("Recent Metadata Change Time      : {}".format(datetime.datetime.fromtimestamp(stat_result.st_ctime)))
File Mode Bits         : 33204
Inode Number           : 51380494
Device Identifier      : 2051
Number of Hard Links   : 1
User Identifier        : 1001
Group Identifier       : 1001
File Size in Bytes     : 15

Recent Access Time               : 2021-08-09 10:11:40.864738
Recent Content Modification Time : 2021-08-09 10:11:41.000735
Recent Metadata Change Time      : 2021-08-09 10:13:04.522583

Functions That Helps With Process Details

As a part of this section, we'll explain different methods which can help us retrieve information about the current process as well as other processes based on the process id.

Retrieve Current Process and Process Group ID Details

We can retrieve current process id using getpid() and getuid() methods. The getuid() method returns current process's real user id.

The getgid() method returns current process's group id. We can retrieve parent process details using getppid() and getpgrp() methods.

The getgroups() method returns a list of additional groups to which the process belongs.

In [176]:
print("Current Process ID                 : {}".format(os.getpid()))
print("Current Process ID                 : {}".format(os.getuid()))
print("Current Group   ID                 : {}".format(os.getgid()))
print("Current Process Group ID           : {}".format(os.getpgrp()))
print("Parent  Process ID                 : {}".format(os.getppid()))
print("Current Process Effective ID       : {}".format(os.geteuid()))
print("List of Groups Process Belongs     : {}".format(os.getgroups()))
Current Process ID                 : 16278
Current Process ID                 : 1001
Current Group   ID                 : 1001
Current Process Group ID           : 16278
Parent  Process ID                 : 13529
Current Process Effective ID       : 1001
List of Groups Process Belongs     : [4, 24, 27, 30, 46, 118, 126, 998, 1001]

Retrieve Process Details Based on the Process ID

We can retrieve the process group id and session id based on the process id as well. The getpgid() and getsid() methods take as an input process id and returns group id and session id of the process represented by that process id.

In [175]:
current_id = os.getpid()

print("Group   ID of Process {} is {}".format(current_id, os.getpgid(current_id)))
print("Session ID of Process {} is {}".format(current_id, os.getsid(current_id)))
Group   ID of Process 16278 is 16278
Session ID of Process 16278 is 16278

This ends our small tutorial explaining how to use a few very commonly needed functions of os module. Please feel free to let us know your views in the comments section.



Sunny Solanki  Sunny Solanki