Updated On : Jan-29,2021 Tags loggging, events
logging - Simple Guide to Log Events in Python

logging - Simple Guide to Logging in Python

In computer science, the log is the process of recording events that have happened when the software or process was running. The process of recording such events is known as logging. The log can be any event of any type, from simple debug messages about function entries, informational messages about function calls/completions, warning messages about failures that can be avoided, error messages which caused the big failure, etc. As the application gets big, it’s recommended to log its events as in case of failures, these logs can help us detect the root cause. Python has a module named logging which lets us log important messages and direct them to different destinations like standard output, files on disks, a socket on network, mails, etc. It even lets us filter log messages based on importance at any point in time. As a part of this tutorial, we'll introduce the basics of how to log messages in python using logging module. We won't go into much detail about directing log messages to different destinations, but we'll present material that will get you started generating log messages easily in python.

The logging module provides 5 important methods to log 5 different kinds of log messages.

  • debug(msg) - It helps us log messages which will only be used by the developers for debugging applications when failure happens.
  • info(msg) - It helps us log messages that will hold information about the normal running of the application like successful completion of functions, etc.
  • warning(msg) - It helps us log messages informing us that some kind of unexpected behavior or some problem has arisen but it’s not that high priority and can be handled later. It won't affect the normal run of applications.
  • error(msg) - It helps us log messages indicating some serious failure that might stop the application from running normally and needs urgent attention. It informs about the failure of some functionality of the application.
  • critical(msg) - It helps us log messages indicating that the application is faced with a fatal error and might not be able to continue normal operations. It'll need immediate attention.

The logging module has internally assigned a numeric value to the criticality of log message type (CRITICAL - 50, ERROR - 40, WARNING - 30, INFO - 20, DEBUG - 10, NOTSET - 0). This value is generally referred to as log level. The loggers generally display log messages whose log level matches or are above the set log level. The default log level set by logging module is WARNING hence it'll only print a warning, error, and critical messages by default.

We'll now explain with simple examples, how we can log messages with different log levels, how we can change the format of log messages getting display, etc.


Example 1

As a part of our first example, we have explained how we can log messages to standard output. We have designed a simple method named addition(a,b) which takes two parameter values, adds them and returns the result. We have kept the logic of converting arguments to float and performing summation in try-except block to catch exceptions and record them. We have added three different kinds of log messages in the function (DEBUG, INFO, and ERROR). We then call the addition function with different types of argument values to see log messages get printed.

We can notice from the output that it only printed the ERROR type of log message and did not print INFO and DEBUG log messages. The reason behind this is that logging module is set by default to print log messages with a log level of WARNING and above as explained earlier.

The default format of log message provided by logging is log_level:logger_name:log_message. We can see that first, it has printed log level as ERROR, then the logger name as root, and at last logging message.

The logging module lets us create more than one logger and they follow a hierarchy like a directory system. The root logger of the hierarchy has the name of root generally. This is the reason all log messages has logger name as root by default. We'll explain in the later example how we can format log messages.

logging_example_1.py

In [ ]:
import logging

def addition(a, b):
    logging.debug("Inside Addition Function")
    try:
        result = float(a) + float(b)
        logging.info("Addition Function Completed Successfully")
        return result
    except Exception as e:
        logging.error("Error Type : {}, Error Message : {}".format(type(e).__name__, e))
        return None


if __name__ == "__main__":
    result = addition(10,20)
    print("Addition of {} & {} is : {}\n".format(10,20, result))

    result = addition("20",20)
    print("Addition of {} & {} is : {}\n".format("'20'",20, result))

    result = addition("A",20)
    print("Addition of {} & {} is : {}".format("A",20, result))

OUTPUT

Addition of 10 & 20 is : 30.0

Addition of 20 & 20 is : 40.0

ERROR:root:Error Type : ValueError, Error Message : could not convert string to float: 'A'
Addition of A & 20 is : None

Example 2

Our second example builds on our first example. We have introduced two if statements in addition(a,b) which checks for type and contents of arguments. If the arguments are string and have digits only then it'll log warning messages informing that this should not happen in the future because even though currently it works fine, in the future this won't work. All the remaining code is exactly the same as our previous example.

When we run the script, we can see that it adds the warning message to the log. Our second call to the addition function passed 20 as a string which results in the logging warning message. We can notice in this example as well that log messages with a log level of WARNING and above got printed.

logging_example_2.py

In [ ]:
import logging

def addition(a, b):
    logging.debug("Inside Addition Function")
    if isinstance(a, str) and a.isdigit():
        logging.warning("Warning : Parameter A is passed as String. Future versions won't support it.")

    if isinstance(b, str) and b.isdigit():
        logging.warning("Warning : Parameter B is passed as String. Future versions won't support it.")

    try:
        result = float(a) + float(b)
        logging.info("Addition Function Completed Successfully")
        return result
    except Exception as e:
        logging.error("Error Type : {}, Error Message : {}".format(type(e).__name__, e))
        return None


if __name__ == "__main__":
    result = addition(10,20)
    print("Addition of {} & {} is : {}\n".format(10,20, result))

    result = addition("20",20)
    print("Addition of {} & {} is : {}\n".format("'20'",20, result))

    result = addition("A",20)
    print("Addition of {} & {} is : {}".format("A",20, result))

OUTPUT

Addition of 10 & 20 is : 30.0

WARNING:root:Warning : Parameter A is passed as String. Future versions won't support it.
Addition of '20' & 20 is : 40.0

ERROR:root:Error Type : ValueError, Error Message : could not convert string to float: 'A'
Addition of A & 20 is : None

Example 3

As a part of our third example, we have demonstrated how we can override the default log level (WARNING) set by logging. The code of this example is exactly the same as our previous example with only the addition of two lines which are related to overriding the default log level set by logging and printing log level.

The logging module has a method named basicConfig() let us override default log settings like log level, log format, date format, log location, etc.

We have set the log level to DEBUG as a part of this example which will result in printing all log messages with level DEBUG and above.

When we run the script, we can notice from the output that it has not printed log messages of all levels.

Please make a note that by default log levels are internally represented by an integer value. We can give level value as an integer as well and all messages with the level that and above will be printed.

logging_example_3.py

In [ ]:
import logging

logging.basicConfig(level=logging.DEBUG)

def addition(a, b):
    logging.debug("Inside Addition Function")
    if isinstance(a, str) and a.isdigit():
        logging.warning("Warning : Parameter A is passed as String. Future versions won't support it.")

    if isinstance(b, str) and b.isdigit():
        logging.warning("Warning : Parameter B is passed as String. Future versions won't support it.")

    try:
        result = float(a) + float(b)
        logging.info("Addition Function Completed Successfully")
        return result
    except Exception as e:
        logging.error("Error Type : {}, Error Message : {}".format(type(e).__name__, e))
        return None


if __name__ == "__main__":
    print("Current Log Level : {}\n".format(logging.DEBUG))

    result = addition(10,20)
    print("Addition of {} & {} is : {}\n".format(10,20, result))

    result = addition("20",20)
    print("Addition of {} & {} is : {}\n".format("'20'",20, result))

    result = addition("A",20)
    print("Addition of {} & {} is : {}".format("A",20, result))

OUTPUT

Current Log Level : 10

DEBUG:root:Inside Addition Function
INFO:root:Addition Function Completed Successfully
Addition of 10 & 20 is : 30.0

DEBUG:root:Inside Addition Function
WARNING:root:Warning : Parameter A is passed as String. Future versions won't support it.
INFO:root:Addition Function Completed Successfully
Addition of '20' & 20 is : 40.0

DEBUG:root:Inside Addition Function
ERROR:root:Error Type : ValueError, Error Message : could not convert string to float: 'A'
Addition of A & 20 is : None

Example 4

As a part of our fourth example, we have demonstrated how we can log messages to a separate file. Our code for this example is almost the same as the last example with the addition of few parameters in a call to basicConfig() method.

We have retrieved file name using file attribute of python removing .py extension. We have then given the filename ending with .log extension to filename parameter of basicConfig() method. The filemode parameter sets the file mode that will be used with the logging file. The default value of parameter filemode is 'a' which will result in appending new log messages to existing log messages every time we run the application. We have set this parameter to 'w' so that it overwrites old log messages and keeps only newly generated log messages in a log file. The choice of this parameter is up to developers.

When we ran the script, we can notice from the output that all log messages are now removed from the standard output. We have displayed the contents of file logging_example_4.log where all log messages have moved.

logging_example_4.py

In [ ]:
import logging

file_name = __file__.split(".")[0]

logging.basicConfig(filename="{}.log".format(file_name), filemode="w", level=logging.INFO)

def addition(a, b):
    logging.debug("Inside Addition Function")
    if isinstance(a, str) and a.isdigit():
        logging.warning("Warning : Parameter A is passed as String. Future versions won't support it.")

    if isinstance(b, str) and b.isdigit():
        logging.warning("Warning : Parameter B is passed as String. Future versions won't support it.")

    try:
        result = float(a) + float(b)
        logging.info("Addition Function Completed Successfully")
        return result
    except Exception as e:
        logging.error("Error Type : {}, Error Message : {}".format(type(e).__name__, e))
        return None


if __name__ == "__main__":
    print("Current Log Level : {}\n".format(logging.INFO))

    result = addition(10,20)
    print("Addition of {} & {} is : {}\n".format(10,20, result))

    result = addition("20",20)
    print("Addition of {} & {} is : {}\n".format("'20'",20, result))

    result = addition("A",20)
    print("Addition of {} & {} is : {}".format("A",20, result))

OUTPUT

Current Log Level : 10

Addition of 10 & 20 is : 30.0

Addition of '20' & 20 is : 40.0

Addition of A & 20 is : None

logging_example_4.log

INFO:root:Addition Function Completed Successfully
WARNING:root:Warning : Parameter A is passed as String. Future versions won't support it.
INFO:root:Addition Function Completed Successfully
ERROR:root:Error Type : ValueError, Error Message : could not convert string to float: 'A'

Example 5

As a part of our fifth example, we have demonstrated how we can format log messages and override default log_level:logger_name:log_message log message format set by logging. We have introduced two new parameters named format and datefmt in basicConfig() function which let us format log message and date in the log message.

The format parameter accepts a python-format string in a format %(variable_name)s. The logging module stores various logging related information like time, level name, module named, function name, line number in file, process & thread details, etc in a variable. We can format python string with these variable names to generate a log message and this format will be followed for each log message.

As a part of this example, we have divided the log message into two lines. The first line of log message consists of log time, log level name, module name, function name, line number, process details, and thread details separated by a colon. The second line of the log message has an actual log message. We have also provided a date format which we want to use to format date using datefmt parameter.

We can see a change in the default log message format when we run the script. Please notice time, module name, function name, line number, process, and thread details included in log messages.

The list of variables that we can include in the format string and their format is available at the below link.

logging_example_5.py

In [ ]:
import logging

logging.basicConfig(format="%(asctime)s : %(levelname)s : %(module)s : %(funcName)s : %(lineno)d : (Process Details : (%(process)d, %(processName)s), Thread Details : (%(thread)d, %(threadName)s))\nLog Message : %(message)s",
                    datefmt="%d-%m-%Y %I:%M:%S",
                    level=logging.INFO)

def addition(a, b):
    logging.debug("Inside Addition Function")
    if isinstance(a, str) and a.isdigit():
        logging.warning("Warning : Parameter A is passed as String. Future versions won't support it.")

    if isinstance(b, str) and b.isdigit():
        logging.warning("Warning : Parameter B is passed as String. Future versions won't support it.")

    try:
        result = float(a) + float(b)
        logging.info("Addition Function Completed Successfully")
        return result
    except Exception as e:
        logging.error("Error Type : {}, Error Message : {}".format(type(e).__name__, e))
        return None


if __name__ == "__main__":
    print("Current Log Level : {}\n".format(logging.INFO))

    result = addition(10,20)
    print("Addition of {} & {} is : {}\n".format(10,20, result))

    result = addition("20",20)
    print("Addition of {} & {} is : {}\n".format("'20'",20, result))

    result = addition("A",20)
    print("Addition of {} & {} is : {}".format("A",20, result))

OUTPUT

Current Log Level : 20

29-01-2021 05:21:59 : INFO : logging_example_5 : addition : 19 : (Process Details : (5418, MainProcess), Thread Details : (140096320595776, MainThread))
Log Message : Addition Function Completed Successfully
Addition of 10 & 20 is : 30.0

29-01-2021 05:21:59 : WARNING : logging_example_5 : addition : 12 : (Process Details : (5418, MainProcess), Thread Details : (140096320595776, MainThread))
Log Message : Warning : Parameter A is passed as String. Future versions won't support it.
29-01-2021 05:21:59 : INFO : logging_example_5 : addition : 19 : (Process Details : (5418, MainProcess), Thread Details : (140096320595776, MainThread))
Log Message : Addition Function Completed Successfully
Addition of '20' & 20 is : 40.0

29-01-2021 05:21:59 : ERROR : logging_example_5 : addition : 22 : (Process Details : (5418, MainProcess), Thread Details : (140096320595776, MainThread))
Log Message : Error Type : ValueError, Error Message : could not convert string to float: 'A'
Addition of A & 20 is : None

Example 6

Our sixth example is exactly the same as our previous example with two changes. The first change is that we have moved the addition function to addition_operations.py file. The second change is that we have changed the date format in our log message. All the remaining code is exactly the same.

The main reason to create this example was to explain how log messages work on applications with multiple files.

arithmetic_operations.py

In [ ]:
import logging

def addition(a, b):
    logging.debug("Inside Addition Function")
    if isinstance(a, str) and a.isdigit():
        logging.warning("Warning : Parameter A is passed as String. Future versions won't support it.")

    if isinstance(b, str) and b.isdigit():
        logging.warning("Warning : Parameter B is passed as String. Future versions won't support it.")

    try:
        result = float(a) + float(b)
        logging.info("Addition Function Completed Successfully")
        return result
    except Exception as e:
        logging.error("Error Type : {}, Error Message : {}".format(type(e).__name__, e))
        return None

logging_example_6.py

In [ ]:
import logging
from arithmetic_operations import addition

logging.basicConfig(format="%(asctime)s : %(levelname)s : %(module)s : %(funcName)s : %(lineno)d : (Process Details : (%(process)d, %(processName)s), Thread Details : (%(thread)d, %(threadName)s))\nLog Message : %(message)s",
                    datefmt="%d-%B,%Y %I:%M:%S %p",
                    level=logging.INFO)


if __name__ == "__main__":
    print("Current Log Level : {}\n".format(logging.INFO))

    logging.info("Main Func Started.")

    result = addition(10,20)
    print("Addition of {} & {} is : {}\n".format(10,20, result))

    result = addition("20",20)
    print("Addition of {} & {} is : {}\n".format("'20'",20, result))

    result = addition("A",20)
    print("Addition of {} & {} is : {}".format("A",20, result))

OUTPUT

Current Log Level : 20

29-January,2021 06:13:36 PM : INFO : logging_example_6 : <module> : 12 : (Process Details : (5978, MainProcess), Thread Details : (140219984185152, MainThread))
Log Message : Main Func Started.
29-January,2021 06:13:36 PM : INFO : arithmetic_operations : addition : 13 : (Process Details : (5978, MainProcess), Thread Details : (140219984185152, MainThread))
Log Message : Addition Function Completed Successfully
Addition of 10 & 20 is : 30.0

29-January,2021 06:13:36 PM : WARNING : arithmetic_operations : addition : 6 : (Process Details : (5978, MainProcess), Thread Details : (140219984185152, MainThread))
Log Message : Warning : Parameter A is passed as String. Future versions won't support it.
29-January,2021 06:13:36 PM : INFO : arithmetic_operations : addition : 13 : (Process Details : (5978, MainProcess), Thread Details : (140219984185152, MainThread))
Log Message : Addition Function Completed Successfully
Addition of '20' & 20 is : 40.0

29-January,2021 06:13:36 PM : ERROR : arithmetic_operations : addition : 16 : (Process Details : (5978, MainProcess), Thread Details : (140219984185152, MainThread))
Log Message : Error Type : ValueError, Error Message : could not convert string to float: 'A'
Addition of A & 20 is : None

This ends our small tutorial explaining a simple way to log messages in Python using logging API. The logging API is very vast and has lots of other functionalities which we'll cover in different tutorials. The main aim of this tutorial is to get beginners started with logging messages in python. Please feel free to let us know your views in the comments section. Let us know if you want to modify anything in the tutorial, add anything or have any questions. We enjoy such technical discussions.



Sunny Solanki  Sunny Solanki