BMW CarIT Logo Typo

Blog

News, ideas and events

Blog

Two Python modules I wouldn’t go without

Background

Recently I was working on a Python based project. Some of the modules were written from scratch.
In this post I would like to share with you two built-in Python modules which are easy to use and should be in every Python programmers arsenal: logging and argparse
There is good and in-depth documentation out there on these two topics, so consider this blogpost as a gentle primer. I definitely recommend reading the linked documents referenced below.

Context

To demonstrate the usefulness of these modules, I came up with a simple staged example using Python 2.7. We will develop a simple command line tool that will perform some calculations for us.

Logging

Most of the time such command line tools start as simple scripts being only a few lines long. So here is our first version. After some coding we have some wanted output and lots of debugging print statements.

worker.py debug output with print statements
def do_some_work(counter):
    print 'do_some_work', counter
    print 'Counter -> Result'
    print '-----------------'
    while counter >= 0:
        result = _internal_calculation(counter)
        print '%4d  ->  %.3f'%(counter, result)
        counter -= 1
 
def _internal_calculation(number):
    print '_internal_calculation', number
    if number == 0:
        print 'Error Zero is not allowed'
        raise Exception('Zero is not allowed')
    result = 1./number
    print 'returning', result
    return result
 
if __name__ == '__main__':
    do_some_work(5)

Letting this code run produced the following output:

python worker.py
do_some_work 5
Counter -> Result
-----------------
_internal_function 5
returning 0.2
   5  ->  0.200
_internal_function 4
returning 0.25
   4  ->  0.250
_internal_function 3
returning 0.333333333333
   3  ->  0.333
_internal_function 2
returning 0.5
   2  ->  0.500
_internal_function 1
returning 1.0
   1  ->  1.000
_internal_function 0
Error Zero is not allowed
Traceback (most recent call last):
  File "worker.py", line 22, in <module>
    do_some_work(5)
  File "worker.py", line 8, in do_some_work
    result = _internal_function(counter)
  File "worker.py", line 16, in _internal_function
    raise Exception('Zero is not allowed')
Exception: Zero is not allowed

The tool output is cluttered with our debugging print statements.

To overcome this situation we will use the Python logging module[1]. The module has some predefined logging levels: CRITICAL, ERROR, WARNING, INFO, DEBUG. To use it we simply import the module with import logging. After the import we can use the basic logging functionality like logging.debug(msg).

Here is a simple example.

simple use of the python logging module
import logging
 
logging.debug('Test %d', 123)
logging.info('Test %d', 1234)
logging.warn('Test %d', 12345)
python test_logging.py
WARNING:root:Test 1234

Here not all of statements are shown on the the console output, because the default log level is WARNING.

In our little example we replace all our debugging print statements with calls to the appropriate log-level function.
We also use the provided configuration function logging.basicConfig([**kwargs]) to customize the output further. For our command line tool all our logging should be sent to a file and with the log level set to DEBUG.
To add information from an exception into the log add the keyword argument exc_info set to True to the logging function call.

worker.py using logging module
import logging
 
def do_some_work(counter):
    print 'do_some_work', counter
    print 'Counter -> Result'
    print '-----------------'
    while counter >= 0:
        try:
            result = _internal_calculation(counter)
            print '%4d  ->  %.3f'%(counter, result)
        except:
            logging.error('bad result for counter:%d',counter,  exc_info=True)
        counter -= 1
 
def _internal_calculation(number):
    logging.debug('_internal_calculation %d', number)
    if number == 0:
        logging.error( 'Error Zero is not allowed')
        raise Exception('Zero is not allowed')
    result = 1./number
    logging.debug( 'returning %f', result)
    return result
 
if __name__ == '__main__':
    logging.basicConfig(filename='simple_app.log', level=logging.DEBUG)
    do_some_work(5)

Letting this code run results in this output

python worker.py
Counter -> Result
-----------------
   5  ->  0.200
   4  ->  0.250
   3  ->  0.333
   2  ->  0.500
   1  ->  1.000

and creates this log file

simple_app.log
INFO:root:do_some_work 5
INFO:root:_internal_calculation 5
DEBUG:root:returning 0.200000
INFO:root:_internal_calculation 4
DEBUG:root:returning 0.250000
INFO:root:_internal_calculation 3
DEBUG:root:returning 0.333333
INFO:root:_internal_calculation 2
DEBUG:root:returning 0.500000
INFO:root:_internal_calculation 1
DEBUG:root:returning 1.000000
INFO:root:_internal_calculation 0
ERROR:root:Error Zero is not allowed
ERROR:root:bad result for counter:0
Traceback (most recent call last):
  File "simple_test_logging.py", line 9, in do_some_work
    result = _internal_calculation(counter)
  File "simple_test_logging.py", line 19, in _internal_calculation
    raise Exception('Zero is not allowed')
Exception: Zero is not allowed

The command line output is now separated from our debugging output. For production use of the tool we can the reduce the amount of output in our logfile by setting level=logging.INFO or level.logging=WARNING.

This is all on the logging topic. We have only covered the tip of the iceberg, there is excellent information and more examples on the official Python websites (see [2], [3])

Links

[1] https://docs.python.org/2/library/logging.html
[2] https://docs.python.org/2/howto/logging.html
[3] https://docs.python.org/2/howto/logging-cookbook.html

Argument parsing

Assuming our little tool works, we would like to extend it to accept two input parameters: The counter value to start at and an optional end value for the iteration.
The simple approach would be to just extend the code to parse the command line parameters sys.argv. If there was only one parameter we could come up with this simple solution:

worker.py parsing the command line option
import sys
 
...
 
if __name__ == '__main__':
    counter =0
    if len(sys.argv) > 1:
        counter = int(sys.argv[1])
    do_some_work(counter)

Introducing the second parameter or a simple help output for the user (which parameter is what, help, etc.) would be much more effort. However this is where the Python argparse module can help. It was introduced in Python 2.7 and should therefore be available to you. If you are bound to an older version of Python you can use the deprecated optparse module [3].

The argparse module is a neat library which handles all issues around argument parsing and outputting help texts on the command line for our little tool. To utilise it, simply import argparse, instantiate an ArgumentParser and use add_arguments.

In a command line script we then let the module perform its magic by calling parse_args(). Without an input value this will default to the sys.argv input of our script.

import logging
import argparse
 
def do_some_work(counter, end):
    logging.info('do_some_work counter %d end %d', counter, end)
    print 'Counter -> Result'
    print '-----------------'
    while counter >= end:
        try:
            result = _internal_calculation(counter)
            print '%4d  ->  %.3f'%(counter, result)
        except:
            logging.error('bad result for counter:%d',counter,  exc_info=True)
        counter -= 1
 
def _internal_calculation(number):
#code omitted
 
if __name__ == '__main__':
    logging.basicConfig(filename='simple_app.log', level=logging.DEBUG)
    parser = argparse.ArgumentParser(description='do calculations on some integers')
    parser.add_argument('-c','--counter', metavar='counter', type=int, help='int value to start counting down', default=5)
    parser.add_argument('-e','--end', metavar='end', type=int, help='int value to stop counting at (included)', default=0)
    args = parser.parse_args()
    do_some_work(args.counter, args.end)
python simple_test_logging.py -c 7 -e 2
Counter -> Result
-----------------
   7  ->  0.143
   6  ->  0.167
   5  ->  0.200
   4  ->  0.250
   3  ->  0.333
   2  ->  0.500

Getting help or showing errors is also done by the argparse module:

python simple_test_logging.py -h
usage: simple_test_logging.py [-h] [-c counter] [-e end]
do calculations on some integers
optional arguments:
  -h, --help            show this help message and exit
  -c counter, --counter counter
                        int value to start counting down
  -e end, --end end     int value to stop counting at (included)

Now we have some nice help output in case the user needs help or enters illegal values.

Links

[1] https://docs.python.org/2/library/argparse.html
[2] https://docs.python.org/2/howto/argparse.html
[3] deprecated: https://docs.python.org/2/library/optparse.html

Conclusion

The two Python modules logging and argparse are easy to use and well documented. There is no excuse for not using them.

For my projects I compiled a simple template to start with:

command_line_tool_template.py
import logging
import argparse
 
def main(arg1, arg2):
    logging.info('%s %s', arg1, arg2)
 
if __name__ == '__main__':
    logging.basicConfig(level=logging.DEBUG)
    parser = argparse.ArgumentParser(description='TODO put description here')
    parser.add_argument('-o','--one', metavar='one', type=int, help='TODO', default=1)
    parser.add_argument('-t','--two', metavar='end', type=int, help='TODO', default=2)
    args = parser.parse_args()
    main(args.one, args.two)