Profiling in Python

Identifying the parts of the code to optimize is of most significance, profiling plays key role in this step.

Profiling is dynamic programming analysis that aims to facilitate optimization by collecting statistics associated with execution of the program, profile is a set of statistics that describes how often and for how long various parts of the program executed.

Popular profiling libraries for python include Time/Timeit, cProfile, Py-Spy, Yappi etc., here we demonstrate profiling with cProfile.

For more details about profiling refer https://docs.python.org/3/library/profile.html#introduction-to-the-profilers

Steps :

• Define profile class (refer https://docs.python.org/3/library/profile.html#profile.Profile)

import cProfile, pstats, i
 
def profile(fnc):
     
    """A decorator that uses cProfile to profile a function"""
     
    def inner(*args, **kwargs):
         
        pr = cProfile.Profile()
        pr.enable()
        retval = fnc(*args, **kwargs)
        pr.disable()
        s = io.StringIO()
        sortby = 'cumulative'
        ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
        ps.print_stats()
        print(s.getvalue())
        return retval
 
    return inner
 
@profile
def func(self):
    # implementation.....        

• Apply the profile decorator to analyze the code snippet.
• Run the program to check the stats.
• By analyzing the time taken for each function call, we can identify the part of the code to optimize.

Sample stats

Observe the total time column to figure out function calls which are taking more compute time. From the above sample stats we can infer that oneOf method from pyparsing module is taking most of the compute time.

Once the overhead is caught, try to tune it or figure out alternative ways to handle the task.