Debugging OpenStack services

Debugging is important aspect and skill that every software engineer must have. Especially when dealing with big projects like OpenStack, it’s always important to understand the workflow before you jump into fixing the things out and it can help you in asking right question in community.

Here are some pointers which I commonly used.

Enable debug mode for an OpenStack service

This is the default choice for almost all OpenStack developers. If you think you are good at regular expressions and can map the things. This gives good trace of what went wrong.

Most OpenStack services use the same configuration options to enable the debug logging that is also used to troubleshoot your OpenStack environment.

To enable the debug mode for an OpenStack service:

1. Log in to each controller node.
2. Locate and open the required OpenStack service configuration file in the /etc directory, for example, /etc/nova/nova.conf.
3. In the DEFAULT section, change the value for the debug parameter to True:

debug = True

4.   Restart the service from screen

Debugging OpenStack services with PDB

PDB is common debug tool and used widely. Same can be used for debugging OpenStack services as well. For the illustration I have used the python debugger to debug Cinder 3par driver code. Below is some scenario that I have tried out with Volume creation task on Cinder volume service.

1)     Stop the cinder volume service running from the screen.

• Go the the screen with Screen –r
• Go to Cinder volume service Ctrl-a p/ Crtl-a n
• Ctrl-C

Add the following to your code where you want to start debugging.

1. Add to Imports : Import pdb
2. Add below line where you want to start debugging from :  pdb.set_trace()
3. Next , Run the Volume service manually from the screen console as below

/usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf

The control will automatically halt at the above line where set_trace is added.

   def create_volume(self, volume):

       pdb.set_trace()

       common = self._login()

       try:

           return common.create_volume(volume)

       finally:

           self._logout(common)

Eg:

> /opt/stack/cinder/cinder/volume/drivers/hpe/hpe_3par_fc.py(184)create_volume()

-> common = self._login()

(Pdb)

(Pdb)

You can step into functions from this prompt, execute line by line, etc (execute whatever is possible with PDB) .

You can run all the PDB commands here.

Enjoy Debugging