Analysis of Custom-defined Software Metrics for an Open-source RTOS
Software analytics distills incomprehensible vast amounts of data that is accumulated during the software development lifecycle into small pieces of valuable information (software metrics/measures or on a higher level, key performance indicators). This helps to ensure maintainable and testable code, to identify vulnerable code or to improve development processes [1]. The history of software analytics goes back to the 1970s when Thomas McCabe introduced the term cyclomatic complexity to manage and control software complexity [2]. The state of the art regarding security metrics is described in [3].
One driver of software analytics is the desire of companies to make use of their available software expertise also in terms of custom-defined metrics [4]. These metrics are then not only utilized for code that has been developed in-house but also for open source software, that has become an important factor in today’s software development.
This post covers how custom measures can be defined and then be used in a dashboard with Vector’s Squore [5]. As an example, results for the Zephyr project [6], an open-source RTOS, especially designed for IoT embedded devices, are visualized in Squore.
Development data can be imported into the Squore platform from many different types of sources, e. g. static analysis tools or dynamic test tools. For the purpose of this post, we simply use Squan, a source code analyzer that comes with Squore, and Cppcheck [7], a static analysis tool for C/C++. When Squan analyzes source code, it determines various base measures for this code. Examples are the number of loops (LOOP), the number of IF statements (IF), the nesting level (NEST) or a stability index (SI), that shows how much the code has changed compared to the previous version.
The XML snippet below indicates how two (admittedly artificial) custom measures can be defined based on Squan base measures for different levels within the project (function level, file level and package level). CUSTOM_COMPLEXITY calculates the sum of LOOP, IF and NEST. WEIGHTED_CUSTOM_COMPLEXITY in addition factors in the extent of source code changes (as SI goes down with a lot of changes, the WEIGHTED_CUSTOM_COMPLEXITY goes up).
When the XML configuration is read into Squore, the Dashboard Editor can be used to put together a custom dashboard that makes use of the newly defined measures. Charts can be added with the Chart Editor that is shown in the screenshot.
After different software versions for the Zephyr project are cloned from github and analyzed with Squore, a simple custom dashboard showing CUSTOM_COMPLEXITY and WEIGHTED_CUSTOM_COMPLEXITY looks like below. Two example conclusions that could be drawn from the graphs might be: 1.) A clean-up has taken place in version 1.6.0 that has reduced CUSTOM_COMPLEXITY. 2.) Major changes in complex software parts in version 1.12.0 made WEIGHTED_CUSTOM_COMPLEXITY rise even though CUSTOM_COMPLEXITY went down.
It is worth mentioning that Squore also comes with full-fledged pre-defined dashboards. The screenshot shows the Code Quality dashboard that makes use of the Cppcheck and Squan results for the different Zephyr versions.
Software development lifecycles produce very large data sets. Valuable information that is hidden in these data sets can be made accessible with the help of software analytics. This allows well-informed, justifiable decision making instead of gut feeling. As the analysis of the Zephyr project has shown, Squore is well-suited for the determination and analysis of non-obvious software characteristics. For continuous quality monitoring and reporting including custom metrics, a DevOps pipeline integration is possible.
[1] http://menzies.us/pdf/18analytics.pdf
[2] https://www.academia.edu/8103905/A_Complexity_Measure
[3] https://collaboration.csc.ncsu.edu/laurie/Papers/SecurityMetricsSMS.pdf
[4] https://youtu.be/vad9F4Wh8fc?t=289
[5] https://www.vector.com/int/en/products/products-a-z/software/squore/
