Accelerating FPGA Development with Automated Builds
When I transitioned from research into industry, there was an amazing difference in development tools used to improve software engineering speed. After learning the ropes with all the tooling the team used, my software development speed doubled. Good development tools make a huge difference in how long engineering takes.
After moving into the firmware engineering team, my old software team lead asked me to try improve the firmware team tool set. Fortunately, my managers were flexible about what I worked on as long as I got my actual tasks done, so I put 10% of my effort to save the firmware team as much time as possible with development and test environment tools.
After 8 months of improving our tooling as my 10% side job, I believe I achieved the doubled development speed for the firmware team members. Of the tools produced, the primary improvement was Jenkins automated builds.
For software engineering, Jenkins' main purpose is to ensure the master branch of the version control is working. However, for FPGA engineering a Jenkins build system can be an enormous development speed improvement, because a major time sink in FPGA development is the extremely long build times. Jenkins can offload that build dead time to a build server and the firmware engineer can continue to be productive in the meantime.
Long build times drive a large part of developer behaviour in FPGA engineering. It creates an enormous amount of dead time that needs to be planned around. The plan of a day of work will often involve thinking around what is going to be built and when the build will finish. Firmware engineers will often check and start new builds during the weekend and in the evenings to minimise the dead time of builds during the working day.
Automated builds allows for 3 FPGA development process accelerations:
- Offloaded Build Verification
- Change Error Isolation
- Parallel Debugging Builds
Offloaded Build Verification - Jenkins running FPGA build scripts can allows for builds to be verified for synthesis/implementation errors and can provide information on timing closure. This is normally a manual task that needs to be conducted by the firmware engineer and can take several hours. During that time it is difficult to do any additional work, as further design changes can disrupt the build steps of synthesis and implementation.
However, with an automated build system the firmware engineer can simply create a change patch-set, start a Jenkins build, and move on to making the next change. Thus removing the time spent waiting for verification builds.
Change Error Isolation - To minimise the number of builds that need to be performed, firmware engineers will try and make many changes before attempting a single build. This has the negative side effect of it becoming difficult to find out what change caused an build error or timing failure. Tracking down the cause of errors within many changes can be a significant time waster.
Jenkins is able to easily both queue and parallel build many minor changes, which is normally a last resort for any firmware engineer to do manually. Development using smaller change sets makes tracking down problems easier during development. Jenkins can also provide a brute force way of finding the problems by testing very small incremental changes and will run tirelessly overnight. This is extremely time consuming for a developer to do manually, due not having to intervene at intervals and make changes between every build.
Parallel Debugging Builds - FPGA designs can be difficult to debug, due to limitations in how many probes can be inserted into the design. When trying to isolate a problem a firmware engineer will often take a guess of where the problem is and insert probes where appropriate and build. If they're wrong they would need to remove the old probes and insert news ones for their next best guess and start the next build.
Almost every firmware engineer has tried to parallel build these kinds of tests, but it is difficult to manage locally. Issues include, computer performance reductions slowing development work, having to make multiple copies of the same project, and difficulty in remembering the contents of the earlier debug builds.
Having automated builds, allows easy parallelism of the test cycle steps. While an debug build is running, the developer can focus on the thinking parts of the work, where they test using earlier debug builds or develop the next debug build.