How to setup a Python-based software development workflow for any organization
Disclaimer : this is a personal story with a few hard feelings attached. I will improve the article to make it more enjoyable later. But if you can bear with me for a moment (I know it's a long read), you'll probably find a thing or two that are worth your time. Thanks in advance.
Introduction
I want to talk today about the first technical challenge I faced in my most recent experience (at LERITY). First, a few words about LERITY. It's a small company (less than 50 employees) transitioning from mainly producing optical equipment for defense systems, to integrating advanced HMI in their solutions.
The issue they face is that they have really high ambitions, but actually miss the mark on the basics. It is true that a small company can not afford a stack based on GitLab, Confluence, JIRA, and Jenkins. At the company, most people do not even know what those things are, or how to use them. And the people who do know, are either leaving, or prevented the opportunity to setup a more modern infrastructure.
Upper management is old (almost retiring), and not tech-savvy. My first act upon taking the job, was to setup, on my own computer, an alternative stack, hoping that eventually, we would find the time to improve the infrastructure more globally. This transition is absolutely necessary because they have a history of failure to retain people at the position of HMI lead. Another engineer was there before me, and he just left at the first opportunity. Not only that, but they also work with young engineers (PhD students) on AI subjects, and there is nobody to guide them.
That responsability was recently given to someone who caused me to lose the job. It's a sad irony in a way, because I delivered the solution under extreme time pressure, going into burnout, working weekends and nights. I got so tired that I missed some administrative deadlines (including provisioning my cloud VPS account, so my website went down), I was forgetting my cup of tea in random places, lost my umbrella in the train, fell sick with fever for two days, because it was the only way my body found, to force me to rest. That same person was promoted for burning me out, using me to promote their own project, and putting in peril the major project of the company (as I was not able to respect previously agreed-upon deadlines or produce quality work under those circumstances).
That same person does not know their left from their right, the difference between scripting and compiling low-level code (writing a library, or an optimized graphics pipeline). That same person has no remorse about the harm they are causing to others, as there was a prior incident with another developer a month prior to my arrival. That same person (but upper management is the same) also does not have the reflex to ask for advice about how to perform the job, despite lacking experience in the area. It's just a disaster being taught to manage people that way, and I expect many people's careers to be destroyed by having to deal with the way that person is encouraged to do things.
So they cashed in, since the customer was happy with the prototype. To their credit, they do have a veteran developer who is extremely good, but mostly a cowboy coder (no test, no documentation, one space indentation, no spaces on a line, but I still love his code because I see the ingenuity and experience behind it). He gave me a basic framework, and I worked out the rest. But I also wrote numerous warnings, to try to alert them to the mistakes they were making. The funny thing is, I got fired for stating the truth, leaving me no time to teach them better ways to manage a software development workflow. And two days after I got fired, they wanted me back as a freelancer (only to be surprised later, that it is much more expensive, but also not always legal). All those emotional reactive decisions, that lead to increased stress and bad results, can be avoided by following a proper process, and using better tools.
A quick anecdote, a childhood friend of mine had the weird idea of using SAP as ERP of a new company of less than 5 people. This idea is so wrong, there are no words to describe it. But here you have it, people who don't know much, sometimes try with the best of intentions to use things not appropriate for them. One of the goals of this article is to explain how to leverage high quality affordable tools. So, what are the cheap alternatives, that a small company, or even an individual developer can use ?
0) Instant messaging
I don't have a preferrence here. In my days, we were having fun with "net send", until system administrators got a bit unhappy with that (and Microsoft removed the command). But there is always a need for company-wide instant messaging tools. We are the generation that grew up using SMS after all. Most organizations using a Microsoft stack will have access to Teams, others might use Slack, Google Chat exists... I mean, anything can work. Worse case, you can fall back on emails, if chatting is not the appropriate medium. Note that you can create working groups in a chat application. It is less formal, and there will be an history of the various discussions. You can post a question and get an answer a couple hours later, when someone has the time to look at it. You can ping someone if attention is required immediately. That is basically how people can work accross the world. On open source projects, there is often an IRC (internet relay chat) channel you can use to reach the core developers. A more modern form is Discord. Communication is a fundamental requirement of any project, and if your organization lacks that, find a way to get it as soon as possible. Most services are free, but you might want to spend a little money to protect your business privacy.
1) Package manager
Under Windows, I use MSYS (MinGW-64, a CygWin derivative), but the standard nowadays is chocolatey. To install software you can then type:
pacman -S <package_name>
chodo install <package_name>
Note: many of the following tools are Python software packages delivered as web sites. This means you can run the integrated server for testing purposes, but for deployment, you'd better install the nginx web server, required modules, SSL certificates, etc.
2) Source code management
The industry standard is git. Too young for CVS, but I used SVN, mercurial, and TFS in the the past. Short summary, always use git. Long answer, all systems have pros and cons, but the most productive, intuitive and supported one is git, so use that.
Then, there is the question of the interface to the SCM. TortoiseGit and TortoiseSVN, or even TortoiseHg provide a visual interface under Windows. But I don't use them at all, the command line is better. If you are just starting out, do try those, but learn the command line tools as soon as possible. Advanced use cases (like history rewriting) will require you to read the documentation, and type. Even more advanced use cases like setting up a build server will also require the use of commands that the build bot will execute.
3) Tracking features, bugs, and milestones
To keep track of features, bugs, and milestones, I propose trac-wiki. It is Python-based, and tries to not get in your way. Bare bones wiki and tracking system, which you can configure to integrate with git, SVN, or even authenticate users with LDAP (Microsoft Active Directory).
The industry standard is GitLab, which can be self-hosted (community edition). Not to be confused with GitHub, a (mostly) open source code repository for git. Other tools I used in the past are Bugzilla (php) and Redmine (Ruby). I am currenty running a kallithea (Python) instance on my personal repository on logiqub.com/repo. The reason I don't recommand kallithea, despite it being based on Python, is because issue tracking is optional. The API to plug it in, might be there, or not, I don't know.
4) Setting up continuous integration and delivery
For build automation, I recommand buildbot. The software industry standard would be Jenkins. buildbot runs a master web server, and a series of workers. The workers are configured with a repository to clone and commands to execute. You can have different builds depending on what you want to do:
Once everything is configured, every time code is committed, a job will be started, and you can know instantly if you are breaking a test or another build (for example 32 bits vs 64 bits).
Under Windows, the MSBuild executable (part of the .NET Framework) is like a XML version of GNU make. It's not too difficult to figure out what it can do, but there are many options, some of which might be compiler dependent, so I can not give too many details. I prefer to use "make" anyway.
I have seen FinalBuilder being used to create installation packages, but that's not the Unix way. Also, fiddling with the Windows registry, not really my cup of tea. I'd suggest to avoid depending on that if possible. But if you really need a graphical tool, you can consider using that (it's not free).
5) Documentation
5.1) Wiki
To document software ardhitecture, design, tool installation and usage, a wiki system works best. 10 years ago MediaWiki (the engine powering Wikipedia) was cool, but it works with php and requires a database. It's best that instead you use MkDocs. The advantage of MkDocs is that you can use flat text files instead. The files use the markdown syntax. Then the build system will generate a full website, with search bar, menus, etc. Files are also searchable offline with unix tools like grep, whereas text in a database is not searchable offline (the database is unreadable withtout using the database engine/server).
Same thing with images, like class diagrams. By the way mermaid can be used for this purpose (but I don't know how mermaid can be installed). If it's suitable, you might instead use a dedicated diagram tool like Dia or yEd, and export diagrams as images in png format.
On a side note, trac already contains a wiki, so you might just use that, but like MediaWiki, it uses its internal database instead of flat files. Flat files are so good that I personally use the Flask-FlatPages Python package on my Flask powered websites.
Another software industry standard, especially when it comes to open source projects is read the docs. It is also powered by Python, but I never installed it, and I find MkDocs websites more appealing visually. But it sure looks like a good option.
5.2) Code documenation
Finally we need a tool that parses the code, and extracts the code structure for us. Most large code bases are difficult to get into, and having source code documentation allows you to build an understanding of class relationships and the design philosophy of an application's programming interface. The actual code tends to clutter everything with too many details, that makes it difficult to get an overview of what is going on.
The industry standard is Doxygen primarily for C/C++/Java projects (but it can work in other cases). For Delphi and FreePascal there is PasDoc.
6) World class projects that using these tools
As you can see, the major technological products of our era also use these tools:
7) What happens if you do not use those tools ?
7.1) Without issue tracking
The very basics of a workflow is to list the tasks you have to do. Because there are an unlimited amount of things to do, but limited time, priorization is essential. This can be done with a simple text file:
Milestone 1
[ ] Feature X
[5] Defect Y
[x] Task Z
A blank check box [ ], means a task was not started yet. A number is used for progression (here [5] means 50%). A cross [x] means that it is done. However, there are limits to such an approach. It's not easy to share. You can put task lists in a spreadsheet, but the more users you have, the more conflicts will happen. A web server and a database will ensure concurrent access, and can also enforce a workflow. GitLab for example, will help with the code review process, where a colleague has to look over what was done before code can be merged, hopefully preventing bugs. This oversight is invaluable to share knowledge between seniors and juniors, and all those comments can be looked at by anyone else, further increasing knowledge sharing.
Without this step, you end in a situation where nobody knows what tasks were given, when they were accepted, worked on, and delivered. If a problem arises, then it's your word, against their word, a waste of time and energy. Issue trackers are good for teamwork, use them!
7.2) Without software code management (version control)
It's possible to version software by taking notes of the features, defects, tasks in a file, and archive everything (source code and binaries) each week. That way, you can just open a zip file with an editor that has a file preview feature (vim can do that), and then look at the notes. If it's a reasonable candidate, unzip the project, launch it, and figure out if it is indeed the version you were looking for.
A SCM like git, will allow you switch to any prior version of the code on the fly. "git diff" help you find out the differences between two versions, (otherwise you'd have to perform the diff manually yourself). "git bisect" will help you find which version introduced a regression. "git stash" will help you save temporary work, in case you are in a rush, and need to do something else right now. "git checkout -b <branch_name>" will create a new branch. That way you can work on several features at the same time. You could also pick up the work that was done by a colleague on his branch, to help him on a difficult problem.
Learning git is one of the best things you can do for your career as a programmer. Doing things like a normal user, means you are going to copy and paste folders, forget which folders contained what, modify the wrong files, sometimes ending in situations where the code doesn't even compile anymore. And if you are not archiving the project every week, you can lose a month worth of progress. Not to mention that by not sharing your code, nobody can have a look, and help you with the problems you are working on. Software code management tools are your friends, use them!
7.3) Without continuous integration and delivery (build automation)
When it comes to build automation, you could just manually trigger all builds, and check the results yourself. However, no programmer can remember everything, not to mention that a code change in one place, a typo, or working too fast, can cause unvoluntary mistakes. Having a clean build process helps tremendously, because you can type "make" and everything gets built. If modifying a file causes another project to fail compilation, you will know instantly. You could avoid sharing files between projects, but then you'd run in the situation where you fix a bug in one project and forget to update it in another. It's not a scalable solution. It's better to reduce duplication, and have a proper build step to catch problems early.
If you have automated tests, you can also guarantee that the software is working on a basic level. This step is especially important in a team, because you might work on something that you didn't build yourself and accidently break things. Having tests that you can run to verify that you are not breaking other people's stuff saves a lot of time. Build automation tools are good, use them!
7.4) Without documentation
Weekly team meetings, brainstorming sessions, were cancelled by the CEO. I am just astonished by that decision, because on one hand it's not surprising coming from him. After all, he doesn't say good morning, shakes hand, or eats with the people who work for him. But on the other hand, I was thinking that any leader would know that gathering people working on the same project is essential to teamwork. Even if the meeting ends up being adjourned because there is nothing to discuss, half an hour a week is not a loss for the benefit the company gets, from people being able to see who is doing what, having difficulties, bouncing off ideas, etc. Without a wiki, or a searchable index of documents, and given that project meetings are cancelled, how are people going to share knowledge ?
That was on the project management side. On the software side, I had one of my previous manager say "the code is the documentation". Obviously, that is an exaggeration. Code can become unreadable quickly, if it is not refactored regularly, and even more so, if it is not tested. Proper documentation is essential to teamwork, because most of time, you only need to leave a hint, to help your fellow colleague take over after you, saving hours of debugging trying to understand why the code does not do what it is supposed to (for example, you planned a feature but didn't implement it complety). When documenting code, describe the intent, the design decision and architecture. The low level details don't matter as much (unless a trick is used, in which case that must be documented as well).
Documentation saves time at the team level. People leave, new ones get hired. But code takes time to read and understand. The better your basic documentation, the faster a new hire can pick things up and be productive. Team meetings, wikis, and documentation generators are excellent. Use them!
Conclusion
Software is not developed today, as it was 20 years ago. A lot has changed, and not taking advantage of open source solutions (which are free) in your organization is a disadvantage compared to the competition.
If you want to learn more about the various techniques you can use to gain an edge, feel free to contact me personnally on LinkedIn. I may not have all the answers, but I have seen many, many things, and the recurring pattern I note in most companies, is that they fail to put a process in place that protects them from human deficiencies.
Our memory is limited, our energy is limited, our knowledge is limited, etc. What makes human beings incredible is the variety of tools we created over our (short) history, and how much we improved them, over the millenia. Software development is a young, rapidly evolving industry. The best developers use and/or create the best tools, because that is what human beings do.
I have a lot of empathy for the people who genuinely do not know about software craftmanship. They just don't know how much they don't know, are a bit afraid of admitting to themselves how bad they really are, how much they have to learn. and they miss out on getting better at creating an environment where programmers want to work. It's not strange that things are done a bit differently in the Silicon Valley. Their model isn't without flaws, but as industry leaders, not only they use the best tools available consistently, but they sponsor, or even create (and give away) the tools they create themselves. And the same thing is true for open source software. If you are a young engineer reading this, to get better very quickly, contribute to any open source project that interests you, to get international experience working with the best developers. They will guide and teach you. The opposite is also true, a company that doesn't have a proper culture will hamper your growth.
Tools matter, so use them! Keep your developers happy, and profit. That's basically the formula that Joel Spolsky was preaching nearly 20 years ago on his blog.
Best Working Conditions -> Best Programmers -> Best Software -> Profit!
I don't blame LERITY for failing to retain me or other developers, but their leadership really needs to ask for advice and listen to the people who have been doing software and project management for years, before making questionable decisions, and shooting themselves in the foot. They want to do innovation, but don't realize they are more than 15 years behind everyone when it comes to software development best practices. It's so different for a programmer, when the CTO or CEO has been writing code himself, and can chime in on technical issues.
To finish this article, I want to quote my friend Jonathan James about leadership :
Leadership involves more than issuing orders or making demands. It's about connecting with your team on a personal level to help them reach their full potential. It's about prioritizing their growth over your ego and building trust instead of instilling fear. True leadership empowers others rather than overpowering them.
... and link this comic which summaries the situation I fell into.
Overall, I am very glad for the short time I spent with LERITY. Wonderful colleagues, fantastic projects:
Extraordinary site. It's a campus full of nature and animals (fish, birds, rabits, sheep, trees, lakes and rivers). If I had the opportunity to finish what I started, I would definitely consider it, given proper managerial support (but that won't happen anytime soon). Heck, this was a trial period, they could have told me that they weren't able to put the needed infrastructure in place, and I would have helped them to prepare a transition, find a good candidate using my network, etc. Even now, I want to take half a day per week, to help them with versioning and building at the very least. Instead, what I take away from this experience, is all the things that you should not do if you want to keep your employees motivated, happy, and finish your projects on time.
Also, I was unsure of my abilities before, but now, I am pretty sure I can hold a senior position or higher. The knowledge gap of twenty years writing software (actively trying to learn as fast as possible), is not something that can be crossed overnight (tens of thousands of hours of experience). Working for a company where there is very little software culture has shown me that. The developers are skilled, don't get me wrong, but the environment makes everything difficult (and sometimes demoralizing). Doing crunch time, all the time, will only create low quality work, exhaustion, and mistakes. It's a lose-lose situation, and management doesn't get that. They think putting people under pressure is how to get the best out of them, because employees are slow and lazy. When in fact, you could give them the best working conditions, training, tools, and benefit from a large increase in morale and productivity.
Post scriptum
But look at me, I am past 40 years old, and now getting white hair worrying about people who don't care about me... why am I this way ? Sigh, I guess I have to. Those systems are meant for national defense. There are human lives at stake, safety is of utmost importance. I have been a soldier. Mixed feeling. I am relieved that I do not have the responsibility for a system that is going to fail in production. But on the other hand, staying silent, when we have seen what happens to companies led by incompetent leaders: Boeing planes crashing, Tesla cars killing, etc.
You can not deliver defense systems by working like this. I have to let this out of my heart, to clear my conscience, as I have done everything I could. If anyone is working with defense system providers, be very mindful of their internal processes, and ask for an audit of their software development workflows. At the very least, if they are serious, they should grant you access to a list of the features and bugs they are working on for your project. If they can't do that, don't work with them, they are not serious people.