Bad Dreams and Software Development

I just woke up from a dream. A bad dream. About software development.

In my dream I’m arguing about secure code development; how to write code that doesn’t introduce security problems into an application. Sometimes this is called Software Assurance. The discussion is not going well. The software developers want me to go away and install a firewall, or write a password policy, or do, you know, something “security-ish”. Something that doesn’t involve them. I want them to pull their head out. The dream was not pretty.

My core argument of my dream was not some buzzword driven monologue about sprint contracts, or STIGs, or code scanners, or settings for yet-another-silver-bullet-tool-to-buy. These are great add-ons, but don’t help if the fundamental engine driving your development is misfiring. My argument was about four fundamental principles that would wipe out 95% of all software security issues. Four fundamental principles that would also reduce rework, increase stability, and, in the long run, reduce development costs. Not that any software effort needs to worry about these issues, in addition to say, security. But just in case anyone else out their wakes up from a similar bad dream at 4:38 in the morning, I’d like to share my argument and trade feedback.

My four fundamental principles are:

1. THINK
2. Step off the Happy Path
3. Get Feedback
4. RTFM

I credit IBM for the first. They used to push “THINK” as logo. Too bad they dropped it (or did they sell it?) The concept is simple and wonderful -- stop periodically and actively engage yourself with what you are doing. THINK. Don’t let momentum or convention take you down a road that leads to misery. People allow this to happen because it’s easy, but in software development easy rarely leads to happy. Technical activities, particularly highly technical activities, do not benefit from a zombie like shuffle to oblivion. Stop. Breath. Lift up your head. Think about what you are trying to accomplish. Why are you doing what you’re doing? How are you doing what you’re doing? I see so many security flaws because people are on auto-pilot. This is particularly bad when different teams have their auto-pilot set to slightly different courses, none of which lead to the desired goal. These discontinuities are fertile ground for security defects to pop up. Attackers want to break you system. The more cracks you have the more wedges attackers can place to pry your system open. Any successful attack is really just a series of mistakes and bugs that an attacker has threaded together to reach their goal. Get your developers to engage, really engage, in their job and think about what they need to do. The end result will be a smooth shiny surface that attackers slip off from instead of grab hold.

Part of that active engagement thinking about how things might go wrong. Engineers and developers love to build things. They are not, by nature, vandals. They love to focus on the most efficient and elegant path from A to B. They assume everything will go their way, because, well, why wouldn’t it? It’s a path of sunshine, clean air, error free connections, unlimited bandwidth, infinite processing capability, perfect end users, and crystal clear interface documents. It is the Happy Path. But the Happy Path doesn’t consider how things fail or what happens when someone doesn’t do what you ask them too. The Happy Path is about success and smooth roads. Unfortunately this often leads to not-so-happy outcomes.

Once, while working on a production system, I noticed a core file appear. Hmmm….. that didn’t seem like a good thing. I pointed the core file out to the lead developer. His face immediately contorted. Our conversation went something like this:

Lead Developer: “Someone entered the date wrong”.

Me: “What???? How did that lead to a core file?”

Lead Developer: “An operator entered the day before the month. We’ve told them not to do that. They have to type the date as MMDDYY. If they switch the day and month, the app can seg-fault and then jack up all the down stream processing. We’ve got to kick them off and restart everything. Maybe rollback the database. How many times to we have to tell them how to enter the date?”

Me: “WTF???????”

I’ll stop the recollection there, but let me state a general opinion of mine: if a user entry causes your code to core, your code is crap. You can guess how the rest of the conversation went. This developer was a huge believer in the Happy Path. So much so that he and his group allowed a simple error to kill and hang multiple processes and possibly corrupt a database. A production database. Why? Because they weren’t willing to consider the possibility that a user might screw up the free form entry of six digits. How many code injection security flaws have been spawned because of the blissful embrace of the Happy Path?

Developers need to look past the Happy Path and think about how things might fail. What happens if this message is mal-formed? What if this user input isn’t what we expect it to be? What happens if I ask for the updated time and the clock say the the time is now 15 minutes in the past? What happens if that UDP message doesn’t arrive? Hmmm???? Every software architecture and design effort should make some effort to consider how their system make break and how to gracefully recover. Sometimes that graceful recovery is a simple stop and exit, but it should never be path off the tracks and into the jungle of unlimited attacks.

What’s the best way to jump off the Happy Path? Get feedback. While this should be fundamental to any software effort, all too often this is a checkbox. People don’t take time to THINK. Code reviews should be a time to really, really look at code. Developers need people to ask them why they are doing something, not how (that’s in the code). Why do they think this will work? Why is this input check good enough? Any person on a review team who shows up with five grammatical errors to fix in the comments and nothing else should be banned if not fired. Developers need to actually talk to other developers about how an interface or API will work. Especially for new and custom interfaces, developers need to sit with their peers and work through the dynamics of how their code will interact with the system. UML is not a way to pad hours during downtime. Developers should be working together with whiteboards, or paper, or a stick and dirt, but they have to communicate and give each other feedback. The output from a code scanner needs to be part of the review process with peers, not just an attachment on a QA form. Early feedback with Security Engineers (“why” are they doing something and “how” are they doing it) can stamp out small fires before they become infernos of leaked data. Developing software in a bubble only works until you walk into someone with a spike. Software efforts will be better if that happens sooner than later. 

And finally, what should people do before they head to that code review or design meeting? RTFM. Don’t assume you know how an API works because you used it three versions ago. The world changes and most vendors are good about describing those changes in a document or man page. Read RFCs and standards. How many code reviews have I been to where htons()/htonl()/ntohs() were not used correctly? Network order is basic stuff that is overlooked because people code to what works, not what specification says. But wait you say, how is that related to security? Simple: when you see developers are not reading and following basic guidance, that is the canary in the mine telling you bigger problems are around the bend. The Government has done some wonderful work in creating STIGs that catch all sort of low hanging fruit. Read them. Microsoft has great documentation on how to use their APIs securely. Make them part of your in-house training. Developers who don’t take time to understand the ground they are building their software on are simply creative typists. No one knows everything, and even those who do need an occasional refresher. Read the manuals. Developers need to constantly retrain themselves. So do Security Engineers. This helps trim off all the low hanging security flaws so you can focus on the upper branches. 

Well, my rant is over. I’ve got two cats who want to be fed, a dog who wants to be walked, and two kids to get up and get ready for swim. I hope your dreams are better than mine, but if they are not, try getting your developers to 1) THINK, 2) Get off the Happy Path, 3) Get Feedback and 4) RTFM. Get these four cylinders firing before you move on to fancy tools and appliances. A turbo-charger is no good if you don’t have a spark. Good luck.

jon <|>