Some thoughts on DevSecOps
In my last musing regarding trusted and onshore electronics, I looked at the future of defense electronics and more specifically reflected on how decisions we are making today will impact that future. In short, asking the question, “are we making the right decisions now for the future we want?”
With that basic question as a reference, another topic foundational to that future is DevSecOps. This article isn’t going to weigh the pros versus the cons—there’s no doubt that the tenets of DevSecOps point us in the right direction. Instead, let’s talk about three things I’m focusing on tackling to ensure that we don’t hamstring ourselves and make the most out of this opportunity to fix past mistakes.
By the way, if this is the first time you’re hearing about DevSecOps, I’d recommend a quick detour to your favorite search engine. Either that or trust me when I sum it all up like this: It’s a set of philosophies for engineering teams to live by that aims to entwine smart development practices with secure and infrastructural methodologies. DevSecOps contrasts itself to the common practice of bespoke systems that treat security as an afterthought and delivery as a one-time affair.
So, what are the three example challenges we face with DevSecOps?
1. Adding hardware to the mix
Up until this point, we’ve seen the vast majority of DoD contracts that included hardware procurement include software too. It’s likely this paradigm will shift in the coming years for a significant amount of programs, but for those that lag in breaking that mold, you have the potential for contradictory objectives. DevSecOps doesn’t account for instances when you need to fold in your co-developed hardware teams and prepare for manufacturing. In general, apps run on hardware platforms that abstract a lot of the hardware details from the software development team. Not quite true in the embedded development world or where your development is done in parallel with hardware development. How do we get the benefits of DevSecOps without divorcing the hardware development teams from the process? Effective hardware abstraction may be the right answer. In that case, I think that effective use of containerization and cloud-native computing is essential. With that said, you are banking on your hardware being secure (which might be another topic I write about).
2. Getting over the upfront costs
There’s a reason we are where we are today. It’s easier and less expensive in the immediate sense to not do DevSecOps. Immediate versus deferred gratification, and the latter never wins when you have Earned Value Management numbers you’re trying to hit. Code coverage needs to be improved, automated tests need to expand, and the code itself reviewed more carefully for cyber vulnerabilities. Test is one of the most expensive aspects of development, but even more expensive is having to reapportion designs because different system components didn’t meet their specifications. This happens all the time and we don’t find out until late in the integration process. Continuous integration and continuous delivery allows this problem to be identified much earlier in the design process. I submit that teams would make their money back before the first major classical integration event.
3. Define for ourselves what it means to be DevSecOps
OK, your program has signed up to adhere to DevSecOps. You take a look at what’s detailed versus what your program already has implemented. If you squint one way, you’re already fully compliant! You take another, more honest look at it and you’re failing miserably. The reality is that with so much left open to interpretation, adherence isn’t being done consistently. Think of it this way, the Constitution is often left open to interpretation, but we have an entire judicial system in place to navigate the many interpretations. To avoid this, we’ll be defining our own standards of compliance with planned milestones for each project to strive toward. As previously mentioned, we know that DevSecOps tenets are the right way to go, so let’s not let subjectivity encumber our ability to achieve what they aim to resolve. Maybe I am being a little too Darwinian, but I think that this is going to be self-policing. Companies that don’t embrace DevSecOps or –at the least– its more basic design tenets may go the way of the dodo. Those that lean in and integrate it into their development process stand a much better chance of being competitive.
Really more like guidelines, anyway:
While I have mentioned the constitution, I don’t mean to make a strong comparison to such an important document. Here, there is no formative, unifying document endorsed by many. Rather, we have a basic concept and theory of bringing together the right people to do the right thing throughout product development. So where are we? *Captain Barbossa voice*, ”it is really more like guidelines anyway.” This is why some bristle with DevSecOps being issued as a “requirement” You get it and are working on it, or you will be left behind. There is a ton of room for innovation and improvement. Mandates kill innovation. This isn’t the time to overprescribe adherence. It is the time to figure out how to make this work for the DoD using real development programs.
What potential benefits do you all see with DevSecOps integrating with the DoD? Are there other problems that are worth discussing? Are design tenets and philosophies worth the discussion at this time. If not, what should be next on the agenda?
In our realm one of work one of the hardest things is being able to do Classified work on closed networks with a DevSecOps mindset. As many are arware, with CMMC being pushed hard in 2019/2020 (and 2021 being where the rubber meets the road), not being prepared to handle DevSecOps on programs will become a problem for a lot of smaller (or unprepared) organizations. Personally, I have seen DevSecOps take a front seat to most dev jobs in DoD the past 10 years. Now the fact the DoD is pushing it is a requirement in many respects, gives them a great leg up in getting them their products more regularly and, us as Contractors, the ability to provided added benefits on a continuous basis. The challenge is overcoming the red-tape. Along with this red-tape, sadly, is that the DoD Engineering workforce must overcome getting out-of their "stuck in ways". While there are an abundance of really smart people, many aren't ready to adjust to the new DevSecOps approach. Helping bring them up to speed with this mindset will offer flexibility in development and a chance at providing faster integration of core technologies into a pipeline that is secure. There are a lot of moving pieces and key pieces to reaching that goal. If you ever need some more insight in this feel free to reach out. Great article. It sure has has sparked some great conversation!
Ignoring Sec for the moment, DevOps philosophies put to practice, in general, encourage frequent, low friction, integration tests and releases. As you mentioned, they are more of a nudge in a direction than a concrete path to follow. When coupled with more modular architectures, they can be a potent force multiplier. For an organization whole-heartedly embracing these ideas, they may inevitably result in a culture shift as they encourage pipelining defined tasks, which are inherently less coupled. Following this approach can lead to reusable infrastructure, changing out widgets of functionality which have defined APIs, accelerating innovation for new use cases. As an added bonus, well defined and bounded activities lend themselves well to containerization, are easier to bid/cost and track, and are more likely to be reusable as-is, reducing future bids and aiding competitiveness. In cases where containerization and container orchestration can be used, whole multi-node physical system tests (Hardware In The Loop) can be orchestrated, run, torn down, and summarized, dramatically reducing the time and cost of testing. This enables more rich exploration of the problem space because you can A/B test algorithms, simulate portions of a system with other components stubbed out (to accelerate test times or because they're not developed yet), and capture the steps needed to run a test with IaC (Infrastructure As Code). By the way, containerization and Infrastructure As Code both capture things in a repeatable way that are often swept aside, forgotten, or inaccurate when projects are bespoke, fast moving or under pressure, and "that guy/gal" has moved on to other things. Also, regarding Hardware In The Loop, I agree that containerization should play a role in that as there are fantastic knobs, dials, and extensibility primitives in, for instance, Kubernetes that can afford engineers a great deal of control while learning a broadly-used, well-tested framework that gets better every day without additional spend. Abstracting hardware is a critical step required to empower the efficiencies that I mentioned above. In many cases, a containerized solution can be employed to interface that hardware to the cluster as if it was just another web/messaging service. As you call out in bullet #2, there is a large upfront cost to this and I have not seen many opportunities to do this right without a shift in the way programs are bid and won. If that large investment is made on the company dime then it could be a strategic advantage, but it'd also be a risk. It can be excruciating to try to get there with 100 steps spread over many programs, all performed by different individuals who may not realize they are fighting the same battle. The customer has to participate in this shift for it to be economically viable. As another aside, leveraging open standards and projects such as the many FREE offerings associated with the CNCF (Open Tracing, Open Policy Agent, Service Meshes, https://www.cncf.io/projects/), affords us a plethora of well-documented, maintained, configurable, ready-to-run tooling that are often superior to more focused, home-grown equivalents. The amount of mature, free, bolt-on functionality is literally astounding. A few man-months (per individual) of exposure and nurture will likely save man-years very quickly if a culture shift is achieved. Looking back at what I've written, I haven't talked at all about Sec! But that's the beauty of it. By standardizing how we deploy "apps" in containers, how we expose hardware to those containers, and how we orchestrate and "bolt on" things like service mesh automatic TLS, auditing, firewalling, etc, the checks we run on containers and the background traffic monitoring become mostly standardized. Enjoying the benefits won't likely come without a culture shift. Without a culture shift, there can be only reduced benefits.
Chris - great stuff. Let’s not forget to include Firmware in the mix. Is it time for Firmware to be just another form of SW to get the benefits of Dev*Ops?
Chris, on your 2nd point, I'll reinforce that DevSecOps is not the right solution for everyone. There is considerable up front cost of infrastructure development to support such tenants as CI/CD, which pays for itself in cases of extended periods of rapid development (years), and when there is complexity such as a large distributed team of developers across organizations. It's often not the right answer for initial proof of concept and rapid innovation. However, once that nugget of goodness is discovered, it's easy to think (in hindsight) those engineers should have used DevSecOps, not realizing that the team would not have achieved that proved concept or generated the key idea as quickly. So we need to figure out how the DevSecOps approach ingests R&D as input, and replaces the typical long-duration formal development cycle - potentially folding in HW co-development - to more rapidly field capabilities that are not outdated by the time they get in the hands of users. Or perhaps DoD sustainment needs to more closely align with development such that DevSecOps becomes that way that systems are maintaned and constantly upgraded, rather than focusing solely on how systems are initially developed and deployed. You could talk about the challenges of having the best Security folks, the best Development folks, and the best Operations folks actually working together as a cohesive integrated team. The organizational impacts of DevSecOps in DoD and its major contractors are at least as compelling as the technical ones.