I have recently been asked a lot about the NIST process mentioned in an earlier post. In a nutshell, NIST was working on a paper describing (software) security as a holistic approach, deeply embedding a security mindset into traditional systems engineering patterns with the goal of building secure and resilient systems, instead of bolting security on in a later stage of the game.

NIST has meanwhile published more current draft materials (http://csrc.nist.gov/publications/drafts/800-160/sp800_160_draft.pdf ). At this time, the link points to a draft dated May 2014. The draft was released about six months ago, and NIST 800-160 starts to pick up with the industry. I had numerous inquiries from major HP customers asking questions on HP’s internal security processes in context of NIST 800-160, and how HP is proactively dealing with security threats not yet known while the product is being built. The language of the requests with terms such as “resiliency”, “trustworthiness”, and “build security in” strongly resemble the language NIST and the Department of Homeland Security have chosen in their publications around software that should be able to not only withstand known attacks, but also be ready to resist new threats or possibly even actively defend against them.

John Diamant is an HP Distinguished Technologist and secure product development strategist in HP. He has done a lot of work on how to design security into a product, and with HP’s Comprehensive Applications Threat Analysis (CATA), John has created a great toolkit to automate parts of the security requirements and threat analysis.

John is working a lot with the US public sector, and he certainly sees a lot of the feedback HP receives around the NIST and DHS security initiatives. He has some very interesting comments on how to create secure and resilient software, and how a Secure Software Development Lifecycle (SSDLC) program fits into this: http://www.securitysolutionswatch.com/Interviews/in_Boardroom_HP_Diamant2.html

This is the last post in a series of posts where we discussed a few Secure Software Development Lifecycle (SSDLC) metrics that I personally find very interesting. The four metrics are:

1. The number of qualified defects found per quarter
2. The number of qualified defects fixed per quarter
3. The difference between [1] and [2], that is, inflow vs outflow
4. The overall number of open qualified security defects

In this post, I will share the last metric of the series, which is a graph of the overall number of open qualified security defects, and their development over time. This data is taken from the same organization and product set discussed in the first post of this series, and it has been anonymized, randomized, and transformed into a relative scale to protect confidential information without affecting the trends that are typically seen when starting a new SSDLC program. All percentages in the graph are relative to the results of the 20 Quarters before the SSDLC program was started.

Overall Number of Open Qualified Security Defects

The graph below shows how the overall number of open qualified security defects developed in the example organization. All percentages in the graph are relative to the results of the first 20 Quarters. The organization started out from a backlog of 100% of open issues, and continuously added to that backlog over the following 9 quarters. The backlog peaked at 242% of open issues, and then started to decrease slightly. This means that R&D teams more than doubled their security defects backlog, despite the great increase of fixes they released in the same time, which is a very impressive achievement.

The graph shows that the teams did a good job of keeping the number of critical issues in the backlog consistently low, and even managed to significantly reduce the number of open critical defects in Quarter 9.

We can also see from the graph that in Quarter 9, the teams managed, the first time in more than 7 years (all for which data is available for this organization), to reduce the size of the overall security backlog. This is obviously very much aligned with the inflow/outflow metric, where Quarter 9 shows a negative inflow (or a net outflow) of security issues in the backlog. This is a big achievement, and a good indication that the SSDLC program was solidly embraced by the organization’s senior leadership as well as by the organization’s engineers.

*Important: Note that an organization maintaining a security issues backlog does not necessarily mean that the organization releases products with known security vulnerabilities. Companies such as Hewlett-Packard have strong business ethics, and do their best to protect their customers from all known and unknown security risks. If a security defect in the backlog affects a version or platform that has not yet been released, or may manifest itself as a vulnerability only under certain operating conditions that are not present in supported setups, then a security defect may be backlogged without increasing the security risk for the customer.

For example, a library that is being used in a product may have a defect in input validation, thus leading to a buffer overflow in the library, ultimately affecting the product using the library. The SSDLC process would discover such an issue and track it as a security defect in the backlog. However, the “supported setup” for this library may be that it is only to be used in a product that performs its own input validation before passing any user provided data to the affected library method. As long as the library is being used in this “supported setup”, there is no security vulnerability in the final product, which means that the security defect in the library does not result in a security vulnerability and hence does not translate into a security risk for the customer. Still, the security defect of the library is being tracked in the backlog, so that a fix for the library can be obtained, or the library can be replaced.

Next Steps

One goal of an SSDLC program is to reduce risk and increase confidence that security issues are properly handled. Backed by this confidence in the SSDLC process and the quick turnaround times on security issues an organization will eventually achieve, the leadership team may define new Service Level Agreements (SLAs) for open security issues.

For example, an organization may establish strict SLAs with time windows in which security defects would have to be fixed (issue qualified, defect fixed, patch tested, patch delivered / publicly available). The organization may split this up in a matrix for low / medium / high / critical severity and the source of the defect (internally found / reported by customer under NDA / publicly known). Ideally, the organization should define another metric on how well they delivered against these SLAs, making them true security professionals!

In the first post of this series, we discussed a few Secure Software Development Lifecycle (SSDLC) metrics that I personally find very interesting. The four metrics are:

1. The number of qualified defects found per quarter
2. The number of qualified defects fixed per quarter
3. The difference between [1] and [2], that is, inflow vs outflow
4. The overall number of open qualified security defects

In this post, I will share some inflow / outflow metrics, and their development over time. This data is taken from the same organization and product set discussed in the first post of this series, and it has been anonymized, randomized, and transformed into a relative scale to protect confidential information without affecting the trends that are typically seen when starting a new SSDLC program. All percentages in the graph are relative to the results of the 20 Quarters before the SSDLC program was started.

Inflow vs Outflow

The inflow / outflow metric gives a good indication on how successful an organization is in dealing with newly found issues: can they qualify and address security defects fast, or are they overwhelmed by the influx of new issues, and just keep piling them onto a backlog?

This graph shows the difference between the number of incoming new defects and the number of defects that have been closed in the same time period. Like in the previous illustrations, the graph shows relative numbers (percentages) in relation to the results of the first 20 Quarters. Unfortunately, this makes the graph a little harder to read, because the percentages do not directly translate into the actual number of issues that have been added to the backlog. In this graph, big positive percentage numbers mean that work is added to the backlog. A negative percentage number (or a number close to 0) is desirable, because this means that the team is ahead of or at least keeping up with the influx of work.

This graph shows two peaks, which is a very common characteristic for organizations where multiple groups contribute to finding security problems, but only one group is fixing them. If we compare this to the two graphs we discussed in the previous post, we can explain the first peak by a large inflow of defects, which the developers are not yet prepared to handle. After about 12 months into the program, the engineers are starting to catch up. This is very common, and a good sign, because it reflects the learning and training that the developers have to go through to know how to deal with the findings, and then the developers applying these learnings and starting to catch up with incoming work. This first peak is usually always present when rolling out a new SSDLC program.

The second peak is very typical for an organization with dedicated QA teams or security pen testing teams. Once these teams have been ramped up, completed their training, and are fully operational, their reporting output picks up significantly – typically also after 12 months. Other than the R&D team (who may also report more defects against their code), they are usually not chartered to fix the issues they discovered. This leads to additional pressure on the developer team, and the developer team must adjust (again) to the higher influx of security issues. Once this adjustment is complete (Quarters 8 and 9), the organization reaches an ideal state of close to zero or negative inflow / outflow difference.

The graph also reveals how an organization is prioritizing work. In this case, the organization is rightly prioritizing work on critical security issues. However, we can also see that they are focusing on medium and low severity problems first, before addressing high severity issues. This may be justified, for instance if crucial resources who can deal with the high severity problems are not available as they are assigned to work on the critical severity defects, or if some of the higher severity defects are not manifesting as security vulnerabilities (e.g. due to defense in depth covering them), allowing delays to fix lower-severity problems that do lead to security vulnerabilities. This metric makes potential prioritization problems visible and actionable.

Edit: continue reading…

In last week’s post, we discussed a few Secure Software Development Lifecycle (SSDLC) metrics that I personally find very interesting. The four metrics are:

1. The number of qualified defects found per quarter
2. The number of qualified defects fixed per quarter
3. The difference between [1] and [2], that is, inflow vs outflow
4. The overall number of open qualified security defects

In this post, I will share metrics and a graph of the number of qualified security defects that have been fixed, and their development over time. As mentioned in the previous post, the data used to plot the graphs has been anonymized, randomized, and transformed into a relative scale. These transformations are necessary to protect confidential information, but have been performed in a way that does not affect the trends that are typically seen when an SSDLC program is rolled out. All percentages in the graph are relative to the results of the 20 Quarters before the SSDLC program was started.

Number of Qualified Defects Fixed per Quarter

The graph below is a hypothetical metric of the same organization discussed previously, in the same time period. It shows the anonymized and transformed results of the efforts the organization went through to close security defects.

As with the previous graph, we see a slow start. However, once the number of newly discovered defects grow, the fix rates start to go up. There can be many reasons for this behavior. Common reasons for this are typically a combination of buy-in to the program by executive management and buy-in from engineering: In a mature software organization, engineers take responsibility for their deliveries, and create pressure to deal with critical issues rather than delaying them. Often, engineering programs are most successful if both factors are present.

In the organization we use in this example, we see that R&D fixed more (about 35%!) qualified security defects in a single quarter than they did in the previous 5 years, which is an efficiency increase of several thousand percent – and this organization was still in the growing phase during Quarter 9!

Edit: continue reading…

When discussing Secure Software Development Lifecycle (SSDLC) programs, I get asked a lot whether such an investment is actually worth the effort, or whether it would be better to invest the available resources differently. It is true that running an SSDLC program actively is a big investment, and – as with any business expense – it is very reasonable to evaluate the ROI of these expenses.

As with any engineering program, having solid data is a must when communicating results to upper management. In particular the C-suite of management will request metrics that actually show that the program works as expected.

While there are dozens of metrics that can be collected when executing the SSDLC, I find four metrics particularly interesting. They deal with the inflow and processing of security defects in an organization’s portfolio. Eventually, most “relevant” findings from an SSDLC activity that are actually tracked somewhere (instead of being fixed immediately) are either security defects or security requirements. Security requirement findings are a lot harder to quantify than security defects, because they are bound to a product’s target market, that is, how a product is being used and deployed. Two identical products that are selling to two different target markets may have very different sets of security requirements. For security defects, this is similar to a certain extent. However, a security defect is usually a security defect, and only the severity rating that is attached to it changes with target markets and deployments. This makes security defects a metric which is very easy to quantify and compare.

The four metrics we will discuss in the following are:

1. The number of qualified defects found per quarter
2. The number of qualified defects fixed per quarter
3. The difference between [1] and [2], that is, inflow vs outflow
4. The overall number of open qualified security defects

During my tenure at HP, I helped several organizations within HP introduce and successfully run SSDLC programs. Such organizations in HP may be business units or groups of product teams. The data provided in this series of posts is constructed based on data I collected from an R&D organization in HP with a few hundred R&D engineers. For obvious reasons, I cannot reveal from which exact organization and what exact product set the data was derived from. To maintain additional confidentiality, I took data that is several years old and not current, changed the numbers in a way that would not change the overall pattern, and then transformed it into a relative scale. These transformations do not affected the trends in the graph, which makes the anonymized (and somewhat randomized) data good enough for this series of posts.

Number of Qualified Defects Found Per Quarter

The figure below shows the number of issues found per quarter over a time period of 10 (i.e. 9 plus 1) quarters, broken up into low, medium, high, and critical severity defects: this particular organization had a history of 20 quarters of security activities. All the results that were achieved in these 5 years are represented in the “20 Quarters” bar. The bars for Quarter 1 to Quarter 9 represent data points after the organization decided to introduce an SSDLC program. Note that all data points are relative to the results of the first 20 Quarters, as the absolute numbers are confidential and cannot be published.

The graph shows that the program started slowly, with the number of newly reported security defects per quarter being in the 10% range of the previous 20 Quarters. Then it gained traction, roughly doubling this number to about 25% per quarter relative to what was reported in the overall 20 Quarters before the program was initiated. After Quarter 7, the program really took off, and started to show fast growth on newly discovered defects.

This is a very typical pattern. Depending on the size of the organization, it usually takes 6 to 9 months after real activities (i.e. more than Powerpoint activities) have started to see any significant results. Once a few big teams or lighthouse projects within the organization are fully on board, and see the first encouraging results for their projects, managers start investing more resources and engineers start to develop a higher level of security awareness. At this point, we often see an exponential growth until a saturation level is reached, and then a slight decline to a stable level.

In this particular example, we see that this organization (which at the time was still in the growing phase) managed to find almost as many (about 75%) qualified security defects in a single quarter than they did in the previous 5 years.

Edit: continue reading…

Security static code analysis is performed together with other static code analysis methods at or before compilation time. The biggest advantage of using static code analysis tools is that they can be used in the very early stages of the projects, long before the code has reached a state where it can be executed and evaluated with traditional quality assurance methods. With the high level of automation that can be achieved, static code analysis is an ideal tool to introduce a minimum code quality level without wasting the precious time of security experts on routine manual reviews. Common automation options in particular include automated execution of code auditing at check-in to SCM, as well as automated failing of a build in centralized build environments (e.g. when using CI systems such as Jenkins).

Static code analysis tools cover a broad range of code analytics options, ranging from trivial pattern matching tools to find comments in the code like “todo” or “fixme”, up to very complex execution path analysis tools. Such advanced tools can observe, for example, a specific variable through the execution path of a method and make deductions on the reliability and sanitation status of data stored in such variables.

Depending on the policies of the organizations and products, any form of static code analysis may be security relevant. While it is common understanding that execution path analysis often provides valuable insights on security, some organizations may decide that enforcing specific code formatting is relevant to their security program. One justifications for this is that a common definition of whitespace across the project’s developer community makes semi-automated code reviews easier.

For these reasons, static code analysis is mostly policy driven, and the (secure) coding policies that an organization defines ultimately drives the selection of tools and their configuration. Consequently, the selection of approved tools and their (project specific) configuration is key in making static code analysis a significant contributing factor to application security. As with any policy, the static code analysis rule set, as well as the list of approved tools need periodic reviews to make sure that the latest advances in security research and subsequent tool improvements are incorporated into a project’s security strategy.

Once a solid approach to architecture threat analysis has been established, most of the remaining security vulnerabilities are coding problems, that is, poor implementation. Examples include injection issues, encoding issues, and other problems such as listed at the OWASP Top 10 Project.

While a checklist of best practices for developers can help with addressing some of these bad coding habits, a more structured and repeatable approach should be established as well. Static application security testing (or “Static Code Analysis” – SCA) can identify most of the code-level vulnerabilities that remain after a thorough architecture threat analysis. However, it is crucial that SCA is executed consistently and automatically.

A common best practices is to analyze any newly written source code prior to compilation or, for scripting languages, prior to promoting it for an intermediate release. Automating this process in a build system / continuous delivery tool chain makes this very scalable and can also ensure that developers follow specific secure coding best practices.

When implementing automated SCA during builds, the project owner needs to make a decision of whether to fail a product build if the SCA process fails. I generally recommend that a developer should be able to run the entire toolchain on their local machine. This allows them to run the entire build locally, as it would be executed on the central build server, with all checks and automated tests, before they commit to source control. This does not only ensure proper execution of security tests, but also other quality assurance tools such as regression testing.

The Hat of Shame (https://www.flickr.com/photos/foca/6935569551)

However, to improve their productivity, the developers must have the option to skip tests locally. To avoid developers committing bad code to source control, and so trigger unnecessary builds on the central build server, they must also have the option to configure the same tools that are used in the build chain for use in their IDE – with the same rules as used in the central project configuration, so that they can execute the SCA while they are writing the code, and justify skipping the SCA build locally before checking in.

Using such a setup ensures that developers can deliver code that meets the individual project’s standards. I generally also recommend failing the build on the central integration build server if committed code does not meet these standards. In most cases, this is caused by a developer not using the toolchain (including the local tools for the IDE) as instructed and so causing unnecessary work and delays for the rest of the team – which means that this developer is entitled to wearing the hat of shame for a while.

The processes I described earlier for security requirements analysis and architecture threat analysis earlier seem very heavy weight, and a question that I get asked frequently is how to use such processes in agile models. At this time, HP is the third largest software company in the world (measured in total software revenue, behind IBM and Microsoft). There is a huge bandwidth of software development models in HP: I have been leading secure software development lifecycle (SSDLC) programs in both HP Software and HP’s Printing and Personal Systems group, working with teams that employed traditional models (“waterfall style”) as well as with teams that used more progressive models (Scrum, XP, SAFe, etc).

With all teams I worked with, it was possible to create an SSDLC program that accommodated the individual team’s working model. As an example, while a team using a traditional waterfall model will perform the requirements and the design analysis in their “planning stage”, an agile team will commonly have already completed these activities in their previous Potentially Shippable Increment (PSI). In other words, while the majority of developers in a team that uses e.g. SAFe may be working on PSI n, part of the team has already started work on the analysis of the requirements and design that will go into PSI n+1.

The steps that need to be performed in a secure development lifecycle program are independent of the development model, but how they are scheduled and executed may be different with every organization. It is important to design the SSDLC program to match a team’s needs, and it is equally important to create metrics for the SSDLC program to match an organization – making sure that the metrics reflect not only the aspects of the SSDLC program, but also fit into the existing model of how an organization is measured.

Deadlines and budgets in software projects are usually tight, and managers often ask me which would be more beneficial for them to meet their goals: Security requirements analysis or architecture threat analysis?

Asking this question is comparing apples to oranges, as these are two different things and both of them provide substantial benefits. The security requirements analysis helps to ask the best questions to get the requirements right. Getting requirements wrong is a common issue, in particular when complex regulations are involved (which is frequently the case with security requirements). Without proper requirements analysis, the product team may end up doing the greatest job of building the wrong thing.

Architecture threat analysis ensures that the product is designed to be robust and resilient from a security perspective. If this step is omitted, the application may be riddled with security vulnerabilities – and hence not meet basic security requirements either.

The security requirements analysis ensures that the team is building the right product from a security perspective, and architecture threat analysis makes sure that they are building the product right.

Once all requirements for a project have been gathered, it commonly enters the “design” phase of the development lifecycle. In this phase, the architect(s) turn the prioritized requirements into an application blueprint for coding. This is a crucial phase during software development, as some decisions made here are irreversible once implementation has started (or, at least, very hard to change without significant investments).

During this phase, the security architect starts analyzing the overall attack surface of the design. This analysis is ideally performed on the design specifications created by the other product architects, before any code is written. The assumption is that the implementation of any software component may be flawed, and the goal is to reduce the opportunities for attackers to exploit a potential vulnerability resulting from such a potential flaw. During this step, the design may change by removing or restructuring specific components, and additional (run-time) requirements that mitigate or reduce specific risks resulting from the design may be introduced. An example for design changes may include layered defenses, while additional requirements may include restricting access to specific services and determining the level of privilege required during execution.

Common approaches to threat analysis include attack surface analysis as well as quantitative and qualitative risk analysis. Unfortunately, skilled security architects who can perform an in-depth threat analysis are rare, which usually makes this particular step the most expensive part of a secure development lifecycle program. The key to success is using the available resources wisely, and creating a repeatable and scaled approach that consistently produces high-quality results.

There are several (more or less) structured brain-storm-based approaches for threat analysis. The more structured ones commonly have some level of tool support (e.g. “threat model tools”) which helps to make the process somewhat more comprehensive, while the less structured ones depend fully on the participants’ creativity and security expertise. The inevitable variability in these factors as well as in the participants’ stamina can produce dramatically different results.

Applying a maximally structured approach to threat analysis commonly allows identifying architectural security risks more consistently, more effectively, more repeatable, and at a significantly lower cost. It also allows determining both the risks from the identified threats, and establishing appropriate mitigations. Most importantly, it also achieves completeness that a brainstorm-based approach commonly cannot guarantee: when employing a structured approach, the security architect can utilize measureable criteria to determine when the threat analysis is complete, and not simply because they “cannot think of any more problems to look at.”

Threat modeling is, for the most part, ad hoc. You think about the threats until you can’t think of any more, then you stop. And then you’re annoyed and surprised when some attacker thinks of an attack you didn’t. – Bruce Schneier