Penetration testing methodologies and frameworks

This article explores major penetration testing methodologies and frameworks. The article first discusses the main features and implementation of the penetration testing methodologies of OSSTMM, NIST SP 800-115, CSE/RCMP TRA-1, PTES, ISSAF, and PCI-DSS. It then discusses the main features and implementation of the penetration testing frameworks of OWASP Testing Guide and MITRE ATT&CK® frameworks.

• Introduction
• OSSTMM (Open Source Security Testing Methodology Manual)
• NIST SP 800-115 (Technical Guide to Information Security Testing and Assessment)
• CSE/RCMP TRA-1 (Harmonized Threat and Risk Assessment Methodology)
• PTES (Penetration Testing Execution Standard)
• ISSAF (Information Systems Security Assessment Framework)
• PCI-DSS (Payment Card Industry Data Security Standard)
• OWASP Testing Guide (Web Security Testing Guide)
• MITRE ATT&CK® framework
• Penetration testing methodologies and frameworks comparison

I obtained my PhD in Digital Transformation and Innovation in April 2020 from the PhD in DTI uOttawa Program, School-EECS, Faculty of Engineering. I did my PhD thesis (titled Technoethics and sensemaking: Risk assessment and knowledge management of ethical hacking in a sociotechnical society) on the topic of ethical hacking sociotechnology (thesis advisory committee: uOttawa professors Rocci Luppicini, Liam Peyton, and Andre Vellino).

You may also be interested in Practical foundations in penetration testing.

Introduction

Security assessments follow structured methodologies to ensure reliable testing. In this context, a methodology is a step‑by‑step process that guides an entire penetration testing engagement from scoping through reporting, while a framework is a structured set of guidelines, checklists, or adversary‑focused knowledge bases that testers apply within or alongside a methodology. This article explores the following penetration testing methodologies: OSSTMM (Open Source Security Testing Methodology Manual) v3.0 (Herzog, 2010), NIST SP 800-115 (NIST Special Publication 800-115: Technical Guide to Information Security Testing and Assessment) (Scarfone et al., 2008), CSE/RCMP TRA-1 (Communications Security Establishment/Royal Canadian Mounted Police Harmonized Threat and Risk Assessment Methodology TRA-1) (2007), PTES (Penetration Testing Execution Standard) (2014), ISSAF (Information System Security Assessment Framework) (Open Information Systems Security Group, 2006), and PCI-DSS (Payment Card Industry Data Security Standard) v4.0 (2022). This article also explores the following penetration testing frameworks: OWASP Testing Guide (Open Web Application Security Project Web Security Testing Guide) v4.2 (2020), and MITRE ATT&CK® v18 (2024) framework.

OSSTMM (Open Source Security Testing Methodology Manual)

This discussion compares the three penetration testing methodologies—OSSTMM 3.0, NIST 800-115, and CSE/RCMP TRA-1—to offer insights into establishing a harmonized penetration testing methodology (see Information Security Assessment Methodologies Table).

The original Open Source Security Testing Methodology Manual (OSSTMM), published on December 18, 2000, is a peer-reviewed manual of security testing and analysis, “a methodology for a thorough security test, known as an OSSTMM audit” by the Institute for Security and Open Methodologies (ISECOM). OSSTMM version 3.0 was published on August 2, 2008. In version 3, OSSTMM encompasses tests from all channels: Human, Physical, Wireless, Telecommunications, and Data Networks. A set of security metrics, Risk Assessment Values (RAVs), provide a tool that can provide a graphical representation of changes in state over time. The primary focus in version 3 has been to move away from solution-based testing, which assumes specific security solutions will be found in a scope and are required for security (like a firewall). Instead, the focus is on a metric for the attack surface (the exposure) of a target or scope, allowing for a factual metric with no bias (the risk-based approach).

The OSSTMM structures its audit process not around linear phases, but through a set of modules corresponding to its core channels: Human, Physical, Wireless, Telecommunications, and Data Networks. The OSSTMM methodology features a single security testing methodology for all the channels. For each module, the manual provides a detailed "task list" that guides the tester on what specific security properties to verify. These tasks are not exploitation steps but are designed to measure the operational security of each channel by checking for specific attributes like trust levels, access controls, security processes, and human vulnerabilities. This modular approach ensures a comprehensive assessment that covers the entire operational attack surface, from social engineering and physical intrusion to network penetration.

The OSSTMM methodology features a battery of security parameters for each channel. These parameters include Posture Review, Logistics, Active Detection Verification, Visibility Audit, Access Verification, Trust Verification, Controls Verification, etc. The actual "testing" is a systematic process of measuring these defined security parameters against each channel. Measurements feed into the Risk Assessment Values (RAVs), as a larger "visibility" score increases the measurable attack surface. By applying this consistent set of operational checks to every channel, OSSTMM generates a factual, data-driven snapshot of security that is agnostic to any specific technology or assumed solution.

NIST SP 800-115 (Technical Guide to Information Security Testing and Assessment)

The purpose of NIST SP 800-115: Technical Guide to Information Security Testing and Assessment (September 2008) is “to provide guidelines for organizations on planning and conducting technical information security testing and assessments, analyzing findings, and developing mitigation strategies” (Scarfone et al., 2008, p. ES-1). NIST SP 800-115 divides penetration testing into four main phases: Planning phase, Discovery phase (addressing Target Identification and Analysis Techniques), Attack phase (addressing Target Vulnerability Validation Techniques), and Reporting. NIST SP 800-115 Section 4 Target Identification and Analysis Techniques focuses on “identifying active devices and their associated ports and services, and analyzing them for potential vulnerabilities” (p. 4-1). It includes Network Discovery which “uses a number of methods to discover active and responding hosts on a network, identify weaknesses, and learn how the network operates.”

Passive (examination) and active (testing) techniques discover devices and active hosts on a network. Passive techniques can use a network sniffer to monitor network traffic and record the IP addresses of the active hosts, and they can report which ports are in use and which operating systems on the network have been discovered–without sending out a single probing packet (Scarfone et al., 2008, p. 4-1). Section 4 also covers Network Port and Service Identification. “Some scanners can help identify the application running on a particular port through a process called service identification” (p. 4-3). Banner grabbing involves “capturing banner information transmitted by the remote port when a connection is initiated. This information can include the application type, application version, and even OS type and version.” The result of network discovery and network port and service identification is “a list of all active devices operating in the address space that responded to the port scanning tool, along with responding ports” (p. 4-3). Port scanners can identify active hosts, operating systems, ports, services, and applications, but they can not identify vulnerabilities. “To identify vulnerable services, the assessor compares identified version numbers of services with a list of known vulnerable versions, or performs automated vulnerability scanning” (p. 4-4).

Vulnerability scanners can be broadly divided into two categories: Web application scanners such as Acunetix, WebInspect, and NetSparker; and network and infrastructure scanners such as Nessus, Qualys, and Metasploit. Vulnerability scanners can check compliance with host application usage and security policies, identify hosts and open ports, identify known vulnerabilities, and provide information on how to mitigate discovered vulnerabilities. Vulnerability scanners often use their own proprietary methods for defining the risk levels. One scanner might use the levels low, medium, and high; another scanner might use the levels informational, low, medium, high, and critical, making it difficult to compare findings among multiple scanners. Vulnerability scanners rely on a repository of signatures which requires the assessors to update these signatures frequently to enable the scanner to recognize the latest vulnerabilities. NIST SP 800-115 Section 5 Target Vulnerability Validation Techniques focuses on using information produced from target identification and analysis to further explore the existence of potential vulnerabilities. "The objective is to prove that a vulnerability exists, and to demonstrate the security exposures that occur when it is exploited” (Scarfone et al., 2008, p. 4-5).

CSE/RCMP TRA-1 (Harmonized Threat and Risk Assessment Methodology)

The CSE/RCMP Harmonized Threat and Risk Assessment Methodology TRA-1 presents a flexible approach which can be automated and serves as a general framework for a harmonized penetration testing methodology by applying a project management frame. The TRA approach provides “a clear rationale for cost-effective risk mitigation strategies and safeguards to meet business requirements; and a transparent audit trail and record of risk management decisions to demonstrate due diligence and accountability, thereby satisfying statutory obligations and policy requirements” (CSE/RCMP, 2007, p. EO-2).

Information Security Assessment Methodologies Table

PTES (Penetration Testing Execution Standard)

The Penetration Testing Execution Standard (PTES) was developed to provide a consistent and comprehensive framework for conducting penetration tests. Its core contribution is the definition of seven distinct phases that guide the entire engagement, from initial contact to final reporting: Pre-engagement (Scope, contracts), Intelligence Gathering (Recon), Threat Modeling (Identify attack vectors), Vulnerability Analysis (Scanning), Exploitation (Gaining access), Post-Exploitation (Persistence, pivoting), and Reporting (Remediation guidance). The penetration testing phases begin with Pre-engagement, where scope and rules of engagement are formally established, and proceed through Intelligence Gathering, Threat Modeling, and Vulnerability Analysis to build a deep understanding of the target environment before any exploitation is attempted. This structured approach ensures that testing is methodical, repeatable, and aligns with client expectations from the outset.

PTES extends its methodology beyond mere technical execution to cover the full lifecycle of an attack. The Exploitation phase focuses on gaining initial access, while the Post-Exploitation phase involves actions like maintaining persistence, lateral movement, and determining the value of the compromised assets. The process culminates in the Reporting phase, which is designed to provide clear, actionable remediation guidance tailored to both technical teams and business stakeholders. This end-to-end standard is particularly well-suited for general penetration testing across network, web, and cloud environments, offering a common language and process for testers and clients alike. PTES was developed by a consortium of security professionals and consultants from various organizations in the industry. The effort was led by a core team including individuals from companies like NetSPI, Chromium Security, and the security consulting firm The Aperture Labs. The current version is PTES 1.0, which was released in 2011. It is important to note that while PTES remains a highly influential and referenced standard, it has not seen a formal version update since its initial release.

ISSAF (Information Systems Security Assessment Framework)

The Information Systems Security Assessment Framework (ISSAF) is a specialized, step-by-step approach to penetration testing developed by the Open Information Systems Security Group (OISSG) in 2006. Its extensive guidebook—which clocks in at over 1,200 pages—lays out a comprehensive framework for assessing network, web application, and database security. The ISSAF’s comprehensible and highly structured approach is easily customizable for individual organizations and testers, allowing for the creation of personalized testing plans and making it a practical choice for those using multiple tools in a coordinated manner.

It is important to note that the ISSAF goes well beyond simple penetration testing. ISSAF details exploitation techniques and covers network, web apps, and databases. The framework also encompasses the creation of educational tools for training individuals with network access and ensures that all testing activities adhere to appropriate legal standards. ISSAF saw a single release in 2006 and has not been updated since, cementing its role as a historical reference rather than a current operational standard. ISSAF remains a valuable resource for understanding the foundational, step-by-step processes of early penetration testing methodologies.

PCI-DSS (Payment Card Industry Data Security Standard)

The Payment Card Industry Data Security Standard (PCI-DSS) Version 4.0 is fundamentally a security standard, not a penetration testing methodology itself. However, it mandates a strict set of testing requirements for any entity that stores, processes, or transmits payment card data. Its focus is the protection of cardholder data and the Cardholder Data Environment (CDE). The standard requires both annual penetration tests and tests following any significant change to the CDE, ensuring that security assessments are a recurring and integral part of the compliance lifecycle.

The PCI-DSS standard specifies the scope and nature of these required tests. Penetration testing must cover all network, application, and segmentation controls, verifying that isolation mechanisms effectively protect the CDE. Furthermore, it requires external vulnerability scans to be performed by an Approved Scanning Vendor (ASV). This prescriptive approach ensures a baseline of security testing rigor across the payment ecosystem, making compliance with PCI-DSS v4.0 mandatory for merchants and payment processors.

OWASP Testing Guide (Web Security Testing Guide)

The OWASP Web Security Testing Guide (WSTG) is a comprehensive resource curated by the OWASP Foundation, specifically designed for testing the security of web applications and services. Unlike broader methodologies like OSSTMM or NIST SP 800-115, the WSTG has a sharp focus on the application layer, providing a detailed, actionable checklist for testers, developers, and security professionals. Its primary goal is to produce a standardized and complete framework for testing web application security, ensuring that common and critical vulnerabilities are systematically identified.

The guide is structured to mirror the phases of a typical application penetration test. It begins with preliminary steps like information gathering and configuration management testing, then moves into a thorough examination of identity management, authentication, and session management controls. The core of the WSTG is its extensive coverage of specific vulnerability classes, most notably the OWASP Top 10 risks such as SQL Injection (SQLi), Cross-Site Scripting (XSS), and Cross-Site Request Forgery (CSRF). For each testing area, the guide provides a clear objective, descriptions of how to test for weaknesses, and guidance on how to interpret the results.

A key strength of the OWASP Testing Guide is its evolution to keep pace with modern application architectures. While it thoroughly covers traditional web applications, it also includes critical testing procedures for APIs (REST and SOAP), serverless architectures, and cloud-native applications. This makes it an indispensable tool not only for dedicated penetration testers but also for DevSecOps teams integrating security into the development lifecycle. By providing a community-driven, open-source set of best practices, the OWASP WSTG establishes a common language and baseline for web application security testing across the industry.

MITRE ATT&CK® framework

MITRE ATT&CK® (Adversarial Tactics, Techniques, and Common Knowledge) is a globally accessible knowledge base of adversary behaviors derived from real-world cyber threat observations (MITRE, 2024). ATT&CK was created by the Mitre Corporation and released in 2013. Often described as an "encyclopedia of hacking," ATT&CK helps defenders understand attacker methodologies and build more effective defenses. It serves as a foundational resource for threat intelligence, detection engineering, red teaming, and security strategy development. While the Lockheed Martin Cyber Kill Chain provides a high-level model of sequential attack stages, MITRE ATT&CK offers a granular taxonomy of the specific Tactics, Techniques, and Procedures (TTPs) that adversaries employ within each stage of an intrusion.

The framework organizes adversary behaviors into distinct matrices tailored to different operational environments. The Enterprise ATT&CK matrix covers Windows, Linux, macOS, and major cloud platforms including AWS, Azure, and Google Cloud. The Mobile ATT&CK matrix addresses threats targeting Android and iOS devices, such as spyware and malicious applications. The ICS ATT&CK matrix focuses on Industrial Control Systems and Operational Technology environments. Within each matrix, behaviors are categorized by Tactics, which represent the adversary's high-level objectives during an operation—for example, Initial Access, Execution, Persistence, Privilege Escalation, and Lateral Movement. Each tactic is associated with specific Techniques, which describe the methods used to achieve those objectives. Phishing (T1566), PowerShell execution (T1059.001), and Pass the Hash (T1550.002) are representative examples. Techniques may be further refined into Sub-Techniques to capture variations in implementation, such as distinguishing between a spearphishing attachment and a spearphishing link. Finally, Procedures document how known threat groups—including APT29, Lazarus, and others—implement these techniques in actual campaigns, providing real-world context for defenders.

Organizations apply MITRE ATT&CK across multiple security functions. Blue teams and Security Operations Center (SOC) analysts map detection rules in SIEM and EDR platforms to specific ATT&CK techniques, enabling systematic gap analyses that answer questions such as "Can we detect Credential Dumping (T1003)?" Incident responders use the framework as a structured playbook to investigate breaches and trace adversary activity. Red teams and penetration testers leverage ATT&CK to design adversary emulation exercises that test defensive controls against documented TTPs, often in collaboration with defenders during purple team engagements. Threat intelligence teams rely on the common ATT&CK taxonomy to track, compare, and communicate adversary behaviors across reports and organizations.

A typical ransomware attack illustrates how an intrusion maps to the ATT&CK framework: Initial Access is achieved via Phishing (T1566), Execution follows using PowerShell (T1059.001), Persistence is established through Registry Run Keys (T1547.001), Lateral Movement employs Pass the Hash (T1550.002), and Impact culminates in Data Encrypted for Impact (T1486). Practitioners can explore the framework directly through the MITRE ATT&CK website and employ companion tools such as the ATT&CK Navigator for visualization, CALDERA for automated adversary simulation, and Atomic Red Team for validating detection coverage against specific techniques.

Penetration testing methodologies and frameworks comparison

Below are the most widely used penetration testing methodologies and frameworks and their key features and use cases.

Penetration Testing Methodologies and Frameworks Comparison Table

Key takeaways

• Penetration testing is guided by both methodologies, which provide a process for the entire engagement (e.g., OSSTMM, NIST SP 800-115, PTES), and frameworks, which provide knowledge bases of attacker behaviors (e.g., MITRE ATT&CK) or specialized checklists (e.g., OWASP WSTG).
• OSSTMM 3.0 provides a scientific, metrics-focused approach to operational security across multiple channels (human, physical, network), generating factual data about an organization's attack surface.
• NIST SP 800-115 offers a high-level, phased approach (Plan, Discover, Attack, Report) that aligns with broader U.S. federal compliance standards and is strong on vulnerability assessment.
• CSE/RCMP TRA-1 applies a project management framework to harmonized threat and risk assessment, emphasizing cost-effective mitigation and audit trails.
• PTES (Penetration Testing Execution Standard) defines a seven-phase lifecycle from pre-engagement through reporting, providing a common language for general penetration testing.
• ISSAF (Information Systems Security Assessment Framework) offers detailed, step-by-step exploitation guidance across network, web, and database layers, though it is now largely a historical reference.
• PCI-DSS v4.0 mandates specific penetration testing and vulnerability scanning requirements for any organization handling payment card data, focusing on the Cardholder Data Environment.
• The OWASP Testing Guide (WSTG) is the definitive standard for web application security testing, offering a detailed, phase-based checklist for finding critical vulnerabilities like those in the OWASP Top 10.
• The MITRE ATT&CK® framework is not a testing methodology but a knowledge base of real-world adversary TTPs (Tactics, Techniques, and Procedures), essential for red teaming, threat hunting, and improving defensive detection capabilities.

References

Communications Security Establishment/Royal Canadian Mounted Police. (2007). Harmonized threat and risk assessment methodology (TRA-1). http://www.rcmp-grc.gc.ca/ts-st/pubs/tra-emr/index-eng.htm

Herzog, P. (2010). OSSTMM 3–The open source security testing methodology manual. Barcelona, España: ISECOM. https://www.isecom.org/OSSTMM.3.pdf

MITRE. (2024). ATT&CK: Adversarial tactics, techniques, and common knowledge. MITRE Corporation. https://attack.mitre.org/

Open Information Systems Security Group. (2006). Information systems security assessment framework (ISSAF). http://www.oissg.org/information-systems-security-assessment-framework-issaf.html

OWASP Foundation. (2020). OWASP web security testing guide (WSTG). https://owasp.org/www-project-web-security-testing-guide/

PCI Security Standards Council. (2022). Payment Card Industry Data Security Standard: Requirements and testing procedures (Version 4.0). https://www.pcisecuritystandards.org/document_library/

PTES (Penetration Testing Execution Standard). (2014, August 16). http://www.pentest-standard.org/index.php/Main_Page

Scarfone, K., Souppaya, M., Cody, A., & Orebaugh, A. (2008). Technical guide to information security testing and assessment (NIST Special Publication 800-115). National Institute of Standards and Technology. http://csrc.nist.gov/publications/nistpubs/800-115/SP800-115.pdf