This article discusses what you can expect at each stage of a high quality security review, with the aim of helping folks understand what to look for when engaging penetration testers.

This is the third article in our Purpose and Execution of Penetration Testing series, following on from The Purpose of Penetration Testing (which I suggest reading first if you haven’t already…) and Timeboxed Penetration Testing.

Variability in Penetration Testing

Before we get started, I figured I’d explain why there can be variability in output between penetration testing firms. Penetration testing and security reviews are, potentially to the chagrin of some of the practitioners, a professional service. This means we (the security consultants performing security reviews) exchange our time (and care) on a project for an amount of money.

Different firms operate on different business models, but the undeniable physics of the situation effectively translates to Time==Money. Our industry standards frame what should generally be done, but these are more general guidelines rather than formulaic lists that can be easily followed by a practitioner. The Timeboxing article I wrote earlier this year expands on how these standards (while useful and good) leave a lot up to the individual practitioner.

Practitioners approach penetration tests differently depending on their background. A Principal-level consultant with 30 years experience in the IT industry, software development history, systems administration, networking and architecture experience, and the last 10 years in the security testing field, will likely approach finding and classifying vulnerabilities differently from a fresh graduate with a degree in comp-sci and a paper in cyber security. I’m not suggesting one is good and the other bad, simply that pentesting is a professional service and the amount of time available, the testing organisation’s business model and internal priorities, and the individual professional doing the work are factors in the output.

If you’ve ever engaged multiple penetration testing firms for the same or similar targets, you’ve probably noticed that the results can vary quite a bit. This is the core reason why.

This is why understanding what makes a quality penetration test matters, and thus this article. Onto the stages! We’re going to discuss what a quality penetration test will entail at each step of the process. During initial scoping stages, during the testing itself, and after the report is delivered.

Pre-engagement - The Scoping Phase

Before any testing can start, there is the pre-engagement and scoping phase. This is where you’ll approach a penetration testing provider with the details of what you’d like to achieve. Usually, it’s something like… “We’re interested in finding out about any security issues in our system/application/network/organisation, can you please provide an estimate for doing this testing”.

A quality penetration test begins with a quality scope, and a quality scope begins with solid communication between the client and the testing firm. Throughout this process, you should feel engaged and like the firm is building a path to understanding your specific system and needs.

The ‘scope’ in the case of the penetration test is the target of the review. Some engagements, like hybrid-security assessments and bespoke security reviews, have a focus instead of a strict scope. This depends on the target and what you’d like to achieve. Scope from the commercial sense also sets how long the engagement will take, which also determines the engagements cost.

The scoping phase should understand the target system (be it an application, network, cloud platform, organisation, embedded device, robot, CNC machine, etcetera) well enough to put together an estimate on how long the job will take at a detailed level. This means determining the amount of functionality, lines of code, breadth of functionality, and the target technologies in use. This level of detailed understanding can’t always happen during scoping for various reasons, and if that’s the case the limitations in scoping need to be clear in the delivered quote.

Every test is a little different in scope. I’m generally wary of templated scoping forms and pre-canned scopes where any engagement of a specific type takes a set amount of time. Personally, I prefer discussing the system with the client directly and then figuring out what data is needed to build an accurate picture of the system. These days, I look at source code and perform cursory analysis of the target system to really understand the target well and determine the amount of time for the review. If the proposed engagement length is too long, then the scoping has provided an accurate picture of the target system which we can use to whittle down with our client and agree on the focal points of the engagement in the limited time-frame.

A quality pre-sales process will leave you confident that the penetration testers understand your environment and the work to be done. You should have a shared agreement on any areas of focus, how the testing will be done, and how time constrained engagements will be prioritised.

Requirements

A quick note on testing requirements. We’ve started including technical testing requirements in our statements-of-work and it’s been helpful in agreeing how the testing will be performed. We generally need access to the target application, network or cloud account as the main requirement. We also use access to any relevant source code repositories and administrative access to servers, networking equipment and cloud accounts depending on the specific target. This is to perform the review accurately and quickly by looking directly at configuration, and occasionally making changes (which are logged, communicated and rolled back… naturally…) to facilitate deeper testing of a target.

These are all technical problems, though. An experienced tester can figure out other ways to achieve a similar goal and communicate the trade-offs with their clients. A good example is testing a third-party application that a client has installed. Source code is almost never available in that situation, so we’ll rely on binary reverse engineering and disassembly techniques instead to achieve a similar outcome and inspect logic.

Testing - Where the findings get found

The core part of penetration testing is the process of finding vulnerabilities and areas for additional security hardening. What does a quality penetration test look like during this engagement phase?

Solid communication is what it looks like. The testers should be communicative from when they start. If the target is particularly complex or large, I like to organise an initial workshop to discuss the environment and put together architecture flow diagrams and attack surface tables.

This is also a good time for the tester and the client to discuss any known vulnerabilities. If there are currently known issues, or areas that are of particular concern to the client, these can be discussed and help contextualise the testing. We achieve more when we work together.

High severity issues in public-facing targets, or issues which are more likely to lead to immediate impact, are communicated immediately. We use interim finding summary documents, but there are various ways to tackle sharing technical details prior to the final report. The important part is anything that’s critical and risky gets notified immediately.

A quality test starts with good attack surface analysis. The tester invests time in understanding what the application or network does before systematically testing what it shouldn’t do. This is key, and one of the few things every penetration testing methodology shares regardless of target. Again, this one loops into communication.

I once had a colleague who lamented a previous penetration testing agency failing to determine a parameter that was encoded with a specific algorithm. I mentioned that if I can’t figure something like that out, I’ll generally reach out to the client’s technical contacts and ask them directly. Their response was “if the penetration tester worked like that, I’d hire them again!”

Quality penetration tests include solid communication, and consultants who aren’t afraid to reach out and discuss your systems to build a really solid understanding.

The Role of Industry Standards in Testing

Industry standards like OWASP, NIST SP 800-115, and OSSTMM serve as important reference points during testing. They let both the tester and the client operate with a base understanding of what industry-standard checks will be done. Unfortunately these standards don’t and likely can’t cover every aspect of methodically reviewing a system. The standards are helpful, but incomplete when applied to a specific target.

The OWASP Top Ten is perhaps the most common example of this pitfall. The OWASP Top Ten is a risk-aware overview of the most common web application security issues, not a comprehensive testing methodology. A quality penetration test methodically reviews your specific system with your organisation’s specific business in mind. The top-ten greatest hits of vulnerabilities as determined by OWASP don’t really have much input into end-to-end vulnerability and security analysis of a specific target, and a test that only checks for the top-ten is flawed from the beginning.

A quality test will cross reference and include appropriate industry standards for assessing the target as part of the overall testing methodology. A good testing methodology will look at prior art and determine how to best apply and extend it for your particular system. These standards should be documented and cross-referenced in both the statement of work and in the final report.

The Bugs Penetration Testing Likely Won’t Find

There are vulnerabilities that a penetration test is unlikely to ever find, thanks to the complexity of computing environments and or modern software supply chains (https://pulsesecurity.co.nz/articles/the-purpose-of-pentesting#the-complexity-problem).

I’d like to share a conversation I’ve had with a few new starters in the penetration testing field over the years:

Denis: What kind of engagements don’t you like doing?

Rookie: Brochure-ware websites! They’re boring and there’s nothing to attack.

Denis: Oh yeah what did you find on the last one you did?

Rookie: It was about 3 months ago now, all I found was an NGINX server header disclosure.

Denis: Okay, NGINX has had memory corruption vulnerabilities patched in the last 3 months. Why didn’t you find those on your test?

Rookie: ….

Denis: ….

Now, I realise this interchange makes me sound like a jerk if you have any experience with penetration testing at all. “Denis, you can’t expect a two day brochure-ware site review to find memory corruption issues in the underlying web server!” Of course not! That’s the point!

The underlying web server has an attack surface, the operating system that runs that web server has an attack surface, the network interface driver that accepts the network packet has an attack surface. Which unknown vulnerabilities ([zero-day][https://en.wikipedia.org/wiki/Zero-day_vulnerability]) in these attack surfaces should the review discover?

We need realistic expectations about what a penetration test can reasonably find. A quality tester will be transparent about these limitations during scoping, and the report should explain the coverage achieved.

The primary focus of a penetration test is generally things the client is directly responsible for. Code and configuration. For applications this means input validation, authentication and access control logic, session management, data handling, and the security of custom-developed functionality. For a network test, this would be the configuration of services, network controls, user controls (passwords, multi-factor adoption), security patching and hardening specific to the tech in that network. Similarly for a cloud-based review, the configuration of the cloud services is the core priority. A quality tester will spend the majority of their time here since these are the areas where the client has direct control and is responsible for the vulnerabilities.

Secondary focus goes to things like third-party libraries and dependencies. The real question the test should answer is “are vulnerable dependencies and security patching being managed effectively”, rather than deep dive into finding zero-day in whichever library. Unless, of course, that’s an agreed priority of the engagement!

Low-level infrastructure concerns like zero-day operating system and server vulnerabilities, hardware security issues, or physical security flaws in the underlying datacentre simply won’t be investigated during a standard application penetration test and that’s by design. The scope defines what gets tested, and whether communicated through the report or other methods, the output of a test should be clear on what was focused on.

Reporting - What makes a good report?

The report is the primary deliverable of a penetration test. This is our tangible output that clients will have have to show their board, their auditors, and their technical teams. A quality report should be useful to all of these audiences, and include enough information that it becomes its own form of internal cyber security training material for technical team members.

We should see good writing across the whole report, which explains technical concepts without falling into the trap of becoming vague or patronizing. Quality technical analysis will also avoid emotive writing without justification. This means calling exploitability “trivial” or impact “significant” without explaining why these adjectives have been included is a no-no.

Note: Denis has a problem with overly emotive or “hyped” writing in cyber security in general, and will complain about it at any given opportunity.

Summaries will cover the engagements goals, the approach and methodologies used for the testing and the high-level outcome of the engagement. We generally discuss notable findings and overall security posture of the target.

Other summaries get added as needed, such as technical summaries where more complex topics that cover multiple findings get discussed. Things like vulnerabilities that are inherent to the chosen architecture, or chaining multiple vulnerabilities together for a higher impact outcome.

By the time a test is completed, a decent security consultant will have a good picture of the target system and generally even more ideas on where the testing can go next. When this happens (more often than not, these days!) we include an Ongoing Security Program summary that details what we feel the next steps for the client’s security program should look like.

Great Technical Findings

We’ve talked about the summaries, but the real bones of a solid technical penetration testing report are in the technical findings. The severity of these findings, funnily enough, isn’t the indicator of a quality penetration test. Rather, how the technical information is communicated.

I have a bunch of maxims on reporting technical findings, and I’ll spare you the complete novel on all of those. A quality finding hits the following marks for me:

• Solid triage and explanation on the technical details, meaning no false-positives.
• Explanation on the impact of the finding, or how likelihoods of future vulnerabilities are practically increased.
• Clear evidence. Sufficient technical detail for a non-penetration-tester to replicate the finding. Screenshots of testing tools like Burp Suite are a pet-peeve of mine since the tools aren’t generally used or accessible to non-practitioners. Technical details should include how to trigger the vulnerability or confirm the missing hardening control with common tooling. If it can’t be reproduced without specific actions, then those steps are either provided in a standalone script or a step-by-step list.
• Triage of the finding’s cause, including source code, disassembly or configuration snippets that show why the vulnerability has occurred.
• Additional resources and cross links to industry standards, vulnerability identifiers and other information that’s helpful in understanding and remediating the finding. It’s unusual that a penetration tester finds an entirely new aspect of cyber security that has never been discussed before and has no supporting resources.
• Actionable recommendations. This one is critical enough that it gets it’s own heading!

A solid technical finding can be picked up by a developer, sysadmin or other IT practitioner and reproduced. The technical details become their own kind of training material, and can be used to help clients’ internal team get more familiar with triggering vulnerabilities themselves. A critical part of the remediation process!

Actionable Recommendations

This is a big one. Recommendations for technical findings need to be implementable and often catered to the individual target. It’s easy for a penetration tester to write ‘implement multi-factor authentication’, but sometimes that recommendation really reads ‘I do not understand the real-world complexities of operating this system’.

If a recommendation cannot be implemented, it’s not worth the paper it’s written on. Actionable recommendations means the tester has understood the target system, and has a clear picture on how to improve it. This can include specific guidance around which library methods to use to handle input, through to softer recommendations on investigating ways to improve resilience to an overall vulnerability class.

What we don’t want to see in a quality penetration testing report is a generic one-line statement like ‘validate user input’ or ‘improve password quality’. We need details and solid guidance to help clients secure their systems well.

Generic Security Hardening Findings

Generic security hardening findings are common misconfigurations with various potential impacts, but generally no immediate exploitability. I’ve heard these classified as “filler” findings. Common examples are missing HTTP security headers, or TLS cipher configurations with known weaknesses. Most automated security tooling will detect these issues, since they’re well understood and easy to test for programatically.

A quality test will include these issues, but they’ll be scored appropriately based on their real impact to the overall security posture. We take a slightly different approach, and consolidate all of these findings that can be detected with a security scanning tool into a general “hardening” finding. The idea being that staying on top of security hardening issues like TLS ciphers is best handled with solid automation and open-source tooling by the client themselves, rather than an engagement with an external penetration tester.

The recommendations in these generic hardening findings suggest the client runs tools like Nuclei, Prowler and testssl.sh themselves. We also contribute features and findings back to open-source projects like Nuclei so the wider community can benefit. Naturally, if these scanners find a meaningful issue with real impact then that warrants its own finding with all the accouterments detailed in the Great Technical Findings section.

What Gets Reported and What Doesn’t

Not everything a tester notices becomes a finding. Vulnerabilities with measurable impact should always be reported. If the tester can quantify how an attacker class can exploit something to cause harm, it goes in the report. But what about things that are suspicious or unsafe without being technically exploitable right now? Our hardening and architectural findings? That’s where professional judgment comes in.

These findings capture things like excessive attack surfaces (exposed endpoints that aren’t used and don’t need to be), insufficient segmentation, or unsafe code patterns (URLs or SQL queries built through string concatenation instead of proper libraries). These aren’t bugs today, but they’re dangerous patterns that will likely cause issues down the road. A quality test includes these when the recommendations are genuinely actionable and meaningful.

Conversely, things with no practical impact and no value in remediation don’t make the cut.

Post-engagement - Improving the system based on the report

You’ve received your report, the invoice has been paid, and we’re done with penetration testing… Right? Wrong!

Technology is complicated, and cyber security is one of the more complicated niches in an already complex field. After the report has been delivered, it’s up to you to take the output of that report and turn it into positive change, but you don’t have to do that with no support. Most penetration testers in New Zealand wear the title of “Security Consultant” and this is where they get to consult!

Recall how one of the points earlier about reporting was ‘no false positives’? Consultants are also human, and make errors. Feedback and open communication is key to getting solid outcomes. If something doesn’t make sense, ask about it! A good tester will either be able to explain the issue better, or issue an updated report without the false-positive.

After the report is delivered and the findings start being triaged and remediated, it’s not unusual for further questions to come up. A common issue, for example, is recommendations for cloud-based findings likely resulting in a huge uptick in costs, and a different workaround needed to be found. Reaching out to the security consultants can help answer questions and benefits both the tester and client. The client gets input on their decisions and remediation early, and the tester gets to understand the real-world applicability of their recommendations and findings.

Confirmation testing and how to approach it well is worthy of its own article, and will be the next article in this series!

What about commodity or automated testing?

We’ve been discussing quality penetration testing thus far. Quality in this case being synonymous with a conscious and measured approach to each step of the engagement. Is there a place for commodity or automated testing?

There are situations where a port scan or output of an automated vulnerability scanning tool is genuinely helpful to a client, especially when the client cannot run this tooling themselves. Automated scanners have broken production networks before. Someone with experience running them can make clients feel more confident and add real value. Seeking some assistance from an expert when running these tools without internal knowledge is understandable.

If having someone come in to run some tools and share the output is legitimately useful, then this is valuable work. We just need to be clear about what they’re getting, and how that differs from a complete test.

Summary

A quality pentest understands your systems from scoping through reporting, communicates reproducible findings with actionable recommendations, and leaves you feeling supported and understood.

A great pentest will leave you with a better understanding of your tech, and a path to making positive change in your cyber-security posture.