Penetration testing and vulnerability research are not the same thing. At Pulse Security, we’ve taken a different approach to certain penetration and security testing engagements. We’ve begun using a vulnerability-research based approach where we collaborate directly with client staff to understand and assess complex or heavily integrated systems. We’re calling it our “hybrid security assessment” service which can include aspects of threat modelling, attacker analysis, network testing, architecture review, application testing, reverse engineering, source code review, and more, as needed to fully understand and assess the security of large and complicated systems.
This has led to a significantly better understanding of a system’s security posture and the vulnerabilities that may affect it. We can find the relevant bugs and provide more useful advice on remediation.
This article is a look at the “hybrid security assessment”. What it means, the benefits it offers and how it differs from traditional penetration testing . I’ll describe an imaginary architecture (though based on real-life engagements we’ve completed) and why a hybrid assessment provides a better understanding of the overall system’s security posture.
Background
Penetration testing is used to discover vulnerabilities within a target system. The issues are documented, potential impacts analysed and then rated based on criticality (read: risk to the system and/or wider business). These vulnerabilities can then be resolved or mitigated, (hopefully) prior to a legitimate attacker discovering and exploiting the issues. However, penetration testers are not malicious attackers, and real-life attackers are not generally bound by a tight timeframe or scope looking at a single interface or system.
The idea to apply a different approach to security assessments came about when we were discussing the differences between how we’d often do penetration testing, versus how we would look at the security of something as part of our research program. A standard, ye-oldie black-box penetration test would involve banging our head against a wall for X amount of time with no further information, and at the conclusion of the engagement write up a report based on our understanding of the system. Whereas when doing security research on a product or system, the consultant with fewer constraints built a better understanding of the system and found more, actually relevant, vulnerabilities.
Due to the deep-dive nature of vulnerability research, the consultant was able to explain complex elements and threats to the system. This was thanks to the consultant being able to disassemble the target, look at source-code/assembly and surrounding systems and figure out what the practical attack surfaces of the thing being researched actually were.
So we figured: why not also work directly with clients and allow for a flexible scope and testing approach? If the aim of the engagement is to figure out the security posture of an application or environment, working directly with client technical staff and getting direct guidance from key individuals helps:
- Build a deeper understanding.
- Better understanding means better attack surface analysis.
- Better attack surface analysis means testing can be targeted to the areas of greatest risk.
This leads to significantly better results, and a more secure system overall. Compared to a traditional penetration test, where the consultants would be given a target, maybe some credentials and be sent on their merry way.
Working directly with the client throughout the review leads to stronger reporting. Given that we build a good understanding of the system together, we can make practical recommendations and suggest specific workarounds. We want to offer better guidance than just a dusty screenshot of some forsaken exploit and a one line recommendation that says something like “validate user input”. There are practical solutions to technical problems, and we want to be able to have those discussions. This relationship ends up being a two-way street and client staff end up learning more about the practical vulnerabilities and risks within their systems. I like to think of it as treating our clients as colleagues, rather than consumers.
But, enough high-level mumbo jumbo, let’s look at the key differences between traditional penetration testing and a hybrid assessment.
Our Imaginary System - ThymeSheet
The imaginary system we’re going to be looking at will be ThymeSheet. ThymeSheet is a SaaS (Software-as-a-Service) platform. It’s a subscription-based service, hosted entirely in AWS (or so we think!) and supporting multiple tenants. Customers sign up to ThymeSheet and set up users, which consist of admin users, management users and standard users.
A traditional penetration test would look at a system like this from the perspectives of an attacker on the internet and as malicious users of various levels. Here’s a quick diagram of what a competent penetration tester could figure out about the system in the first half hour or less:
And now lets compare that to what the actual system looks like[1], the original pen-test scope is highlighted in red:
A real-life attacker isn’t limited by penetration test scope. If that source-repository server was on the Internet, then that is just as valid a target as the main ThymeSheet application’s frontend. Similarly, the attacker may discover additional API endpoints or different integration points that cross boundaries. Again, the goal of vulnerability research is to discover and understand the vulnerabilities within the system, which in this case extend far beyond just the intended ingress path for users.
Traditional penetration testing generally won’t consider the exposure of a compromised developer or internal user. What does a compromised developer account mean for the platform and are the defences in place adequate in relation to the risks presented? The aim of the hybrid security assessment is to answer these questions from a low-level technical perspective and provide data and metrics that can be used to make risk and/or impact-based decisions. More on this soon.
[1] This is purposefully high level and missing certain interfaces. For those interested, these diagrams were made with pytm.
The Hybrid Assessment
The goal of the hybrid security assessment is to understand the vulnerabilities and assess a complex target environment against real-world attacks. The engagement is designed to allow our consultants and the client to establish the attack surface of complex systems and uncover security issues affecting the system and the surrounding environment. The engagement can include aspects of network testing, architecture review, application testing, reverse engineering, source review and others as required by the specific target.
The assessment will first establish the attack surface of the target system and perform high-level threat modelling. We establish a list of potential attacker categories to guide the review. This does require a greater level of access to perform the review when compared to traditional penetration testing, specifically the availability of technical contacts within the organisation, management level access to target systems and source-code.
Using our ThymeSheet example, we would be provided with administrative access to the AWS environment (like our devops person) and access to source code repositories and build pipelines (like a developer), along with normal user accounts and other more common access.
The Kick-Off Meeting
The hybrid assessment typically begins with a kick-off meeting. Key staff members (both technical and not) jump into a 30 minute to hour long initial session where we discuss the platform, it’s design and any particular concerns the client staff may have.
I had one client refer to the initial meeting as “confessing our sins”. I kind of like this analogy. You understand the system and areas that might need improvement, or things that were rushed at the time. There, there, let it all out. By putting any concerns on the table early on, high-risk areas can be identified and the engagement better guided. We want to bring guidance (absolution though resolution, maybe?) to the table, rather than judgement and the same set of parroted “best practice” suggestions.
In the case of the ThymeSheet system, the imaginary client tabled the relationship between legacy on-premise systems and the Kubernetes cluster running in AWS. The devops folks in the kick-off meeting also pointed out that the monitoring stack was running in the AWS admin account, which felt suspect and warranted further investigation.
Both of these things were areas of concern for ThymeSheet. Our job now is to map these concerns back to attacker types and threat models and figure out what the real-world exposure and potential impact would be.
Threat Modelling and Attacker Analysis
Next up the threat modelling an attacker analysis stage begins. This involves understanding the system’s boundaries, who the potential attackers are and beginning to build attack trees. Without a practical knowledge of how the system actually works, these are very high level at the beginning. As the vulnerability discovery phase happens, the threat models and attack trees are refined and given better context. Practically, this means the models and trees become more accurate as the engagement progresses.
For example, we consider the risks posed by malicious developers, or compromised developer accounts. In certain situations, simply creating a new branch in a repository is enough to extract the credentials used by the deployment systems. Which raises the question, what permissions are granted to the deployment systems, and can a compromised or malicious developer gain access to the production systems by compromising the deployment pipeline? Are developer accounts adequately defended?
As the consultant gains a better understanding of the system through vulnerability hunting, the threat models are updated, and new attack surfaces detailed. Practically, this means the models and trees become more accurate as the engagement moves along.
Vulnerability Discovery
Once the initial attack surfaces have been defined, the vulnerability discovery process starts. This is where the actual hacking (tm) happens. The consultant researches specific components to discover the vulnerabilities, while referencing the threat-model and attacker analysis work to understand the impacts. Because access to systems has been provided, along with source code, the vulnerability hunting phase generally progresses faster than when compared with a traditional black-box penetration test. The consultant develops a solid understanding of the system internals at this point, the vulnerabilities are given context and can be scored appropriately. This is no longer just a web application pen-test, or a network pen-test, or an AWS pen-test. The consultant chooses the best methodology and attack techniques based on the attack surface and threat models. Practically, this means that a hybrid assessment can include elements from our entire arsenal of various services (except incident response, hopefully!).
Collaboration is often key during the vulnerability research phase. Developing an understanding for a business’s specific risk profiles and potential impacts doesn’t happen in a day. Understanding each organisation’s specific risk profiles is a nuanced process, and realistically this is what we want to be looking at when understanding and scoring vulnerabilities. What is the impact and what does it mean for your business? With a collaborative model, it’s simple. When we find something noteworthy that needs more information or might be high risk, we can discuss it and figure out what the vulnerability means for the business.
Looking at ThymeSheet for example, there is a low-severity bug (according to CVSS) where an attacker on the internet can disclose all customer domains integrated with the Azure-based sign-on flows. Upon discussing it with the project contacts, it turns out the Timesheeting As A Service industry is cut-throat and exposing this data to competitors is a significant concern to the business. CVSS low, however business impact; much greater than low.
The Report
The key deliverable at the conclusion of the hybrid assessment is a detailed technical report.
Hybrid security review reports include specific details on each issue discovered during the review, including bespoke recommendations and full proof-of-concept exploit code. This is a priority for us. By providing complete code and step-by-step instructions on how to replicate each vulnerability, the report becomes a valuable training resource for up-skilling internal staff. Technical staff members can step through the exploit process themselves and learn how the attacks work, which subsequently leads to a stronger understanding, better confidence in possible fixes and avoiding similar issues in future code.
A side-effect of the deep-dive nature of a hybrid assessment is a lengthy report usually detailing a significant amount of vulnerabilities above and beyond what would be seen in a traditional penetration test. In a magical world of infinite security and development budgets this wouldn’t be an issue, but when constrained by costs and the fact the sun will eventually burn out, prioritisation and architectural-level decisions come into play. If you’re looking at completing a hybrid review, a report read-out workshop where we can discuss the prioritisation and potential fixes is strongly recommended.
Looking at our ThymeSheet example, we found that the legacy Accountomatic5000 system was exposed to the internet and vulnerable to several critical issues. Patching the entirely unsupported Accountomatic5000 was not going to happen, so the read-out workshop focused on limiting the exposure through attack surface management, restricting the ingress access to only the bare minimum necessary and building high-confidence indicators of compromise for ongoing detection. There’s no value in making recommendations that can’t be implemented so let’s figure out together what course of action is realistically best.
Summary
Regular penetration testing as strictly defined attacker types has its place in a mature security program. However, as things start to get a little weirder and security concerns extend beyond just the front door into a system, the vulnerability-research based approach has proven the better option in a lot of cases. Especially when it comes to understanding the practical security posture of your own strangely titled SaaS timesheeting platform.
Hopefully this article has given some insights into our vulnerability-research based hybrid security assessments and how they help build a better understanding of what vulnerabilities the system may be harbouring. This article is intended to be a high level first look at these types of engagements, so if you have questions for us, feel free to spin an email to [email protected].