What is a pentest or a red team? Before we go any further, we should define our terms, though you may find they’re often used interchangeably:
- Pentest: A penetration test, colloquially known as a pentest, attempts to use attacker methods to assess the exploitability, number, and impact of vulnerabilities so the target organization can remediate. It’s often thought of as a test of the technical controls of a security program.
- Red Team: A red team, by contrast, is not entirely vulnerability-focussed, but rather goal-driven, and a test of the overall posture of a security program. They use vulnerabilities and attack methods to achieve the goal, but also test the social and procedural controls such as detection and response capability, procedures around phishing (fraudulently contacting employees) and so on to simulate how a real attacker might go after important information using any means available. Red team tests are much more a test of the security posture as a whole than pentests.
Pentesting or red-teaming serves a number of valuable purposes. It can help establish confidence in your security practices, or highlight areas that need to be shored up. It can provide evidence needed for certifications or customer security questionnaires. It can help build a case for investing in tools and approaches that keep your users’ data safe.
Much of the security industry, for better or worse, focuses on selling services related to adversary simulation, and so much of the marketing and sales around security services does as well. Knowing what you’re hoping to get out of a pentest can help you navigate the marketing hype and find a firm that pushes your application’s security past its limit.
Are you ready for a pentest?
Well before contacting a vendor or responding positively to a sales inquiry, an organization should ask itself: is it even ready for a pentest? There are a number of presentations, blog posts, tweets, and podcast comments about readiness for bug bounties; pentests, like bug bounties, largely serve to illustrate gaps, and to generate new work without doing anything to clear the backlog of existing bugs. It’s important to realistically assess your security program’s ability to triage and schedule fixes for new bugs that come in, especially since some may be quite serious and require major architectural changes. It is also important to realistically assess the maturity of your organization’s infrastructure security. Are you comfortable with your organization’s inventory management? Your story about vulnerability management? Access controls? Think about the things that keep you up at night regarding your security posture, as those will be the first things that pentesters will find.
What do you want to get out of it?
Unfavourable answers to the questions of security readiness may in fact be the purpose of seeking a pentest. It’s important to determine what value you intend to derive from a pentest or red team engagement. You, as the red team coordinator, may wish to think about who should be the primary consumer of the delivered report as you decide the scope, target, and other details. There are several reasons to engage offensive services and different forms of value you can derive from each.
Making a case for investing in security
You may be seeking approval for security headcount or products from executives or decision makers, but are having a difficult time communicating the scope of the problem or the return on investment. In these cases a pentest report can be a fantastic tool, with graphs full of “CRITICAL vulnerability” in bright red ink and a narrative structure in bullet points showing a simulated attacker starting from the internet and getting access to whatever crown jewels your organization is most concerned with. Here at Slack, our utmost concern is your customer data, but your organization may find quarterly financial reports to be the most important bits of digital information, or journalistic sources, or customer metadata. It depends entirely on the needs of your group.
Evidence for certifications and customers
You may be looking for a pentest report to respond to a customer or regulatory request, since many companies will ask for, if not demand, your most recent pentest as part of their procurement process. A pentest report from a trusted and competent vendor may be the difference between closing a deal and not. Additionally, in a regulated space, you may be required to periodically engage pentesters to maintain certifications. Slack has many certifications, including HIPAA, SOC2 and ISO27001, but it’s important to note that regulatory requirements are a minimum rather than a goal, and your pentest alignments should still reflect a desire to do security right (which, if done well, should make audits easy to pass).
Establishing confidence in your systems and processes
You may be confident in your security policies, threat modelling, and mitigations and practices. You may feel like you’re able to fend off all but the most sophisticated attacker. And you might want to test that assumption. That’s great! Pentests, and even more so red team engagements, can provide a level of confidence that your organization proves a difficult target to determined, skilled, professional attackers. Assuming, of course, you’ve selected your vendor’s skill strengths for your environment. A vendor that excels at penetrating Windows environments may be able to stumble through your Mac office with a reasonable degree of success, but isn’t given the opportunity to show their strengths.
Finding the right pentest vendor
So how do you find a vendor, anyway? It is important to point out that organizations don’t innovate. People innovate. Organizations can, at best, support the people that comprise them and seek out people who have the skills and interests to further their goals. And red teams are made of people with motivations and feelings and personal lives outside their job. Some of the best hackers in the world may work for boutique, expensive firms tailored to your environment, with a small staff and a selective customer base, because it regularly engages them with exciting work But some of the best may also enjoy the stability (both in workload and pay) and the flexibility that a bigger company affords them.
The other thing to consider when choosing a vendor is your own environment. Are you primarily a top-down Windows-on-AD based shop? Or do you issue MacBooks to everyone? Is your production infrastructure a cheffed AWS machine, or have you gone all-in on Kubernetes? You’ll want to ensure that your simulated intruders are skilled with your particular infrastructure. The less time they spend ramping up, the more value you’ll get out of their time.
Another great strategy is to find individuals doing talks and writing blog posts about your particular infrastructure needs and reach out to their employers (or the people themselves). Often people working at an organization will bring other people with similar expertise with them, so it’s a good marker that the consulting firm will work for your environment. A boutique firm can work against your goals, if your goals for a pentest are customer- or compliance-facing. Your customers or compliance auditors may not be familiar with or trust the specialized firm you find. Luckily many of the large, well-known vendors employ great people with a wide range of skills. It is important in this case to make your needs known early on in the contract negotiations, so the best people for your environment at the contracting firm can be scheduled.
Scoping
The initial scoping call usually includes both technical and financial components, usually with the latter following agreement on the former (you figure out what work needs to be done and then figure out what it’ll cost). To get the best engagement possible you’ll want to prepare.
The big questions are going to be about scope and target: What systems are off-limits (perhaps because they contain sensitive data, or are so mission-critical that any disruption could impact the business) and what are the “crown jewels” that the team should aim to access as a metric for success? You should also think about how involved you’d like your own teams to be. Are you testing the defenders and incident response teams’ capabilities? Is this a “purple team” exercise, with the defenders and attackers working closely together to write new detection rules?
Here at Slack we like to give our red team engagements a wide-open scope, but keep a referee who’s invested in the success of the red team that won’t leak operational information to the defending blue team. One approach we’ve used to get the most out of the limited time we have with the red team is to give them documentation describing our architecture and all the systems in it. This way our pentest team has access to information that a real attacker with months of recon time might get.
During the engagement
Once the engagement begins, it can proceed in a few ways. You may choose to keep it as a pure black box adversary simulation, giving neither the defending party nor the attackers any information on how the engagement proceeds. On the other hand, it may make your internal teams more comfortable if someone knowledgeable receives periodic updates. It’s common to give daily updates of where the pentesters have been and what their plans are. At Slack we take it a step further and integrate our referee in to the red team directly by inviting them to shared private channels that have a constant stream of updates from the pentesters, along with suggestions from our referee on where to proceed. This has the added bonus of better simulating a long-term attacker that quietly collects information about an organization.
Reviewing the results
After the engagement is complete, the pentest team will deliver a final report that includes a high-level overview, explanation of their procedures and tools, along with a list of vulnerabilities found along the way. You may wish to discuss the particular findings for your environment, as the rubric pentesters might use is quite general. It’s possible that you’ll want to reclassify some vulnerabilities based on your mitigations and architecture, or based on the privileged position you may have given the attackers. Once the report is delivered, someone in your organization will need to be responsible for the project management tasks of filing bugs and following up with affected teams to remediate.
Rinse and repeat
Pentests can show you the weaknesses of your systems, and give you and your customers peace of mind that you’re ready to weather attacks. But each test only assesses a particular moment in time — as your systems change and mature, it’s likely that the surface area for attack will also change, so regular pentests should be scheduled to exercise not only the protections you have in place, but also test against new intrusion approaches.
Hiring pentesters or red teams can seem like a daunting task, especially in the world of high-pressure sales, but it doesn’t have to be difficult. It can be as simple as:
- Know what you want to get out of the engagement
- Hire the right vendor
- Set them up for success
- Ensure you close any holes they find
If you’re a security engineer and you’re interested in helping us keep Slack data safe, take a look at our Careers page and apply today!
