Understand the differences between penetration testing and vulnerability scanning, and do both. Pen testing is a great way to test a company’s defensive capabilities against attack. But it can only test known vulnerabilities, meaning that a successful pen test doesn’t guarantee full protection against all possible threats. Vulnerability scanning is a crucial capability because it finds unknown vulnerabilities. And it can be easy to do if you use automated vulnerability scanning software.
As a part of our series about “5 Things You Need To Know To Tighten Up Your Company’s Approach to Data Privacy and Cybersecurity”, I had the pleasure of interviewing Ray Overby, CTO & Co-Founder at Key Resources. He’s a mainframe security and vulnerability expert, and a 30-year veteran in the specialized and highly demanding arena of mainframe information security. His special focus is mainframe vulnerability management and consultations to match organizations’ strategic business needs. www.krisecurity.com
Thank you so much for joining us in this interview series! Before we dig in, our readers would like to get to know you. Can you tell us a bit about how you grew up?
I was raised in a military family, moving from one base to another until I was 10 years old. My father was a Chief Petty Officer in the Navy and served on a destroyer in the Atlantic and Pacific. We ended up in Northern Illinois at the Great Lakes Naval Station in the late ’60s. My mom took a job at Abbott Pharmaceuticals second shift and worked there for 33 years. I have a brother who is eight years older. We were raised to be self-sufficient and learned early to get our chores completed before going out. You worked in order to enjoy life. I enjoyed sports and began wrestling and playing baseball in sixth grade. I wrestled in high school and won a state championship my junior year.
Is there a particular story that inspired you to pursue a career in cybersecurity? We’d love to hear it.
The first year we were living in Illinois I attended a computer programming camp at the Naval base where they taught us Basic and some Assembler. We had an opportunity to run our programs on the Naval Ordnance Research Calculator (NORC), which was considered one of the fastest processors when it was first built in 1954. I was hooked and decided that I wanted to be a software developer. I attended the Naval camp and others every year after that through 12th grade. I was lucky to land a job right out of college with SKK, Inc. and became a developer for the first commercially available security package for IBM mainframes. I have been in security since 1980.
Can you share the most interesting story that happened to you since you began this fascinating career?
When we incorporated Key Resources in 1988 it was with the intention of only doing security consulting and services. In 2007, we were asked to do a comprehensive mainframe penetration test for a Fortune 50 financial institution. They had been hacked. It was never made public and they wanted to know what other vulnerabilities might be lurking in the weeds.
We arrived onsite on a Monday morning and were given cubicles, but our credentials were not yet available. I settled in and began browsing things I could get at on their systems without a set of credentials. I was on the network in less than 30 minutes and found an exit that allowed me to create an ID and escalate my credentials. I then turned off all encryption for a millisecond.
I was shocked! You weren’t supposed to find vulnerabilities in the operating system layer on the mainframe, much less exploit it. It wasn’t heard of and no one thought about it. I immediately informed the VP of Operations that I needed to have a meeting with him and anyone else he deemed appropriate to discuss what I found. Less than an hour onsite and I was in a meeting room with eight VPs and their staff. I executed my REXX exec and turned off the encryption. The lead systems programmer was on the phone. I asked him to verify that I had indeed turned off the encryption. He confirmed this, and the room erupted into chaos. I turned to my partner and said “I don’t think they are going to let us leave. They may bury us in the atrium.” And there started our journey into vulnerability management software development.
None of us are able to achieve success without some help along the way. Is there a particular person to whom you are grateful who helped get you to where you are? Can you share a story about that?
Barry Schrager is considered the father of mainframe security to many in the industry. He was my first boss at SKK, Inc. and my mentor. He taught me how to think like a hacker before the term was acknowledged. I learned how to analyze software methodologies and write technically elegant software. And I learned how to think like an entrepreneur, not an employee. When SKK was acquired by Computer Associates in 1987, it was Barry who convinced me to go out on my own and use my skillset to help organizations build strong internal security teams.
Are you working on any exciting new projects now? How do you think that will help people?
We are looking at how to provide commercially available software and configuration vulnerability management for HPC systems. These systems are beginning to be used in industrial and commercial sectors and will drive the evolution of High-Performance Computing. Commercial applications such as data mining, oil exploration, web search engines, medical imaging, and financial modeling require the processing and analysis of large amounts of data in sophisticated ways.
What advice would you give to your colleagues to help them to thrive and not “burn out”?
Have a set of rules that dictate your work environment. For example, I eat breakfast with my wife every morning before I turn on my computer (including Saturday and Sunday). I finish my workday at 8pm Monday-Thursday and finish at 3pm on Friday so I can go out to dinner with my family (now we eat on our property outside). I take three 2-week vacations each year. I have a hobby (late date nickels). And I read a book a week.
Let’s now shift to the main focus of our interview. The Cybersecurity industry, as it is today, is such an exciting arena. What are the 3 things that most excite you about the Cybersecurity industry? Can you explain?
- The ability to make a difference. In the early days of Key Resources, people would ask me why I work so hard. My answer was that the work is too important to ignore. My experience with that Fortune 50 client showed me the mainframe industry has a problem that requires a programmatic solution. For a long while, no one listened. Even worse, for a while there was what I call a “Conspiracy of Silence” in our industry, where some of the top vendors and leaders actively denied the existence of vulnerabilities. I found that both irresponsible and risky. So much about our world depends on the mainframe — credit card transactions, personal health data, our government records, and so on. We built a solution and a business that is all about making transactions safer and making it more difficult for the hackers to compromise data. Over the years, we’ve identified 100+ unique kinds of vulnerabilities on every mainframe platform worldwide. Through that work, I believe we’ve made a positive difference to cybersecurity.
- Along those same lines, I’m also excited by the opportunity to change minds and evangelize for mainframe security. Mainframe is a closed industry. Vulnerabilities are not talked about. But they are real, and their risks are huge. Over the years I’ve found vulnerabilities that, if they had led to breaches on any other kind of IT system, would have made the evening news. But mainframe threats are not talked about in the same way risks to distributed systems are. Still, data is pretty hard to ignore. I’ve had interactions and shared data that swayed skeptical clients or very highly respected peers, who re-evaluated their position and ultimately admitted they had the wrong perspective on mainframe security. It’s gratifying to see minds change, not because it proves I was “right,” but because the stakes are so high that we need more people willing to take this issue seriously.
Finally, I’m excited that cybersecurity is constantly changing, and there’s always something to learn. The mainframe is being pulled into the 21st century. It’s not just a back-office system anymore. People are connecting into the mainframe environment through the internet and through an assortment of avenues that we never anticipated. Just think of connected home appliances — as my appliances get smarter, I have more TCP/IP network nodes in my house. Theoretically, I could exploit those nodes to break into a corporate mainframe. It sounds laughable, but TCP/IP is TCP/IP whether it’s on my mobile phone or my fridge. Today’s IT environment is changing so fast you constantly need to learn new things, and I find that exciting. I have no reason to be bored.
Looking ahead to the near future, are there critical threats on the horizon that you think companies need to start preparing for?
Zero-day vulnerabilities on the mainframe are a major critical threat companies will face as the mainframe plays a more prominent role in modern IT. Most companies take mainframe security for granted because these systems have a reputation for reliability and impenetrability. But mainframes are increasingly being connected to public-facing IT systems, and hackers know the mainframe is where some of the most sensitive information, including financial records, customer data, and personally identifiable information (PII), is stored. So, hackers are pursuing mainframe attacks using the same techniques that work for other IT systems (e.g. social engineering), while exploiting zero-day vulnerabilities that are present because of poorly written or configured operating system code. Once they’re in, hackers can do just about anything they want to the mainframe — from stealing data to crashing the whole system — so the risk to enterprises is significant.
Do you have a story from your experience about a cybersecurity breach that you helped fix or stop? What were the main takeaways from that story?
The story I told earlier about the Fortune 50 company is one of the most significant of my career. I was part of a team brought in to test the security of that company’s mainframe. The speed and ease with which I was able to find a vulnerability — just by snooping around a bit without any special tools or even user credentials — opened my eyes as to how vulnerable the mainframe environment was. Even more crystalizing was the client’s reaction — they weren’t happy at all! They simply never expected we would find any issue. It showed me that changing minds wouldn’t be an easy task. We would need to overcome a lot of skepticism and doubt. But, happily, we have found ways to open people’s eyes.
What are the main cybersecurity tools that you use on a frequent basis? For the benefit of our readers can you briefly explain what they do?
The mainframe doesn’t have a lot of cybersecurity tools per se. There are the fundamental Enterprise Security Managers: CA ACF2®, CA Top Secret®, or RACF, which protect mainframe applications and databases. But these tools can’t detect vulnerabilities in mainframe operating system code, which are particularly risky.
That’s why Key Resources developed the z/Assure® suite. We have several products that cover different aspects of mainframe security. z/Assure VAP is an automated tool to identify unknown mainframe vulnerabilities. z/Assure CAM helps IT risk officers and compliance managers assess their mainframe configuration for deviations from corporate security policy.
But ultimately, mainframe security often involves writing your own scripts to check things out. We’re also noticing a lot of compelling new Linux-based tools that hackers can use to get into the corporate network and exploit the mainframe. Companies can use those same tools to test their systems. Metasploit is an interesting one that is very useful for penetration testing.
How does someone who doesn’t have a large team deal with this? How would you articulate when a company can suffice with “over the counter” software, and when they need to move to a contract with a cybersecurity agency, or hire their own Chief Information Security Officer?
A lot of that depends on industry compliance requirements, as well as company resources. For small companies, resources can be sparse, and employees wear a lot of hats. Many times, these types of companies need to piecemeal their defenses and roll their own program rather than buying a more mature, supported tool. Easy-to-use tools, potentially even freeware and shareware, can sufficiently tend to cybersecurity even on inflexible budgets. If the budget allows, bringing in an outside expert is worth it, but it would still make sense to test out the right toolset first.
As you know, breaches or hacks can occur even for those who are best prepared, and no one will be aware of it for a while. Are there 3 or 4 signs that a lay person can see or look for that might indicate that something might be “amiss”?
Some signs that an external threat has gained access to the mainframe is if major, unexpected changes are made to the mainframe operating system. It could be a new account, privilege changes on existing accounts, or signs of configuration changes. Another sign could be that logging is disabled — hackers don’t want the system keeping a log of their activity while they’re trying to gain access or give themselves privileges within the system.
System instability vulnerabilities, if exploited, can cause z/OS service issues. If an address space crashes or starts behaving erratically, that should be a clear red flag that someone is attempting to crash the entire system.
At Key Resources, we focus on code-based vulnerabilities — flaws in operating system code that allow a program to bypass security controls. Unfortunately, if a hacker exploits one of these code-based vulnerabilities, they could gain access to all the data, applications and users on the entire mainframe. That could be catastrophic, and it’s very difficult to trace, especially if the hacker gains system administrator privileges. That’s why it’s crucial to defend your system proactively rather than be reactive to this threat.
After a company is made aware of a data or security breach, what are the most important things they should do to protect themselves further, as well as protect their customers?
Since the mainframe typically stores mission-critical, sensitive information, a breach can be catastrophic for an organization, their employees, and their customers’ or users’ data. At the earliest sign of a breach, communication is crucial. The security team needs to notify employees and anyone able to help, which could include management, tech specialists, client service managers, and your PR and communications teams. External communication is also important — you may need to mail notices to your clients, issue an official media release, or speak to the press. This step requires total transparency to prevent the issue being downplayed.
Then, it’s all about understanding the root of the issue so you can uncover the cause of the breach. Most often the breach is caused by external hacking, but sometimes involves a negligent employee. Find the source of the problem before you take steps to attack it. Patches protect against other vulnerabilities once they are found and installing them quickly reduces the risk of additional attacks. Finally, enlisting a third-party consultant to audit the system helps ensure that a breach won’t happen again.
Compliance regulations add an important sense of urgency to cybersecurity, creating a checkbox that needs to be checked in order to work in regions with these requirements. Those mentioned invariably call out penetration testing and vulnerability analysis. Since mainframe vulnerabilities only occur in cases of improper design or bad code, binary code scanners based on Linux and Windows could never result in a security vulnerability. But thanks to these compliance requirements, everyone in mainframe is going to have their OS scanned. We’re the first ones to do it and know where exactly to look, resulting in 100% accurate vulnerability conclusions.
What are the most common data security and cybersecurity mistakes you have seen companies make?
One of the industry’s most persistent problems is an attitude problem. The belief among security leaders that mainframes are impenetrable and immune to external attacks can lead to severe business risks. CISOs and CIOs, who may not have deep knowledge of mainframe environments, typically take mainframe security for granted, if they even bother to think about the topic in the first place. I described this “Conspiracy of Silence” above, in which industry leaders avoid publicly disclosing vulnerabilities. This suppresses any talk of mainframe vulnerabilities, leading to a lack of independent research to shed light on the risks.
This attitude leads to a related issue: leaving the task of securing the mainframe to multiple areas of IT, who operate in silos from one another, such as the security, operations, and systems programming teams. Rather than splintering the responsibility for mainframe security, companies should hire a mainframe architect or bring in a mainframe specialist to do routine scanning.
Since the COVID19 Pandemic began and companies have become more dispersed, have you seen an uptick in cybersecurity or privacy errors? Can you explain?
The uptick in remote workers increased the potential for mainframe vulnerabilities. But with more and more connections tapping into the mainframe from home, there is greater opportunity for hackers to jump-in. Users are more likely to have mainframe security credentials compromised or exploited, connecting hackers to a buffet of highly sensitive information. Two-factor authentication plus internal encryption can help battle these potential breaches, but in most cases mainframers just don’t think the expense is necessary due to their castle and moat mentality. Security practitioners outside of the mainframe already know this because they get pummeled with potential hacks every day. Mainframes aren’t quite there yet, but work-from-home has made this mindset switch more urgent.
Here is the main question of our interview: What are the “5 Things Every Company Needs To Know To Tighten Up Its Approach to Data Privacy and Cybersecurity” and why? (Please share a story or example for each.)
The main thing companies should know about their approach to mainframe security is that the old assumptions don’t apply anymore. The mainframe is not Fort Knox, it’s not inherently secure, and in fact, modern IT means it’s more connected and exposed than ever. So, you need a modern approach to mainframe security. We make 5 recommendations:
- Designate a mainframe security architect. This is the “guardian” of your mainframe, the person accountable for designing and maintaining your mainframe security policies.
- Ensure separation of duties. Don’t put all the responsibility for security on your security architect. A security professional should not be responsible for reporting on their own activity — that’s an obvious conflict of interest. So, while your security architect designs the mainframe security processes, the reporting of mainframe security should be left to someone else, whether they work in-house or independently.
- Make scanning application and operating systems code your number one priority. Mainframe vulnerabilities exist. But you can’t correct them if you can’t find them. Get the tools and support you need to make a difference.
- Check for excessive access risks. This is right out of the Zero Trust security playbook. The reality is employee user credentials are a major vulnerability. There’s a reason hackers love them. They’re relatively easy to find or steal, and they provide easy access to your systems. And many times, those credentials have more privileges than what’s actually required for the user to do his or her job. It’s such a basic oversight, but it’s an easy one to exploit and even easier to fix. Fortunately, you can automate excessive access checking to find problem accounts.
- Understand the differences between penetration testing and vulnerability scanning, and do both. Pen testing is a great way to test a company’s defensive capabilities against attack. But it can only test known vulnerabilities, meaning that a successful pen test doesn’t guarantee full protection against all possible threats. Vulnerability scanning is a crucial capability because it finds unknown vulnerabilities. And it can be easy to do if you use automated vulnerability scanning software.
You are a person of enormous influence. If you could inspire a movement that would bring the most amount of good to the most amount of people, what would that be? You never know what your idea can trigger. 🙂 (Think, simple, fast, effective and something everyone can do!)
A movement to provide funds to educate our high school students about diversity through work-life travel. For many years Europe, Australia and Japan have been sending their young people out on sabbaticals (either before college, during college, or before entering the workforce). There are many young people who can benefit from spending time in other countries working or learning for a year. Most universities in the U.S. have semester abroad programs, but the students attend an affiliate or wholly owned facilities that are Americanized. Our oldest son attended George Washington University studying international economics. He applied to go over to Switzerland for his junior year to study at Fribourg University under one of the top economists in the world. He was turned down by GW because it was not a US-based university. He went anyway. His dorm floor included Iranian, Hungarian, German, French, and Texan students! He learned more in that year than he did in his other three years of college. And he has maintained his friendships with all those students over the years.