As the world of edge computing becomes more distributed, billions of connected devices live on the edge, which need to be secured, managed and automated. For many businesses, this means deploying a VMware and cloud infrastructure and using VMware vSphere, for example, to encrypt private information.  While it is easy enough to encrypt data on the edge, key management has proven to be a challenge.

I recently sat down with Patrick Townsend, Founder and CEO of Townsend Security to talk about deployments on the edge, achieving a strong security posture with key management, and other ways that businesses can better secure their private data. 

Patrick, Townsend Security has had key management solutions for VMware for a number of years. What is special about Edge computing?

Well, Edge computing is fascinating.  It isn’t really that different from how we currently think about computing and data security in the cloud or on-premise. By moving applications closer to the end user, Edge computing brings a better, faster user experience to the end user.  So, if you are running an application in the cloud, perhaps in a retail or healthcare environment, the delay over the network can degrade the experience or inhibit the ability to collect a lot of data, for example, from IoT devices. Edge computing is a natural evolution of making things more efficient with a better user experience.  However, Edge computing also brings new security challenges too. If we are collecting data that is sensitive in nature, it is just as sensitive out on the Edge as it is in our data center.

So what is special or different about Edge computing from a security point of view?

There are a number of challenges.  How do we deploy applications in a secure way?  How do we do application patching? One of the most important security efforts that we make is to keep everything patched and up to date.  When you have Edge computing, there are a lot more environments in distant locations. The security process really becomes more complex when we move to Edge computing.  Those challenges can be solved, but they represent things that we really need to pay attention to.

At the same time, as we are pushing applications out to the Edge, compliance regulations are getting more stringent.  Just look at the California Consumer Privacy Act (CCPA) and GDPR before it. Both of those are making the protection of sensitive data much more important.  The risks of data loss to an organization are escalating, and at the same time, we are pushing data to more and more places - so we have a big security challenge that we need to step up to.

Protecting data data in a centralized IT data center is a challenge, but one that we are used to.  Edge computing brings unique problems with it. For example, let’s say you pushed some data out to a dozen Edge computing environments.  You’ll need to encrypt that data to meet compliance, but where is the key manager? Is it back at your central on-site data center? If so, you may have just lost the advantage of pushing everything to the Edge.  Encryption and key management also need to be pushed out to the Edge in order to meet security best practices, just as you would in on-premise environments.

In terms of the cloud, can you give some examples of Edge environments?

Sure.  In the Cloud, we try to deploy applications close to the end-customer which gives us better response times and a better customer experience.  So, in AWS or Azure, we can move applications closer to where the end-customer lives. CSPs are making this easier by automating some of the deployment tasks.  By pushing applications to the Edge, you get really close to the physical location of the customer. For example, if you live in Sweden, you don’t need to connect to a key manager that is sitting back in Silicon Valley.  You should connect to a key manager that is near you. When moving to the Edge, encryption and key management need to move with you.  

By the way, you may have noticed that VMware has been working much more closely with Cloud Service Providers to provide true VMware infrastructure on cloud platforms.  For example, on Azure, you can deploy a full bare metal stack - VMware in the cloud and managed the way you want. But again, when you push those VMware environments to the Edge, what about the encryption key management?  The good news is, that with our new Alliance Key Manager for Edge Computing, we can make that easy and affordable to accomplish.

How about some examples of non-Cloud Edge environments?

Almost all of us use VMware on premise, and it isn’t really all that different to what we are currently doing.  Think of a medium or large retail organization with hundreds or thousands of storefronts. When you walk into the store, there is a very good chance that there is a local VMware node out there that is running many applications.  Think about a large box store with retail, pharmacy, and automobile services. The VMware environment in a single store might support dozens or hundreds of specialized applications. How do you protect data in that environment? Sometimes when we think of Edge computing we think of “just” the cloud, but this isn’t the case.  Again, just like with the Cloud, it doesn’t need to be difficult to push encryption and key management to the Edge, it just needs to get done.

How do compliance requirements impact Edge computing?

Well, compliance requirements, which are getting stronger as we speak, make the challenge of Edge computing even more important to address.  If you think about it, when we have centralized IT processing, we have one environment to protect. It may be a very data rich environment with sensitive data that cybercriminals may want to steal - but it something that isn’t that difficult to protect.  What if we have 500 of those environments out there across on-prem and cloud locations? The attack surface has been dramatically increased. The data is still important and still a target, but now we have a lot more to deal with. I think people are waking up to the security challenge and need to focus just as much effort on securing data at the Edge as we do at the centralized, on-premise data center.  We have to deploy all of our security defenses at the Edge in the same way that we do with core IT systems. The data is the same.

How is Townsend Security trying to help resolve this challenge.

The barriers to getting Edge data protection right are only party technical.  Key management vendors have not adapted to the new reality of the Edge. The huge expense of traditional KMS solutions is the primary barrier to protecting data at the Edge.  For small businesses, they can even be completely priced out of the market around doing encryption and key management right. Large organizations have been priced out as well.  When there are hundreds or thousands of endpoints that need protected, vendors need to step up to help these businesses secure their data.  

At Townsend Security, we have two distinct advantages.  First, our Alliance Key Manager for Edge Computing solution is virtualized, automated, and VMware Ready.  For example, our key management solution has been certified by VMware for vSphere key management - to protect VMs and vSAN storage.  We are seeing many organizations deploy VMware at the Edge. Second, we have the ability to flexibly license and price our solutions for the Edge.  Enterprises can now deploy full VMware VM encryption and key management at Edge with an affordable solution.

To hear this conversation in its entirety, download our podcast Living on the Edge and hear Patrick Townsend further discuss deployments on the edge, achieving a strong security posture with key management, and other ways that businesses can secure their private data.

Historically, encryption and key management have been deployed at the application or database level. There are even several databases who’s “Enterprise” edition (like Microsoft SQL Server or MongoDB, for example) include options for encryption and external key management built right in the database. Unfortunately, these types of databases are the exception, rather than the rule. If you were to examine an organization's IT infrastructure, you are more likely to find a wide variety of databases and applications, some natively supporting encryption, some not, and many containing unprotected private information or personally identifiable information (PII). Simply put, their encryption strategy has been limited due to cost and resources required to properly protect private information. 

Fortunately, these same enterprises have deployed VMware infrastructure, and starting with vSphere 6.5 and vSAN 6.6, are able to encrypt sensitive workloads in VMware using the advanced cryptographic features in vCenter. To put it a little more simply, businesses can protect their sensitive information in their internal applications and databases that don’t natively support transparent encryption with tools offered by VMware.

I recently sat down with security expert and CEO, Patrick Townsend, to talk about how enterprises can leverage VMware’s vSphere and vSAN to encrypt private data - regardless of whether their applications or databases support encryption. 

Hi Patrick. Let’s jump right in. With the introduction of vSphere encryption in 6.5 and vSAN 6.6, it has become much easier for businesses to encrypt private data. In the past they have relied on encryption at the application level or used the encryption that comes with their database. With so many enterprises deploying VMware, they no longer need to let their application or database limit their encryption strategy.

That’s absolutely correct. There are databases like Microsoft SQL Server and MongoDB EA, for example, that have encryption built right in - which makes it easy. But there are other times when encryption can be much more difficult. SQL Server Standard edition and the Community edition of MySQL, for example, do NOT support encryption. So, you have these widely used databases, with lots of unprotected data because that can be a challenge to encrypt. Using vSphere and vSAN encryption is a great way to address these gaps in an organization's encryption strategy with industry standards-based encryption. 

Sometimes the barrier to encryption is the cost of upgrading databases to “Enterprise” editions. Almost all of us are running VMware in our infrastructure anyway, so in many cases we already have the tools we need - the encryption support is there, we just need to use it. VMware even provides excellent guidance for encrypting databases, like Oracle and SQL Server, for example.

So, one of the most obvious questions. How is performance?

This is always a concern that people bring up. I can say that VMware has done a great job with performance in both encrypted VMs and vSAN - and performance continues to improve. These days, you can even deploy a large database on vSAN. This is a technology that has matured and gained the trust of customers, and they are adopting it at a rapid rate. There is also some really good material from VMware about performance expectations - white papers, solutions briefs, etc. Furthermore, both vSphere and vSAN take advantage of the Intel AES-NI on-chip accelerator for encryption, which provides a great performance boost.

Of course the key manager is the critical component that ensures the encrypted data stays encrypted. Without proper key management, it is like leaving the keys to your house under the welcome mat. What should our readers be looking for in a key manager?

Here is something that I think VMware did right. You must use a key manager in order to activate vSphere encryption of VMs or vSAN encryption. Within vSphere you are able to create a KMS cluster, define failover key managers, multiple KMS clusters, etc. They did a great job. Furthermore, VMware based their interface on the Key Management Interoperability Protocol (KMIP) industry standard. Other databases vendors, for example, allow local storage of encryption keys. That is really such a BAD security practice, so I am glad that VMware saw implications of that. If you are going to use VMware data-at-rest encryption, you are going to use proper encryption key management and that will be much better from a security perspective. I also think that this reflects on VMware as a company and their concern for their enterprise customers.

What to look for in a key manager? All enterprise level key managers are validated to FIPS 140-2 by the National Institute of Standards and Technology (NIST). Be absolutely sure you key management vendor has completed this validation. Secondly, your key manager should support the KMIP protocol. Finally, if you are taking credit cards for payments, look for a PCI validation. We validated our Alliance Key Manager with both Coalfire and VMware, as a joint project. This helps our customers easily get through an audit, which can be quite difficult.

While I have you, I was hoping you could also offer some clarification on the term KMS. For example, VMware defines KMS as a Key Management Server. Amazon defines their KMS as a “Key Management Service.” How should our readers be thinking about a KMS in regards to VMware encryption?

Ah, the chaos of three letter acronyms. KMS, in general terms, means Key Management Server. It is a broad term covering key management devices that manage the entire lifecycle of a key - from creation to destruction. You are right, Amazon does call their key management service KMS, which can lead to some confusion. This service is NOT to be confused with a key management server - and does not give you full control over the entire key lifecycle. It is a shared administrative environment where you share access to the keys with Amazon.

You need to approach cloud service provider (CSP) implementations of key management services with trepidation. It is important for YOU to hold exclusive access to your keys and that only you have the only administrative control. Cloud lockin can be another concern as well.

To hear this conversation in its entirety, download our podcast Don’t Let Your Application or Database Limit Your Encryption Strategy and hear Patrick Townsend further discuss Encrypting applications and databases that don't natively support encryption, encryption performance, and other fundamental features of an enterprise grade key manager.

Recently, we here at Townsend Security had the opportunity to poll the fans of our Newsletter to see how folks are doing with encryption and key management for their data-at-rest. We conduct this survey, and surveys like it, so that the larger InfoSec community can get a snapshot of how businesses, in general, are doing in securing their sensitive data. Below are a few key findings, hope you enjoy!

Overall Results

First, the good news: 73% of respondents report that they encrypt their sensitive data while at-rest. This makes sense as all the respondents are fans of our Newsletter; the group is a little self-selecting in that they have already expressed an interest in data security. Of course, we would like to see the number at 100%, but that would mean our work is already done—and we know we still have a long way to go.

To give a bit of perspective, we conducted two additional surveys that represent a more general audience that we published, here and here. In those two surveys the adoption rate for encryption is closer to 50%. So, hats off to our fans for being above the curve!

Now, the bad news: Only 50% of respondents say that they use proper key management to secure their encrypted data-at-rest (again, a little self-selecting in that, as part of the reason they like our Newsletter is that they are learning more about key management). Interestingly, even if you adjust the data to only look at those who replied that they do use encryption, the number only jumps to 66%.

As a comparison with the wider community, only about 30% of respondents in our other two polls (referenced above) said that they use encryption key management to securely manage their keys.

Expert Weigh-In: Patrick Townsend, CEO of Townsend Security
"Encryption is not enough. In order for encryption to be secure, the keys must be properly managed—100% of the time. If you don’t properly manage your encryption keys, it is like placing your house keys under your welcome mat. Every good thief knows to look in the obvious places for easy entry. Hackers do as well."

Encryption and Key Management Use, per Database/Blob Storage

It is no surprise to see that, overall, if a database/blob storage reports a rise/fall in the use of encryption, there is also a corresponding rise/fall in the use of proper key management. What is interesting, however, is the databases/blob storage where the respondents reported the widest gap in adoption of key management in comparison to the adoption of encryption. Here are the top five databases and their corresponding adoption gaps:

What may or may not be surprising is that SharePoint leads the pack in lack of key management adoption (compared to encryption adoption) and SQL Server Enterprise Edition comes in third. SharePoint is built on top of Microsoft SQL Server as its datastore (for structured data, at least). For SQL Server 2008 Enterprise edition and up, you now have the ability to not only take advantage of SQL Server’s Transparent Data Encryption (in SharePoint and SQL Server), but you also can leverage the power of a third-party encryption key manager using Extensible Key Management (EKM). This means it is incredibly easy to not only deploy encryption but also proper manage the encryption keys.

What is less surprising is the other three that made the top of the list. All these come with free editions that do not come with encryption libraries, let alone the ability to properly manage the keys. So anyone spinning up a free version of these databases will, by their very nature, not be able to secure their data.

Expert Weigh-In: Tim Roncevich, Partner at CyberGuard Compliance
"Many Enterprise editions of databases come with robust AES encryption and a way for a third-party vendor to manage the encryption keys. If you are storing sensitive data in an open-source, or free, version of a database, upgrade today. Hacks similar to the Cathay Pacific breach of 2018 were due to the company not upgrading to the Enterprise edition to take advantage of the encryption and key management that were available to them."

What Virtualization Do You Use?

About three-quarters of respondents said that they use VMware in their environment. The other quarter reported using Hyper-V, Red Hat Virtualization, Citrix Hypervisor, or KVM. Less than 10% said they used multiple virtualization platforms.

This is great news for the majority of businesses, then, in terms of encryption and key management. VMware’s vSphere 6.5 and up come with encryption ready to use. Not only that, but using a third-party encryption key manager is easy to set up and deploy.

Expert Weigh-In: Sharon Kleinerman, Director of Sales at Townsend Security
"For those organizations struggling to secure their data-at-rest with encryption and key management, doing so has never been easier. If you have VMware 6.5 and above, you simply set up your third-party key manager through vSphere’s KMS Cluster KMIP interface, tell vSphere which VMs you want encrypted, and your data is encrypted. Same with vSAN. It really is easy to encrypt with VMware as your virtual environment."

Backup & Recovery Solutions

Backup and recovery solutions are an integral part of business continuity. In fact, Allied Market Research estimates that the market will grow by almost 25% year over year through 2023. In the next few years, however, Gartner estimates that 50% of companies will augment or replace their current backup solution with another.

Our findings fall roughly in line with Gartner’s research. According to our survey, about 40% of respondents say they will, or don’t know if they will, replace their current backup and recovery solution.

Expert Weigh-In: Steve Brown, Partner at Rutter Networking Technologies
"For those thinking of switching your B&R solution, it is important to make sure that the solution you are switching to provides encryption and a way to manage your encryption keys. Encryption should not be an afterthought. Instead, it should be one of the main drivers as to why you would either stay with your current solution or look farther afield."

Conclusion

The rate at which data breaches are happening is not slowing down. We all know this. But the adoption rate of best practices is still lagging. While it is heartening to see our blog’s fanbase beating the overall average for using encryption and key management to secure sensitive data-at-rest, We still have a long way to go.

The good news, it is easier than ever to adopt best practices. If you are thinking about truly defending yourself with a defense-in-depth strategy, talk to us today.

In our practice here at Townsend Security we engage with a lot of startups and mature ISVs who are trying to grow their business and customer base, leverage their technologies into new opportunities, and grow or migrate to the cloud. We know how difficult it is to start and grow a company, and what a wide set of business challenges have to be overcome. Our hats are off to every entrepreneur who has created a successful company, and every ISV who has kept it going!

I want to share a few thoughts on some pitfalls that can damage your ability to grow your company with a focus on the encryption of sensitive data. Too many promising companies flounder because of poor security implementations, and failing to get encryption right can lead to lost opportunities - maybe even the loss of that breakout sale you need to land a global company. Some early thought and planning about data security can help you weather your migration up the food chain and avoid such losses.

Number 1: Failure to encrypt sensitive data

The single biggest failure of data security is not doing it at all. Even in this age of massive public data breaches, and the damage that they do to companies of all sizes, most startups and ISVs are not implementing encryption of sensitive data. When product managers and developers work on their next big idea, they focus on exciting features in their product and often ignore the work it takes to implement encryption. They instead rely on access control lists and other mechanisms to protect data. These are, of course, important things to do. But the failure to encrypt sensitive data leaves a big hole in your security strategy.

What can go wrong if you haven’t implemented encryption? LOTS !!!

• The publicity around a data breach can tarnish your reputation and kill opportunities.
• The lack of encryption may cause compliance regulation failures making it impossible to enter new markets.
• You may not be able to pass a security review of your software by that large global Enterprise.
• You may not be able to enter government channels where encryption is a mandate.
• If your customer experiences a data breach you may encounter substantial litigation costs that damage your financial resources and delay critical development.
• You may fail to secure that next round of funding when an investor discovers the security gaps in your product.

When these kinds of events damage your ability to grow your company, it can be hard to mitigate them in a timely fashion. And you often won’t know about these dangers until you get fairly far down the road with your business plan.

Number 2: Failure to get key management right

For startups and ISVs who DO understand the need for encryption of sensitive data, the next biggest pitfall is the failure to protect encryption keys properly. Almost every database that supports encryption also supports the ability to protect the database encryption keys with a key manager. But that doesn’t mean that good key management is the default! In most cases the default database key management option is to store the encryption keys on the same server as the sensitive data. Sometimes the database will even store the encryption key locally and in the clear! So getting encryption key management right is critical to your security strategy. It won’t help to have encryption of your data enabled, and then have a cybercriminal steal your data along with the encryption key.

Related to key management here are some things to look for when you consider databases for your application:

• Does your database have built-in encryption? Relying on third-party encryption solutions at the file/folder level will certainly cause deployment and scalability problems.
• Does your database support integration with third-party key managers? If there is no easy way to integrate proper key management into the database, this will also cause deployment and technology delays.
• Does your database support open standards for key management? For example, the Key Management Interoperability Protocol (KMIP) defines how applications like databases can easily integrate a key manager.
• Does your database support key management failover? Remember that protecting encryption keys with a key manager also brings along the question of high availability and failover.

HINT:

If you are a startup be sure to choose a database that supports built-in encryption and proper key management. You have lots of good choices in both commercial and open source solutions. So go with a database with native, built-in encryption and key management!

Number 3: Failure to get FIPS 140-2 right

There are important standards and certifications for key management solutions. The most important of these is the National Institute of Standards and Technology’s (NIST) FIPS 140-2 standard. In addition to being a published standard, there is also a validation process for key management systems. The standard, and the validation to that standard, are critically important to your data security strategy. All professional key management solutions have been validated to the FIPS 140-2 standard and you should be sure to deploy a validated key management solution. This will help you avoid failing a security audit by that important new customer!

In addition to ensuring that your key manager is validated to FIPS 140-2, be sure that the entire key management solution is validated. There are many cases where the encryption library alone is validated to FIPS 140-2, but the key management application is not. It is good to have validated encryption, but that is just the start! Encryption key management has its own validation points and you will need both.

Snake Oil Alert !!!

Unfortunately, there are some key management solutions that make unwarranted claims about FIPS 140-2 compliance and validation. Here are a few warning signs to look for when you evaluate a key management solution:

• A vendor makes compliance claims, but there is no validation. Some vendors claim to be “FIPS 140-2 compliant” but in fact have never completed a FIPS 140-2 validation. Security is hard, and unsubstantiated claims should be a red flag.
• A vendor claims FIPS 140-2 compliance, but the validation is “in process”, but not complete. A security product can be “in process” for many months or even years. A claim of FIPS 140-2 compliance without actual completion should also be a red flag.
• A vendor makes some claims of FIPS 140-2 validation, but research shows that the key management solution was not validated by that vendor.
• A vendor makes a claim of FIPS 140-2 compliance, but the solution is only compliant when backed by a third party validated key management solution. In this case the vendor solution itself is not validated, but relies on the validation of another solution. You may be fooled into thinking that the solution itself is compliant when it is not. Especially watch for this pitfall with open source solutions.

You can always check a vendor’s claims of FIPS 140-2 compliance. Ask for the NIST FIPS 140-2 certificate number, and then Google it. NIST makes the validation certificate available to the public on their website. Copy and paste this into Google search:

NIST FIPS 140-2 certificate number 1449

That was easy!

Number 4:  Failure to make encryption and key management easy and invisible

Now that you are on the road to getting encryption and key management right, it is important to also make it easy and invisible. Your customers have a lot on their agendas, and becoming a key management expert is probably not one of them. So even if you follow the above advice and implement encryption and key management, do your customers a favor and make key management easy. The best way to do this is to bundle a key management solution into your product, and make key management automatic. You can still enable the configuration of an external key management system (some customers will want this), but you can really make it easy for most of your customers if you automate the key management tasks.

Automating key management is a great competitive advantage! One of our partners in the archival and backup space implemented this strategy and make great competitive wins on this feature alone! Their message was simple:

“We have encryption and key management. It is FIPS 140-2 validated. It is completely automatic so you don’t have to spend time fiddling around with a complex key management system.”

This strategy won them a lot of competitive deals and it was easy to talk about - and it shortened the sales cycle.  Of course, be sure that your key management solution supports this type of integration and automation!

Number 5:  Failure to segment customer data

As you move to the cloud and create shared, multi-tenant SaaS solutions, be sure to plan for and architect data segmentation into your solution. You will encounter large customers who will not want to have their data in the same space as other customers. They will want the additional security of segmenting their data into a virtual private cloud. With planning, your technical team can meet this kind of requirement, and help you close that very large deal.

Of course, a data segmentation plan requires a key management segmentation plan. For the same reasons customers want to segment their data, they don’t want to share key management with other customers. And they want to maintain full control of the key management implementation. So be sure to plan for customer-specific deployments of encryption key management and failover key management servers. A properly implemented data and key management segmentation plan will even allow for on-premise deployments that are “cloud ready.”

Number 6:  Failure to develop new market opportunities

Think about Amazon (the company) for a moment. At one point in their history they were an online bookstore. Today the company is very different. Amazon first leveraged its technologies to sell all kinds of products, and then created Amazon Web Services (AWS) to enable all of us to benefit from cloud technologies.

Are you thinking like Amazon? If not, you might be missing some big opportunities. Now that you have secure applications, are there lateral opportunities or technology licensing opportunities available to you? When you approach new opportunities and partners, don’t be afraid to talk about security. Regardless of what you’ve heard:

SECURITY SELLS!

Over the last several years, encryption key management has attained “essential infrastructure” status. When done properly, key management can protect encrypted data - and in the event of a data breach, can even provide a company with an exemption for a breach notification.

I recently sat down with Patrick Townsend, Founder and CEO of Townsend Security to talk about encryption key management and the importance of standards, what to look for in a key manager, and fundamental features of an enterprise grade key manager. While Townsend Security isn’t alone in the key management game, there are a few telltale signs that differentiate a key manager that meets industry standards and one that will leave you with a breach notification on your hands. Let’s jump in.

Hi Patrick. Townsend Security has been in the key management game for close to 10 years now. Back when Alliance Key Manager was brought to market, people weren’t really sure what key management was. Now it is considered essential infrastructure.

Yes, it sure is. Things have changed so dramatically over the last few years. Encrypting sensitive data is now a requirement for businesses who need to meet data privacy compliance regulations. And it is also achieving visibility at the executive level as a part of a GRC (governance, risk, compliance) plan. The landscape really has changed a great deal, and I think organizations are stepping up and doing a better job.

We are also seeing databases implementing encryption directly into the database engine. MongoDB Enterprise is an excellent example of this by incorporating AES encryption into the WiredTiger storage engine and providing KMIP for good key management. Microsoft SQL Server Enterprise has also built the encryption engine right into the database with transparent data encryption (TDE) and provides extensible key management (EKM) for vendors like us to plug in to and allow users to properly manage encryption keys.

Let’s spend a minute on compliance. What regulations require key management?

It is true that there are a lot of compliance regulations and most of us fall under multiple. Certainly, GDPR and the new Australian law have come on board and are affecting the way companies all over the world, even outside of the EU, who are now taking steps to protect that data. Almost all regulations require the protection of data, which encryption can help with. Regarding encryption, from a best practices point of view, if you don’t have proper key management, you don’t have encryption.

PCI DSS and the payment industry has now been with us for a decade and that also drives the protection of a lot of sensitive data. While PCI DSS is a private regulation between merchants who accept credit cards and the industry, it is a strong regulation with strong penalties. Because of the longevity and focus on security, PCI DSS often informs other regulations and corporate governance boards.

While we live in a very complex set of compliance requirements, there are fortunately some similarities. When we set out to protect sensitive data, and do it right, which I am sure you know by now means based on standards and security best practices, we find business meeting a broad set of regulations.

As I mentioned earlier, key management is considered essential infrastructure. Back when Townsend Security first entered the market, there were just a few options for the enterprise to choose from. Today, options have dramatically increased. With that said, there are a few ways to distinguish a key manager that meets industry standards and one that will leave you with a breach notification on your hands.

There are a few key quality indicators when assessing key management systems. First off, a key management server must be validated to FIPS 140-2. That is the US and Canadian standard for cryptographic modules and it is just a fundamental indicator of the seriousness of a company and its ability to build good key management solutions. Unfortunately, there are claims of FIPS 140-2 compliance by vendors who have systems which have actually not completed a validation. It’s always a good idea to ask your key management vendor what their certificate number is so that you can look it up yourself on the NIST website. Additionally, key management vendors should be able meet KMIP requirements by demonstrating interoperability. These sorts of validations and certifications show that a key management vendor is stepping up to the technical requirements that customers expect.

Finally, there are individual specific validations to regulations. Our key manager, for example, has been through a PCI DSS validation with a QSA auditor and with VMware. You want to avoid solutions that are simply “key storage” which haven’t been validated and there has not been external review and validation of the cryptographic approach. It is incredibly easy to get encryption and key management wrong from a technical point of view.

Third-party attestations and validations are critically important. You are putting your digital assets, not to mention your company’s reputation, at risk when you use a poor quality key management server (KMS). Fortunately, today there are very affordable solutions available everywhere from a traditional hardware security module, to VMware, to the cloud. 10 years ago, key management servers where terribly expensive.

Yeah, I have been seeing key managers hitting the market that range from open source to cloud service provider (CSP) provided. Some key managers will even claim FIPS compliance when a simple check with NIST shows them nowhere to be found. You also get providers who claim FIPS compliance because they are using a module that has been through a validation, which as I am sure you can tell us more about, doesn’t make your solution compliant.

That’s right. Unfortunately, there is still a fair amount of snake oil in the industry. People claim FIPS 140-2 compliance, but haven’t fully been through the process. As you mentioned, claiming one component as being FIPS 140-2 compliant does not make your entire solution compliant. I like to say, “I drive a Toyota. If I put a Jaguar hood ornament on my car, it doesn't turn my car into a Jaguar. It is still a Toyota!” There is a lot of language used by some solution providers that just isn’t accurate and should be verified before choosing a solution.

You also talked about governance. Let’s take a look at the recent Marriott/Starwood breach. There was an interesting statement where they said “we can’t be absolutely sure that the encryption keys weren’t lost.” If you are using enterprise level key management systems, you have monitoring and audit trails that will show if keys have been retrieved. By them not knowing, it is easy to surmise that they weren’t adequately protecting our information. This was a failure of governance in regards to critical security infrastructure. We might not be talking about the breach right now had they had industry standard key management in place.

Changing gears a little, we have in the past seen cloud vendors offer key management systems in their shared environments. Multi tenant key management presents a concern for enterprises. Just last week I had a conversation with a security professional from a global enterprise who said, “for us, trust is paramount to our brand. We will never allow encryption keys to be stored and accessed by a cloud vendor.” I thought it was a very interesting statement and completely lines up with other enterprises that I speak with.

There is also a bit of confusion on the term KMS. For example, VMware defines KMS as a Key Management Server. Amazon, on the other hand, defines their KMS as a “Key Management Service.”

Well, there is a bit of confusion on this isn’t there? A true enterprise key management system (KMS), is responsible for the entire lifecycle of an encryption key - from generation where admins create provably strong keys, to storage, to provisioning applications and users who need the keys, all the way through to the archival and destruction of the keys. Key management systems manage all of those phases according to industry standards.  

A key management service can better be defined as storage service and offers a mix of capabilities. Some are services that are shared, multi-tenant environments. CSP key managers often work this way (think AWS KMS, Azure Key Vault). Many services don’t give you full access or control over a key’s complete lifecycle. Some services allow you to bring your own key, but then bring that into their own infrastructure, where you then share administrative access and control.

It is very important to know that if you are truly trying to do security the right way, you need a key management system that is built to industry standards and validated to industry standards. One last point, it is possible to do key management correctly in the cloud. There are third-party key management offerings that can be found in the marketplace that meet all the standards that we have been talking about, and are dedicated to you and only you. Our Alliance Key Manager is one, for example.

A feature that we are very proud of with our key manager is that it runs the same software in the Cloud, VMware, or as a traditional hardware security module (HSM). We have customers setting up hybrid cloud/on-prem deployments or even cross-cloud. The way we license our key managers works really well for the modern enterprise which needs a predictable, low TCO. With Alliance Key Manager, there are never added fees for additional databases or applications.

To hear this conversation in its entirety, download our podcast Don’t Forget About FIPS 140-2 and Other Fundamental Key Management Features and hear Patrick Townsend further what enterprises should look for in a key manager, the importance of standards, and other fundamental features of an enterprise grade key manager.

Townsend Security, today announced that its Alliance Key Manager for VMware has achieved VMware Ready status. This designation indicates that after a detailed validation process Alliance Key Manager for VMware has achieved VMware’s highest level of endorsement and is supported on VMware ESXi  (all supported versions, vSphere 6.5 and later, and vSAN 6.6 and later) for production environments.

“We are pleased that Townsend Security and Alliance Key Manager for VMware qualifies for the VMware Ready logo, signifying to customers that it has met specific VMware interoperability standards and works effectively with VMware cloud infrastructure. This signifies to customers that Alliance Key Manager for VMware can be deployed in production environments with confidence and can speed time to value within customer environments,” said Kristen Edwards, director, Technology Alliance Partner Program, VMware.

By using Alliance Key Manager for VMware with VMware ESXi (all supported versions, vSphere 6.5 and later, and vSAN 6.6 and later) organizations can centrally manage their encryption keys with an affordable FIPS 140-2 compliant encryption key manager. Further, they can use native vSphere and vSAN encryption to protect VMware images and digital assets at no additional cost. VMware customers can deploy multiple, redundant key servers as a part of the KMS Cluster configuration for maximum resilience and high availability.

“By achieving VMware Ready status with Alliance Key Manager for VMware, Townsend Security has been able to work with VMware to bring affordable encryption key management to VMware customers and the many databases and applications they run in VMware,” said Patrick Townsend, CEO of Townsend Security. “Meeting data security compliance in VMware vSphere is now easier than ever.”

The VMware Ready program is a co-branding benefit of the Technology Alliance Partner (TAP) program that makes it easy for customers to identify partner products certified to work with VMware cloud infrastructure. Customers can use these products and solutions to lower project risks and realize cost savings over custom built solutions. With thousands of members worldwide, the VMware TAP program includes best-of-breed technology partners with the shared commitment to bring the best expertise and business solution for each unique customer need.

Townsend Security and Alliance Key Manager for VMware can be found within the online VMware Solution Exchange (VSX) at https://tsec.io/VMwareReadyPR. The VMware Solution Exchange is an online marketplace where VMware partners and developers can publish rich marketing content and downloadable software for our customers.

VMware, VSXi, vSphere, vSAN and VMware Ready are registered trademarks or trademarks of VMware, Inc. in the United States and other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

This last November (Nov. 6-9, 2018) we had a chance to participate in the 20th annual PASS Summit in Seattle as an exhibitor. It was a great time as SQL Server professionals from around the world attended. We had an opportunity to ask them about their company's encryption and key management practices. Below are the results as well as some expert weigh-in on the findings. Enjoy!

A special thanks to our contributors for their expertise and guidance. You all are clear-minded professionals that have a lot to offer those looking to better secure their data:

-Sebastian Meine, Ph.D., sqlity.net
-Steve Brown, Rutter Networking Technologies
-Tim Roncevich, CyberGuard Compliance
-Sharon Kleinerman, Townsend Security
-Patrick Townsend, Townsend Security

If you are looking to protect your encryption keys for your sensitive data in SQL Server, you need a FIPS 140-2 compliant centralized key manager that:

• Never charges you additional fees for connecting a new end-point.
• Never limits the number of end-points based on the model of the KMS.
• Never limits the number of encryption keys generated or stored.
• Never forces you to pay extra fees for software patches.
• Never forces you to pay extra fees for routine software upgrades.
• Always gives you unmatched customer service.
• Always protects your keys, 24/7.

You need Alliance Key Manager for SQL Server.

Both encryption and proper key management are a crucial part of defending your sensitive data. Consider the massive Marriott/Starwood Resorts data breach that was announced at the end of November, 2018. Attackers were able to first gain access to their systems in 2014, almost a full four years before they were detected in September of 2018.

To date, it is estimated that over 500 million records were compromised. 327 million of which contain personally identifiable information (PII) like names, physical addresses, birthdays, etc. In a statement put out by their parent company, Marriott, they noted that, “for some, the information also includes payment card numbers and payment card expiration dates, but the payment card numbers were encrypted using Advanced Encryption Standard encryption (AES-128). There are two components needed to decrypt the payment card numbers, and at this point, Marriott has not been able to rule out the possibility that both were taken.” (italics and bolding mine)

So, while the information was encrypted, the company cannot rule out that “two components needed to decrypt” were not taken. One of these needed components would be the encryption keys. No one knows for sure why they they cannot rule this out, but if the encryption keys were properly managed, the potential for them being stolen along with the data would be highly unlikely. Encryption is crucial to securing sensitive information. Just as important: properly managing the encryption keys.

But encryption, and by extension key management, have a bad reputation for being difficult to deploy in the enterprise. Encryption projects used to be costly as well as time and personnel intensive. While many databases came with encryption as part of their native libraries, it was a largely manual process not for the faint-of-heart—and add on top of that finding ways to properly manage the encryption keys; most developers would put it off for another day.

The good news, much has changed in the last 10+ years. Deploying encryption and key management can take a fraction of the time it normally took. But that doesn’t mean to you don’t have to work smart. Coming at an encryption project in a haphazard way could cost you time, money, and sensitive data not getting the proper protection it needs.

Follow along for three tips you need to get your encryption project off on the right direction.

Create a Unified Policy for Encrypting Data-at-Rest

The first step in any project is to get agreement between stakeholders as to what data-at-rest should be encrypted. Some of the obvious contenders would be any cardholder data (CHD) you have in you environment, personal health information (PHI) or personally identifiable information (PII) that needs to be kept safe. Many time this information falls under a compliance regulation like PCI DSS or HIPAA. The choice to encrypt this data is clear.

But less clear is data, that if exposed, would leave your company exposed to brand damage, lawsuits, or loss of competitive advantage. Whether it's the plans for a new product, proprietary schematics for an existing product, or information that exposes your business processes, business has a lot of information they want kept secret.

In fact, Deloitte estimates that Intellectual Property data can constitute more than 80 percent of a enterprise company’s value. Below is a short (and certainly not exhaustive) list of items that your company should be encrypting:

• Product/Solution Documents: If your product or service relies on proprietary information to give you a competitive advantage in the marketplace, you need to encrypt anything that would give your competition a window into how your products or solutions work.
• Research and Development (R&D) Data: In the same vein, any R&D you are conducting is your advantage in tomorrow's competitive landscape. Don't let it be stolen from you because you did not properly secure it.
• Financial Reports: If you don’t want your competitors spying on your financial information, encrypt it.
• Legal Documentation: There is a lot of documentation, that if made public, could tarnish a company's reputation. Harassment settlements, sexual misconduct accusations, financial misdealings, even benign partner agreements—all these need to be kept private and out of the public's eye.

Just as crucial as protecting PII and IP data is protecting any client data you are responsible for. Making sure that client data is safe is of utmost importance. Just one breach could cause major reputational damage and see a loss of current and future contract revenue.

Use Existing Encryption

Most databases now (like SQL Server and MongoDB), as part of their Enterprise editions, come with Transparent Data Encryption (TDE). This encryption encrypts the entire database while at-rest and normally uses either AES or 3DES encryption. This typically takes less time to deploy than column level encryption as there is less configuration to do.

If upgrading to an Enterprise edition is not in the budget, some third party encryption products may be an option. NetLib, for example, file level encryption for SQL Server. It supports all versions of SQL Server from 2000 to 2017 and can save you the upgrade costs from going from Standard or Dev editions to Enterprise.

Learn More: SQL Server Encryption Survey 2018

Use an Established Third Party Vendor for Encryption Key Management

The most important part of a data encryption strategy is the protection of the

encryption keys. Because encryption key management is crucial to data protection, the National Institute of Standards and Technology (NIST) provides guidelines on best practices for key management. NIST Special Publication SP-800-57 provides recommendations for encryption key management.

What this means to you is that managing your encryption keys is not a simple process. Why? Because you must create a defense-in-depth system for your encryption keys. Remember, hackers don’t break your encryption, they steal your keys. And the only way you keep your keys safe is to layer them in protection so that only authenticated and authorized personnel have access to them.

The good news is, there are vendors at the ready who already comply with the highest standards and have both virtual and hardware options. The real trick also is finding one that is both dedicated and affordable. Of course, there are some key manager options provided by public clouds, but they are not a dedicated key manager (i.e. you own the key manager) but rather your keys are housed are a multi-tenant environment as well as the public cloud provider having administrative access. This can get tricky as a public key manager may be compelled by the government to hand over your keys, without your knowledge, and unlock your data. Worse yet, if your data is stored in the same public cloud, the cloud provider would have access both to your keys and data. Unsafe! To learn more, check out our recent blog post on PCI SSC guidance around this issue.

Here at Townsend Security we do things differently. Not only is our Alliance Key Manager a dedicated key manager, you can purchase our key management platform of your choice and use it the way you want. It is that simple. From that point on:

• We never charge you fees for connecting a new end-point.
• We never limit the number of end-points based on the model of the KMS.
• We never limit the number of encryption keys generated or stored.
• We never force you to pay extra fees for software patches.
• We never force you to pay extra fees for routine software upgrades.

We do things a little differently here at Townsend Security and we think that makes encryption just a little easier.

In April of this year the Payment Card Industry Security Standards Council (PCI SSC) released a new document on cloud guidance called “Information Supplement: PCI SSC Cloud Computing Guidelines”. It was an update of the first version of the guidance issued in 2013. While this is not a set of mandatory rules, it is a core guidance document and recommendations in PCI guidance documents often end up as requirements under the PCI Data Security Standard (PCI-DSS) and PCI Payment Application Data Security Standard (PCI PA-DSS). So it is worth understanding the guidance and it is wise to align your IT and business processes with the guidance. It is better to get things right at the beginning rather than have to rip-and-tear to fix things later.

There is another reason to pay attention to the PCI cloud guidance: The PCI standards often set the expectations for security best practices in other regulations, and reflect evolving industry standards such as those developed by the National Institute of Standards and Technology (NIST). Even if you are not processing credit card payments, you should be paying attention to this guidance.

When it comes to encryption key management in the cloud, there is a bombshell in this document (essentially describing services such as Amazon KMS and Microsoft Key Vault). What does the new PCI guidance say about encryption and key management? Let’s parse it out and see where it goes.

Take a look at appendix E.10 “Data Encryption and Cryptographic Key Management”.

Appendix E.10 starts by describing the shared, multi-tenant architecture of cloud services (pretty much states the obvious). And then makes this statement:

“If a Customer shares encryption keys with the Provider, or engages the Provider as a key custodian, details of Provider access permissions and processes will also need to be reviewed and verified.

This consideration is particularly critical if cryptographic keys are stored or hosted by a third-party Provider that also hosts the encrypted data. If Provider personnel have access to a Customer’s keys and the Customer’s encrypted data, the Customer may have unintentionally granted the Provider ability to decrypt its sensitive data.”

In fact, all cloud service providers such as Amazon (AWS), Microsoft (Azure), and Google (GCP) have access to both your data and your encryption keys if you are using their key management services. This includes AWS Key Management Service (KMS), Azure’s Key Vault, and Google’s Customer Customer-Managed Encryption Keys (CMEK). Perhaps unknowingly, you HAVE granted your cloud provider the ability to decrypt your sensitive data.

Here is how the PCI SSC sees the risk:

“Any data that is decrypted in the cloud may be inadvertently captured in clear text in process memory or VMs via cloud maintenance functions (such as snapshots, backups, monitoring tools, etc.). To avoid this risk, Customers may choose to keep all encryption/decryption operations and key management on their own premises, and use a public cloud only for storage of the encrypted data.

Applicable controls must be applied to the encryption, decryption, and key management processes to ensure that data can only be retrieved (decrypted) by those who are authorized with a defined business need.”

Wow, that’s a pretty strong statement about not allowing your cloud provider have access to your encryption keys and sensitive data. It is hard to imagine a scenario where the cloud service provider has a “defined business need” to access your sensitive data.

Pointing back to the perceived risks of the cloud provider, here is the key point in the PCI cloud guidance:

Because compromise of a Provider could result in unauthorized access to multiple data stores, it is recommended that cryptographic keys used to encrypt/decrypt sensitive data be stored and managed independently from the cloud service where the data is located.

Wow, there you have it. Don’t use the cloud service providers key management service because there is too much risk. This recommendation affects a very large number of users in the cloud.

Where do we go from here?

Fortunately, there are solutions available now to solve this problem (we have one). Let’s outline some options. There will be pluses and minuses for each one. But the good news is that there are multiple solutions to this issue.

1. Deploy your own dedicated key manager in your own on-premise data center

It is fairly easy to deploy an encryption key manager in your own data center and enable its use by cloud applications. Most enterprise key managers use a secure TLS-encrypted session to interoperate with the key manager. Once you enable an outbound TCP port to your key manager, you can easily use the on-premise key manager. Note that this could be a hardware security module (HSM) or a virtual key management appliance running in VMware.

Remember that you probably do not have to retrieve the encryption key from the key server to your cloud application - most key managers support on-board encryption and decryption services. This alleviates the risk of an exposure of the encryption key in cloud memory. Performance will be the important factor to weigh in this regard. While the key manager may be quite efficient in the encryption or decryption operation, the communications lag times may mitigate against this approach.

2. Deploy your own dedicated key manager in a hosted platform

If your organization does not have on-premise infrastructure for a key manager, don’t despair. It is really straightforward to deploy a key manager in a hosted environment. A hosting provider can provide a home for a hardware security module, or for a software appliance. Establishing the firewall rules may take a bit more work, but this is an approach that has worked well for our customers.

3. Deploy your own dedicated key manager in a different cloud

One creative option to separate the encryption keys from the protected data is to deploy the key manager in a different cloud platform. You could, for example, deploy your application data in Amazon Web Services, and deploy the key manager in Microsoft Azure. This helps mitigate the risk of one cloud service provider having access to both your encryption keys and your protected data - one of the key concerns expressed in the PCI guidance.

Note that this solution will probably require that you work with the firewall rules in both cloud provider platforms. The good news is that this is not complicated - we have customers doing this today.

4. Deploy your own dedicated key manager in a separate cloud instance

Lastly, it is possible to deploy a dedicated key management solution in the same cloud as your protected data, but completely avoid the use of the cloud provider’s encryption key management infrastructure. The key server runs in its own virtual machine or EC2 instance and encryption key management is exclusively dedicated to you. If you take this approach, but sure that your key management vendor is not using the cloud providers encryption key management infrastructure! Encryption keys and key management should only be accessible to you and not to your vendor or cloud provider.

I know that some cloud customers are reluctant to take this approach due to concerns about the ability of the cloud provider to access all of the customer applications and data on their platform, including a key management system running in the cloud. Personally I think the risk is minimal, but if you have that concern see the previous alternatives.

In summary, it would be prudent to avoid the use of cloud service provider key management services such as AWS KMS, Azure Key Vault, and Google Customer-Managed Encryption Keys (CMEK). These services will put you at odds with the PCI cloud security recommendations, and likely put you in variance with future regulations. Not a good place to be.

My advice? Get encryption key management right from the beginning. If you are using a cloud provider’s KMS, start your migration now. You have readily-available choices that are affordable. Get started now!

Our Alliance Key Manager is validated to PCI-DSS and available in cloud, VMware, and HSM platforms. You can do this! Get started here.

Patrick

Like most cloud service providers, Google provides options for customers to encrypt data in the Google Cloud Platform (GCP) for Google application and storage services.  Of course, core to any encryption strategy is how the encryption keys are managed. Properly managing encryption keys is central to overall encryption security. In this blog let’s look at your options for managing encryption keys in GCE and the implications for security, key custody (ownership), and business flexibility. We will also look at some alternatives to Google-provided encryption and key management.

Google has three models for encryption key management:

• Google managed encryption and encryption keys (you have no access to the keys)

• Customer-Managed Encryption Keys, or CMEK (you and Google share key management and access)

• Customer-Supplied Encryption Keys, or CSEK (you supply the key, Google does not store or manage)

It is important to understand key management in each of these areas.

Google Managed Encryption and Encryption Keys

Google automatically encrypts your data in storage for you. This is the default implementation of storage encryption. With this option Google creates and manages the encryption keys and you do not have access to the encryption keys nor to any key management functions. This basic level of encryption support ensures that your data is encrypted at rest, encrypted in Google-managed backups, and encrypted as it is replicated through the Google cloud infrastructure.

It is important to note that Google has full access to the encryption keys that protect your data and will not provide them to you. However, Google may provide these encryption keys and protected data to law enforcement or intelligence agencies as dictated by law in your country, and it may not notify you if encryption keys are disclosed under these circumstances.

Customer-Managed Encryption Keys (CMEK)

With the CMEK approach you and Google share in the administration and management of encryption keys. You have the ability to manage the creation and use of the encryption keys for protecting data in storage, and you have the ability to revoke or delete an encryption key. You do not have the ability to export an encryption key, so Google maintains exclusive custody of your encryption keys.

It is important to note that Google has full access to the encryption keys that protect your data and will not provide them to you. Google may provide these encryption keys and your protected data to law enforcement or intelligence agencies as dictated by law in your country, and it may not notify you if encryption keys are disclosed under these circumstances.

Customer-Supplied Encryption Keys (CSEK)

With the CSEK approach you provide the encryption keys to Google and Google does not create, store or manage the keys. You must generate the keys and manage them yourself outside of the Google Cloud Platform. This is normally accomplished using an Enterprise Key Management System. When you create a new virtual image you will have the option of specifying the encryption key in raw, Base64 encoded format, or you can specify the key using RSA key wrapping. Google does not store this encryption key. This means that you must supply the encryption key each time you start the GCP service. To some extent this can be automated using command line tools.

With CSEK Google does not store or manage your encryption keys in any way. Google cannot provide the key to law enforcement or intelligence agencies as it has no possession of the key.

Google Key Management Summary

In many ways the Google Cloud Key Management Service (Cloud KMS) is primitive compared to other cloud service provide key management services, and especially when compared to Enterprise Key Manager Systems which provide dedicated, non-shared, control over the entire encryption key lifecycle.  The cloud services that are protected by Google Cloud KMS are also limited to the storage of data. Additionally Google Cloud KMS does not support open standards for key management such as the OASIS Key Management Interoperability Protocol (KMIP). If data protection, encryption key custody, and exclusive access to encryption keys are important to you, you may wish to explore other cloud service providers and independent key management vendors (Townsend Security can help).

Encryption and Key Management with MongoDB in Google Cloud Platform

One notable exception to the limitations of Google Cloud Key Management is the implementation of MongoDB Enterprise Advanced on the Google Cloud Platform. MongoDB Enterprise Advanced supports the KMIP interface to key management, so you have many choices for the deployment of encryption key management outside of GCP. These choices include the ability to deploy key management:

• As hardware security module (HSM) in your own data center.
• As a hardware security module in a hosted data center.
• As a VMware virtual machine in your own data center or a hosting provider that supports VMware and vCloud.
• As a dedicated cloud instance in another cloud platform.

At the present time in July of 2018, MongoDB Atlas does not support the KMIP interface, so Atlas users will not have the ability to use their own dedicated key management system.

Resources:

Google Customer Managed Encryption Keys (CMEK):

https://cloud.google.com/storage/docs/encryption/customer-managed-keys

Google Customer Supplied Encryption Keys (CSEK):

https://cloud.google.com/storage/docs/encryption/customer-supplied-keys