Townsend Security Data Privacy Blog

As blockchain technologies make their way towards general acceptance in private and public sector IT systems, the critical issues of governance, risk management and compliance come into play - and blockchain teams are maturing to address these areas. One important gap to fill involves the proper protection of sensitive data in a blockchain deployment. It seems odd to discuss data protection in the context of blockchain. Isn’t blockchain based on cryptography? Yes, it is, but there remains a gap in the area of data protection. Let’s delve into this in more detail.

Blockchain’s innovative way of linking transactions and guaranteeing their immutability in a distributed ledger is based on well known and respected cryptographic algorithms and processes. The ability to extend this level of assurance across a large number of widely distributed nodes is clearly an amazing extension of modern computing. While there have been security lapses in public blockchain implementations, these have generally been related to improperly securing credentials and mistakes in implementing chaincode. Blockchain methodologies are standing up well to external attacks.

One important aspect of blockchain is its transparency. That is, everyone has perfect visibility into the transactions on a ledger and their current validity. This transparency is a core feature of blockchain - and that leads to a problem:

Some data that we want to put on the blockchain is sensitive, and we may not want to expose it to others.

There are lots of reasons why we might not want some information on the blockchain ledger to be transparent:

• An organization’s reputation suffers when they lose or expose sensitive information. This is true for both public and private organizations and a significant loss of reputation is difficult to mitigate.
• Even little bits of data in blockchain transactions needs to be protected. When sensitive data in a blockchain ledger are aggregated, it can indicate the direction of a business’s activity and leak important information about strategic developments to it competitors.
• Compliance regulations prevent storing sensitive personal information in the clear. The PCI Data Security Standard mandates that credit card (Primary Account Numbers) be encrypted. The New York Department of Financial Services (23 NYCRR 500) requires the encryption of certain sensitive information. The EU General Data Protection Regulation (GDPR) mandates the protection of sensitive information of “Data Subjects”. here are other regulations that require or recommend protection of sensitive data.
• Digital assets that represent intellectual property need to be protected from cybercriminals and state actors. The loss of key intellectual property can be devastating to a startup or mature enterprise.

Therefore, it is critical for organizations to design proper data privacy into blockchain projects from the very beginning. It is painful and potentially impossible to fix data privacy gaffs after the fact.

Some blockchain advocates suggest that the solution to this conundrum is to not place sensitive information on the blockchain at all. But this is an impossible goal. Data on a blockchain may not specifically identify an individual, but may contain enough information that it can be combined with previously leaked information to form a full picture of an individual. Remember that hackers are really good at data aggregation. Losing a little sensitive information can lead to an embarrassing loss of a lot of information.

Other blockchain advocates suggest that the answer to this problem is to store sensitive data off of the blockchain altogether. But does this really solve any problem? This approach loses the many advantages of blockchain technology, and doesn’t do anything to solve the data protection puzzle. “Out of sight, out of mind” is not a solution to any problem.

Some blockchain implementations attempt to achieve privacy through “add on” features. Hyperledger channels and collections are two examples of this. These facilities use access controls to attempt to achieve this. As good as these facilities are, access controls will not address the data protection requirements of compliance regulations, nor provide other protections that encryption provides.

For all of the reasons we encrypt sensitive data in traditional databases, we need to encrypt sensitive data on a blockchain. This doesn’t mean that we have to encrypt everything that we put on the blockchain ledger, but it means we have to have the same intelligence in regard to sensitive data on blockchains that we have in the most secure systems today.

Fortunately, we can accomplish data protection on blockchains and maintain their usefulness. In fact, not only CAN we accomplish this, we MUST accomplish this in order to preserve the usefulness of blockchain technology.

If we are going to encrypt data that we put on a blockchain, we have to address a few requirements that are specific to blockchains:

• We have to use industry standard encryption algorithms, such as AES, to meet compliance regulations.
• We have to manage encryption keys using industry standards and best practices. This means storing encryption keys away from the blockchain ledger and doing so in a provably standard and secure way.
• We have to make encryption keys available to the users and smart contracts that need them. This is a challenge in a distributed blockchain environment.
• We must authenticate user’s authority to use encryption keys.
• We must have a mechanism for restricting access to encryption keys, and for granting and revoking access to those keys.
• We know how to accomplish these tasks in a traditional, centralized IT system. Years of work have produced standardized approaches to encryption. But blockchain presents real challenges to meeting these challenges.

Fortunately, innovation in the area of protecting data on a blockchain ledger is advancing.

At BlockNKey we built a key orchestration system architected from the ground up for distributed ledger technology. NIST compliant encryption and key management, a key vault, and key access control are built into each registered blockchain node. Cryptographic keys grant permission to whomever is permitted access to the data, how it’s accessed and when it’s accessible. This enables multi-party access to the appropriate data in real time through verified and validated access points. BlockNKey is compatible with public and private blockchains while enabling proper data security with easy to use REST APIs. It will even help you if you are storing sensitive data “off chain”.

Townsend Security has partnered with BlockNKey to bring an encryption and key management solution to blockchain users. More information here.

Many VMware customers know that they can encrypt their virtual machines that are managed with vSphere and other VMware tools. VMware vSAN encryption can also provide important protections for data-at-rest in vSAN virtual disks. I wanted to share some thoughts I’ve received from our VMware customers and partners on some of the benefits of using vSAN storage with encryption enabled.

A Simple Way to Encrypt

When you have a large database, it can be inefficient to store the data in a directory or folder directly in your virtual machine. vSAN can be much easier to manage from an administrative and recovery point of view and your VMware applications can easily connect to the vSAN volume. vSAN is configured using the VMware tools you already know how to use and managing vSAN storage is easy.

Did you know that you can enable vSAN encryption to protect that database with sensitive data? You can enable vSAN encryption on existing virtual disks or on new virtual disks that you create. The process is simple and does not require any downtime for your application - and vSAN encryption enables the use of a KMIP compatible key manager like our Alliance Key Manager so that you stay lined up with industry standards and security best practices. It is an easy way to improve your overall security posture.

A Simple Way to Meet Compliance

Many of our VMware customers are struggling to implement encryption on their databases to meet compliance regulations and to protect the organization’s digital assets. Although encryption and key management have become much easier over the years, it can still seem like a daunting task. VMware vSAN encryption to the rescue! It is easy to implement with the tools you already have, and you can deploy an affordable key management solution such as our Alliance Key Manager to fully meet compliance requirements and security best practices. You configure key management directly through the KMS Cluster facility in vSphere, and then activate vSAN encryption. Alliance Key Manager does not impose any limits on the number of virtual disks you protect, nor on the number of nodes that connect to the key manager.

A Simple Way to Save Money

Some databases, such as Oracle and Microsoft SQL Server, require expensive license upgrades to enable encryption capabilities. This cost can be out of reach for many small to medium size organizations. Using vSAN encryption is an affordable way to achieve a better security posture using the tools and the IT professionals you already have.

You might be wondering if VMware supports the deployment of these databases on vSAN volumes. The answer is absolutely YES! You will find substantial documentation from VMware on doing exactly this. The documentation includes reference architectures and analysis of performance impacts. You can confidently move forward with vSAN encryption knowing that VMware has invested time and effort to make sure you are successful.

Lastly, we know that some VMware users have deployed the free version of vSphere. There are some costs associated with upgrading to the paid tier of vSphere in order to get the ability to encrypt VMs and vSAN. If this is where you are today, talk to us about how we can help with the uplift to the next level of vSphere capability.

Resources:
vSAN Documentation
Oracle Database on VMware vSAN Solution Overview
Architecting Microsoft SQL Server on VMware vSphere
Pointers to our AKM for vSphere/vSAN Solution Brief 

One of the easiest ways to implement encryption controls in your VMware infrastructure is to activate vSphere and vSAN encryption. With vSphere VM encryption you can insure that all VM images are encrypted at rest, and with vSAN encryption you can set up virtual disks that are fully encrypted protecting any files that you place there. vSphere encryption was implemented in version 6.5, and vSAN encryption was implemented in version 6.6. All subsequent versions of vSphere and vSAN include these capabilities. (Note that you must be on the Enterprise or Platinum edition).

In both vSphere and vSAN the key manager is integrated using the open standard Key Management Interoperability Protocol, or KMIP. This means that any key management solution that supports the necessary KMIP interface can work as a vSphere or vSAN key manager. Our Alliance Key Manager solution implements this support, and is already in use by our VMware customers. 

The most common question we get about these new encryption features is: How do I manage failover for the key managers?

This is a great question as VMware is a part of your critical infrastructure, and key management has to work with your high availability strategy. There are two parts to this question and lets dig into both of them:

Defining Key Managers to vSphere KMS Cluster

Key managers are defined to vSphere using the option to configure the KMS Cluster. A KMS Cluster configuration allows you to define more than one key manager. So you have a readily available path for failover. The first key manager configuration is the primary key manager, and all subsequent key managers in the KMS Cluster are failover key managers. vSphere will always use the first key manager you define and treat it as the primary. 

In the event vSphere cannot connect to the primary key manager, it will try to connect to the second key manager in the KMS Cluster configuration. If that one fails it will try the third one, and so forth. The failover order is the order in which you define key managers in the KMS Cluster, so you should keep that in mind as you define the key managers.

While vSphere allows you to create multiple KMS Cluster definitions, very few VMware customers need multiple definitions. Just put your key manager definitions in a single KMS Cluster and you are set to go. 

If you have failover clusters for VMware, be sure to define the KMS Cluster for the failover environment, too!

Implementing Key Mirroring in Alliance Key Manager

Now that you have failover key managers defined to the KMS Cluster, you need to activate key mirroring between the primary key manager and each failover key manager. This is really easy to do, and you don’t need any third party products to implement key mirroring with Alliance Key Manager. Real time, active-active key mirroring is built right into the solution. You can SSH into the key manager, provide credentials, and then take the menu option to set up the primary or secondary key server. Answer a few questions and you will have key mirroring enabled between two or more Alliance Key Manager servers.

Our Alliance Key Manager solution implements full support for vSphere and vSAN encryption key management and has everything you need to get started. Adding encryption to your VMware environment is easy. VMware did a great job with this implementation of key management support and you can easily realize the benefits of protecting VMware infrastructure.

Alliance Key Manager documentation for vSphere can be found here.

You can download Alliance Key Manager and get started right away. Here is where to go to start the process.

Townsend Security will help you get started with vSphere and vSAN encryption. There is no charge for the evaluation or evaluation licenses and you will get access to the Townsend Security support team to ensure you have a successful project.

Patrick

In our practice here at Townsend Security we engage with a lot of startups and mature ISVs who are trying to grow their business and customer base, leverage their technologies into new opportunities, and grow or migrate to the cloud. We know how difficult it is to start and grow a company, and what a wide set of business challenges have to be overcome. Our hats are off to every entrepreneur who has created a successful company, and every ISV who has kept it going!

I want to share a few thoughts on some pitfalls that can damage your ability to grow your company with a focus on the encryption of sensitive data. Too many promising companies flounder because of poor security implementations, and failing to get encryption right can lead to lost opportunities - maybe even the loss of that breakout sale you need to land a global company. Some early thought and planning about data security can help you weather your migration up the food chain and avoid such losses.

Number 1: Failure to encrypt sensitive data

The single biggest failure of data security is not doing it at all. Even in this age of massive public data breaches, and the damage that they do to companies of all sizes, most startups and ISVs are not implementing encryption of sensitive data. When product managers and developers work on their next big idea, they focus on exciting features in their product and often ignore the work it takes to implement encryption. They instead rely on access control lists and other mechanisms to protect data. These are, of course, important things to do. But the failure to encrypt sensitive data leaves a big hole in your security strategy.

What can go wrong if you haven’t implemented encryption? LOTS !!!

• The publicity around a data breach can tarnish your reputation and kill opportunities.
• The lack of encryption may cause compliance regulation failures making it impossible to enter new markets.
• You may not be able to pass a security review of your software by that large global Enterprise.
• You may not be able to enter government channels where encryption is a mandate.
• If your customer experiences a data breach you may encounter substantial litigation costs that damage your financial resources and delay critical development.
• You may fail to secure that next round of funding when an investor discovers the security gaps in your product.

When these kinds of events damage your ability to grow your company, it can be hard to mitigate them in a timely fashion. And you often won’t know about these dangers until you get fairly far down the road with your business plan.

Number 2: Failure to get key management right

For startups and ISVs who DO understand the need for encryption of sensitive data, the next biggest pitfall is the failure to protect encryption keys properly. Almost every database that supports encryption also supports the ability to protect the database encryption keys with a key manager. But that doesn’t mean that good key management is the default! In most cases the default database key management option is to store the encryption keys on the same server as the sensitive data. Sometimes the database will even store the encryption key locally and in the clear! So getting encryption key management right is critical to your security strategy. It won’t help to have encryption of your data enabled, and then have a cybercriminal steal your data along with the encryption key.

Related to key management here are some things to look for when you consider databases for your application:

• Does your database have built-in encryption? Relying on third-party encryption solutions at the file/folder level will certainly cause deployment and scalability problems.
• Does your database support integration with third-party key managers? If there is no easy way to integrate proper key management into the database, this will also cause deployment and technology delays.
• Does your database support open standards for key management? For example, the Key Management Interoperability Protocol (KMIP) defines how applications like databases can easily integrate a key manager.
• Does your database support key management failover? Remember that protecting encryption keys with a key manager also brings along the question of high availability and failover.

HINT:

If you are a startup be sure to choose a database that supports built-in encryption and proper key management. You have lots of good choices in both commercial and open source solutions. So go with a database with native, built-in encryption and key management!

Number 3: Failure to get FIPS 140-2 right

There are important standards and certifications for key management solutions. The most important of these is the National Institute of Standards and Technology’s (NIST) FIPS 140-2 standard. In addition to being a published standard, there is also a validation process for key management systems. The standard, and the validation to that standard, are critically important to your data security strategy. All professional key management solutions have been validated to the FIPS 140-2 standard and you should be sure to deploy a validated key management solution. This will help you avoid failing a security audit by that important new customer!

In addition to ensuring that your key manager is validated to FIPS 140-2, be sure that the entire key management solution is validated. There are many cases where the encryption library alone is validated to FIPS 140-2, but the key management application is not. It is good to have validated encryption, but that is just the start! Encryption key management has its own validation points and you will need both.

Snake Oil Alert !!!

Unfortunately, there are some key management solutions that make unwarranted claims about FIPS 140-2 compliance and validation. Here are a few warning signs to look for when you evaluate a key management solution:

• A vendor makes compliance claims, but there is no validation. Some vendors claim to be “FIPS 140-2 compliant” but in fact have never completed a FIPS 140-2 validation. Security is hard, and unsubstantiated claims should be a red flag.
• A vendor claims FIPS 140-2 compliance, but the validation is “in process”, but not complete. A security product can be “in process” for many months or even years. A claim of FIPS 140-2 compliance without actual completion should also be a red flag.
• A vendor makes some claims of FIPS 140-2 validation, but research shows that the key management solution was not validated by that vendor.
• A vendor makes a claim of FIPS 140-2 compliance, but the solution is only compliant when backed by a third party validated key management solution. In this case the vendor solution itself is not validated, but relies on the validation of another solution. You may be fooled into thinking that the solution itself is compliant when it is not. Especially watch for this pitfall with open source solutions.

You can always check a vendor’s claims of FIPS 140-2 compliance. Ask for the NIST FIPS 140-2 certificate number, and then Google it. NIST makes the validation certificate available to the public on their website. Copy and paste this into Google search:

NIST FIPS 140-2 certificate number 1449

That was easy!

Number 4:  Failure to make encryption and key management easy and invisible

Now that you are on the road to getting encryption and key management right, it is important to also make it easy and invisible. Your customers have a lot on their agendas, and becoming a key management expert is probably not one of them. So even if you follow the above advice and implement encryption and key management, do your customers a favor and make key management easy. The best way to do this is to bundle a key management solution into your product, and make key management automatic. You can still enable the configuration of an external key management system (some customers will want this), but you can really make it easy for most of your customers if you automate the key management tasks.

Automating key management is a great competitive advantage! One of our partners in the archival and backup space implemented this strategy and make great competitive wins on this feature alone! Their message was simple:

“We have encryption and key management. It is FIPS 140-2 validated. It is completely automatic so you don’t have to spend time fiddling around with a complex key management system.”

This strategy won them a lot of competitive deals and it was easy to talk about - and it shortened the sales cycle.  Of course, be sure that your key management solution supports this type of integration and automation!

Number 5:  Failure to segment customer data

As you move to the cloud and create shared, multi-tenant SaaS solutions, be sure to plan for and architect data segmentation into your solution. You will encounter large customers who will not want to have their data in the same space as other customers. They will want the additional security of segmenting their data into a virtual private cloud. With planning, your technical team can meet this kind of requirement, and help you close that very large deal.

Of course, a data segmentation plan requires a key management segmentation plan. For the same reasons customers want to segment their data, they don’t want to share key management with other customers. And they want to maintain full control of the key management implementation. So be sure to plan for customer-specific deployments of encryption key management and failover key management servers. A properly implemented data and key management segmentation plan will even allow for on-premise deployments that are “cloud ready.”

Number 6:  Failure to develop new market opportunities

Think about Amazon (the company) for a moment. At one point in their history they were an online bookstore. Today the company is very different. Amazon first leveraged its technologies to sell all kinds of products, and then created Amazon Web Services (AWS) to enable all of us to benefit from cloud technologies.

Are you thinking like Amazon? If not, you might be missing some big opportunities. Now that you have secure applications, are there lateral opportunities or technology licensing opportunities available to you? When you approach new opportunities and partners, don’t be afraid to talk about security. Regardless of what you’ve heard:

SECURITY SELLS!

If you are new to encryption you might be asking yourself, "what is the difference between RSA encryption and AES encryption, and when should you use them?" It’s a great newbie question, so let’s go exploring.

AES stands for Advanced Encryption Standard and is in wide use around the world. It falls into a class of encryption methods called “symmetric” encryption. That is, the same secret (an encryption key) is used to encrypt the data, and also used to decrypt the data. AES encryption is probably the most widely used encryption method for protecting data at rest. You will find it used in self-encrypting disk drives, database encryption, storage encryption, and so forth. It’s been around since about 2002, and it is an international standard. Roughly speaking, when you encrypt with AES you put data and the secret encryption key into software that implements AES encryption, and out comes the encrypted data. When you want to use that data you put the encrypted data and the same encryption key into the software, and out comes the original data that you can use.

There are other symmetric key encryption algorithms, and we’ll discuss that a bit below.

RSA encryption is named after the three inventors of the encryption method: Ron Rivest, Adi Shamir, and Leonard Adleman. RSA falls into a class of encryption methods called “asymmetric” encryption. The name asymmetric follows from the fact that there are two related secrets, or keys, used for encryption. One is called a public key, and the other is called a private key. The keys are related in the sense that if you encrypt with the public key, you can only decrypt with the related private key. And the reverse is true, too - If you encrypt with the private key, you can only decrypt with the associated public key. The math is pretty amazing and involves very large prime numbers and factorization. RSA keys are usually used when you have two physically separate endpoints. RSA encryption is often used in web browsers to connect to your favorite websites, in VPN connections, and in many other applications. We use asymmetric encryption every day.

There are other asymmetric encryption algorithms, and we’ll mention a few later.

So, when do we use AES encryption?

AES encryption is great when we have a constrained environment. For example, if we encrypt data in a database, we will decrypt data when we need to access the database. Another example is hard drive encryption - we encrypt the data written to the disk, and decrypt it when we read from the disk. Encryption and decryption will take place on the same platform and in the same context. AES encryption is great for this particular use case. That is why it is commonly used for protecting data at rest.

When do we use RSA encryption?

RSA encryption is really great when we have two physically or geographically different end-points. If I am encrypting data in San Francisco, and you are decrypting it in Dubai, I am likely to use RSA encryption because it is ideal for two separate end-points. I can encrypt data with an RSA public key at the originating end-point, send it over an unsecure web connection, and decrypt it with the RSA private key at the destination end-point, and not worry about who might intercept it in the middle. The unique public / private key aspects of asymmetric encryption helps us be secure when we are separated by many miles of insecurity and hostile internet territory.

Performance and how this affects the use of RSA encryption

RSA encryption is great for protecting the transfer of data across geographic boundaries. But we have a bit of a problem with RSA encryption - it is really poor from a performance perspective. I might want to send you my sensitive file, but encrypting that with RSA is going to be difficult due to the low performance of RSA encryption. No problem! You can combine RSA encryption with AES symmetric encryption to achieve the security of RSA with the performance of AES. This is normally done by generating a temporary, or session, AES key and protecting it with RSA encryption.

Other symmetric algorithms

AES is not the only symmetric encryption method. The older, and still standard, Triple DES (Data Encryption Standard) method is still in wide use. Triple DES is an accepted standard even though it is older than AES. However, for any new applications you should avoid the use of TDES (also called TDEA) encryption and it is likely to be deprecated as a standard soon.  Other encryption algorithms exist, such as Two Fish, Blow Fish, Ghost, and others. While they may be good encryption algorithms, they have not achieved the status of accepted standards, and so you should avoid them.

Other asymmetric algorithms

RSA is the granddaddy of asymmetric algorithms. But is is not the only accepted standard for asymmetric encryption. Elliptic Curve Cryptography (ECC) is also in wide use (usually combined with a symmetric algorithm) and is an accepted standard for asymmetric encryption. It performs better than RSA, but still lags AES in terms of performance. You should feel comfortable using ECC for asymmetric encryption needs.

AES encryption and modes of encryption

While AES encryption is the most commonly adopted encryption method, you should be aware that there are multiple modes of operation that can be used with AES. These are also specified in the standards. The raw AES mode of operation is called Electronic Code Book, or ECB. Because raw AES in ECB mode can leak pattern information when encrypting large amounts of data, it is common to use a mode of encryption that incorporates an initialization vector. The Cipher Block Chaining (CBC) mode of AES encryption is very common, as is Counter (CTR) mode. For storage devices it is common to find the XTS mode of encryption used. If data corruption is of concern, you might find the Galois Counter Mode (GCM) in use.

The evolving world of encryption

The world of encryption is always evolving. Cryptographers are working on new algorithms and improvements to existing algorithms to meet the challenges of high performance computing and quantum computing. It is an exciting time for cryptography and encryption key management. For now, you should always stick to published standards like AES, RSA and others mentioned here. Doing so brings the benefits of a consensus among a world-wide group of cryptographers, and keeps you in alignment with many compliance regulations.

Please let me know if you have any questions.

Patrick

MongoDB is revolutionizing the world of ‘Database’ with its scalable, secure, replicating database. With MongoDB Enterprise Advanced customers get the ability to protect sensitive data with built-in encryption, and a convenient, standards-based interface for encryption key management based on the industry standard Key Management Interoperability Protocol, or KMIP. Here at Townsend Security we fully support the MongoDB Enterprise Advanced key management architecture, and have certified our Alliance Key Manager with MongoDB for both Intel and IBM Power architectures. And, Alliance Key Manager supports real-time key mirroring between one or more failover key servers to ensure that you never have a service interruption because of a failed connection to a key server!

Customers often ask us how to design a resilient key management strategy with MongoDB. So let’s look at three common scenarios.

First, a note about the use of a load balancer

The key management interface for MongoDB Enterprise Advanced only allows for the definition of one key management server. In order to define one or more failover key servers, the following scenarios use a load balancer to allow connecting to the failover key server. As their name implies, load balancers are normally used to balance workload requests among multiple servers, and are available in all platforms where MongoDB is supported (VMware, cloud, etc.). In this case you use a load balancer to provide a failover mechanism. Future versions of MongoDB may support the definition of multiple key servers in the MongoDB configuration file - if that occurs the use of a load balancer would not be required.

Scenario 1: A primary and one local secondary replication set

Imagine a simple implementation of MongoDB that involves one primary node and one secondary node to provide for data recovery and high availability. In this scenario the secondary node is deployed locally, or in the cloud across regions or availability zones,  to the primary node. However, many MongoDB customers deploy the secondary node in a remote location to improve business recovery (see the next session).

Our goal in this scenario is to minimize or eliminate business interruption in the event the primary node fails or the key server for the primary node fails. To achieve this, we leverage the KMIP interface of MongoDB, a load balancer, and the key mirroring capabilities of Alliance Key Manager. 

From the mongod.conf configuration file:

It is important to note that we don’t have to worry about redundancy and failover of MongoDB itself. This is very nicely handled by MongoDB. What we are striving to achieve is to implement an equally resilient interface to key servers.

The primary MongoDB node has a KMIP configuration that points to the load balancer. The load balancer itself is configured to connect to the primary key server and a failover key server. Note that the failover key server is actually the key server for the secondary MongoDB node. In this scenario each key server functions as the primary key server for the local node, and as the failover key server for the other node. In the event the primary key server cannot be reached due to a network failure or a key server failure, the load balancer uses the secondary key server.

Alliance Key Manager supports real-time key mirroring between one or more failover key servers. In the above scenario if the primary key server fails, the secondary key server will have a copy of the necessary encryption key. Note that Alliance Key Manager can mirror keys to more than one failover key server if needed.

The result of this type of implementation is a fully redundant key server deployment to protect both the primary MongoDB node as well as a fully redundant key server deployment for the secondary MongoDB node.

Scenario 2: A primary and multiple secondary replication sets

MongoDB customers often deploy more complex implementations with multiple secondary replication sets. This might be done to provide high performance for data ingestion (writes) into the primary node, while secondary nodes provide high performance query support and data redundancy. This type of scenario involves multiple nodes each with its own availability and recovery requirements. There can be many key managers involved in supporting this environment.

This scenario also leverages the use of load balancers to provide for automatic failover to a secondary key server. See the notes above about load balancers.

Primary and secondary MongoDB nodes have KMIP configurations that point to a load balancer. The load balancers are configured to connect to the defined failover key server when needed. Note that the failover key server is a different key server on another MongoDB node. In this scenario each key server functions as the primary key server for the local node, and as the failover key server for the other node. In the event the primary key server cannot be reached due to a network failure or a key server failure, the load balancer uses the secondary key server.

Alliance Key Manager supports real-time key mirroring between one or more failover key servers. In the above scenario if a key server fails, the secondary key server will have a copy of the necessary encryption key. Note that Alliance Key Manager can mirror keys to more than one failover key server if needed.

The result of this type of implementation is a fully redundant key server deployment to protect both of the MongoDB nodes.

Scenario 3: Complex, geographically distributed nodes

In complex MongoDB deployments that involve many geographically dispersed nodes, the same principles can be used to create highly redundant and resilient key management services. In this case the key managers are mirroring encryption keys and access policy across geographically dispersed locations. An encryption key management server and its backup mirrored server are available in all regions. 

This scenario uses multiple key management servers to provide a very high level of resilience and redundancy. The primary MongoDB node runs a primary and secondary server in order to ensure a high level of hot failover support. The remote MongoDB nodes also run a primary and secondary key server for the same reason - to provide the maximum availability of the secondary MongoDB databases. It is also important to note that this key server deployment leverages the ability of Alliance Key Manager to replicate keys to more than one additional key server.

In this example there may be many geographic regions that participate in the MongoDB replication group. Note that it is possible that multiple MongoDB nodes can share one set of key servers. There is no requirement that each MongoDB node needs to have its own key server.

This more complex MongoDB configuration demonstrates the scalability of Alliance Key Manager to serve even very complex deployments of MongoDB. And you can start with a simple deployment of MongoDB and scale up the key management deployment as you grow MongoDB.

Summary

Hopefully the above scenarios will help you understand and plan your MongoDB encryption key management needs, and provide a template for the growth of your MongoDB platforms. With Alliance Key Manager it is easy to start with a simple MongoDB deployment and then scale it as you invest more deeply in MongoDB technology. You won’t paint yourself in a corner with your early configurations!

Important note about MongoDB Atlas:

At the time this blog was written there is no support for key management using KMIP for MongoDB Atlas, the cloud-based solution. The MongoDB Atlas service only supports AWS Key Management Service (KMS) and Azure Key Vault.  Full encryption key management support is available by running MongoDB Enterprise Advanced as an AWS EC2 instance, Microsoft Azure virtual machine, or Google Cloud Platform virtual machine. We will update this blog if MongoDB Atlas implements the KMIP interface for key management.

Resources

Alliance Key Manager for MongoDB

MongoDB Security Guide

MongoDB technical guides

In April of this year the Payment Card Industry Security Standards Council (PCI SSC) released a new document on cloud guidance called “Information Supplement: PCI SSC Cloud Computing Guidelines”. It was an update of the first version of the guidance issued in 2013. While this is not a set of mandatory rules, it is a core guidance document and recommendations in PCI guidance documents often end up as requirements under the PCI Data Security Standard (PCI-DSS) and PCI Payment Application Data Security Standard (PCI PA-DSS). So it is worth understanding the guidance and it is wise to align your IT and business processes with the guidance. It is better to get things right at the beginning rather than have to rip-and-tear to fix things later.

There is another reason to pay attention to the PCI cloud guidance: The PCI standards often set the expectations for security best practices in other regulations, and reflect evolving industry standards such as those developed by the National Institute of Standards and Technology (NIST). Even if you are not processing credit card payments, you should be paying attention to this guidance.

When it comes to encryption key management in the cloud, there is a bombshell in this document (essentially describing services such as Amazon KMS and Microsoft Key Vault). What does the new PCI guidance say about encryption and key management? Let’s parse it out and see where it goes.

Take a look at appendix E.10 “Data Encryption and Cryptographic Key Management”.

Appendix E.10 starts by describing the shared, multi-tenant architecture of cloud services (pretty much states the obvious). And then makes this statement:

“If a Customer shares encryption keys with the Provider, or engages the Provider as a key custodian, details of Provider access permissions and processes will also need to be reviewed and verified.

This consideration is particularly critical if cryptographic keys are stored or hosted by a third-party Provider that also hosts the encrypted data. If Provider personnel have access to a Customer’s keys and the Customer’s encrypted data, the Customer may have unintentionally granted the Provider ability to decrypt its sensitive data.”

In fact, all cloud service providers such as Amazon (AWS), Microsoft (Azure), and Google (GCP) have access to both your data and your encryption keys if you are using their key management services. This includes AWS Key Management Service (KMS), Azure’s Key Vault, and Google’s Customer Customer-Managed Encryption Keys (CMEK). Perhaps unknowingly, you HAVE granted your cloud provider the ability to decrypt your sensitive data.

Here is how the PCI SSC sees the risk:

“Any data that is decrypted in the cloud may be inadvertently captured in clear text in process memory or VMs via cloud maintenance functions (such as snapshots, backups, monitoring tools, etc.). To avoid this risk, Customers may choose to keep all encryption/decryption operations and key management on their own premises, and use a public cloud only for storage of the encrypted data.

Applicable controls must be applied to the encryption, decryption, and key management processes to ensure that data can only be retrieved (decrypted) by those who are authorized with a defined business need.”

Wow, that’s a pretty strong statement about not allowing your cloud provider have access to your encryption keys and sensitive data. It is hard to imagine a scenario where the cloud service provider has a “defined business need” to access your sensitive data.

Pointing back to the perceived risks of the cloud provider, here is the key point in the PCI cloud guidance:

Because compromise of a Provider could result in unauthorized access to multiple data stores, it is recommended that cryptographic keys used to encrypt/decrypt sensitive data be stored and managed independently from the cloud service where the data is located.

Wow, there you have it. Don’t use the cloud service providers key management service because there is too much risk. This recommendation affects a very large number of users in the cloud.

Where do we go from here?

Fortunately, there are solutions available now to solve this problem (we have one). Let’s outline some options. There will be pluses and minuses for each one. But the good news is that there are multiple solutions to this issue.

1. Deploy your own dedicated key manager in your own on-premise data center

It is fairly easy to deploy an encryption key manager in your own data center and enable its use by cloud applications. Most enterprise key managers use a secure TLS-encrypted session to interoperate with the key manager. Once you enable an outbound TCP port to your key manager, you can easily use the on-premise key manager. Note that this could be a hardware security module (HSM) or a virtual key management appliance running in VMware.

Remember that you probably do not have to retrieve the encryption key from the key server to your cloud application - most key managers support on-board encryption and decryption services. This alleviates the risk of an exposure of the encryption key in cloud memory. Performance will be the important factor to weigh in this regard. While the key manager may be quite efficient in the encryption or decryption operation, the communications lag times may mitigate against this approach.

2. Deploy your own dedicated key manager in a hosted platform

If your organization does not have on-premise infrastructure for a key manager, don’t despair. It is really straightforward to deploy a key manager in a hosted environment. A hosting provider can provide a home for a hardware security module, or for a software appliance. Establishing the firewall rules may take a bit more work, but this is an approach that has worked well for our customers.

3. Deploy your own dedicated key manager in a different cloud

One creative option to separate the encryption keys from the protected data is to deploy the key manager in a different cloud platform. You could, for example, deploy your application data in Amazon Web Services, and deploy the key manager in Microsoft Azure. This helps mitigate the risk of one cloud service provider having access to both your encryption keys and your protected data - one of the key concerns expressed in the PCI guidance.

Note that this solution will probably require that you work with the firewall rules in both cloud provider platforms. The good news is that this is not complicated - we have customers doing this today.

4. Deploy your own dedicated key manager in a separate cloud instance

Lastly, it is possible to deploy a dedicated key management solution in the same cloud as your protected data, but completely avoid the use of the cloud provider’s encryption key management infrastructure. The key server runs in its own virtual machine or EC2 instance and encryption key management is exclusively dedicated to you. If you take this approach, but sure that your key management vendor is not using the cloud providers encryption key management infrastructure! Encryption keys and key management should only be accessible to you and not to your vendor or cloud provider.

I know that some cloud customers are reluctant to take this approach due to concerns about the ability of the cloud provider to access all of the customer applications and data on their platform, including a key management system running in the cloud. Personally I think the risk is minimal, but if you have that concern see the previous alternatives.

In summary, it would be prudent to avoid the use of cloud service provider key management services such as AWS KMS, Azure Key Vault, and Google Customer-Managed Encryption Keys (CMEK). These services will put you at odds with the PCI cloud security recommendations, and likely put you in variance with future regulations. Not a good place to be.

My advice? Get encryption key management right from the beginning. If you are using a cloud provider’s KMS, start your migration now. You have readily-available choices that are affordable. Get started now!

Our Alliance Key Manager is validated to PCI-DSS and available in cloud, VMware, and HSM platforms. You can do this! Get started here.

Patrick

Take a little imaginary trip with me.

You want to buy a new car and you are dreaming of all of the great places you can go and hikes you can take. You’ve done your research, you’ve identified a favorite model, you know the mileage and quality ratings, and you’ve made up you mind! That dream car is going to be yours.

You spend the obligatory two hours at the dealer and drive away with your dream vehicle. You drive it home and park it in the driveway.

Time to head out to your first destination! You pile your hiking gear into the car and head for the Appalachians for your first hike.

Suddenly, your car stops dead in the road and says “Sorry, you need to pay for this destination. It is not a part of your original car purchase plan. That will be an extra $5,000 please. Credit cards accepted!”

WTF?

You didn’t read the fine print. This car purchase plan only includes 5 destinations and the Appalachians aren’t in the plan. Each new destination is going to cost you a bundle.

Welcome to Key Management System licensing.

Most Enterprise Key Management Systems (KMS) work with this licensing model. You will usually find that you need to license each end-point that connects to the key manager. One end-point may be included in the KMS pricing, but you need to purchase additional “license packs” to attach additional systems. Or, you find that your KMS only supports a limited number of encryption keys. As soon as you roll your keys to meet compliance requirements, it is suddenly time to purchase capacity for more keys! Or, for that new encryption project, you may find that you need to purchase new software to enable encryption for that environment.

And it can get really bad when you discover that the KMS system comes in different hardware models and you suddenly need to purchase an entirely different hardware model and do full replacements. That’s painful.

This typical KMS licensing model works really well for the companies that sell them. For companies who need to deploy encryption?

Not so much …

It means going to management for new budget approval every time you want to extend your security through better encryption. Over time this can add up to hundreds of thousands or even millions of dollars in new license and maintenance fees.

There has to be a better way, right?

There is. Here at Townsend Security we do things differently with our Alliance Key Manager licensing. You purchase the KMS platform of your choice and use it the way you want. It is that simple. From that point on:

• We never charge you fees for connecting a new end-point that needs a KMS.
• We never limit the number of end-points based on the model of the KMS.
• We never limit the number of encryption keys generated or stored on the KMS.
• We never force you to pay extra fees for software patches.
• We never force you to pay extra fees for routine software upgrades.
• If you have a hardware HSM, we do not force you to re-purchase the KMS at end of life. Just replace the hardware and keep on keepin’ on!

Fortunately, the story about purchasing a car was completely made up. But the nightmare of KMS licensing is real. Just know that it doesn’t have to be that way. Take a look at our KMS offerings and see a new path forward.

Pinch yourself. And get ready for that dream trip.

We provide Alliance Key Manager as a hardware security module (HSM), as a VMware software appliance, and as a cloud instance (Azure, AWS) - All running the same key management software with the same interfaces and applications.

Patrick

Like most cloud service providers, Google provides options for customers to encrypt data in the Google Cloud Platform (GCP) for Google application and storage services.  Of course, core to any encryption strategy is how the encryption keys are managed. Properly managing encryption keys is central to overall encryption security. In this blog let’s look at your options for managing encryption keys in GCE and the implications for security, key custody (ownership), and business flexibility. We will also look at some alternatives to Google-provided encryption and key management.

Google has three models for encryption key management:

• Google managed encryption and encryption keys (you have no access to the keys)

• Customer-Managed Encryption Keys, or CMEK (you and Google share key management and access)

• Customer-Supplied Encryption Keys, or CSEK (you supply the key, Google does not store or manage)

It is important to understand key management in each of these areas.

Google Managed Encryption and Encryption Keys

Google automatically encrypts your data in storage for you. This is the default implementation of storage encryption. With this option Google creates and manages the encryption keys and you do not have access to the encryption keys nor to any key management functions. This basic level of encryption support ensures that your data is encrypted at rest, encrypted in Google-managed backups, and encrypted as it is replicated through the Google cloud infrastructure.

It is important to note that Google has full access to the encryption keys that protect your data and will not provide them to you. However, Google may provide these encryption keys and protected data to law enforcement or intelligence agencies as dictated by law in your country, and it may not notify you if encryption keys are disclosed under these circumstances.

Customer-Managed Encryption Keys (CMEK)

With the CMEK approach you and Google share in the administration and management of encryption keys. You have the ability to manage the creation and use of the encryption keys for protecting data in storage, and you have the ability to revoke or delete an encryption key. You do not have the ability to export an encryption key, so Google maintains exclusive custody of your encryption keys.

It is important to note that Google has full access to the encryption keys that protect your data and will not provide them to you. Google may provide these encryption keys and your protected data to law enforcement or intelligence agencies as dictated by law in your country, and it may not notify you if encryption keys are disclosed under these circumstances.

Customer-Supplied Encryption Keys (CSEK)

With the CSEK approach you provide the encryption keys to Google and Google does not create, store or manage the keys. You must generate the keys and manage them yourself outside of the Google Cloud Platform. This is normally accomplished using an Enterprise Key Management System. When you create a new virtual image you will have the option of specifying the encryption key in raw, Base64 encoded format, or you can specify the key using RSA key wrapping. Google does not store this encryption key. This means that you must supply the encryption key each time you start the GCP service. To some extent this can be automated using command line tools.

With CSEK Google does not store or manage your encryption keys in any way. Google cannot provide the key to law enforcement or intelligence agencies as it has no possession of the key.

Google Key Management Summary

In many ways the Google Cloud Key Management Service (Cloud KMS) is primitive compared to other cloud service provide key management services, and especially when compared to Enterprise Key Manager Systems which provide dedicated, non-shared, control over the entire encryption key lifecycle.  The cloud services that are protected by Google Cloud KMS are also limited to the storage of data. Additionally Google Cloud KMS does not support open standards for key management such as the OASIS Key Management Interoperability Protocol (KMIP). If data protection, encryption key custody, and exclusive access to encryption keys are important to you, you may wish to explore other cloud service providers and independent key management vendors (Townsend Security can help).

Encryption and Key Management with MongoDB in Google Cloud Platform

One notable exception to the limitations of Google Cloud Key Management is the implementation of MongoDB Enterprise Advanced on the Google Cloud Platform. MongoDB Enterprise Advanced supports the KMIP interface to key management, so you have many choices for the deployment of encryption key management outside of GCP. These choices include the ability to deploy key management:

• As hardware security module (HSM) in your own data center.
• As a hardware security module in a hosted data center.
• As a VMware virtual machine in your own data center or a hosting provider that supports VMware and vCloud.
• As a dedicated cloud instance in another cloud platform.

At the present time in July of 2018, MongoDB Atlas does not support the KMIP interface, so Atlas users will not have the ability to use their own dedicated key management system.

Resources:

Google Customer Managed Encryption Keys (CMEK):

https://cloud.google.com/storage/docs/encryption/customer-managed-keys

Google Customer Supplied Encryption Keys (CSEK):

https://cloud.google.com/storage/docs/encryption/customer-supplied-keys

A rose is a rose is a rose - let's talk KMS.

In IT technology and security, the language and three-letter acronyms we use can sometimes make things more confusing than they need to be. That’s true with encryption key management systems and services that are available today. So let’s sort out the language, TLAs, and try to reduce some of the confusion in this area.

HINT: In the following discussion, pay special attention to the words “system” and “service”. I’ve added emphasis below to help with clarity.

Enterprise Key Management System

An Enterprise Key Management System is a security appliance (hardware or software) that manages encryption keys through their entire lifecycle - key creation, key activation, key use, key expiration or retirement, key escrow, and key destruction. The “Enterprise” part of this descriptive phrase is often dropped, and these types of system are often referred to as Key Management Systems. The word “Enterprise” is often used to indicate that the key management system can be used for a wide variety of purposes within an organization.

Key Management Systems may be hardware devices (usually hardware security modules, or HSMs), software appliances (think VMware virtual machines), or cloud instances. Their use is dedicated to a single organization and usually managed by security professionals within that organization providing the organization exclusive custody of the encryption keys. Key Management Systems are usually validated to the FIPS 140-2 standard by the National Institute of Standards and Technology (NIST).

Category: “Key Management System”; or “Enterprise Key Management System”

Three-Letter Acronym: Usually “KMS”; less frequently “EKMS”

Here at Townsend Security we provide our Alliance Key Manager as a Key Management System available as a Hardware Security Module (HSM), VMware virtual appliance, and as a dedicated cloud instance (Azure, AWS) and is FIPS 140-2 compliant.

Cloud Service Provider - Key Management Service

A cloud service provider’s Key Management Service is generally a multi-tenant, encryption key storage service managed by the cloud service provider that provides a subset of encryption key lifecycle management. Administrative duties for encryption keys are a shared responsibility of the cloud service provider and the organization that uses the keys. This means that the organization is sharing custody (ownership and access) to encryption keys.

Cloud-based Key Management Services are not FIPS 140-2 validated, but may be partially backed by Key Management Systems which are. Examples of cloud service provider key management services include Amazon Web Services Key Management Service (AWS KMS), Microsoft Azure’s Key Vault (Azure KV), and Google Cloud Platform’s Customer-Managed Encryption keys (GCP CMEK, also known as GCP Cloud KMS).

The cloud service provider’s use of the “KMS” acronym adds substantially to the confusion. Just remember that cloud service providers implement a multi-tenant, shared “service” and not a dedicated key management “system”.

Category: “Key Management Service”; or “Cloud Key Management Service”

Three-Letter Acronym: Usually “KMS”, “KV” or “CMEK”; less frequently “Cloud KMS”

Townsend Security is not a cloud service provider and does not provide multi-tenant, shared custody key management services.

ISV Solutions for the Cloud - Key Management Systems

Independent Software Vendors (ISVs) provide Key Management Systems that are not managed or accessed by the cloud service provider or the ISV, and which are dedicated to the user organization. This means that the cloud user is not sharing custody (ownership and access) to encryption keys and has full and exclusive management, ownership, and access to encryption keys. ISV cloud Key Management Systems provide support for the entire key management lifecycle including key creation (key establishment), key activation, key distribution, key backup and escrow, and key destruction.

ISV Key Management Systems are not FIPS 140-2 validated, but may be based on software that is FIPS 140-2 compliant.  

An ISV’s use of the “KMS” acronym more closely reflects the Enterprise Key Management System described above. These are key management systems that are dedicated to a single organization and which support the entire lifecycle of key creation, key distribution, and key escrow.

Category: “Key Management System”; or “Cloud Key Management System”

Three-Letter Acronym: Usually “KMS” or “Cloud KMS”

Townsend Security provides its Alliance Key Manager solution as a dedicated, single tenant, Key Management System on the Amazon Web Services and Microsoft Azure cloud platforms. The solution can be located in the relevant Marketplace.

Cloud Service Provider - Bring Your Own Key

One small variation on the theme of cloud service provider Key Management Services involves the ability to import your own encryption key to the cloud service. In this case the cloud service is not creating the encryption key for you, but is allowing you to bring an encryption key that you have created, or which you manage outside of the cloud, to the cloud’s key management service. In this case the key you bring is imported into the multi-tenant, shared key management service and is then available through the cloud service provider key management service interface. Although you have exclusive control over the creation of the key and may store it outside of the cloud, it is important to remember that the storage and management of the imported encryption key is a shared responsibility between you and the cloud service provider. You do NOT have exclusive control of the key after it is imported to the cloud.

Category: “Key Management Service”; or “Cloud Key Management Service”; often “Bring Your Own Key” or “Customer-Supplied Encryption Key”

Three-Letter Acronym: “BYOK” or “CSEK” (Google Cloud Platform)

Townsend Security fully supports bringing your own key to its Alliance Key Manager Key Management System for the Azure and AWS cloud platforms. Key custody remains fully with the organization and is not shared with the cloud service provider nor with Townsend Security.

In summary: A rose is a rose is a rose.

However, a Key Management Service is NOT a Key Management System.

When discussing encryption key management care must be taken to understand the meaning of KMS. It can be quite different depending on the context. I hope this blog helps clarify the language and issues around key management.

Patrick

(with apologies to Gertrude Stein)