As developers we bring a lot of tools to our projects. As our skills develop these toolsets become quite extensive. The best developers I know are efficient – we don’t like to solve a problem more than once. So we build up our toolkits to make our projects go faster and, more importantly, help us focus on new and innovative problems that are really exciting.
Unfortunately we all know about technical debt. Our toolkits can get out of date and embed some bad habits. As our industry evolves this is especially true when it comes to secure programming. Our toolkits don’t necessarily reflect current security best practices. We’ll dive into this in the sections below.
In this page we want to share with you some thoughts on how to approach data security (encryption and key management) both from a design point of view as well as from an implementation point of view. In this page we won’t delve into specific programming languages – we trust experienced developers to implement good data protection practices in their languages of choice. Armed with an understanding of good encryption and key management practices, your code and your solutions will stand the test of time and you will help keep your organization out of the headlines!
This page is primarily for developers. Others may benefit from these comments, but we are addressing the community we grew up in and love to hang out with. This group of bright women and men who work hard every day to make their organizations innovative, competitive, responsible, successful and useful to their customers, employees and stakeholders. We know that we can be cranky and cantankerous, sometimes we communicate poorly, and we are often overly rigid - but we’ve also met some of the most passionate and compassionate people you could ever hope to meet among this group. We’ve never met another group who strive so hard to achieve excellence and do the right thing. This page is for you. (And, it goes without saying, we are sure that you will let us know if you disagree with anything here!)
So, let’s step up our game when it comes to protecting data at rest!
|Click here to view this eBook offline|
Things we do for security
As developers we do a lot of things to protect our applications. We guard against SQL injection attacks, we practice safe programming techniques (no buffer overwrites in our code!), and a lot of other things, too. For example, we take seriously the recommendations of the Open Web Application Security Project (OWASP), we implement ACLs to protect access to data, we implement data masking, and we embed audit trails into our applications.
Why We Encrypt
There are a lot of reasons why we encrypt data at rest including, but not limited to:
Governmental and private compliance regulations either mandate data protection or strongly recommend it. These include the PCI Data Security Standard (PCI-DSS), the EU General Data Protection Regulation (GDPR), The Health Insurance Portability and Accountability Act (HIPAA), Notifiable Data Breaches law (NDB), the Data Protection Act, and many others. Not building your applications to meet compliance regulations can subject your organization and its customers to fines and sanctions. Not a great place to be!
Reputational risk can be a major concern of organizational stakeholders. Governance leaders in public and private organizations attempt to minimize the risk to their reputations caused by a significant data breach. You can help by embedding data protection right in your code. It’s unfortunate, but failures of security and reputational damage are going to find their way back to our desks! See more here.
“Understanding the cybersecurity posture of an investment is critical to assessing the value of the investment and considering reputational, financial, and legal harm that could befall the company,”
-- Jake Olcott, a vice president with cybersecurity firm BitSight.
Intellectual Property Protection
Intellectual property protection is often high on the list of reasons for organizations who want to protect data with encryption. State actors, unscrupulous competitors, and others want to appropriate intellectual property to further their aims.
Legal Liability Mitigation
Legal liability mitigation can be an important driver for data protection. Data breaches can result in fines, lawsuits, and unplanned expenses related to recovery can be massive and accumulate over several years.
Startups to Unicorns, and Enterprises of All Sizes
As technology startups and investors mature they are increasingly aware of the risk that poor security represents to their investments of time, money, and human resources. In today’s environment, an investment can go up in smoke with an embarrassing data breach or a new privacy regulation. An obvious recent example is the data breach experience by Facebook that led to a near-term loss of $130 Billion in valuation. While this is an extreme example of the loss of value, it is not an uncommon occurrence.
No entrepreneur wants to wake up and face this type of catastrophic impact on his or her new venture. The best way to avoid this event is to build security, including encryption, in at the beginning.
Here is what one entrepreneur said about learning this lesson the hard way (emphasis added):
“When it comes to enterprise, do product first and sales second. This actually applies to all products, but it’s particularly important when it comes to enterprise customers. … when you scale beyond your initial use cases to a broad enterprise sale, enterprises quickly nail down any new vendors to align with their procurement processes and technical requirements. Particularly if you’re going above a budgetary threshold, procurement requirements like credit checks, references, financial statements, business cases, budget cycles and open tenders can severely delay an enterprise sale. In addition, you will have to do technical and security assessments which are usually the same requirements for big vendors like SAP or Oracle and entirely not suitable for a small cloud-based startup. These hurdles like SSO, data ownership, encryption and integrations all hamper your ability to generate revenue quickly.”
As developers we know that adding security after the fact is very difficult. Don’t get caught short. Shift left.
Encryption Design Principles
What Data Needs to be Encrypted?
Many compliance regulations define sensitive personal data quite broadly. The EU General Data Protection Regulation (GDPR) is an example. When you think about encrypting sensitive data, think broadly. It is more than a credit card number or bank account number. Email addresses, phone numbers, medical diagnostic information, racial or ethnic origin and a lot more should be included in the category of sensitive information that needs encryption protection.
Why Use Standard Encryption Algorithms?
All of the mainstream relational and NoSQL databases use industry standard encryption algorithms such as 256-bit AES. AES (also known as Rijndael) is an industry standard. It is highly performant, internationally accepted, and is likely Quantum Computing resistant. Make 256-bit AES encryption your go-to standard. Avoid non-standard algorithms or algorithms that will soon be deprecated. There are some good encryption algorithms out there, but you will want to stick with accepted standards. Avoid algorithms like:
- Triple DES (3DES)
There are some cool encryption algorithms out there, but they are not accepted standards and you don’t want to have to argue about that (you will lose the argument).
Move Up the Stack as Much as Possible
Build in encryption at every layer as best you can. Generally, the higher up the application stack you can go, the better your security posture will be for encryption. Modern implementations of encryption are very efficient, so don’t worry about duplicating encryption at various layers. Think of the stack along these lines (high in the stack, to low in the stack):
Self-Encrypting hard drives or SSD
There are tradeoffs for all of these approaches. If you implement encryption at the application layer you get very granular control of encryption and key management, and the ability to implement data masking on decryption, user access controls and much more. But it is typically more expensive in terms of time to implement when you approach from the application layer. If you have the choice, move up the stack in terms of where you implement encryption!
At the other end of the spectrum, self-encrypting drives (SEDs) are easy, but provide only a minimum of security. SEDs may also not meet certain compliance regulations such as PCI-DSS and GDPR. When you use self-encrypting drives you are protecting from the loss of a hard drive. But this is a relatively rare occurrence and easy to mitigate.
For the best security posture you should implement encryption at the application layer and at the database layer. These approaches will provide you with a defensible approach to encryption security.
Use Databases that Support Encryption
When you make a decision about the database for your application, choose one that natively supports encryption without the need for a third party add-on. For example, MongoDB Enterprise, Microsoft SQL Server, MySQL and Oracle Database all natively support built-in encryption. You have lots of choices for encrypting data at the database level, so choose a database that natively supports encryption and provides for flexible encryption key management. Save your future self from a big database reengineering effort. Your future self will thank you!
Why Encryption Key Management is Critical
You’ve probably heard the expression that “A chain is only as strong as its weakest link.” In the world of data protection and encryption, this means that your encryption strategy is only as good as your ability to protect encryption keys. Encryption keys are THE secret that has to be protected. If you are using strong encryption keys, you must protect them with industry best practice methods.
These methods will completely fail and will prevent you from being successful with Enterprise customers:
- Defining the encryption key in your application code
- Storing the encryption key in a database file
- Storing the encryption key in the Master database
- Storing the encryption key on removable media (USB, etc.)
- Storing the encryption key on the same server as the data
- Storing the encryption key on a mounted drive
- Sharing encryption key storage with other users, companies, etc.
- Storing parts of a key on two or more servers (split keys)
Use an encryption key management system that is designed for this purpose, and which meets industry standards such as FIPS 140-2 and the OASIS KMIP standard. Store the encryption keys away from the protected data on a separate server (hardware or virtual). Make sure that your end-customer has exclusive access to encryption keys and avoid multi-tenant, shared key systems.
A word of caution: When you review encryption key management solutions do not be satisfied with general statements about “FIPS 140-2 compliance”. All professional key management systems have completed the actual validation to FIPS 140-2 through a NIST certification and have been assigned a certificate number. Avoid “snake oil” claims about FIPS 140-2 compliance:
The National Institute of Standards and Technology (NIST) publishes a list of all vendors who have completed a validation. You can find it here.
Alliance Key Manager has been validated to the standard, certificate number 1449.
Encryption Key Management for Applications
In this section we will discuss some core design principles as you implement encryption in your on-premise or cloud solutions. How you manage encryption keys will be critical to your success as a developer whose applications will serve Enterprise customers.
Store Keys Away from the Protected Data
As noted above you should ensure that you store encryption keys away from the data that they protect. Attackers are adept at discovering when encryption keys are stored on the same server (hardware or virtual) as the protected data. This is true even when the locally stored key is protected by layered encryption. For example, Microsoft SQL Server offers the ability to locally store a key and protects the key with layered key encryption keys. However, attackers have learned how to recover the key in just seconds.
Protect Keys Using Security Best Practices
Security best practices requires that you use a key management system specifically designed for creating, storing and managing encryption keys through the complete use lifecycle of the key. Generally this means using a dedicated key management server that is network attached to your application using secure TLS encrypted connections.
Use Strong Keys
It is amazingly difficult to generate strong encryption keys, and it is easy to get this wrong. Professional key management systems that have been certified to the FIPS 140-2 standard will generate strong encryption keys. Be sure to avoid these techniques for encryption key generation:
- Internet-based random password generators
- Operating system utilities like /dev/random in Linux
- Windows random number APIs
In addition to generating strong keys, enterprise key management systems will securely store them based on industry standards. This helps avoid the accidental loss of encryption keys that can occur when generated by applications.
Customers and users should have the option of maintaining exclusive control of their encryption keys. This means that all key management functions should be available exclusively to the end customer. In cloud environments this means avoiding key management or key service offerings by the cloud service provider that are always shared key custody implementations. Customers should have the option of using a dedicated cloud key management service, or storing the encryption keys outside of the cloud in their own on-premise architecture.
Embedding Key Management
When evaluating your VM solution alongside your encryption key management solution, it’s important to look for certain certifications and validations. One of these is from National Institute of Standards and Technology (NIST): NIST FIPS-197 validates AES encryption. VMware encrypts and decrypts according to NIST-validation. It also manages encryption keys according to NIST guidelines.
Flexible Key Server Integrations
If you design the interface to key management systems correctly, you can provide a key management solution out of the box, and you can also give your customers the flexibility to use their own key management solutions. An embedded key management solution does not have to lock you in to a specific key manager or key management vendor.
Hardware Security Module (HSM) Support
While most of your customers will want to use virtualized key servers (VMware, Cloud, etc.) some may want to deploy Hardware Security Modules (HSMs) in their environment. By basing your interface to a key management solution on the KMIP standard, you can easily provide HSM support to your end customer. Be aware, however, that not all key management vendors provide the full range of platforms – HSMs, VMware or KVM, and cloud instances.
As mentioned previously, when you develop for enterprise customers they will want to deploy a key management solution in their existing on-premise infrastructure. In today’s world, that usually means VMware. Be sure the key management vendor you select provides an option to deploy the key manager as a virtual machine in addition to cloud and hardware options.
Applying Patches and OS Upgrades
Key management systems are a part of your critical infrastructure. If you embed a key management solution with your application, be sure your vendor has an adequate mechanism to apply application and OS upgrades.
When you embed a key management solution in your application, it becomes a critical part of your application infrastructure. Ask your key management vendor if they will escrow their source code as a part of your partnership.
Designing & Implementing a Key Management Interface
As a developer you will want to implement an easy-to-use interface for the customer to configure and register a key management solution. Configuring for key management will be similar to configuring any secure web service, and there are some common principles and objectives to consider.
You will want to provide your customer with a convenient way to configure an encryption key management solution. While the configuration should be easy to define and maintain you will also want it to be secure. In Linux, a common approach is to define the configuration parameters in a .CONF text configuration file. If you take this approach be sure to change the ownership of the configuration file to a user ID that cannot sign on to the system, and protect the directory holding the configuration file with appropriate restrictions.
On Windows you can take the same approach as on Linux, but you also have the alternative of storing configuration parameters in the Windows Registry.
Common Key Server Configuration Fields
While there are differences in how different key management systems are configured, there are also some common fields that need to be configured. If you are using the KMIP interface for the key server (see below), this becomes much easier. Since most key servers now implement support for KMIP, this is the recommended method for configuring and connecting to a key server.
Note that the following values will be specified for the primary key server and for one or more secondary failover key servers.
Here are the common values needed for your key server configuration:
The host name of the primary key server. This should be a DNS name that is discoverable through DNS services, or which is defined in the local <hosts> file.
Or, IP address
As an alternative to the host name you can allow the user to enter an IP address for the primary key server. Host names
The port number for the key server encryption key service. For KMIP servers the default is 5696, but should be configurable.
Certificate Authority (CA) certificate
Almost all key management systems authenticate using PKI infrastructure and require that the connecting client application (your solution) present the key server’s CA certificate. This value provides the reference to the key server’s CA certificate. This may be fully qualified path to an X509 certificate in PEM format, a fingerprint, or other type of reference.
Key server certificate and private key
Again, most key management systems will authenticate the specified client-side end-point using a key server generated client certificate and private key. This value provides the reference to the a signed client-side certificate and private key. This may be fully qualified path to an X509 certificate in PEM format, a fingerprint, or other type of reference.
Key identifier or key name
The identifier or name of the encryption key. This is typically a character string.
This is the version of the encryption key, or sometimes call the key instance. This is not used in a KMIP specification, but is required by some key management systems. This is typically a character string.
User ID (optional)
While is it not common, some key management systems require a user ID and password for authentication that is in addition to the certificate-based authentication. This should be a string value and may not be required by the key manager.
The password for the key manager if it is required.
KMIP for General Interface
The OASIS open standard for interfacing with a key management system is the Key Management Interoperability Protocol, or KMIP. Several years after its first release, the KMIP protocol has now been implemented on all major key management systems (there are some exceptions). If you are implementing a key management interface in your application, you will support the largest number of key managers if you implement the KMIP interface.
Avoid the tendency to chase the latest version of KMIP. While there have been good enhancements since the first release of the standard, if you don’t need those enhancements you should just use version 1.1. You can find the documentation of KMIP on the OASIS standards website, and a description of the functions and data requirements needed to interface to a key server.
It is important to note that there is one group of vendors who have not embraced the KMIP standard – the largest cloud service providers. For reasons that are not clear, the major cloud platforms only expose proprietary interfaces to their key management services. Of course, implementing key management using a proprietary protocol leads to cloud vendor lock-in which may not be in your interest. Fortunately, there are good key management solutions available in the cloud that implement the KMIP interface and help you avoid cloud provider lock-in.
Key Server Failover
Key management systems become an integral part of your mission critical infrastructure. As a developer you should implement an interface to the key manager that allows for automatic failover to one or more secondary key servers in the event it is not possible to connect with the primary key server. This can be handled in your configuration files. It is recommended that you allow for the definition of at least one primary and one secondary key server. An optimal design would provide for multiple secondary, or failover key servers.
Example: A good example of a flexible key management implementation that provides for failover is the VMware vSphere KMS Cluster configuration. vSphere KMS Clusters support multiple key server configurations. If vSphere cannot connect to the primary key server, it delays for a few seconds and tries the next key server in the configuration. It continues this process until it has processed all key servers, then attempts the sequence again. This gives vSphere customers maximum flexibility with their key server configurations.
Protecting KMS Credentials
All key management systems use credentials to authenticate and authorize a user application to the key server. You should implement strong protections for both the configuration file that references these credentials, as well as the certificates, keys, and passwords used for key server authentication. The techniques you use for this should be consistent with the recommendations of your operating system vendor, and security best practices.
When you retrieve a key from the key server it will be in memory in your application. If you are going to use the encryption key for multiple operations you may wish to store (cache) the key in application memory in order to achieve better performance. While there is no perfect way to protect encryption keys in memory, it is a good security practice to use the best tools you can to protect these keys. In a Windows environment you can use the Data Protection API (DPAPI) to protect the encryption key. In non-Windows environments consult your application development language reference for ways to protect the data in memory.
Diagnostic logs are invaluable in any security application. They help your customer support team respond to application problems, and provide the information needed to properly monitor your application. Continuous monitoring is a critical security control, and continuous monitoring needs real-time input from logs your application creates.
Be sure that you provide a basic level of system logging for all operations. This should include all operations that are undertaken with the key manager, as well as all operations that affect the configuration and operation of you application. Be sure to log both successful as well as failed operations! All are critical to a proper continuous monitoring exercise.
Caution: Be sure that you do not log cryptographic material in your system logs, or restrict such logging to “debug” modes of logging. Cyber criminals will search logs for this type of information as a way to bypass encryption key management security controls.
It is wise to allow for variable levels of logging, and this is normally specified in the configuration file. A normal level of logging will produce the logs needed for a continuous monitoring system. High levels of logging will provide your customer support team and developers with the information they need to resolve problems. A debug level of logging lets you solve these really difficult problems that are intermittent or hard to analyze. Here are some recommended values for your logging configuration:
None: No logging.
Basic: Minimal logging.
Verbose: Verbose logging of all application operations
- Debug: Highly verbose logging with data dumps
Hint: If you enable options for verbose or debug levels of logging, it is recommended that you warn the user on application startup about this level of logging. These levels of logging can consume a lot of disk space, and may expose sensitive information. Customers should be warned if they are enabled at application startup.
Application logs can consume a great deal of space if not managed properly. On Linux systems be sure to enable “logrotate” on your application logs. On Windows, write application logs to the Windows Event Manager. Don’t let your application logs crash your application!
Write in Syslog Format
Your application logs will probably be forwarded to a SIEM solution or a log collection server as a part of a continuous monitoring strategy. You can make your logs more effective by implementing the following features:
Support Syslog, CEF, and LEEF Formats
Write your logs in Syslog format. The basic format is based on RFC 3164. All SIEM solutions are able to readily digest log entries in Syslog format, and you will spend less time with your customers helping them set up their SIEM solutions.
Normalize the Data
Most Syslog messages are written in raw text format. You can make your logging messages far more effective if you normalize field data in Key-Value format. Instead of this:
“The IP address is 22.214.171.124”
Write the information like this:
ip_addr = 126.96.36.199
Serialize the Hash
Cyber criminals often try to hide their tracks by deleting system logging messages. You can help minimize this risk by serializing and hashing your Syslog messages. Use a fairly large number for serialization, and increment by 1 for each Syslog message that you create. Then use a strong hash algorithm such as SHA-256 to hash the entire message, and append the hash. In the event of a breach of your system you will be able to detect deleted and altered system log messages.
Protect Log Files
Your log files may contain sensitive information, and can be subject to corruption by malicious users. Be sure to implement access controls and file integrity monitoring for your log files and audit trails.
While system logs often serve the purpose of an audit trail, a separate audit trail in a user-friendly format can be very helpful to customers. Consider creating an audit trail of all key manager activity in CSV format. The audit trail should contain the operation name, key server name, key server IP address, port number, key name for retrieval, date, time, user, and the success or failure of the operation. Any other information that is returned by the key server should also be logged. However, never log actual encryption key values or other cryptographically sensitive data (key hashes, encrypted data, plaintext data, etc.).
Your Application Code and Key Management Operations
If you can’t leverage the native capabilities of database encryption that are present in MongoDB, SQL Server, Oracle, and other databases, you may need to implement your own interface to a key server. The following sections describe two common models for this type of implementation.
Model 1 - Protect Local Data Encryption Key with Key Manager
When you want to implement a Transparent Data Encryption strategy using one data encryption key to protect the data space, you can efficiently implement this by creating the Data Encryption Key, protecting it with a Key Encryption Key on the key manager, and storing the protected Data Encryption Key in or near the local data store.
This approach often gives very good performance, but does not provide a granular approach for key management for specific fields or users. The following section outlines a basic pseudocode approach to initializing and protecting a Data Encryption Key (DEK), and then unlocking it (decrypting it) with the Key Encryption Key (KEK) when needed. Note that this is a high level design and that you will want to perform application logging, error recovery, and key manager failover in your own implementation.
Initializing and Protecting a Data Encryption Key
- Retrieve configurable options
- Initialize application logs
Connect to key manager using TLS
- Request symmetric key (DEK) on key manager
- Request symmetric key (KEK) on key manager
- Retrieve DEK from key manager
- Send DEK to key manager to encrypt with KEK
- Retrieve encrypted DEK from key manager
- Store encrypted DEK locally
- Erase unprotected DEK in memory
- Close logs
- Write audit trail
Unlocking a Data Encryption Key for Use
- Retrieve configurable options
- Initialize application logs
Connect to key manager using TLS
- Send protected DEK to key manager for decryption
- Retrieve decrypted DEK
- Protect DEK in memory
- Use DEK for local encryption and decryption tasks
- Delete DEK from memory
- Close logs
- Write audit trail
Active MonitoringModel 2 - Protect Data Encryption Keys with Key Manager
This model presents a more granular approach to the generation and use of Data Encryption Keys. For example, if you want to use a different encryption key for each user, you can dynamically create encryption keys as needed. You might use a customer number or other unique identifier for the user as the basis for a data encryption key name. There are many possibilities for this type of implementation, but they are usually characterized by the use of unique keys for the person or document that is being protected.
Register new user (or document)
Retrieve configurable options
Initialize application logs
Connect to key manager using TLS
Create unique key identifier
Request key manager to create a new key
Store (associate) new key name with new user
Close application log
Create audit trail
Protect user (or document)
Retrieve configurable options
Initialize application logs
Connect to key manager using TLS
Use unique key identifier to retrieve key from key manager
Perform encryption or decryption task
Clear key from memory
Close application log
Create audit trail
Compliance regulations and security best practices require that you periodically rotate (change) your encryption keys. If you are using an underlying database or data store that already implements encryption and key management, refer to the documentation on the best way to achieve key rotation. Note also that after key rotation you will probably want to retain older versions of keys that may be protecting data archives or backups.
Key rotation in Model 1 above is a fairly straightforward process of changing the Key Encryption Key (KEK) that protects the Data Encryption Key (DEK). Following the logic above you can request that the DEK be decrypted by the key manager, request the creation of a new key, and then protect the DEK with the new KEK.
When using Model 2 above to perform more granular encryption key management you can perform key rotation easily if you store the name of the encryption key and key version (key instance) with the protected data. Encryption key names and key versions are not sensitive data and this can be an effective way to enable dynamic key rotation without bringing down the database.
Alliance Key Manager
“A very cost effective solution in terms of performance, manageability, security, and availability. As a result, my company was quickly able to implement full database encryption leveraging the AKM as our key management solution in weeks. Comparable solutions could have taken months.”
The solution offers unparalleled security, flexibility and affordability for all enterprises. With no client-side software to install, customers can deploy Alliance Key Manager and install the PKI certificates on the database server to easily begin retrieving encryption keys.
Alliance Key Manager is FIPS 140-2 compliant and in use by over 3,000 organizations worldwide. The solution is available as a hardware security module (HSM), VMware instance, and in the cloud (Amazon Web Services, Microsoft Azure, and VMware vCloud). Townsend Security offers a 30-day, fully-functional evaluation of Alliance Key Manager.