Townsend Security Data Privacy Blog

Some years ago, during an “in between” period of my life, I drove a taxi in Houston, Texas. It was one of those enriching life experiences (this means it left scars), and a recent security newsletter from Bruce Schneier had me thinking about it again.

All of us drivers loved to take a customer to Gilley’s, a famous honky-tonk out in Pasadena.  Gilley’s was a huge place with live country music, line dancing, a mechanical bull, a real rodeo arena, and lots of Texans (most with quite a few long necks behind them). It ran well into the early morning hours and was always busy. It was a good distance from downtown Houston or the Houston airport and a ride to or from Gilley’s was going to be a good fare and usually a good tip.

Here’s the security angle – Gilley’s could be a bit dangerous starting from about 10 or 11 at night. There was a whole lot of drinking going on (I know you will be surprised by that), and some roughneck or cowboy or soldier was going to take an unfortunate interest in someone else’s girlfriend. Or maybe someone liked the wrong football team. Or whatever – there was no shortage of things that could cause a fight. A shooting or brawl was not that uncommon at Gilley’s.  Every driver I knew carried some type of “protection” under the seat. Mine was a short tire iron. But some carried serious heat. But you never wanted to be in a position of actually having to defend yourself – you were probably going to get some serious hurt on you.

Every night when you were driving taxi you had to make a decision about taking a late night run to Gilley’s. A lot of drivers just wouldn’t go out there after 11pm. Some drew the line at 1pm, or wouldn’t go out there when the place was closing.  But if you’ve had a bad day, that run might help you get profitable before sunrise. So, you were always making a security assessment – how much risk were you willing to bear?

Now here is what I was thinking about: When I think of Pasadena, Texas, my impression is still tinged with that original experience. For all I know, Pasadena may have changed into a yuppie paradise with 5-star restaurants and day spas. I’ve seen other neighborhoods transform (good or bad) over time. South of Market in San Francisco now has a Whole Foods, and China Basin is definitely not as dangerous. So things change over time. And a person’s personal security posture will change, too, if there is adequate information about the neighborhood.

Now let’s bring these chickens home to roost.

Things have changed in the world of IT. We used to feel safe behind our firewalls and DLP systems and anti-virus software. We carefully avoided upgrading our operating systems and software to avoid buggy releases. This made complete sense at the time.

But now the attacks come in from infected PDF files and infected web sites. A USB thumb drive can carry the danger. Systems that we thought were relatively safe like Macs, mobile phones, or IBM Mainframes and AS/400s now are as much of a threat as anything outside the firewall. Criminals now routinely use weaknesses in unpatched systems to steal sensitive data. The threat landscape has changed. We need to change, too.

So, when you think about that OS or software upgrade you should give more weight to staying current, and perhaps a little less weight to avoiding some bugs. I know the risks of doing software upgrades, and that you have to make a judgment call. But out of date software is honey to the bad guys. It’s time to re-think the security posture - the neighborhood is not the same.

Patrick

No, I’m not from Texas (Hat tip to Lyle Lovett)

Learn how we have made encryption and key management easier and more affordable than ever with Alliance Key Manager.

The last few days has seen a number of new reports about a security vulnerability in RSA public/private keys in use on the Internet. The vulnerability has to do with duplicate keys, and not with any weakness in the cryptographic algorithm itself. But it is disturbing information because public/private key encryption is crucial to the security of web sites and a number of other secure applications and services.

The news is based on the work of academic researchers and you can read the paper here.

While the researchers did not identify the cause of the duplicate keys, a number of us are guessing that the problem lies in random number generation. It is really easy to create bad random number generators, and hard to get it right and prove that you have it right. So any sloppiness in your engineering processes related to RNG can lead to this type of problem. There are not a lot of applications in use that create RSA public/private keys and X509 certificates, so the problem may be limited to a small number of these applications. But at this time there is no indication of which RSA key generation routines may be at fault.

How bad is this problem and should you worry about it?

At this point I don’t think there are any known attacks or breaches based on this vulnerability. It may have happened, but it hasn’t been reported yet. However, while the original researchers did not disclose their methods for identifying the duplicate RSA keys, it doesn’t seem hard to think of ways to do this. That being said, I am not sure how easy it would be to mount an effective attack even if you know about the duplicate keys. A lot is still unknown about this vulnerability.

I think Bruce Schneier has a good take on this issue. If you are concerned about this potential problem, you can read his comments here.

Are there bad random number generators in the wild?

You bet. Some years ago we found one on the IBM i (AS/400, iSeries) platform. In the early days of the IBM i platform you could use a system API named CEERAN0 to generate random numbers. We were shocked to learn how poor this RNG application is. It would start generating collisions within about 30,000+ cycles. That is really bad. It turns out that IBM also provides a cryptographically secure RNG application, but the older one still exists and we’ve seen it used in vendor and customer applications.

So, the obvious question is how can you know if a random number generator is good? One place to start is with the National Institute of Standards and Technology (NIST). NIST publishes guidelines on proper RNG methods, and a certification program. You can read the NIST recommendations for certifying random number generators here (warning: heavy lifting ahead).

NIST also has a certification program for random number generators and vendors like us can submit our work to independent labs that perform NIST testing. All of our cryptographic solutions have been through this testing. It is also important to note that encryption key management systems that undergo FIPS 140-2 certification also go through full RNG testing. Our Alliance Key Manager if FIPS 140-2 certified and the RNG routines were NIST certified as a part of that process. We’ve also certified our RNG implementations on a variety of platforms including the IBM i. The list of vendors who have completed certification are here.

While I don’t think NIST certifications are a perfect indicator of good cryptographic implementations, I certainly wouldn’t accept any encryption key management or cryptographic solution that had not been through an independent certification process.

Proper random number generation is crucial to secure cryptographic systems. You can’t leave RNG to chance (sorry about that).

When more information is available on the RSA vulnerability I’ll give you an update. 

For more information on the importance of encryption key management, download our white paper "Key Management in the Multi-Platform Envrionment" and learn how to overcome the challenges of deploying encryption key management in business applications.

Stay safe.

Patrick

I’ve been writing about encryption performance lately because our customers and potential customers have been asking about the impact of encryption on the overall performance on their systems.  It’s good that they are asking these questions as a poorly performing encryption library can have severe impact on your application environment. This is especially true on an IBM Enterprise platform like the IBM i (formerly known as AS/400 and iSeries) where customers often run multiple applications.

While it is common in the Microsoft, UNIX, and Linux worlds to segment different applications onto different physical servers, it is common in the IBM i world to run many applications on the same server. You typically find CRM, ERP, web, and many other applications happily co-existing on one IBM i server. But this means that a poorly performing encryption library will have a ripple impact on all of these applications, and not just one.

IBM provides a no-charge, AES software encryption library on the IBM i platform that developers can use to encrypt data. It implements all of the standard AES key sizes (128, 192, and 256) along with a variety of other encryption algorithms, both open and proprietary.  I don’t believe the software library has been independently certified to the NIST standards, but I believe that it properly implements the AES encryption algorithm.

But how does it perform?

We did a simple little comparison test of encrypting 1 million credit card numbers on an entry level IBM i model 515 server with a single processor. We compared the native IBM AES library with our own AES encryption library which is NIST certified and optimized for encryption.  The difference is very large. Our IBM i encryption library clocked in at 116 times faster than the native IBM i library. Note that this is an informal test and not independently verified, but practical experience by our customers is very similar.

What does this mean in terms of application performance when you add encryption to the mix? The math is pretty simple. An encryption task that takes 10 minutes with our library will take several hours with the IBM library. That’s painful. And all of the other applications that share this system will also feel the pain.

The problem is not limited to just an occasional developer at an individual customer site. Some vendors of IBM i software use the IBM encryption libraries, too. So you can be inadvertently using the poorly performing libraries without knowing it.

Often I see IBM i customers trying to fix an encryption performance problem by adding additional processors to their servers. This can be expensive, and usually involves software license upgrade fees. It can also not have the impact that you might think. Due to the way that encryption works, adding a second processor usually will not double your encryption throughput. Another bit of disappointment and extra cost.

It is usually not hard to fix an encryption performance problem if you catch it early. If you’ve take a modular approach to the implementation, you can usually swap out one module for another without too much difficulty. You just don’t want to be doing that for hundreds of applications.

For more information on AES encryption, download our white paper "AES Encryption and Related Concepts" and learn about how proper encryption and key management work together to secure your data.

Patrick

Before you deploy an encryption solution, there is one often-overlooked consideration to be aware of – performance.  A slow encryption solution can change a “job-well-done” into “we need to get this solution off our servers and go buy from Townsend Security STAT!”  This actually happened to a retail customer of ours.  Their initial encryption implementation was so slow that it prevented them from being able to use it in production.  True story.

So are there any performance tests you can do before you decide on a FIELDPROC encryption solution?  You bet.

Before you begin, you need to decide how many fields in a table you have to encrypt.  If you need to meet the PCI DSS compliance regulation, you might only need to encrypt one field (credit card number).  If you are protecting PHI (Protected Health Information) in the medical segment or PII (Personally Identifiable Information) for privacy notification laws, then you may have several fields in a table that need to be protected. Every column that you need encrypted is going to add to the overall performance burden.

A good first performance test is with just with one column.  We recommend creating a table with one million records, implement FIELDPROC, and then seeing how long it takes to encrypt the data in that table.  These results will give you an idea how your system will perform when you are only encrypting one field (a credit card number to meet PCI DSS, for example).

Next, if you need to encrypt several fields (ZIP code, phone number, credit card number, etc.), do a test on a table with that many fields.  You will learn a lot very quickly about the performance of FIELDPROC and your encryption solution.  If you do these tests, and we think it is absolutely important that organizations try this test before they deploy a FIELDPROC encryption solution, you will learn a lot about how the encryption will impact your production environment. 

Summary

Look at what you need to protect, try and create as close to a real-world test as you can, and see how your performance results are.  It is very simple to do.  We can even provide you with a sample database and table with a million records so that you can create and test on your machine.

Listen to our podcast “IBM i FIELDPROC Performance: Speed Matters” for more information on encryption performance with FIELDPROC on IBM i 7.1

A number of studies show that only about 25 percent of companies and organizations have deployed encryption of data at rest to meet privacy regulations, and we seem to be stalled at about that level.  We are now about 10 years past the really big data losses that led to the emphasis on protecting data, why are we making so little progress?

I think one of the main reasons is the level of difficulty in deploying most data encryption solutions. Most organizations still see an encryption project as requiring lots of time, money, and human resources to accomplish. As humans I think we all have a tendency to avoid the hard and painful things we know we need to do (I plead guilty). And this is an impediment to getting our data protected with the right encryption and key management technologies.

Vendors of data protection technologies have been slow to address this part of the equation. We have our heads in the technical side of things trying to be sure that we implement secure solutions that meet best practices, and working towards the difficult product certifications that we have to accomplish. The user experience is not usually the thing most on our minds. So, I think we’ve been a part of the problem.

It is also true that developers who are good at the user experience are generally lousy at security. You just don’t go about security development in the same way you mash up a new web service.  Most of the new web-based security solutions that promise to make things so much easier look from the outside really terrible in terms of encryption, key management, and interface security.

It is up to those of us who make security solutions to make them easier to use. Here at Townsend Security we are trying to channel Steve Job’s focus on the user experience.  Once you have the foundational security applications done and certified, it is time to look at how to make them easier to use. This year we implemented our SQL Server EKM encryption key management solution that makes it easy to secure Microsoft data. We also introduced IBM i FIELDPROC automatic encryption which is making data protection a lot easier for AS/400 customers.  I am convinced we are on the right track in this regard, and you will find us trying to make other environments easier to secure as we go forward.

Best wishes for the New Year!

Patrick Townsend

Learn how we have made encryption key management easier and more affordable than ever with Alliance Key Manager.

In our final installment of questions from “Encryption Key Management with Microsoft SQL Server 2008” we tackle the topics of encryption key management in High Availability (HA) and disaster recover scenarios.  If you missed the first two blogs in the series, you can always go back and read about “The Performance Impacts of TDE on Microsoft SQL Server” and “How often do I need to rotate encryption keys on my SQL Server?”  If you still have any questions about encryption or encryption key management on your Microsoft SQL Server, feel free to send them our way.  It is always a pleasure to hear from members of the data privacy community.  Here is the last question in this series:

Can you speak to how encryption key management works in High Availability (HA) and disaster recovery scenarios?

An encryption key management solution becomes a part of your critical infrastructure if you are properly protecting keys.  If your key manager goes down, your applications stop functioning until you have key management back up.  We really address those concerns in a number of ways.  One way is that our key manager is built for redundancy.  We know it is hardware, and if it is hardware it can fail, so we implement a hardware platform that is resilient and has a lot of redundancy built in.  The first layer of keeping an encryption key manager up and running consistently is to have a good hardware platform.

Secondly, we build real-time mirroring of keys and policy around keys directly into the product.  You can actually deploy a high-availability failover key server with all key activity and access controls mirrored in real-time to a fail-over key server.  If you use SQL Server, you can define right in our software one or more fail-over servers.  For example, in SQL Server, through the configuration panels you can define multiple key servers for fail-over.  If you have a failed server, a hardware problem, or network outage, you can actually define fail-over servers and that will take place in real time.

Furthermore, there is a lot of resilience in the EKM provider software that we install on the Windows server so that if there is a software failure, there are some self-healing mechanisms there too.  If our module were to be killed off by the security administrator, it will get re-started automatically.  So there are a lot of redundancy and self-healing components built in to both the software and configuration components on the key server.

We have done a lot to help customers deal with the concern about resilience of a key manager because it is critical infrastructure, and we fully support that through real-time mirroring.  It is not an operating system feature.  The key server itself has implemented this mirroring capability.  It is itself self-healing.  So if two key servers are mirroring to each other and the network goes down, they will queue up those mirroring transactions, and when the network comes back, it will re-commit those changes.  Alliance Key Manager is a robust facility for making sure you have good backups of your encryption keys.

This concludes our series of questions from the recent webinar “Encryption Key Management with Microsoft SQL Server 2008.”  We encourage you to view the webcast and learn even more about how easily and affordably you can begin meeting data privacy compliance requirements with encryption and encryption key management.

Now that IBM i 7.1 has been available for over a year, more and more companies are finally adopting the latest OS.  It is a great release and we encourage your organization to upgrade.  As a data privacy company, the main reason we are excited about this release is because it finally allowed us to bring you automatic encryption – encryption with no application changes!  The days of modifying your applications to meet compliance regulations (PCI DSS, HIPAA/HITECH, GLBA/FFIEC, etc.) are over.  If this sounds to good to be true, read on.

With the introduction of the FIELDPROC exit point, IBM i administrators now have something similar to what Microsoft SQL and Oracle users have had for some time.  FIELDPROC allows you to implement encryption without changing your applications.  As we attend industry events, one of the top questions we get asked is “Great!  What are the performance impacts?”  This is where the answer is “Depends.”

Any time you are doing encryption in a database environment, there are considerations about performance.  With FIELDPROC, you really have to pay attention to this question because it is an automatic facility and every time a row or record in the database is accessed, the FIELDPROC program is going to get called to do encryption or decryption.  For example, if you have 10 million records in a table and you read that entire table, you are going to make 10 million calls to a FIELDPROC program to do decryption – even if you aren’t using that particular field.  We have heard horror stories from people who have implemented poor FIELDPROC solutions and were not aware about how important investing in a proven encryption solution is.  We are very happy with the performance of our FIELDPROC solution. 

Our FIELDPROC solution uses our own NIST-certified AES encryption libraries (which is very important in many compliance requirements).  They are very highly-optimized, very fast, and have clocked in at under one second for 1 million encryptions (for more details on these tests, listen to our podcast on the topic). And as you know, the encryption library is only half of the encryption process.  The other part is encryption key management.  We have an encryption key management appliance that is FIPS 140-2 certified (again, important for meeting compliance regulations) and implements best practices for encryption key management.  Aside from your server, these components are the two things that effect encryption performance of FIELDPROC the most.

Listen to our podcast “IBM i FIELDPROC Performance: Speed Matters” for more information on encryption performance with FIELDPROC on IBM i 7.1

Yesterday, our partner, Symantec, released a survey on enterprise encryption practices.  The study contained both good news and bad shows when it comes to encryption and key management.  

First the good news, in a recent interview with eWeek Tim Matthews, senior director of product marketing at Symantec said, “About 48 percent of the survey participants reported their organization had increased their use of encryption over the past two years, with one third reporting "somewhat to extremely frequent" deployments of "rogue" projects without any centralized management oversight.”

Now the bad news, Tim Matthews goes on to say, “Business groups and employees are often independently encrypting the data without involving the IT department. While the move to encrypt is a good thing, these unauthorized deployments are a challenge for IT because the data is lost and irretrievable if the employee loses the key, forgets the passphrase or leaves the company without passing on custody of the encryption keys. If IT doesn't have the key, it also becomes harder to properly backup the data or to access the information as part of an e-discovery request”

We understand the difficulties in managing encryption keys and have been helping companies for years with this challenge, so when we saw this statistic about poor key management, it did not come as a big surprise to us, because we hear about the challenges first hand.  The study showed:   “About 52 percent of the respondents said they have had serious key management problems, with about a third claiming that keys were lost or misplaced keys and another third citing key failure. A little over a quarter, or 26 percent for the participants, said former employees refused to hand over keys when they left the company.”

We have posted several different checklists for companies to help them in their selection of key management vendor. One of our more popular lists is the 10 questions that you should ask your potential key management vendor. Given the news in the study, we thought it was worth showing this list again:

1. Is your key manager FIPS 140 certified?  What is the certificate number?
2. How would you describe the encryption key payload as retrieved from the key server?  Is it simple or complex?
3. Is there a common key retrieval application interface on all platforms?  What are the differences?
4. What platforms do you support for key retrieval?  (Note any gaps in platform coverage for your company)
5. Do you provide working sample code for the platforms I need? (Windows, Linux, UNIX, IBM i, IBM z)
6. Do you supply binary libraries for all Enterprise servers?
7. Do you have a Java key retrieval class and examples? Is it standard Java or JNI?
8. Do you charge separate license fees for each client operating system?
9. Do you require that we purchase consulting services from you?  Why?
10. I am an independent software vendor (ISV), can you brand the solution and certify the solution for us?

We encourage you to download and read the study from Symantec. We are sure you will find it very informative.  Then once you have read the study and have decided you need to learn more about encryption key management, listen to this informative 15-minute podcast.

Last week we began a series of blogs with questions from our recent webinar titled “Encryption Key Management with Microsoft SQL Server 2008”.  If you missed part one, you can view it now.  We talked about the performance impacts of TDE (Transparent Data Encryption) on Microsoft SQL Server 2008.  This time we will talk about how often you need to rotate your encryption keys on your SQL Server and whether your database gets locked during Cell Level or Column Level encryption during the re-encryption process.  Additionally, we will touch on how we can help users that have both Microsoft SQL and Oracle environments.  Without further delay, here are the questions:

How often do keys need to be changed?  During Cell Level or Column Level encryption, will the database be locked during the re-encryption process? 

With Cell Level encryption you can roll keys without bringing down your database.  When you establish a symmetric key in Alliance Key Manager, our encryption key management appliance, you can establish how frequently you want that key to be rolled over, rotated, or changed (which all mean generate a new key) for use by the database.  Alliance Key Manager implements support for both automated and manual key rollover, and if you are using Cell Level encryption in Microsoft SQL Server, you can take advantage of that.  The broader question about how often you should change keys in your database is a really interesting one.  There is no simple answer to that.  If you are familiar with PCI DSS you know that particular question got a little more complicated in version 2.0.  Basically, you have to establish a crypto-period for your keys.  I think NIST would say two years is probably the maximum for the usage period of a key.  Five years on the outside for a key to be used, even for the decryption of previously protected data.  The bottom line is you have to establish a crypto-period for your keys and then implement that as policy.  We support that process by letting you define how frequently keys should be changed and then managing that process for you automatically.

With TDE, the answer is a little bit different.  TDE requires some manual steps to roll keys, but we fully support multiple keys and rolling keys with our encryption key management appliance. 

Does your Alliance Key Manager Enterprise Edition serve keys to Oracle databases as well?

Yes.  We provide encryption keys across a wide set of databases, applications, and servers.  We have customers today who are protecting Oracle databases using our Alliance Key Manager to store encryption keys.  We provide sample code that shows how to do that - same thing with MySQL, DB2 from IBM, and almost every standard database that you might run into.  We also make it easy for customers who are using a variety of development languages to implement encryption. 

If you are using SQL Server EKM, you really get a big win because it doesn’t take any programming at all to implement TDE.  So that is the “easy button” in terms of encryption.  Additionally, if you have written applications in Java that use SQL Server, Oracle, or MySQL, we provide Java examples - full applications in fact - on how to use them.  If you programmed in C#,  VBNET, Perl, Python, or PHP, we have sample code that really helps customers get up and running very quickly with our key management solution.

Stay tuned for our next post on encryption key management in High Availability (HA) and disaster recovery scenarios.  In the meantime, you can view a recording of “Encryption Key Management with Microsoft SQL Server 2008”

Townsend Security hosts a monthly webinar on a wide variety of topics ranging from encryption and key management best practices to what’s new with compliance regulations such as PCI DSS.  If you haven’t had a chance to attend any of our webinars yet, sign up for our newsletter and we’ll make sure you know what is coming up next.

During our recent packed webinar, “Encryption Key Management with Microsoft SQL Server” we were asked some great questions from the audience and would like to share them with you on our blog.  Upcoming questions include “How often do I need to rotate encryption keys on my SQL Server?” and “How does encryption key management work in High Availability (HA) and disaster recovery scenarios?”

Here is part one in our series of questions from this webinar.  As always, if you still have any questions, send them our way and we will promptly get back to you.

What are the performance impacts of TDE encryption on Microsoft SQL Server?

Encryption really has a reputation for being CPU intensive.  There is good news on this front for Microsoft customers because I think the people in the SQL Server group did a really great job at performance tuning data encryption within the SQL Server environment.  If you look at what Microsoft says about turning on TDE (Transparent Data Encryption), they say you will pay a 2-4% performance penalty when encryption is enabled.  That is actually very excellent performance.  In our experience, it is actually closer to 2%.  More complex environments may get a little higher than that.

Microsoft will tell you that there is a little more performance penalty around Cell Level or Column Level encryption.  I think that is probably true too, but still I think they have done a great job of making a good solution that performs quite well.

Stay tuned for our next post on “How often do I need to rotate encryption keys on my SQL Server?” and “During Cell Level or Column Level encryption, will the database be locked during the re-encryption process?”  In the meantime, view a recording of “Encryption Key Management with Microsoft SQL Server”.