Considerations for cloud service providers and consumers

Continuing my tentative steps into cloud security, I went to a talk given by Rafal Los of HP (http://h30499.www3.hp.com/t5/Following-the-White-Rabbit-A/bg-p/sws-119) last night at the Chicago Cloud Security Alliance chapter meeting. The purpose of the talk was to understand cloud security from two perspectives; as a consumer and as a provider of cloud computing services. The talk drew quite a bit of discussion from the crowd, mainly due to disagreements on terminology and over different approaches to managing cloud providers.

Some key takeaways for me:

  • Cloud service providers pretty much cover the entire stack, from infrastructure all the way to software. However, you as a smart consumer still need some in-house expertise on the entire stack so you can adequately manage your providers.
  • Transparency is key for a cloud provider, but transparency means more than just sales sheets and sanitized ISO/ITIL compliant security policies. Think open Bugzilla style issue trackers that customers can follow to see issues affecting the service offered by their cloud providers.
  • Good lawyers are needed by both cloud providers and cloud consumers to manage liability (yes, even cloud consumers are exposed to some new liabilities when using cloud services.)
  • Vendor lock in to a cloud provider is scary to consumers; again good in house expertise is needed to design your cloud strategy to migrate easily between providers.

Overall, it was a useful, thought provoking discussion that provided insight into areas of cloud computing I hadn’t thought of before. For any Chicago locals interested in the Chicago CSA, their website can be found here.

Musings on Enterprise Identity Management

Funny how I learn something new everytime I visit a new customer. Seeing different environments and how different customers solve the same problems is eye opening. More importantly, the more environments I see the more I begin to understand how misconceptions I had held in the past.

As a little background, I broke into the infosec biz in the opposite way that many of my peers have. My first formal role in infosec was as a security systems engineer for Motorola. My job was to basically look at security standards, product management requirements, and security threats and design security features. Problem is, when you take that direction you miss out on the real world, hands on perspective of infosec and have difficulty understanding exactly how organizations infosec functions (if it even exists!) It’s way too easy to fall into the ivory tower mentality. Add a couple of years of grad school in infosec, and you have the makings of a great infosec philosopher, but not a practitioner. Consulting has really opened my eyes and is helping me understand what I never fully grasped before.

With that out of the way, during my last couple of engagements I’ve learned a ton about enterprise identity management and have corrected a major misconception I had held in the past. Earlier in my career, I was faced with solving the problem of figuring out how to integrate enterprise identity management systems with wireless equipment; so that system administrators could use their existing credentials to log into the management interfaces of the wireless infrastructure. At the time, the solution that struck me was to use a AAA server with RADIUS; after all RADIUS is easily extensible, simple to develop and support, and is flexible. What I completely failed to realize at the time is that a AAA server is in fact not intended to act as an identity management system. Sure, AAA stands for Authentication, Authorization, and Accounting, but the AAA server is generally just a relay and is not the actual identity management system.

As I have since learned, identity management is actually centered around one of the oldest protocols in the book, LDAP or Lightweight Directory Access Protocol. LDAP is a directory service that is designed to provide extremely fast lookups and extensive search capabilities. LDAP gets it speed from the hierarchical design of objects that are stored within the directory. All objects descend from a single root node, with each child node having one or more child nodes and one or more sibling nodes. Here’s a figure of a basic tree structure of an LDAP directory.

Each node in the directory is uniquely identifiable by use of a distinguished name. A distinguished name is made up by simply concatenating the identifiers for each ascendent node to the node being named. In this figure, the DN for me would be CN=Walter,OU=Person,OU=Users,DC=jdt,DC=com. Each node also has a collection of attributes associated with it that contain information such as date of creation, address, member groups, object identifier and any other information that is pertinent to the object itself. Lookups are done by connecting to the directory server and issuing queries. LDAP queries are interesting in that they are written in a reverse polish notation style; e.g. if you wanted to search for my name in the query above quickly, you could search for a person named Walter with the following query: (&(OU=Users)(CN=Walter)).

To make LDAP directories generally useful, they usually provide some information to LDAP clients without requiring the client to authenticate. Examples would be a web based corporate address book that is accessible from a organizations intranet. However, in order to gain access more sensitive information, LDAP clients usually authenticate to a directory via a bind operation. In fact, this is how many network operating systems such as Active Directory, Novell etc. authenticate users; a bind against a corporate LDAP directory is performed using the user’s credentials. Once the bind is successful, the user is granted access to the local system and further authenticated LDAP queries can be performed to obtain additional information needed to grant the user access to resources.

LDAP itself does not provide any integrity or encryption mechanisms to protect information that is transmitted between the directory and LDAP clients. However, bind operations usually use authentication protocols which protect credentials that are transmitted such as NTLM or GSSAPI. To provide integrity and confidentiality for LDAP data, LDAP may also be transmitted over a TLS secured port (LDAPS). Yet another use case for an enterprise PKI that I had absolutely no idea about.

So, to complete the original story, what I’ve since discovered when it comes to centralized user management, enterprise identity management, don’t think of AAA services. Rather, think of LDAP directory services. As a matter of fact, AAA servers usually have plug-ins (such as FreeRADIUS’s rlm_ldap plugin) that let them use LDAP directories as a data source when performing authentication and authorization services.

Adding more value to your customers

Looking back at my first job, I realize that some of the same concepts which I used to sell camera equipment in a retail store still apply to my consulting job that I have today. Being responsive to customer needs, explaining the benefits of what they are considering purchasing (or already have purchased), and in general going over and above the minimum requirements for your job are all perfectly valid concepts that apply even to security consultants.

As an example, in my current engagement I’m deploying a part of a tokenization solution to satisfy PCI requirements that my customer is required to satisfy. The scope of the engagement from my perspective is rather limited, but I’ve taken a deliberate effort to go above and beyond the minimum requirements outlined in the SOW (without increasing my scope, a fine line to walk for sure!) Specifically, while I was in the customer’s data center performing some configuration tasks, I looked around and noticed that there didn’t appear to be any cameras in position to observe the equipment I was configuring. As the customer explained that the equipment was considered ‘in scope’ of the PCI DSS requirements, I pointed out that requirement 9.1.1 of the DSS requires the use of a video camera to monitor equipment (and yes, I know that the requirement is an and/or requirement). The customer contact I was working with was not part of the physical security team and couldn’t confirm whether or not the area was in fact monitored, but he took it as an action item to follow up on.

While in all likelihood this will turn out to be a non-issue, the customer expressed appreciation at my observation. This in turn leads to a stronger sense of trust between us and, in my opinion, enhances the overall value of the engagement. So next time you find yourself in a position where you can offer a little extra advice to your customers, consider going the extra mile. You won’t regret it!

A primer on HSMs

A big part of my job is to advise customers how to protect high value secret keys such as root CA private keys, tokenization key encryption keys, etc. Solutions range from the relatively simple read-only private key file, passphrase protected stored with minimal permissions to storing keys on dedicated, purpose built hardware devices. Purpose built hardware devices used for secure key storage are known as hardware security modules (HSM). A HSM can be thought of as both a secure key storage device and a hardware implementation of crypto algorithms. Unlike purpose built encryption hardware such as SSL accelerators, a HSM is not designed as a high throughput, low latency device designed to convert plaintext to ciphertext at high speeds. Rather, it is designed to limit the exposure of key material stored within it. This is accomplished by performing operations that require access to the key material within the HSM itself (such as digitally signing data). In some cases where necessary, HSMs can also provide the keys to authorized devices/users. Applications which wish to use HSMs typically use a vendor provided driver that integrates with various platforms (for example, Microsoft provides a Cryptographic Service Provider interface that can be used to integrate applications with HSMs).

In general, the design philosophy behind HSMs is that it should fail closed; meaning that if an attacker or unauthorized user attempts to repeatedly gain access to the HSM key material, the HSM will zeroize all stored keys. This design makes sense because if an attacker were to gain physical access to the HSM, it is preferable that the secret keys be destroyed rather than possibly be exposed. However, since the HSM protects high value keys, it is imperative that organizations which make use of HSMs have a robust key backup scheme in place such as sharing the key across multiple HSMs that are physically separated.

HSMs typically have a strong role based authentication mechanism built in that is designed to differentiate between key owners and HSM administrators. This separation of duties between key owners and administrators is crucial as it prevents HSM administrators from gaining access to key material. Authentication can be provided via the use of passphrases, or in some HSMs, via the use of individual hardware keys that are physically presented to the HSM.

Since access to key material and to the HSM itself needs to be carefully audited and tracked, a principle that is implemented in some HSMs is the concept of witness keys. A witness key is a separate set of keys that are distributed to other people (usually from different departments/organizations than HSM administrators or key owners) that must be presented to the HSM before access will be granted to the HSM and/or key material stored within the HSM. Witness key systems are also known as MofN systems, where ‘M’ witness keys out of a total of ‘N’ existing witness keys must be presented before access can be granted.

In summary, proper protection of high value keys is an important role that any security organization should take very seriously. HSMs can be a viable solution to help ensure that key material is stored in a safe, controlled fashion.

Use of publicly trusted certificates in enterprise networks

Recently I’ve been involved in discussions in the Mozilla security policy mail list where the topic of permitting privately owned PKIs to be subordinates of publicly trusted CAs is being debated. It’s an interesting debate because I see valid reasons to both permit and forbid private PKIs from being subordinates of public CAs.

First, a little background. Previous visitors that have read my post on certificate validation will recall that a standard PKI hierarchy has a single CA (root CA) that is issuing certificates to subscribers. In this model, subscribers are assumed to have a copy of the root CA certificate in their devices. That certificate is then used to validate that certificates presented to the subscriber have been issued by the CA. In an enterprise network environment, a simple flat hierarchy where the root CA directly issues all subscriber certificates, all devices which will need to access resources protected by subscriber certificates would then need to be provisioned with the root CA certificate. In some simple environments, this is easy. For example, in a standard Microsoft Active Directory network, a domain controller could be configured to both act as a CA and could add the root CA certificate to group policy. Computers which are domain members could then fetch the root CA certificate as part of the domain login process.

However, as the set of devices which are used in enterprise networks continue to grow (iPads, iPhones) and as other web browsers (Mozilla, Opera) gain even more acceptance, the problem of provisioning root CA certificates gets more complicated. These devices/software applications obviously cannot be joined to the domain to obtain the root CA certificate. Therefore, the enterprise network administrator will have to provision the root CA certificate on these other devices using some other centralized certificate management solution.

Several commercial certificate authorities offer a service where they will issue a subordinate CA certificate to enterprise customers (full disclosure: my company offers this service as well.) In this case, an enterprise PKI CA certificate would be signed by a root CA certificate that is already embedded in most browsers, operating systems and devices. This permits the enterprise root CA to be trusted by those devices/applications without a need to provision an additional root CA certificate in them.

Such an approach has pros and cons to it. Let’s analyze this approach in the context of 4 common enterprise PKI use cases as outlined in my SANS gold paper using a simple pros/cons approach.

802.1x WLAN

Pros:

  • The WLAN authenticator (typically a AAA server) is automatically trusted by wireless clients which makes it easy to seamlessly support both enterprise and guest WiFi use cases in the same WLAN infrastructure.

Cons:

  • Opens up WiFi clients to rogue AP attacks (if an attacker can obtain a valid certificate, he can easily setup a rogue AP that would be automatically trusted by enterprise WiFi clients which would enable him to exfiltrate credentials from those clients.)

HTTPS

Pros:

  • Enables enterprises to secure Intranet websites with HTTPS without requiring users to ‘click through’ standard browser warnings.
  • Allows any device/web browser software to access the Intranet website without requiring additional user actions to provision the root CA certificate.

Cons:

  • Enterprise sub CAs could potentially issue valid certificates for Internet websites. A rogue enterprise CA administrator could easily issue certificates that do not belong to his enterprise domain. These certificates could be resold to attackers who could use them to setup MITM attacks for other HTTPS protected Internet sites.

S/MIME

Pros:

  • Digitally signed emails could be validated by email recipients that are outside of the enterprise.

Cons:

  • If an attacker is able to obtain a valid S/MIME certificate, he/she could send very convincing spear-phishing emails to enterprise users. It is unlikely that emails with valid digital signatures would be marked as spam by spam filtering software.

Infrastructure (routers, VPN gateways etc)

Pros:

  • Could possibly simplify the provisioning of new network infrastructure components.
  • Makes it easier to support additional VPN client types.

Cons:

  • Makes it much easier for an attacker who has physical access to the enterprise network to setup rogue devices; especially since many network devices which support certificate based authentication use the certificate as the sole means of identifying valid peers.

In my opinion, the risks of using certificates rooted to a public CA for enterprise WLAN, S/MIME and infrastructure authentication outweigh the benefits. For securing Intranet websites with HTTPS, the benefits outweigh the risks because HTTPS protected websites could easily require the user to present credentials that are valid for the enterprise network. That being said, there are certainly additional controls that could be implemented in S/MIME and WLAN deployments to mitigate the risks pointed out above.

What do you think? Are there other risks or use cases for publicly trusted enterprise PKIs?

HOWTO: Getting the list of root certificates supported by Opera

I’ve recently been given a task to determine the set of browsers and operating systems that a particular commonly trusted root certificate is embedded into. The methods to find this set vary depending on the browser vendor; I’ve found myself downloading various browsers to inspect their trust stores, digging into OS keystores, and when I’m lucky parsing a list that is publicly available. With Opera, I was happy to see that they publish their root certificate lists. Unfortunately, the format of the online repository leaves a little to be desired. The certificates are stored as a set of XML files that are named using a scheme which is most likely a hash or fingerprint of the certificate record (I’ve confirmed they are not named with the actual certificate thumbprint or serial number, even though that would seem to be a logical choice for the naming convention).

The schema of the XML files is simple enough; each file contains 3 fields of note: an issuer field a shortname field, and a certificate-data field. For my purposes, the field that I needed to analyze was the ‘Issuer’ field. Since there are 255 certificates listed in the latest directory with the Opera site, I decided to automate my search using a combination of python, wget, and openssl commands. As this is my first foray into python, I ran into a few speedbumps along the way (especially since my previous scripting language of choice was Perl). In the end, I used the following script to automate my search. Hopefully this will save someone else time in the future.

#!/usr/bin/python

import os
import sys
import subprocess
import re

url = sys.argv[1]
cmd = 'wget --quiet -O - %s' % (url)

print "Fetching the list of certs from ", url

# get the list of filenames
page = subprocess.Popen(cmd,shell=True, \
 stdout=subprocess.PIPE).communicate()[0]

p = re.compile(r".*\"\>(.*xml)<")
matches = p.findall(page)

for m in matches:
 certsrc = url + '/' + m
 cmd = 'wget --quiet -O - %s' % (certsrc)
 cert = subprocess.Popen(cmd,shell=True,\
 stdout = subprocess.PIPE).communicate()[0]
 p = re.compile(r"<issuer>\n(.*)\n</issuer>",re.MULTILINE)

 issuer = p.search(cert)
 print "%s issuer: %s" % (m,issuer.group(1))

You are responsible for your own data security

The recent outrage over Dropbox’s change in their terms and conditions appears to stem from a general perception by Dropbox users that the service will never reveal their personal information to anyone (Dropbox employees, law enforcement officials, and other Dropbox users.) I find the general reaction by users and the broader security community rather amusing. It should come as no surprise to anyone that information which is stored on your behalf by a service provider can be turned over to law enforcement officials with proper warrants. In any case, the lesson that all Dropbox (and really anyone who uses cloud based storage services) should take to heart is that if you really want to store your data securely in the cloud, you have to be responsible for your own data security.

I’ve been a Dropbox user for about 2 years now and regularly store sensitive personal information in my Dropbox folder. However, I don’t just trust that Dropbox is going to implement data security best practices to protect my data. Rather, I encrypt my sensitive files before storing them in my Dropbox folder. Solutions I’ve used are Truecrypt to create small data partitions that I can mount on my devices and 7zip with it’s built in AES encryption function. With these solutions, I alone am responsible for controlling when my encryption keys are disclosed. Even if Dropbox is required to share my files with law enforcement or if there is a major data breach in their service, my files are still safe because I control my private data encryption keys.

While it is perfectly reasonable to trust service providers with your information, it is not reasonable to always assume that you will have full control of your information. Users need to take responsibility for their own data protection.

Who can you trust? PKI under attack

Last month, a major CA (Comodo) suffered a security breach when a certificate reseller account was compromised and used to fraudulently issue SSL certificates for high value domains, including www.google.com and login.yahoo.com. Adding to the raft of security breaches suffered by other security vendors (RSA, Barracuda), this breach has rightly so sparked additional attention to the inherent security of PKI and security systems in general. In this particular attack, the CA itself was not directly compromised but rather the account of a registration authority (RA) was breached. In standard PKI workflows, the CA is responsible for receiving certificate enrollment requests, forwarding the requests to a RA (also known as a vettor), and issuing certificates once the RA has approved the request. For online certificate enrollment systems operated by CAs and occasionally bundled into domain name registrations, certificate resellers may host a certificate enrollment site as well as have access to a ‘certificate vettor’ interface both provided by a CA. This vettor interface is an administrative interface that permits the RA to examine the details of the certificate request which they can use to perform identity validation and other checks as required by the CA’s certificate policies/practice statements. This model means the CA can outsource the vetting and enrollment processing to partners/resellers.

In the Comodo incident, it’s quite possible the attacker was able to obtain credentials to the reseller’s vettor interface (I’m not with Comodo so this is speculation on my part). This would enable them to submit certificate enrollment requests for domains they didn’t own and immediately approve them. Luckily, the fraudulent certificates were discovered 15 minutes after issuance according to Comodo. Comodo isn’t saying how the fraudulent certificates were discovered, but most likely there are audit tools that scan issued certificates against a whitelist of known high value domains.

The implication of this attack and it’s effect on the industry has served to cause site operators to seriously question the value of CAs and look at alternative trust models. As explained in my other post, the current CA trust model requires users to trust their browser, operating system, and device vendors to include a list of ‘trusted’ CA root certificates in their products. Site operators which wish to use SSL certificates to secure their websites must purchase certificates from one (or more) of these CAs, during which they prove domain ownership and, in the case of extended validation certificates, additional authentication information. In this model, note that end users (who have the most to lose from accessing fraudulent websites), have no power to choose who they trust (the decisions are made for them by site operators and browser vendors). Nominally, this is actually a good thing since end users generally can’t be expected to have enough information to make sensible trust decisions.

The biggest weakness that this trust model has exposed as a result of the Comodo incident is the difficulty in providing an effective way for fraudulently issued certificates to be revoked. While both CRLs and OCSP were used to revoke the certificates, the fact is that many end user devices do not regularly fetch certificate revocation information due to complexities with maintaining effective access to CRL/OCSP responders and continuing debate on how CRL/OCSP failures should be handled in browsers.

With that background, a couple of alternative trust schemes are being proposed within the IETF. Perhaps the most promising scheme is DANE (DNS based Authentication of Named Entities). The cool thing about DANE is that it leverages another major security protocol overhaul that’s been happening for the last 8 years or so (DNSSEC). Essentially, DANE provides a mechanism for a domain registrar to add a certificate/public key for a given website to that site’s DNS record. Since DNS lookups themselves will be secured using the underlying DNSSEC mechanisms (and the contents of the DNS records will be authentic and integrity protected via DNSSEC services), the DNS records are as safe a place as any to store key material used to secure a site (or any client resource that can use keys to secure connections). The controversy that I’m seeing with the DANE proposals is that the trust model doesn’t require a CA to issue certificates used to secure a site. In fact, DANE fully supports the notion of self signed certificates. The idea is that if you trust that the owner of the domain has already passed necessary validation to get a DNSSEC secured DNS record, why can’t you simply trust the owner to provision a key for his/her site especially since that key is protected by DNSSEC resources?

There are still lots of open questions and discussions around DANE, and the CA vendors are making strong arguments that their services are still required even in a DANE enabled ecosystem (the anti-fraud checks performed by CAs before issuing certificates are unlikely to be performed by DNS registrars and obviously a site operator would not need to perform such checks).

I know that I’ll be following the development of DANE and discussions on the IETF DANE lists.

The evils(?) of self signed certificates

I consider myself largely a lurker on most security mail lists that I subscribe to, including the closed SANS Advisory Board mail list. However, a recent discussion on that list prompted me to think about the supposed risks of self signed certificates, especially in an enterprise network.

Like nearly any other security technology out there, self signed certificates can be evil or perfectly acceptable depending on the context in which they are used. In a nutshell, self signed certificates alone provide confidentiality and integrity to any protocols which use them to establish connections. Self signed certificates alone do not provide authentication in the traditional sense of a PKI, because the certificate has not been issued by another trusted party (such as a centrally managed PKI infrastructure).

So, the question is, does this lack of authentication make self signed certificates evil? I don’t believe that it does. A certificate can be authenticated by other means. For example, if the self signed certificate is installed on a end user system by some other trusted mechanism (such as a centralized directory/management server), the centralized management server is vouching for the self signed certificate. In another extreme example, a user could contact the administrator of a system that is using a self signed certificate and verify the certificate fingerprint being presented by the self signed certificate.

The real risks of self signed certificates are that end users will become accustomed to simply adding self signed certificates to their browser’s trusted certificate store and will ignore the security warnings presented by their browsers over time. The sanctity of browser errors generated by untrusted certificates must be preserved to ensure that users view those errors as exceptional events and respond accordingly.

The Chicken and the Egg Problem; Trusting Wireless Networks

It’s been quite a while since I’ve posted; since then my job has changed and I’m back to system design work. In my new role, one of the problems that I’ve had to solve recently is a familiar chicken and egg problem that anyone who’s had to deploy wireless networks that use enterprise authentication methods (EAP-TLS/EAP-TTLS) has had to face.

The issue surfaces when using an enteprise authentication method that relies on X.509 certificates for server authentication (like EAP-TLS, EAP-TTLS, and PEAP). When a wireless device (here wireless can mean WLAN and wireless broadband technologies) connects to an AP and begins the authentication process, the supplicant on the device needs to verify that the authenticator (typically the RADIUS server behind a wireless AP) is a trusted endpoint. This is done by verifying that the authenticator’s X.509 certificate has been issued by a trusted root certificate authority and is valid (digital signature is valid, cert hasn’t expired, plus any other local policy checks). However, these checks won’t properly handle the case where the authenticator’s certificate has been revoked by the issuing CA. While this is a low risk in most enterprises, in high security or mobile deployments (like the system I’m currently working on), the risk of the AAA server being compromised may be great enough to justify implementing some sort of countermeasures. The typical solution for checking for revoked certificates is to use certificate revocation lists or online certificate status protocol (OCSP). However, both of these methods are problematic in a wireless system for different reasons:

  • CRLs have a defined validity period after which the CRL should no longer be trusted (certificates revoked after the CRL expiration date will naturally not appear in the CRL). While this is normally not a problem, if a wirelss device does not attach to the network regularly, it could potentially fail to download new CRLs and cause the existing CRLs on the device to expire.
  • OCSP is a simple request/response protocol where a device can send a request to a OCSP responder immediately after obtaining a certificate to check that the certificate is valid. However, during the wireless network entry process, the wireless device has to trust the authenticator’s certificate to even obtain an IP address which means that the device cannot use OCSP to validate the certificate until it has already ‘trusted’ the certificate (hence, the chicken and egg problem).

There are a couple of solutions to this problem; ask the authenticator to contact an OCSP responder on the device’s behalf to find out if the authenticator’s certificate has been revoked, or connect to the network temporarily to download CRLs and then reconnect to the network to validate the authenticator’s certificate with the newly downloaded credentials.

The first solution above is implemented via a technique called OCSP Stapling (RFC 4366). OCSP stapling defines a means for an authenticator to contact an OCSP responder to get a OCSP response containing the server certificate’s revocation status. Wireless supplicants which support OCSP stapling will then expect to get a OCSP response in the certificate payload messages that they obtain from servers. However, OCSP stapling is relatively new and has not been widely deployed.

The second solution is somewhat more attractive than OCSP stapling because it does not require the authenticator to implement any additional functionality and requires only minor changes in the wireless supplicant. In this solution, the wireless supplicant will first connect to the network and will trust the authenticator’s certificate for the purposes of obtaining CRLs (or to optionally contact a OCSP responder and cache the response). After the supplicant has obtained the revocation information, the supplicant will immediately disconnect from the network and will peform another wireless association and authentication sequence. However, this time the supplicant can use the freshly obtained revocation information to validate that the authenticator’s certificate has not been revoked before permitting full network entry.

Neither of these solutions are perfect of course; they both require the wireless supplicant to trust the authenticator to a certain extent and don’t protect against the threat where the wireless device is associating with a “complete” rogue wireless network (where the AAA server, DNS server, CRL server and OCSP responder functions have all been spoofed). However, in both cases the overall system security has been improved rather significantly by providing a way for wireless network operators to revoke AAA server certificates with assurance that wireless supplicants will just ‘do the right thing’.