SSL, certificate, and key management enhancements for even stronger security in WebSphere Application Server V6.1

[뤼순신]

SSL, certificate, and key management enhancements for even stronger security in WebSphere Application Server V6.1

http://www.ibm.com/developerworks/websphere/techjournal/0612_birk/0612_birk.html

Peter Birk and Keys Botzum
Published on December 06, 2006

Share this page

FacebookTwitterLinked InGoogle+E-mail this page

3

From the IBM WebSphere Developer Technical Journal.

In previous releases of IBM WebSphere Application Server, SSL management is based on a collection of SSL configurations that are referenced by an alias name from end points throughout the cell. Each alias is prefixed with the node name reflecting where it was created. The SSL runtime for a given end point reads the SSL configuration that it references and creates the SSL sockets from this information. A set of default self-signed certificates located in the DummyServerKeyFile.jks and DummyClientKeyFile.jks keystores are shipped with each installation. While this is simple, it is not secure and should not be used this way in production. Managing certificates requires the use of an external key management tool called IKeyMan. There are references to SSL configurations in every server.xml throughout the system, so the ability to reorganize the SSL configurations when topologies change can be difficult. Applications do not have the ability to access these SSL configurations.

In WebSphere Application Server V6.1, SSL management adds an entire new level of simplicity, security, and flexibility.

• Unique self-signed certificates are created for each profile during profile creation. When a profile is federated into a cell, trust is automatically established with the cell. These self-signed certificates are managed by a fully integrated certificate management infrastructure that replaces the external IKeyMan tool. The expiration of these certificates is monitored on a pre-defined schedule with notifications to system logs and email-sending capabilities. The certificates will be automatically replaced before expiration, by default, and, there will of course be a warning prior to the certificate replacement.

• Additional JSSE key and trust management function is added to the SSL runtime for even more customized key selection and trust decisions. Management of the SSL configurations is tightly integrated in the runtime in such a way that JSSE system properties are no longer required for the default socket factories. While the SSL configurations can still be referenced by the alias name, they can also be managed by scope using inheritance to broaden or tighten the association of an SSL configuration for use by a specific end point, cluster, server, node, node group, or the entire cell. In effect, SSL trust zones can be established more easily in the new centralized management panels.

Build security into applications by design

Secure workloads against the latest threats, simplify access control, and comply with regulatory requirements efficiently with Security services in IBM Bluemix.

Sign up for a free trial

There's much more to describe, so read on for the details.

Information on what to be careful about and what the known limitations and problems are for the various features will be discussed in this article. Examples will be provided when appropriate, and the use of screen shots for showing what is being discussed will be used when necessary. This article is made up of the following sections, each of which focuses on different areas of consideration:

1. SSL configuration -- Relevant to all WebSphere users and explains the facilities for configuring and using SSL in WebSphere Application Server. Understanding this information is critical for all WebSphere administrators.

2. Certificate management -- Describes some of the more advanced WebSphere Application Server features and describes the more complex issues related to certificate management. Most administrators will not need to address these issues, but we include them because of their importance.

3. Programmatic SSL usage considerations -- Brief highlights of some changes and features related to using SSL from application code.

4. Key and certificate generation at initial profile creation -- Describes how certificates are initially created for profiles.

5. Node federation and migration -- Discusses adding nodes to existing cells and migration issues.

6. Managing the lifecycle of cryptographic keys -- Discusses features in WebSphere Application Server that manage the change of cryptographic keys over time.

This article assumes basic knowledge of how SSL works. It is beneficial to reference WebSphere Application Server V6.1 Information Center articles for the basic information about these new features. Additionally, some of the terminology used in this article is defined in the Terminology section at the end of the article.

SSL configuration

For an overall view of the new SSL management capabilities in the administrative console, see Figure 1, which shows the main SSL panel that is reached by selecting the SSL certificate and key management link on the console. It is important to note that the configuration links under Related Items will only show those configuration objects that are cell-scoped (for WebSphere Application Server Network Deployment) or node-scoped (for WebSphere Application Server base), meaning they are visible to every process in the cell or node, respectively. The way scoping works: when you scope something to the cell, everything below it can view and use that configuration object; however, the cell-scoped views cannot see objects that are scoped below it (this is intentional for run time isolation). Therefore, these links provide a subset of the configuration. To see configuration objects that are scoped lower than this, you will need to select the Manage end point security configurations link, which shows a topology view that enable scoping of configuration objects. This will be discussed in more detail later on.

Figure 1. Main SSL certificate and key management panel

View image at full size

What is an SSL configuration?

An SSL configuration is used to encapsulate all the things needed by JSSE (the SSL implementation used by WebSphere Application Server) to securely connect to (or be connected from) another SSL-enabled end point. The same SSL configuration can be used for either outbound (client) or inbound (server) connections. Figure 2 shows the SSL configuration panel. One advanced feature that was not present in the previous releases of WebSphere Application Server is the ability to plug in custom trust and key managers. This will be discussed later in more detail.

Figure 2. SSL configuration panel

View image at full size

Before going into more detail on the SSL configuration, we will briefly summarize some of the more important SSL concepts and terms that will be used throughout this article:

• A key store contains the personal certificates that can be used as the identity for the SSL end point referencing the key store. If more than one certificate is present, a certificate alias on the SSL configuration specifies one of the personal certificates. When an SSL connection is made (on either the client or the server side), certificates may be exchanged. The personal certificate referenced by the SSL configuration and stored in the key store is the certificate that will be used.

• A personal certificate represents the identity of the end point and contains a public and private key for signing/encrypting data.

• A trust store contains the signer certificates which this end point trusts when either making connections (from an outbound end point) or accepting connections (for an inbound end point).

• A signer certificate represents a certificate and public key associated with some personal certificate. The purpose of the signer certificate is to verify personal certificates. By accepting the signer certificate into an end point's trust store, you are allowing the owner of the private key to establish connections with this end point; that is, the signer certificate explicitly trusts connections made to or by the owner of the associated personal certificate. The signer certificate is typically made completely public by the owner of the personal certificate, but it's up to the receiving entity to determine if it is a trusted signer prior to adding it to the trust store.

The out-of-the-box SSL setup for a WebSphere Application Server Network Deployment (hereafter referred to as Network Deployment) cell has a cell-scoped (meaning visible to all processes in the cell) SSL configuration called CellDefaultSSLSettings. This configuration contains node-scoped (visible to all end points on that specific node) SSL configurations called NodeDefaultSSLSettings for every node in the cell. SSL configurations contain many resources, which we will discuss shortly, but the two most important parts of an SSL configuration are the key store and trust store, which contain personal certificates and signing certificates, respectively. To ensure that all nodes can communicate with each other and with the deployment manager, there is a default cell level trust store, which is pointed to by all of the default node level SSL configurations, and includes the signers for all node level personal certificates, as well as the deployment manager. In general, each node has its own personal certificate stored in a node level key store. The deployment manager also has its own personal certificate, which is stored in its own key store. Later we will discuss how new nodes are federated into the cell.

Figure 3 shows the key store configuration from a node perspective:

• The collection of key stores shown here are those that can be used (as key or trust stores) by end points in this node. Other nodes may have a different selection of key stores.
• The topology view enables you to view configurations at various scopes to see the selection capabilities at those scopes.
• CellDefaultKeyStore and CellDefaultTrustStore are present when the deployment manager profile is created.
• NodeDefaultKeyStore and NodeDefaultTrustStore key store configurations are added when the node is federated.

Figure 3. Key store configuration panel from a node scope

View image at full size

Figure 4 shows the default personal self-signed certificate that is created for every unique profile. As mentioned earlier, the signer for this certificate can be found in the CellDefaultTrustStore after federation into the cell. For situations that arise where a signer is missing, the CellDefaultTrustStore is probably the trust store you should add the signer to since most SSL configurations by default use this trust store. The error message that appears from an SSL connection will typically point to the trust store that needs the missing signer when a handshake fails for this reason.

Figure 4. Default personal certificate for every profile

View image at full size

Quality of Protection settings

In addition to the personal and signer certificate configuration, there are other parts of an SSL configuration that need to be considered.

• The SSL handshake protocol determines the syntax of how a handshake is made. Typically, the more advanced the protocol version, the more secure the handshake. TLSv1 is the most secure handshake and therefore should be used whenever possible. However, TLSv1 does not interoperate with SSLv3 which, unfortunately, is the more commonly used protocol. To address this common problem, the IBMJSSE2 provider includes another protocol named the SSL_TLS. This protocol performs a handshake first using TLSv1; if that does not work, it falls back to SSLv3 and so on. This is the default SSL handshake protocol for WebSphere Application Server V6.1 SSL configurations due to its ability to interoperate well with most protocols at the other end of the connection. For security reasons, WebSphere Application Server will not use SSLv2 by default unless explicitly configured to do so.

• A cipher suite group is a WebSphere-specific configuration attribute which groups cipher suites based on strong, medium, or weak strength. The default is "Strong," which typically chooses 128-bit cipher suites. The "Medium" group chooses 40-bit cipher suites, and "Weak" typically just performs digital signing (no data encryption). Figure 5 shows the quality of protection capabilities. It's important to realize that these cipher groups are proper subsets of the available cipher suites. Therefore, setting one side of a connection to "Strong" and another side to "Medium" will not find a common cipher suite and the handshake will fail. If you want a particular combination of ciphers, use the Custom setting and apply those specifically that you want. Otherwise, "Strong" is the recommended setting for all end points.

• Client authentication determines whether SSL client end points are required to send a certificate to the server. For this to work, the server must have in its trust store a signing certificate that can validate the client certificate. In some scenarios, it's more secure when both sides of a connection authenticate using certificates; however, this can impose a management burden, which is why client authentication is off by default. When using certificates that are generated by a certificate authority (CA) or signed by a common signer certificate, it is highly recommended to enable SSL client authentication, since the trust is already established based on the CA root certificate. However, if self-signed certificates are being used, adding a large number of signers to the trust store can cause scalability issues by slowing down the handshake during certificate validation.

Figure 5. Quality of Protection panel for SSL configurations

View image at full size

Centrally-managed SSL configuration selection capabilities

Managing SSL configurations centrally is a new capability that makes it possible to control the run time usage of SSL configurations based on topology. This run time association is set from a topology panel that shows all management scopes in the entire cell. These definitions control the effective SSL configuration used for any inbound or outbound end point. The effective SSL configuration is based on the lowest scope defined for a particular end point, all the way up the hierarchy, based on precedence rules of inheritance. Centrally managing SSL configurations is desirable for the following reasons:

• Easier to take into account the proper trust requirements for all end points in the environment.
• Easier to ensure the same handshake protocol and cipher requirements are specified.
• Enable a more knowledgeable security administrator to be responsible for all SSL changes.
• Can maintain fewer SSL configurations with more flexibility because of the inheritance rules defined in the centrally-managed capabilities.
• Much simpler to make SSL changes based upon topology changes.
• The centralized management can be used to rapidly establish trust zones.

Figure 6 shows how the default SSL configuration is displayed in the centrally-managed topology view. From this view you can see all of the overrides that are specified that define the inheritance rules. To see the effective SSL configuration based on these overrides, you can click on the link for any process or end point and see the effective SSL configuration for that entity. For the centrally-managed configurations to be effective, however, there must not be any direct alias references located at an end point configuration. For example, if an SSLChannel has a direct reference to an SSL configuration, that will override the centrally-managed associations. One can control whether an end point is centrally-managed by marking it as such at each end point configuration, as shown in Figure 9.

Figure 6. Overview of the centrally-managed topology view

View image at full size

Figure 7 shows how you can determine what the inherited SSL configuration currently is for the selected scope and also how to override the inherited SSL configuration by specifying a different configuration directly at the scope being visited. Again, notice the Related Items link on the right; when visiting these links and creating any new configuration objects, you will be creating them at the management scope that you are currently visiting. This means these configuration items are only visible and selectable at this scope and below. If you go back to the cell-scoped link, you will not see these new configuration items, as they are at lower scopes. To see all configuration items that apply to a particular inheritance hierarchy, view the Related Items link from an end point panel, as shown in Figure 7.

Figure 7. Override of the inherited SSL configuration at end point WC_defaulthost_secure

View image at full size

Dynamic outbound SSL configuration selection capabilities

There are some situations where a static SSL configuration for a given end point is not sufficient to meet the requirements for a particular connection. For example, you may have an IIOP static SSL configuration for all internal WebSphere Application Server communications, but there may be an application within the environment that needs to connect to some third-party IIOP server. While it's probably possible to alter the static configuration, used for WebSphere server-to-server communications, it may be more desirable for satisfying the requirements of the third-party server to use a separate SSL configuration so as to not modify the original static SSL configuration's behavior.

Figure 8 shows the creation of a dynamic outbound end point SSL configuration. This configuration object enables the specification of one or more selection criteria to be associated with a specific SSL configuration and certificate alias. When the selection of an SSL configuration occurs for a particular outbound connection using the JSSEHelper API, the connection information passed into the API is used to check the selection criteria to determine if a match is made. If so, the dynamic SSL configuration is used and the selection search is ended. Dynamic selection hits and misses are cached to improve performance.

Figure 8. Creating a dynamic outbound selection configuration

View image at full size

Direct selection SSL configuration

If you decide to manage your SSL configurations using direct selection, Figure 9 shows how this can be done. The example shows the selection for an LDAP registry, but the same concept applies to all SSL end points where the choice is provided. This feature works exactly as it has in previous releases, subject to SSL configuration selection rules that we will discuss shortly.

Figure 9. Direct selection choice of an SSL configuration at end point configurations

View image at full size

Programmatic SSL configuration selection capabilities

While not the focus of this article, SSL configuration selection can also be controlled programmatically. First, application programs can explicitly use the JSSEHelper API provided by IBM to access SSL configurations, as well as to obtain SSLSocketFactories using those configurations. SSL configurations can also be specified on a thread using the JSSEHelper API prior to an outbound connection. This is useful when using existing code that naively uses the default JSSE provider. We will discuss this more later.

SSL configuration precedence rules

By now, it should be clear that there are multiple ways of controlling which SSL configuration is used. It is quite possible that more than one configuration might apply for a particular usage of SSL. To resolve this ambiguity, the WebSphere Application Server runtime uses the following precedence rules when selecting which SSL configuration to use. This applies for WebSphere Application Server SSL usage, the usage of SSL from application code when using the default JSSE provider, and when application code uses the JSSEHelper API directly. The precedence rules are (the first match in this list wins):

1. thread-based selection -- Properties have been specified on the thread programmatically by an application that is directly using the IBM JSSEHelper API. Later implicit usage of JSSE (that is, opening an HTTPS connection with a URL provider) implicitly uses the thread specific settings set by the JSSEHelper API.

2. dynamic outbound selection -- A dynamic outbound configuration has been found that matches this particular outbound connection information. This applies for both application code and WebSphere Application Server's own usage.

3. direct selection -- A WebSphere Application Server SSL configuration has been directly specified for this end point. This can be done administratively for WebSphere Application Server SSL usage. Application programs can also reference SSL configuration directly using the JSSEHelper API.

4. scoped selection -- None of the previous apply, so the topology-based SSL configuration selection method applies.

Although it may be confusing at times, this approach gives you tremendous flexibility. One item to be very conscious of is the fact that dynamic end point specification overrides both direct end point specification and scoped selection. This gives the administrator tremendous ability to change how SSL is used, but can lead to surprising results.

Now that we have covered the key features related to SSL configuration and selection, we will turn our attention to the features related to key store and certificate management.

Certificate management

Default certificate configuration

In previous releases, WebSphere Application Server included default certificates in key stores called DummyServerTrustFile.jks and DummyServerKeyFile.jks, located in the profile's /etc directory, which included the dummy certificate (including a common public and private key). These files are still included with WebSphere Application Server V6.1 for backwards compatibility for applications that are referencing them at this location; however, they are no longer used.

During profile creation, a unique self-signed certificate is generated for each profile and is located in the key.p12 key store. The signer certificate is extracted from this personal certificate and added to the trust.p12 key store. These key stores are created in a location in the configuration repository, either in the cell or node directories, enabling them to be managed from the administrative console or wsadmin, and synchronized to the proper nodes. A typical default WebSphere Application Server configuration will contain key stores and trust stores like this:

• Cell-scoped key and trust stores
  • C:\WebSphere\AppServer\profiles\Dmgr01\config\cells\Machine40Cell01\key.p12
  • C:\WebSphere\AppServer\profiles\Dmgr01\config\cells\Machine40Cell01\trust.p12
• Node-scoped key and trust stores
  • C:\WebSphere\AppServer\profiles\Dmgr01\config\cells\Machine40Cell01\nodes\Machine40Node01\key.p12
  • C:\WebSphere\AppServer\profiles\Dmgr01\config\cells\Machine40Cell01\nodes\Machine40Node01\trust.p12

These are the default key stores created during profile creation for a cell profile. You can configure key stores at any location of any type that are supported in the key store configuration panels to further customize your SSL configurations. The types of key stores supported depend upon your platform.

Since the key stores are now managed by WebSphere Application Server as first class parts of the admin configuration, there are a number of enhancements you can now leverage. Some of the more interesting features are discussed next.

Basic certificate management

You can manage certificates and their key stores from within the administrative console. The key management utilities in WebSphere Application Server V6.1 are based upon the IKeyMan functionality from previous releases. The same terminology is used and similar views are displayed. The fundamental certificate management capabilities include creating self-signed certificates, certificate signing requests, signer certificate management, and so on.

Obtaining signer certificate from a remote port

Normally, to configure the WebSphere Application Server runtime to use SSL to another server you must manually obtain the other server's signing certificate and then import it into the appropriate trust store. However, there is a feature in WebSphere Application Server now to make this easier. Instead, the WebSphere Application Server administrator can obtain the signing certificate directly from the server by using the retrieve from port option (an SSL connection is opened to get that certificate). The administrator is asked to validate that the certificate is correct by verifying the SHA digest and then, optionally, store it in the local trust store. This greatly simplifies the signer exchange process for setting up SSL to external servers, such as LDAP. Figure 10 shows the setup of the remote host and SSL port information to which WebSphere Application Server will connect to retrieve the signer certificate. As you can see in the diagram, the administrator inputs the hostname and port and then clicks the Retrieve signer information button.

Figure 10. Remote signer certificate retrieval‎

View image at full size

If the signer certificate was successfully retrieved, the certificate information -- including the SHA digest of the certificate that was used to ensure it has not been modified in transit -- is displayed for verification. Figure 11 shows the panel that enables the administrator to accept the retrieved signer certificate before it gets added to the trust store, in this case NodeDefaultTrustStore.

Figure 11. Validating a remotely-retrieved signer certificate

View image at full size

Managing certificates for Web server plug-ins

Do not confuse the management of the key store file stored in the WebSphere Application Server configuration repository with the management of the actual Web server plug-in's key store file. Here we are discussing how the WebSphere copy of the plug-in key store file is maintained. You are still responsible for ensuring that this updated key store file is copied appropriately to the Web server whenever it is changed. This task is typically performed manually, although the WebSphere Application Server Web server management feature can also be used -- but, for security reasons, it is rarely appropriate.

WebSphere Application Server also provides for managing the Web server plug-in's key file. When a Web server definition is created, a CMSKeyStore configuration is created that references a CMS key store (also known as a KDB file) and CMS password stash file (also known as an STH file), located in the config/cells/${CELL_NAME}/nodes/${NODE_NAME}/servers/${WEB_SERVER_NAME} directory. By default, these are named "plugin-key.kdb" and "plugin-key.sth", respectively. When created, the key store includes the node's personal certificate and all signers that currently exist in the cell's common trust store (the trust.p12 file in the cell directory). This enables the Web server plug-in to initiate SSL connections out-of-the-box with all internal HTTPS ports opened in the application servers within the current cell. When a new node is federated after the Web server definition has been created, the signer for the new node needs to be added into existing plug-in key store. With WebSphere Application Server V6.1.0.4, this file will be updated automatically as part of the federation; prior to that, this task must be performed manually, so we strongly recommend you get that fix.

If for some reason you want to manage the plug-in keystore manually, you can do so using the same certificate management capabilities as any other key store. The Web server plug-in configuration panels link back to the certificate management panels, located under SSL certificate and key management. Figure 12 shows the Web server Plug-in properties panel that enables you to edit the plug-in key store.

Figure 12. Plug-in properties panel for a Web server definition

View image at full size

Replacing certificates and signers

In any environment using certificates, it is necessary from time to time to update personal certificates and their corresponding signers (for self-signed certificates). This can, of course, be done manually by looking at every trust store in the cell for the corresponding signer. Instead, WebSphere Application Server provides function to simplify this task.

Figure 12 shows the Replace certificate panel, shown after selecting the Replace button, with a single self-signed certificate selected in the Personal certificates collection view. Once presented with this panel, you can specify a different certificate in the same key store that should be used to replace the personal certificate just selected. The old certificate will be removed and all existing SSL configurations that point to that certificate (via the alias) will be updated to use the new certificate's alias. In addition, all trust stores that contained the old signer will be updated to contain the new signer. WebSphere Application Server finds the old signer by matching on the SHA signature on the old signer and searching all trust stores in the cell. Finally, the process can optionally delete the old signer and the old personal certificate. It is generally a good idea to leave the old signer in the trust store until you are certain all corresponding personal certificates have been removed.

Figure 13. Replacement of self-signed certificate

View image at full size

While the certificates are being changed, there will likely be transient SSL problems because of key incompatibility. To minimize problems, if dynamic SSL updates are enabled, the SSL server authentication is temporarily disabled for 90 seconds to allow time for node synchronization of the key stores to complete, and for every running process to load updated information. Therefore, it's important that you perform dynamic SSL configuration updates during times when there is little traffic on the network. Should a node not get updated in the time needed for the synchronization to complete, or it is not started when the changes are made, stop the node and perform a manual syncNode to get the new changes synchronized to the node. If dynamic SSL updates are not enabled, servers will pick up the changes when they are restarted.

Certificate expiration monitor and dynamic run time updates

The behavior described here will be implemented in WebSphere Application Server by APAR PK34093 (likely in V6.1.0.6), and so you are strongly encouraged to upgrade to this fix level when it is available.

As should not be surprising, certificates expire. Should a certificate expire, SSL communication using that certificate will be impossible, which will almost certainly result in a system outage. WebSphere Application Server tries hard to prevent these outages, and when it cannot prevent them, it tries to at least warn you before they occur. The certificate expiration monitor task runs on a configurable schedule which, by default, is every 14 days. The Next start date field for the monitor is persistent in the configuration and is updated with a new date each time it runs. It will execute in the deployment manager process in a Network Deployment environment, or -- if standalone -- in the WebSphere Application Server base process.

When executing, the expiration monitor will search through all KeyStore objects configured in the cell, looking for any personal certificates that will expire within the expiration threshold (90 days being the default; this is configurable via a custom property). If it finds any, it will issue a notification warning of the impending expiration. Notifications are always sent to the serious event stream to all registered listeners. By default, this is the admin console and SystemOut.log. Notifications can also be sent via email using an SMTP server.

In addition to notifications, WebSphere Application Server will attempt to replace self-signed certificates before they expire. By default, the expiration monitor will execute the certificate replacement task (mentioned in the previous section) against any self-signed certificates 15 days before expiration (this is configurable). The task creates a new certificate using the certificate information from the old one, and updates every trust store in the cell that contained the old signer with the new signer certificate. By default, the old signer certificate will be deleted.

The expiration monitor marks any SSL configuration as "modified" whenever the monitor changes the key store or trust store referenced by the configuration. The configuration changes are saved once the expiration update task is completed, causing a ripple to occur throughout the runtime. The first thing that happens is the temporary disabling of SSL server authentication (for 90 seconds) to enable these changes to occur without requiring a server restart. In cases where you do not want this to occur, consider disabling the Dynamically update the run time when SSL configuration changes occur option located at the bottom of the SSL certificate and key management panel in the admin console.

Unfortunately, automatically replacing certificates is not a panacea. WebSphere Application Server cannot update certificates in key stores that are not under its control. In particular, this means that a Web server plug-in that is using the previous soon-to-expire signing certificate will stop working when the corresponding personal certificate is replaced. It also means that if WebSphere Application Server was using the personal certificate to authenticate with some other system, the certificate replacement will cause an outage. Keep in mind that this outage would have occurred anyway -- it is just occurring 15 days sooner, and after WebSphere Application Server has sent multiple warnings of this impending outage. WebSphere Application Server is simply doing its best.

To reduce the risk of an outage from the default WebSphere-created certificates, know that, by default, they are good for 15 years. Of course, a long certificate lifetime does raise some risks, and for those customers concerned about such risks, we encourage you to manually replace the default certificates more frequently.

It should be obvious that letting WebSphere Application Server automatically change the expiring certificates in a production environment is risky, since it could potentially cause a short- or long-term outage. Instead, you should change certificates manually when you are notified of their impending expiration. Automatic replacement is primarily intended to simplify management for less complex environments, and for development systems where brief outages are acceptable. For most production environments, we recommend that you instead monitor and act on the expiration notification messages and disable automatic replacement of self-signed certificates. Figure 14 shows the configuration panel for the certificate expiration monitor.

Figure 14. Certificate expiration monitor configuration panel

View image at full size

Troubleshooting SSL issues

Troubleshooting an SSL connection problem can be quite difficult. Fortunately, a great deal of effort was put into WebSphere Application Server V6.1 for improving the quality of error messages, making it easier to diagnose (and ultimately fix) SSL-related issues. Listing 1 shows an example of one of the most common errors that occurs when attempting to establish an SSL connection from a client to a server. In this case, the server has sent a certificate that is not recognized by the client; that is, the client's trust store does not contain the corresponding signer. The error message provides detailed information on this error, which should make it easier to correct. Notice that it indicates the host:port, the missing signer, the SSL configuration, and even the trust store that is being used. This tells the administrator precisely what needs to be done: in this case, the missing signer needs to be obtained and added to the trust.p12 trust store specified.

Listing 1. Error that occurs during outbound connection when missing a signer certificate

1 2 3 4 5 6 7 8

CWPKI0022E: SSL HANDSHAKE FAILURE:  A signer with SubjectDN "CN=192.168.1.204, O=IBM, C=US" was sent from target host:port "192.168.1.12:9403".  The signer may need to be added to local trust store "c:/WebSphere/AppServer/profiles/Dmgr01/etc/trust.p12" located in SSL configuration alias "DefaultSSLSettings" loaded from SSL configuration file file:c:\WASX_a0633.07\AppServer\profiles\Dmgr01/properties/ssl.client.props".  The extended error message from the SSL handshake exception is: "No trusted certificate found".

We have now covered the normal certificate/key store management issues that are relevant to most WebSphere Application Server administrators. We will now going to turn our attention to more advanced issues related to certificate management. As these topics are quite complex, we will only highlight these other features. See Related topics to learn more.

Advanced certificate and key management issues

Trust manager enhancements

When using certificates, a trust manager is responsible for performing certificate validation to determine if the certificate is acceptable and meets whatever security requirements are in force. WebSphere Application Server provides two trust managers:

• The default trust manager, IbmX509, performs fundamental certificate validation, including the certificate signature validation (ensuring it has not been modified) and certificate expiration validation (ensuring it has not expired). This trust manager does not perform hostname verification by default, although you can set the com.ibm.ssl.performURLHostNameVerification=true property in the security custom properties to enable this function for URL connections only. If this is done, the trust manager will ensure that for URL connections, the hostname specified on the connection matches the SubjectDN in the certificate returned by the server (just as Web browsers do).

• The other JSSE trust manager you can choose is the IbmPKIX trust manager. In addition to performing the functions listed above by default (except that hostname verification still needs to be explicitly enabled for HTTPS URL connections), this trust manager performs advanced certificate revocation list (CRL) checking using either CRL distribution point (DP) extensions (supported by most CAs) or online certificate status protocol (OCSP) system properties. When the IbmPKIX trust manager is selected as your default, CRL DP processing is enabled automatically. This will perform CRL checking for all certificates that contain CRL distribution points. If CRL checking should occur more frequently, system properties can be set in the process configuration to enable OCSP. (Although this information is for an IBM JDK, it applies to any WebSphere Application Server installation, since IBM adds cryptographic functionality, including PKIX, to any JDK bundled with WebSphere Application Server.)

If you need more specialized certificate validation for your environment, you can even write your own custom trust manager. You can plug in any number of custom trust managers to execute after the IBM-provided trust managers have run. When implementing a custom trust manager, there are two interfaces you should consider:

• The typical JSSE trust manager interface is javax.net.ssl.X509TrustManager, which provides the basic methods that will be called by the JSSE runtime during an SSL handshake.

• To obtain additional information about the SSL context, you can optionally implement the com.ibm.wsspi.ssl.TrustManagerExtendedInfo interface.

Advanced signer exchange management for clients using the trust manager infrastructure

By default, the trust store used by clients includes the trusted signer for the local profile. If a client needs to communicate with other WebSphere servers besides the one from the initial WebSphere Application Server base profile, then some additional work needs to be done to establish this trust. When deciding how to establish trust between V6.1 clients and servers, consider these possibilities, in order by level of simplicity:

1. Enable the signer exchange prompt by setting the com.ibm.ssl.enableSignerExchangePrompt property to true in the ssl.client.props file, located in the profile's /properties directory. This is enabled by default. After establishing a new connection where the client does not already have the signer in the local trust store, this prompts the user (when enabled) to accept the signer. The user will be presented with information about the certificate, including the SHA digest of the certificate. This is the same concept that you see in a browser client for trusting certificates. For more information see Secure installation for client signer retrieval‎ in the Information Center.

2. Download all the signers from the common cell trust store and then update the client trust store with these signers. This is accomplished by using the retrieveSigners.bat (or .sh) script.

3. Extract the signer certificate from the server manually and import it into the trust store of the client.

4. Use programming APIs in your application client for programmatically establishing trust for specific connections -- or permanently for all connections (by saving the signer in the trust store). This eliminates manual intervention, which the signer exchange prompt and retrieveSigners script inherently requires. This capability is intended for automated test environments or any environment that is trusted enough to enable the deferral of SSL server authentication. This approach should not be used in a production environment. For more information, review the com.ibm.wsspi.ssl.RetrieveSignersHelper SPI in the Information Center.

Listing 2 shows the programmatic calls necessary to implement option 4 above. This effectively disables SSL server authentication for these calls and should be considered only in trusted environments. This example shows three separate approaches:

1. Programmatically calling the retrieveSigner function to bulk load all cell certificates.
2. Temporarily accepting the next server certificate.
3. Accepting the next server certificate and storing it persistently for future use.

Listing 2. Using RetrieveSignersHelper SPI for establishing trust in a trusted environment

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

import com.ibm.wsspi.ssl.RetrieveSignersHelper;   A)/*** Obtain an instance of the helper class. ***/   RetrieveSignersHelper rsHelper = RetrieveSignersHelper.getInstance();   /*** Retrieve all cell level signers programmatically by passing in all of the same parameters you would pass in from the command line.  These arguments will download all signers from the "CellDefaultTrustStore" into the "ClientDefaultTrustStore".   The -autoAcceptBootstrapSigner is important to specify here in that it accepts the signer necessary to make the original connection to begin with.   ***/   String[] args = new String[] {"CellDefaultTrustStore", "ClientDefaultTrustStore", "-autoAcceptBootstrapSigner"};   rsHelper.callRetrieveSigners (args);     B) /*** An alternative to calling the retrieveSigners capability up front, is to allow trust in specific connections or the capability to download and possibly store the signer for a particular connection.  Here we accept temporarily the next server certificate presented by the next SSL communication on this thread. */ rsHelper.autoAcceptSignerForThisConnectionOnly();   /*** OR ** This is just like the above accept that the signer is stored persistently in the trust store.  It will thus be used for all future requests. ***/   rsHelper.autoAcceptSignerAndStoreInTrustStore();   /*** Note that these two calls apply only to the next SSL communication. If multiple servers need to be accepted these APIs will need to be called prior to each communication. ***/

Customized key managers

The key manager for JSSE can be customized in WebSphere Application Server V6.1. Only one key manager is permitted at any one time, so it will either be the default IbmX509 key manager (which knows how to select a certificate from the WebSphere-configured key stores), or a custom key manager that you configure. A pluggable key manager can be used on clients or servers.

It may be desirable to define a custom key manager if you wish to override how the runtime finds certificates. The most likely scenario is on the client side; perhaps there is a need to support smart cards or enable the user to select the certificate to use for a particular connection.

Using RetrieveSigners script to update trust stores for server environment

We just discussed using the RetrieveSigners script to download signers from a server to a client. The RetrieveSigners script determines what trust store to update by reading the configuration information in the ssl.client.props file. While the script is intended to update the client side trust stores, there are some situations where you may want to download signers from the trust store of a server (or cell) into the trust store of another server (or cell). You might do this if, for example, you are trying to configure cross cell communication.

Of course, you can manually export the signers from one cell and import them into the second cell's trust store. However, you can also "trick" the RetrieveSigners script to do this for you.

To download signers into a server trust store, you can edit the ssl.client.props trustStoreName property to temporarily point to the appropriate trust store; most likely, this is the common trust.p12 located in the cell directory. Listing 3 shows an example of updating the ssl.client.props to reference the cell level trust store. Of course, this example must be run on the DeploymentManager node so that the master copy of the cell trust file is actually updated.

Listing 3. Downloading signers into server trust store using RetrieveSigners

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

/*** Edit the properties/ssl.client.props file prior to running RetrieveSigners.   Change from:   # TrustStore information com.ibm.ssl.trustStoreName=ClientDefaultTrustStore com.ibm.ssl.trustStore=${user.root}/etc/trust.p12   Change to:   # TrustStore information com.ibm.ssl.trustStoreName=ClientDefaultTrustStore #com.ibm.ssl.trustStore=${user.root}/etc/trust.p12 com.ibm.ssl.trustStore=${user.root}/config/cells/Machine40Cell01/trust.p12   Now run the RetrieveSigners script pointing to a different Cell or Server that contains a different trust.p12 than this one.   ***/   C:\WebSphere\AppServer\profiles\Dmgr01\bin> retrieveSigners.bat CellDefaultTrustStore ClientDefaultTrustStore –autoAcceptBootstrapSigner –host otherdmgr.raleigh.ibm.com –port 8879 –user myadmin –password myadminpwd   CWPKI0308I: Adding signer alias "CN=machine40.austin.ibm.com, O=IBM, C=US" to local keystore "ClientDefaultTrustStore" with the following SHA digest:            A7:87:EA:E0:FA:26:38:F6:00:F7:CD:0C:88:6A:85:62:BC:D7:17:3D CWPKI0308I: Adding signer alias "dummyclientsigner" to local keystore            "ClientDefaultTrustStore" with the following SHA digest:            0B:3F:C9:E0:70:54:58:F7:FD:81:80:70:83:A6:D0:92:38:7A:54:CD CWPKI0308I: Adding signer alias "agentcert" to local keystore            "ClientDefaultTrustStore" with the following SHA digest:            88:12:C9:6D:3F:0E:9A:72:25:87:35:4F:BB:42:6D:A5:A8:62:89:D5 CWPKI0308I: Adding signer alias "dmgrcert" to local keystore            "ClientDefaultTrustStore" with the following SHA digest:            98:B3:A1:9C:16:DF:F8:A9:C2:31:12:A8:29:CC:D5:82:EE:F2:3A:F2 CWPKI0308I: Adding signer alias "dummyserversigner" to local keystore            "ClientDefaultTrustStore" with the following SHA digest:            FB:38:FE:E6:CF:89:BA:01:67:8F:C2:30:74:84:E2:40:2C:B4:B5:65

Once you have completed downloading the necessary signers, you will, of course, want to undo the changes to the ssl.client.props file. Also, don't forget to do a full node synchronization to force the updated trust file to be replicated to other nodes in the cell.

Isolating a key store configuration to a particular node

Ordinarily, key stores are managed by the deployment manager in the configuration repository and replicated from there to various nodes in the cell. There are situations where you may want to isolate (or scope) a key store so it is only stored on a single node. For example, the key store may contain sensitive keys that should only be seen by that node and should not even be replicated by the deployment manager. This is a very rare situation, but if this requirement applies, this can be addressed.

Figure 15 shows the creation of a key store configuration that is node-scoped and remotely managed. The reason that it is remotely managed is because it is not stored in the WebSphere configuration repository, making it impossible for the deployment manager to directly manipulate the key store. Instead, the deployment manager uses remote JMX calls to ask the node agent on the target node to manage the key store.

To create a key store configuration that is only visible to end points on a specific node:

1. Use the topology view from the Manage end point security configurations link.
2. Select the appropriate management scope for the node in question, then click on New.
3. From the panel shown in Figure 15, you can now create a new key store for this node. Because the path to the key store in this example is c:\temp, the key store will not be in the configuration repository and thus won't be replicated by the deployment manager. For the deployment manager to manage this key store at all, it must be remotely managed.

We've shown here a simple example with a file-based key store. However, the same applies for cryptographic token devices (for example, hardware key stores). Since these physical devices are accessible only from the node, hardware key stores must be remotely managed, and should be created from the Node link in the topology view or lower, depending upon your isolation needs.

Figure 15. Creating a remotely-managed keys

View image at full size

Programmatic SSL usage considerations

Although the focus of this article is not on the programmatic use of SSL, we felt it was important to briefly highlight some of the changes that might impact existing applications that use SSL. These changes are generally for the better, but as a WebSphere user you still need to be aware of them. There are two key areas of change:

1. New programming APIs enable application developers to directly use the WebSphere SSL configurations.
2. The behavior of the default JSSE provider has changed, as described earlier.

JSSE APIs

WebSphere Application Server V6.1 introduces a new helper API called com.ibm.websphere.ssl.JSSEHelper that acts as glue between the JSSE programming model and the WebSphere SSL configuration and runtime. Applications can take advantage of this API the same as the WebSphere SSL runtime, or they can continue using JSSE APIs directly and manage the configuration themselves, as done in previous releases. Figure 16 shows the interaction between JSSEHelper, SSL configuration, and the JSSE programming model for obtaining constructs such as an SSLContext, SSLSocketFactory, SSLServerSocketFactory, and HTTPS URLStreamHandler. There are several ways to leverage this API. We will describe just a few use cases.

Figure 16. JSSEHelper's role in the SSL runtime

View image at full size

If an application is about to make an SSL connection using a piece of code that implicitly uses the default SSL socket factory (for example, a URL connection), the application can still control what SSL configuration is used by using the thread-based SSL selection feature. Essentially, the JSSEHelper class can set the SSL configuration that should be used on the next SSL connection from the current thread. Listing 4 shows an example.

Listing 4. Example of programmatically specifying SSL properties on the thread

1 2 3 4 5 6 7 8 9

import com.ibm.websphere.ssl.JSSEHelper;   /*** For this example, just hard code an alias.  ***/ String alias = "CellDefaultSSLSettings"; java.util.Properties props = JSSEHelper.getProperties(alias); JSSEHelper.setSSLPropertiesOnThread (props);   /*** Make any URL connection here, for example, and the properties will be picked up by the WebSphere socket factory.  ***/

In Listing 4, the CellDefaultSSLSettings are going to be used to make a secure outbound URL connection. This works because the default URLStreamHandler for the https protocol calls SSLSocketFactory.getDefault(). This will then use the WebSphere JSSEHelper API to determine the configuration to use.

In our second example, we will use the JSSEHelper getProperties() API to specify an SSL configuration to use. SSL configuration selection follows the rules discussed earlier in the SSL Configuration Precedence Rules section. In Listing 5, notice that we create various housekeeping objects in advance and also provide an SSL listener. The listener is is a WebSphere-specific extension, and will be notified when the SSL configuration is changed, giving you the opportunity to do something about it. Null can be used instead if event notification is not desired.

Listing 5. Example of JSSEHelper to obtain an SSL Socket Factory

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

import com.ibm.websphere.ssl.JSSEHelper;   String alias = "aliasforsomeconfiguration"; String host = "myhost.ibm.com"; String port = "443";   /*** Create a map of information to tell the SSL runtime the connection information***/   final HashMap connectionInfo = new HashMap(); connectionInfo.put(JSSEHelper.CONNECTION_INFO_DIRECTION,      JSSEHelper.DIRECTION_OUTBOUND); connectionInfo.put(JSSEHelper.CONNECTION_INFO_REMOTE_HOST, host); connectionInfo.put(JSSEHelper.CONNECTION_INFO_REMOTE_PORT, port);               /*** An optional listener can be passed into the API to get notifications when the returned configuration changes.  ***/ MySSLConfigListener myListener = new MySSLConfigListener();   /*** obtain the SSL connection ****/ javax.net.ssl.SSLSocketFactory sslFactory = JSSEHelper.getInstance().getSSLSocketFactory      (alias, connectionInfo, myListener);

The third example (Listing 6) is essentially similar to the previous example, except that here, rather than obtaining an SSL factory directly, properties about an existing configuration are obtained. These properties could then be modified in some custom way and then the JSSEHelper could be used again to obtain a slightly custom SSL socket factory.

Listing 6. Getting SSL configuration properties

1 2 3 4 5

/*** Obtain the SSL connection information as a properties object  ***/   java.util.Properties props = JSSEHelper.getInstance().getProperties      (alias, connectionInfo, myListener);

The fourth example (Listing 7) shows how to use the JSSEHelper to select an end point, using the dynamic selection capabilities. We do not specify an alias and instead rely on the WebSphere runtime to find an appropriate SSL configuration. As you might imagine, more combinations are possible. The JSSEHelper API is quite flexible.

Listing 7. Example of JSSEHelper's SSL configuration dynamic selection capabilities

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

import com.ibm.websphere.ssl.JSSEHelper;   String host = "myhost.ibm.com"; String port = "443"; /*** The endpointName is arbitrary but can be used for the dynamic outbound configuration. ***/ String endpointName = "ConnToMyHost";    /*** Create a map of information to tell the SSL runtime the connection information for making a dynamic outbound selection decision.   Pass in "null" if you do not care about the dynamic outbound configurations being considered.  ***/   final HashMap connectionInfo = new HashMap(); connectionInfo.put(JSSEHelper.CONNECTION_INFO_DIRECTION, JSSEHelper.DIRECTION_OUTBOUND); connectionInfo.put(JSSEHelper.CONNECTION_INFO_REMOTE_HOST, host); connectionInfo.put(JSSEHelper.CONNECTION_INFO_REMOTE_PORT, port); connectionInfo.put(Constants.CONNECTION_INFO_ENDPOINT_NAME, endpointName);               /*** An optional listener  **/ MySSLConfigListener myListener = new MySSLConfigListener();   javax.net.ssl.SSLSocketFactory sslFactory = JSSEHelper.getInstance().getSSLSocketFactory (null, connectionInfo, myListener);

Readers interested in how JSSE worked in previous releases should refer to the developerWorks article Using the Java Secure Socket Extension in WebSphere Application Server.

Changes to the default SSLSocketFactory and SSLServerSocketFactory

The key change that might impact existing applications is the behavior of the Java SSLSocketFactory.getDefault() method. Many applications use this API, either explicitly or implicitly. It is used implicitly in many places where an SSL connection is being made but no SSL factory is specified; this happens with URL providers, LDAP providers, some Web services, and in many other places.

In a standalone JDK, the SSLSocketFactory.getDefault() method returns a default SSLSocketFactory, which is initialized by the JDK to use the default key and trust stores. These are typically set as JDK system properties, as described in the previously mentioned JSSE article. In a WebSphere server process, system properties impact the entire JDK and all applications running inside the JDK.

In WebSphere Application Server V6.1, the default JDK socket factories have been replaced by WebSphere implementations that have more control over how the configuration information is interpreted, and then can create the JSSE socket factories based on this configuration, rather than using static system properties for everything. Not only does this give the administrator more control -- even of naive applications -- but it also prevents conflicts with the WebSphere Application Server runtime's use of SSL.

A naive program that uses SSLSocketFactory.getDefault() will actually be using the WebSphere SSL socket factory. This factory honors all of the SSL configuration selection rules we described in the SSL Configuration Precedence Rules section, with one exception: if the JDK system properties for key stores are set, they will override what WebSphere Application Server would have done. This is for backward compatibility, so do not depend on this behavior; better to let WebSphere Application Server manage your SSL configurations and key stores.

Key and certificate generation at initial profile creation

So far, we have discussed the steady state default configuration of keys, certificates, and SSL. An obvious question to ask is how these items are created in the first place. As you might imagine, they are created during the profile creation process. Profile creation tasks establish the default configuration for the profile being created. Some of the configuration information is specified in the profile creation panels while other information is derived from the environment. Most of the information is based on templates for the type of profile that is being created. The templates and the environment establish the profile's default configuration documents.

One of the major requirements for the WebSphere Application Server V6.1 release is to remove the default certificates (also known as the dummy key files), and replace them with real self-signed certificates generated during profile creation (unique private keys for the profile). This default configuration can be used in a production environment, as it is secure by default. There are three initial configuration profiles, each with slightly different initial configurations:

• A Base Application Server profile is created with a key.p12 (PKCS12 format) key store and trust.p12 (PKCS12 format) trust store in this configuration directory: ${USER_INSTALL_ROOT}/config/cells/${CELL_NAME}/nodes/${NODE_NAME}/. The default password for these key stores (and those for the other profiles) is always "WebAS." Notice that this approach provides for some isolation of key information between nodes. Should this base server be federated into a Network Deployment cell, the self-signed certificate for this node will be unique to it, and its certificate will be in a node-specific directory.

• A Deployment Manager profile is created with key.p12 and trust.p12 key stores containing a self-signed personal certificate and signer, respectively, in this directory: ${USER_INSTALL_ROOT}/config/cells/${CELL_NAME}/. The trust.p12 key store located in this directory is referred to as the common trust store.

• A Cell profile (containing both a deployment manager and a node) is created with a single self-signed certificate for both the deployment manager and the node, but these will be stored in separate key stores (node-scoped key stores for the node, cell-scoped key stores for the deployment manager). In this case, this means that the deployment manager and one node in the cell happen to share the same self-signed certificates. This may be surprising and can lead to unexpected behavior when replacing certificates. When replacing a self-signed certificate for a node that shares the same certificate as the cell or the cell key store, do not delete the signer in the cell-scoped trust.p12 -- as it is still being used by the node -- unless you replace the self-signed certificate in both locations (node and cell key stores). To avoid this unexpected sharing, it may be desirable to simply create a deployment manager profile and then a base application server profile.

For all profiles, when these self-signed certificates are created they are also added into the key.p12 and trust.p12 in the ${PROFILE_ROOT}/etc directory. These key stores are used by clients (for example, wsadmin) started from this profile. These certificates provide them with the trust needed to communicate with servers in the same profile without requiring any signer exchanges to occur.

While this configuration will not suit every environment, it will be sufficient for most. However, as with almost everything else in WebSphere Application Server, if you don't like the defaults, you can change them.

Customizing self-signed certificate creation

During profile creation, the self-signed certificate that is created for the profile is created with the following default values:

Listing 8. Properties for Customizing Self-Signed Certificate Creation

1 2 3 4

com.ibm.ssl.defaultCertReqAlias=default com.ibm.ssl.defaultCertReqSubjectDN=cn=${hostname},o=IBM,c=US com.ibm.ssl.defaultCertReqDays=5475 com.ibm.ssl.defaultCertReqKeySize=1024

The value of defaultCertReqDays will be changed to 5475 days (15 years) with WebSphere Application Server APAR PK34093 (likely in V6.1.0.6). Earlier versions used a default value of 365 days, which could cause production issues due to the short lifetime.

These property settings will not be found in the server configuration templates, but they can be overridden by specifying new values as custom properties. To override the default values used during profile creation, edit the security.xml file in the ${WAS_INSTALL_ROOT}/profileTemplates directory and specify only those properties that you want to override in the custom properties section. For example, to modify these properties for a "default" profile (Base Application Server), you can add these properties to the following security.xml file prior to creating the profile: ${WAS_INSTALL_ROOT}\profileTemplates\default\documents\config\cells\BaseApplicationServerCell\security.xml. In this example, we decided to change the certificate lifetime for new profiles to 1825 days (5 years), overriding the IBM provided default:

Listing 9. Property Change Necessary to Override Default Self-Signed Certificate Expiration Period

1 2 3

<properties xmi:id="Property_45"      name="com.ibm.ssl.defaultCertReqDays"      value="1825" required="false"/>

When a base server profile is created, this setting will ensure that the initial self-signed certificate will have a lifetime of five years. Depending on your security policies, you may wish to change the certificate lifetime from the IBM defined default of 15 years. Shorter lifetimes improve security, but remember that certificate expiration can cause outages if not properly managed.

Setting the SubjectDN of the created certificates is also fairly straightforward, but notice that, by default, the certificate includes the hostname of the server. The ${hostname} variable is resolved using the JDK java.net.InetAddress.getLocalHost().getCanonicalHostName() API. If this does not resolve to a value that you want (or perhaps you prefer that hostnames not be embedded in certificates), specify a value for the SubjectDN that does not reference the ${hostname} variable.

Web server definition KeyStore creation

When creating a profile, it is also possible to create a Web server definition at the same time, as shown in Figure 17. When this occurs, the profile creation process will put the profile's self-signed certificate into a CMS keystore located in the Web server's server directory in the profile configuration repository. This means that the Web server will share the same self-signed certificate as the initial node.

Figure 17. JSSEHelper's role in the SSL runtime

View image at full size

LTPA key generation

In addition to the various SSL self-signed certificates that are created during profile creation, the default LTPA keys are also generated during profile creation time. One significant change from previous versions of WebSphere Application Server is that the LTPA encryption keys are no longer stored in the security.xml file, but rather they are managed in a key store. Thus multiple versions of LTPA keys to be active simultaneously, enabling validation of older tokens using the old keys, and providing a seamless way to generate new keys without causing problems in the runtime. This will be discussed more in the Managing the Lifecycle of Cryptographic Keys section.

The LTPA key store is named ltpa.jceks and is in the JCEKS format, which makes it possible to store keys as raw "secret" keys. As with other key stores, the default password for the LTPA key store is "WebAS". The location of the key store depends on the profile type:

• For a Base Application Server profile, the ltpa.jceks key store is located in the node directory of the profile configuration repository.
• For a Deployment Manager, the ltpa.jceks key store is located in the cell directory of the profile configuration repository.

Troubleshooting profile creation key and certificate generation

As with any other activity, it is possible for things to go wrong. During profile creation, regardless of whether security is enabled in the profile, an AdminTask is executed to generate the keys and certificates previously discussed; the AdminTask is either PrepareKeysForCellProfile (called when a Cell profile is created) or PrepareKeysForSingleProfile (called for all other profile types). Both of these tasks create a log file in the ${WAS_INSTALL_ROOT}/ logs/manageprofiles/${PROFILE_NAME}/keyGeneration.log. Successful execution of this task will result in the last line of this log file being "KeyStore creation and certificate exchange successful for Cell profile." If this statement or the keyGeneration.log file does not appear, then a problem occurred during profile creation either before or during this task execution.

As it turns out, the security tasks just discussed also happen to be the first configuration task to invoke the wsadmin client during profile creation. As a result, failures in this task often have nothing to do with security, but rather are an indication of a problem with wsadmin. If a keyGeneration.log has not been created, the problem is likely not related to the key generation functions. If a keyGeneration.log has been created, then it may be related to the key generation function, but also may be a problem in wsadmin. To further debug these issues, look in the file ${WAS_INSTALL_ROOT}/logs/manageprofiles/${PROFILE_NAME}_create.log. This is the generic profile creation log file. Search for FAIL somewhere in the log.

Additionally, you can enable trace to get more information. To enable more tracing, edit the ${WAS_INSTALL_ROOT}/profileTemplates/<profile_type>/documents/properties/wsadmin.properties file to enable trace by uncommenting the following line:

#com.ibm.ws.scripting.traceString=com.ibm.*=all=enabled

Of course, you'll want to edit this file prior to starting profile creation. This will produce a wsadmin.trace log file in the ${PROFILE_ROOT}/logs/ directory as part of the profile creation process, the failure of which is likely captured in this trace file unless there's a problem with the profile management tool itself.

Now that we have discussed the initial configuration state of new profiles in WebSphere Application Server V6.1, we will turn our attention to what occurs when nodes are added to existing cells.

Node federation and migration

In this section, we will cover the Network Deployment node federation step that causes a Base Application Server (standalone server) to become a node in a Network Deployment cell. For the default SSL configuration, some automated steps occur during this process to ensure the self-signed certificate created for the Base Application Server profile is trusted by other servers that already exist in the cell, and vice versa.

The goal is to ensure that all servers trust each other by default, so that no additional steps need to be taken by an administrator when using a default out-of-the-box configuration. This is not always possible when more complex mixed-version and/or mixed-platform environments are included. Some manual signer exchange may need to occur, but messages emitted from the WebSphere Application Server V6.1 runtime makes it easier to determine which keystore is missing the required signer, based on the Subject DN from the remote certificate.

This section will first discuss the steps that occur during node federation of a typical homogeneous environment in an all WebSphere Application Server v6.1 cell (no back-level nodes or mixed-platform nodes present). We will then address issues that may need manual intervention due to the complexities of heterogeneous node platforms and/or versions.

Security configuration merge of V6.1 nodes

When a V6.1 base server is federated into an existing cell, the node's personal certificate is added to the cell level trust store, and the node's security configuration is updated to reference the cell level trust store. This ensures that the new node can communicate with other nodes in the cell. Other changes are also done to update the security configuration of the node to ensure it is properly a member of the cell.

Once the federation is complete, the configuration repository is synchronized to the node prior to the node agent being started. This ensures the node agent and the application servers have access to the correct updated configuration information (including the signers and LTPA keys) needed to securely communicate to the rest of the cell. Other cell members are notified of the updated SSL configuration information. If the SSL dynamic configuration changes check box is enabled, all keystore and SSL caches are cleared and the new configurations are loaded by all of the processes across the cell. Thus, the new node should seamlessly become part of the cell without having to restart other servers. Figure 18 shows the Dynamically update the run time when SSL configuration changes occur check box that needs to be enabled to ensure a seamless federation that does not require any server restarts. When this option is disabled, the deployment manager and other nodes may need to be restarted for communications to occur to the new node.

Figure 18. Dynamically update the run time when SSL configuration changes occur

View image at full size

Back-level node federation

When the node being federated is from an earlier version of WebSphere Application Server, the node is unaware of the new self-signed certificate scheme. It is also unaware of the new SSL configuration objects, the new LTPA key store, and so on. As much as possible, the node federation process tries to address this using transformation code that transforms the security.xml file from the V6.1 format into the format the back-level node is expecting.

During federation of a back-level node, if the node is still using the default DummyServerKeyFile.jks and DummyServerTrustFile.jks, these key stores will be automatically replaced with the cell-scoped key.p12 and trust.p12 key stores, respectively. This enables the back-level node to communicate with the V6.1 deployment manager and other nodes in the cell. However, if the back-level node is not using the default key and trust stores, then a manual signer exchange will need to occur before successful node synchronization will occur. This manual process is essentially the same as what would have been done when adding nodes to an existing cell in previous versions of WebSphere Application Server with non-default key stores.

Mixed-platform node federation

For most WebSphere Application Server platforms there are no significant differences here. However, because WebSphere Application Server on z/OS® leverages the z/OS advanced security features (including certificate management), there are some additional complexities when z/OS nodes and non-z/OS nodes are mixed in the same cell.

There are two issues that are unique to z/OS:

• z/OS already contains a PKI environment using RACF (or some other SAF product) to generate certificates using a CA.
• The z/OS RACF key stores are not writable, which makes it impossible for WebSphere Application Server to update the key stores as part of the federation process. For the node agent to successfully communicate with the deployment manager after federation, you will need to manually add the distributed signer certificates (from the cell trust store) into the RACF keyrings, and the RACF signer into the cell trust store of the distributed cell.

If the z/OS node or dmgr profile was created using the family security option (that is, it is using the key.p12 and trust.p12 keystores instead of the RACF keyrings), then you still may need to manually add the trust.p12 signer into the z/OS Daemon SSSL configuration keyring. In WebSphere Application Server V6.1, the z/OS Daemon process contains the only end point still using the z/OS System SSL (SSSL) facility for the SSL runtime. All other end points use the JDK IBMJSSE2 SSL implementation. However, the IBMJSSE2 SSL configurations on z/OS typically still use the RACF keyrings containing RACF generated certificates. This is something configured in the customization dialogs when creating a profile in z/OS.

Migration

When migrating nodes (that is, converting them from one version to another version), different rules are followed. One of the fundamental migration rules is to not change the behavior of the configuration, whenever possible. Thus, migrated cells do not take advantage of the automated self-signed certificate generation. As little change as possible is made to migrated cells. Migration changes the format of the old-style SSL configurations to the new format in the security.xml file. Migration does not change the certificate or key stores that are referenced. If the deployment manager was using the dummy certificates pre-migration, then it will still be using the dummy certificates post-migration. Of course, there still are advanced certificate management features which can be used to simplify the creation of certificates after migration. Create unique certificates in place of the dummy certificates for any uses in production environments.

Because there is a mixture of SSL configurations in a migrated environment, whenever federating a new node into a migrated cell, a manual syncNode many need to occur in order to get the new node synchronizing properly. In some cases, manual signer exchanges will be required to get things working properly. This is a fundamental SSL management requirement, and quite frequently needs a human to make the trust decisions. Whenever it believes the certificate and/or key store configuration has been customized, the migration and merge code will always err on the side of no signer exchange so as to not make trust decisions that are unexpected.

Managing the life cycle of cryptographic keys

One fundamental principle of cryptographic keys is that they should be changed from time to time. Recognizing this need, WebSphere Application Server now includes functionality for managing the lifecycle of cryptographic keys. This new function is used to manage the lifecycle of the LTPA cryptographic keys. However, this functionality is also exposed for application use.

Previous releases stored only one set of LTPA keys. When those keys changed in the runtime, the runtime would remove the old keys and then use only the new keys. Since it is generally a good idea to change the LTPA encryption keys periodically, this behavior would result in outages. When the keys are changed, all existing LTPA tokens (for example, those cached in Web browsers) are no longer usable, as they cannot be decrypted.

In WebSphere Application Server V6.1, a generic key management lifecycle infrastructure has been created not only for use by LTPA, but also for applications that need to perform cryptography. This framework will help manage the key references that are active and regenerate new keys on a scheduled timeframe. For LTPA, the default configuration is to generate new keys every 90 days and recognize two LTPA encryption keys at any given time. The new encryption keys are used for generating/validating new tokens, and the old one for validating old tokens. Since LTPA tokens are normally only valid for a few hours, this approach ensures that updating the encryption keys results in no downtime at the price of a bit of extra cryptographic work. As you might imagine, WebSphere Application Server first tries to decrypt LTPA tokens using the new key and then, if that fails, tries the old key. This has some cost but it is fairly small, since there are only two key versions supported and presumably a rapidly declining number of LTPA tokens created with the previous key version.

Figure 19. Key lifecycle management infrastructure

View image at full size

Figure 19 shows how the com.ibm.websphere.crypto.KeySetHelper API interacts with the key management runtime and configuration. The com.ibm.websphere.crypto.KeySetHelper API is used internally by the LTPA token components. It can also be used by applications to request keys, although we won't be discussing that further. There are a few fundamental parts to this:

• Key stores that hold multiple versions of the cryptographic keys.
• Key generators that generate the keys. KeyPairGenerator for public/private key pairs and KeyGenerator for secret keys.
• Key sets that contain and manage the keys and use key generators to generate them.
• Key set groups that schedule and manage the generation of new keys across multiple key sets.

We will look at this now in a bit more detail.

Using KeySets to manage keys

KeySets are the configuration objects for keeping track of a specific key type. The key type can be a symmetric secret key or an asymmetric key pair (public and private). The KeySet manages the KeyStore the keys will be stored in, the aliases which are actively being referenced, and the implementation class that determines the details of how the key(s) are generated.

What a KeySet does not do is determine attributes of the key, such as the key length, algorithm, and so on; it only knows if the key is a key pair or secret key so it knows how to store the keys. Key pairs are special because they have two associated keys and need to be stored under a similar alias name for retrieval‎. A key pair is managed as one key reference by the KeySet.

Figure 20 shows the information that is tracked in a KeySet configuration. Once the KeySet is configured, it is typically placed in a KeySetGroup, which controls the scheduling of key generation, among other things, discussed below.

Figure 20. KeySet configuration panel

View image at full size

Using KeySetGroups to organize KeySets

KeySetGroups manage one or more KeySets. This is the case because multiple key sets may be related in some way and they all need to be updated at the same time. KeySetGroups can be used to update the contained KeySets, either manually or automatically on a scheduled basis.

When scheduled updates are in use, the scheduler will create a "next start date" value that is saved in the configuration and visible in the console Automatic key generation occurs in the deployment manager (for Network Deployment) or Base Application Server (for standalone) once this next start date is reached. If either of these processes are not started once the "next start date" time has elapsed, the key generation will be executed the next time the processes startup and a new "next start date" will be calculated. Figure 21 shows the panel for configuring a KeySetGroup.

Figure 21. KeySetGroup configuration panel

View image at full size

Use of KeySetGroup for managing keys by LTPA

Careful readers may notice that the LTPA password is no longer required when configuring LTPA. Instead, you only need to specify an LTPA password when importing or exporting LTPA keys. The LTPA password is only used to encrypt the keys when exported into a properties file that is being used to configure cross-cell SSO.

As already mentioned, LTPA keys are auto-generated during profile creation. After that, the LTPA keys are managed using either CellLTPAKeySetGroup (for Network Deployment) or the NodeLTPAKeySetGroup (for WebSphere Application Server base). Figure 22 shows the authentication mechanisms panel, available by navigating to Secure administration, applications, and infrastructure => Authentication mechanisms and expiration. Notice that the key set group in use is indicated.

Figure 22. LTPA reference to the CellLTPAKeySetGroup

View image at full size

Configuring LTPA cross-cell single sign-on

Unfortunately in life, nothing is free. While the new approach to LTPA key management is an enormous improvement in security, it does have implications for cross-cell single sign-on (SSO). Cross-cell SSO relies on the different cells sharing the same LTPA encryption keys -- that's how the cells can trust each other's LTPA tokens.

To configure cross-cell SSO, as with previous releases, you must export the currently active LTPA keys from one cell and import them into another. However, there is one very important configuration change that must be made: automatic key regeneration must be disabled. If you don't do this, when the cells automatically generate new versions of their LTPA encryption keys, they will lose the ability to communicate, since their LTPA encryption keys aren't the same.

Look again at Figure 22 and notice the Automatically generate keys check box for the CellLTPAKeySetGroup. Disable this check box and save the configuration in both cells prior to exchanging the LTPA keys. You can still manually generate new LTPA encryption keys from time to time in one cell and then import them into the other cell. This way, you can maintain SSO and change the keys.

Conclusion

This article provided an overview of the changes made in WebSphere Application Server V6.1 to enhance the SSL runtime and configuration. By default, WebSphere Application Server is more secure than ever before, and makes it easier for you to adapt to topology changes by centrally managing the configuration and making the SSL configuration more topology aware. These changes have also established a foundation for future enhancements to further simplify and enrich the SSL, certificate, and key management functions.

Terminology

Definitions to some of the fundamental terminology used in this article is provided below.

• SSL -- Secure socket layer; a protocol used to establish authenticity, integrity, and confidentiality over various other protocols using signing and encryption.

• Authenticity -- The process of determining that one side of the connection or the other is in fact who they say they are, based on proof established by what they have (a signed certificate). By default, SSL clients authenticate the server, but the SSL servers do not authenticate the clients. You can enable "mutual" authentication, which authenticates certificates in both directions.

• Integrity -- The process of verifying that the data, when transmitted, is not modified in flight. This is done by creating a digest of the data at the sending side and comparing the digest of the data at the receiving side. The two digests must match exactly to ensure integrity.

• Confidentiality -- The process of ensuring that someone cannot read the information on the wire, thus the need to encrypt. The encryption process creates cipher text from plain text. The cipher text is sent across the wire and is not humanly readable. The cipher text is then decrypted using a shared key, negotiated during the SSL handshake, to return to plain text that is humanly readable (at least in the format it started in).

• X509Certificate -- An object that represents the identity of some entity or person. This contains a distinguished name (DN) and other extensions (attributes) that can be used, such as e-mail, and so on. The X509Certificate is associated with a public and private key: the public key is inside the certificate while the private key is maintained in tight control by the entity which the certificate represents. The private key should never be shared, but it is perfectly fine to give the public key to another entity for them to trust you.

• A key store (in JSSE terms) stores the personal certificate, which represents the X509Certificate, public key, and private key. This is the representation of the identity of this entity.

• A trust store (in JSSE terms) stores the X509Certificate and public key only (also referred to as a signer certificate). The trust store must contain all signer certificates from all other entities that it is trusting to make connections to or with. Without the signer of the remote entity, an SSLHandshakeException occurs with a message stating "No trusted certificate found."

• A key manager is code that selects the certificate from the specified key store during the SSL handshake, based on the signers available by the target server sent over during the handshake. Some key managers make certificate selections based on configured properties. WebSphere Application Server uses com.ibm.ssl.keyStoreClientAlias (for client certificate selection) and com.ibm.ssl.keyStoreServerAlias (for server certificate selection).

• A trust manager is code that makes trust decisions during a handshake. The default IbmX509 trust manager performs certificate validation such as signature validation and expiration checks. The default IbmPKIX trust manager performs the same validation, plus more advanced certificate revocation checking (CRL), which determines if the certificate has been removed by the certificate authority (CA). The CRL distribution point (DP) extensions can be used to obtain a CRL when a certificate is received. This automatically occurs when the IbmPKIX trust manager is selected and this extension exists. Additionally, the Online Certificate Status Protocol (OCSP) can be used to perform an online check of certificate validity; this requires additional system properties to be specified, as documented in the JDK CertPath API docs on IBM developerWorks.