Add Failover and Load balancing to Your CXF application with Open Source

In the next blogs I'd like to describe how easy it is to add other enterprise features to your existing CXF based applications. My previous blogs were always very focused on security. The enterprise features I'd like to talk are:

• load-balancing
• failover
• end-to-end monitoring

In this blog, I'll address load-balancing and failover whereas end-to-end monitoring is addressed in the next blog.

The assumption is that the web services are deployed on several machines. You want to either load balance the requests among the deployed services dynamically or failover in case a service is not reachable anymore.

To show how this can be implemented I'll use the example wsclientWebapp of the Apache CXF Fediz project which has been released recently.

This example already supports Web SSO (IDP), WS-Security and STS. The architecture is described in a previous blog. More information on how to build, deploy and test the wsclientWebapp example are described in its README.

This example has the following gaps to be deployed in a distributed environment:

• URL's of the web service is configured on the service consumer side which is difficult to manage across the environments and for all web service consumers
• If an error occurs you have to analyze log files to drill down the root cause of the issue. Usually, the log files of all involved applications are distributed in the network. It's difficult to correlate the messages in the different log files.

These issues might be manageable within the scope of a project but not on the enterprise level as your application might consume several services which are used by other applications as well. As mentioned at the beginning, the second gap will be addressed by implementing end-to-end monitoring which is described in the next blog.

Talend built Apache licensed open source components which address the previously mentioned gaps and add load balancing and failover capabilities to your CXF application.

Update your CXF application

Service Providers register its endpoints (URL) at the Talend Service Locator and service consumer can lookup for an service endpoint (compare to DNS or UDDI). The Service Locator is Apache licensed and re-uses other proven open source projects like Zookeeper. Besides the server component, the CXF Failover/Clustering feature has been extended to integrate with the Service Locator.

What do you have to do in your application?

A) Web Service Provider

Make the following changes in the maven project wsclientWebapp/webservice/service

1) add POM dependency to Talend Locator library

This library hooks into the CXF Failover/Load Balancing functionality to integrate with the Service Locator server.

  <dependency>
    <groupId>org.talend.esb</groupId>
    <artifactId>locator</artifactId>
    <version>5.1.1</version>
  </dependency>

2) add a file locator.properties to your maven project src/main/resources

This file contains information where the Service Locator server is running.

# Configured zookeeper instances (divided by a comma if several instances uses). 
# The service locator client will pick an instance
# to connect to the service locator until a connection is established. 
locator.endpoints=localhost:2181

# Endpoint prefix property is needed because the services
# in a container where the endpoints
# are relative to the container.
endpoint.https.prefix=https://localhost:10443/fedizservice

connection.timeout=5000
session.timeout=5000

3) Update Spring configuration file applicationContext.xml

The highlighted lines must be added to the example application.

...
   <import resource="classpath:META-INF/cxf/cxf.xml" />
   <import resource="classpath:META-INF/tesb/locator/beans.xml" />
...
   <!-- GreeterService -->
   <jaxws:endpoint id="GreeterService"
      implementor="org.apache.cxf.fediz.examples.service.GreeterImpl"
      wsdlLocation="WEB-INF/wsdl/hello_world.wsdl"
      serviceName="svc:GreeterService"
      xmlns:svc="http://apache.org/hello_world_soap_http"
      address="/GreeterService">

      <jaxws:properties>
         <entry key="ws-security.signature.properties" value="stsKeystore.properties" />
         <entry key="org.talend.tesb.endpoint.secured" value="true"/>
      </jaxws:properties>

      <!-- Talend feature -->
      <jaxws:features>
         <bean class="org.talend.esb.servicelocator.cxf.LocatorFeature" />
      </jaxws:features>
   </jaxws:endpoint>

4) Run "mvn clean install"

Maven builds a new WAR package. This package is now able to register its endpoint at the service locator.

B) Web Service Consumer

Make the following changes in the maven project wsclientWebapp/webapp

1) add POM dependency to Talend Locator library

This library hooks into the CXF Failover/Load Balancing functionality to integrate with the Service Locator server.

Some transitive dependencies have to be excluded as they are already part of the fediz package which is available in the parent classloader of tomcat.

  <dependency>
    <groupId>org.talend.esb</groupId>
    <artifactId>locator</artifactId>
    <version>5.1.1</version>
      <exclusions>
        <exclusion>
          <artifactId>xmlsec</artifactId>
          <groupId>org.apache.santuario</groupId>
        </exclusion>
        <exclusion>
          <artifactId>wss4j</artifactId>
          <groupId>org.apache.ws.security</groupId>
        </exclusion>
        <exclusion>
          <artifactId>ehcache-core</artifactId>
          <groupId>net.sf.ehcache</groupId>
        </exclusion>
      </exclusions>
    </dependency>

2) add a file locator.properties to your maven project src/main/resources

This file contains the network address where the Service Locator server is running.

# Configured zookeeper instances (divided by a comma if several instances uses). 
# The service locator client will pick an instance
# to connect to the service locator until a connection is established. 
locator.endpoints=localhost:2181

connection.timeout=5000
session.timeout=5000

There are different options to configure the cache of endpoints. Please check the Talend_ESB_InfrastructureServices manual for all configuration options.

3) Update Spring configuration file applicationContext.xml

The highlighted lines must be added to the example application.

...
   <import resource="classpath:META-INF/cxf/cxf.xml" />
   <import resource="classpath:META-INF/tesb/locator/beans.xml" />
...
  <jaxws:client id="HelloServiceClient" serviceName="svc:GreeterService"
    xmlns:svc="http://apache.org/hello_world_soap_http"
    serviceClass="org.apache.hello_world_soap_http.Greeter"
    address="locator://whatever"
    wsdlLocation="WEB-INF/wsdl/hello_world.wsdl">
    <jaxws:properties>
      <entry key="ws-security.sts.client">
        <bean class="org.apache.cxf.ws.security.trust.STSClient">
          <constructor-arg ref="cxf" />
          <property name="wsdlLocation" value="https://localhost:9443/fedizidpsts/STSServiceTransport?wsdl" />
          <property name="serviceName"
            value="{http://docs.oasis-open.org/ws-sx/ws-trust/200512/}SecurityTokenService" />
          <property name="endpointName"
            value="{http://docs.oasis-open.org/ws-sx/ws-trust/200512/}Transport_Port" />
          <property name="onBehalfOf" ref="delegationCallbackHandler" />
          <property name="enableAppliesTo" value="true" />
        </bean>
      </entry>
    <entry key="ws-security.cache.issued.token.in.endpoint" value="false" />
    </jaxws:properties>
    <jaxws:features>
      <bean class="org.talend.esb.servicelocator.cxf.LocatorFeature" />
    </jaxws:features>
  </jaxws:client>

4) Run "mvn clean install"

Maven builds a new WAR package. This package is now able to lookup the endpoint at the service locator.

You can also apply the following patch to the sources of the fediz example which enables the integration with the Talend Service Locator.

Deploy Service Locator

Last but not least, the Talend Service Locator must be deployed and started which is shipped as part of the Talend ESB. Follow these steps to start the Service Locator:

1. Download the Talend ESB Standard Edition (SE) here. The Standard Edition is Apache licensed.
2. Unzip the file
3. Run <install-dir>/container/trun
4. Execute the following command in the console

   tesb:start-all

5. You can log the most recent logs with the command

   log:display

The Service Locator is running now.

More information about the Talend ESB is available here:

• Software and documentation
• Webinars
• Whitepapers, etc.

Test the application

You can test the application as described in the README of the Fediz example wsclientWebapp.

Enter the URL https://localhost:8443/fedizhelloworld/secure/service.jsp and click on the button Call Service after login (User: alice, Password: ecila). This triggers a web service call where the web service consumer will lookup the service provider URL at the Talend Service Locator (and caches this information). You will see log statements like this:

...
Jul 5, 2012 11:21:08 PM org.apache.cxf.clustering.FailoverTargetSelector setStrategy
INFO: Using failover strategy org.talend.esb.servicelocator.cxf.internal.EvenDistributionSelectionStrategy@9235c2
Jul 5, 2012 11:21:08 PM org.talend.esb.servicelocator.cxf.internal.LocatorClientEnabler enable
INFO: Client enabled with strategy org.talend.esb.servicelocator.cxf.internal.EvenDistributionSelectionStrategy.
Jul 5, 2012 11:21:08 PM org.talend.esb.servicelocator.cxf.internal.LocatorTargetSelector prepare
INFO: Found address with locator protocol, mapping it to physical address.
Jul 5, 2012 11:21:08 PM org.talend.esb.servicelocator.cxf.internal.LocatorTargetSelector prepare
INFO: Using strategy org.talend.esb.servicelocator.cxf.internal.EvenDistributionSelectionStrategy.
Jul 5, 2012 11:21:08 PM org.talend.esb.servicelocator.cxf.internal.ReloadSelectionStrategy getPrimaryAddress
INFO: Get address for service {http://apache.org/hello_world_soap_http}GreeterService using strategy org.talend.esb.servicelocator.cxf.internal.EvenDistributionSelectionStrategy selecting from [https://localhost:10443/fedizservice/GreeterService] selected = https://localhost:10443/fedizservice/GreeterService
...

We removed the endpoint URL https://localhost:10443/fedizservice/GreeterService in the configuration of the service consumer. Instead a lookup at the Service Locator resolves this URL.

How can you see which services are registered at the Service Locator

You can use the Zookeeper Client shipped with the Talend ESB to access the Service Locator (Zookeeper Service) which is illustrated next:

/projects/talend/TESB_SE-V5.1.1/zookeeper/bin$ ./zkCli.sh
[zk: localhost:2181(CONNECTED) 0] ls /
[cxf-locator, zookeeper]
[zk: localhost:2181(CONNECTED) 1] ls /cxf-locator
[{http:%2F%2Fapache.org%2Fhello_world_soap_http}GreeterService]
[zk: localhost:2181(CONNECTED) 2] ls /cxf-locator/{http:%2F%2Fapache.org%2Fhello_world_soap_http}GreeterService
[https:%2F%2Flocalhost:10443%2Ffedizservice%2FGreeterService]
[zk: localhost:2181(CONNECTED) 3] 

This is not a very nice UI to see the registered services. After the blog about End-to-End monitoring I'll describe the Talend Administration Console (TAC) which allows to see the registered service and the messages exchanged in a graphical way.

This blog showed how easy it is to add failover and load-balancing capabilities to your CXF application with open source components and without changing a single line of code. I very much like the approach convention over configuration.