Kanban from a Trench

Introduction

After forming part of a software project team that got underway with a significant project at the BBC Worldwide in 2013, I felt that it would be useful to release a few notes about my experience of Kanban and how it helps to deliver good quality software to customers.

Here are some notes that I prepared in order to deliver a Kanban motivational speech to an engineering community on a subsequent project outside of the BBC.

Kanban from a Trench

Kanban is technique for managing the software development process. It doesn’t tell us how to develop software but provides techniques that help us to enforce what we, (as a software development team), agree is the right formula to develop acceptable quality software for our customers.

As an example, we may determine that an adequate software engineering process requires the following:

  • Well understood and agreed requirements to be the foundation of onward development.
  • Development is sympathetic to an evolving well understood architecture that’s published for the entire team to work from.
  • We practice JIT development/engineering – only do what is required when it’s required.
  • A set of coherent, well planned and engineered tests are built and maintained – from unit tests to UAT.
  • Deployment and release management is actively managed through planning and coordination.
  • Feature development priority adjustments are possible in order that our agile team can respond quickly to change instigated by key stake holders.
  • Peer review of all team assets – code, documentation, user manuals and tests – as a minimum.
  • Feature designs are documented with light-weight descriptions and pictures – bullet points and photos of whiteboards. Sometimes, formal UML diagrams can be justified, i.e. for state or sequence representations.
  • The development process must evolve with the team, product and organisational aspirations.

The Kanban Board

For the Kanban projects in which I have been involved, our project board typically consists of five key areas:

  • a) a team pool (a collection of names),
  • b) a group of associated columns that represent our software engineering process,
  • c) some streams,
  • d) a set of feature tasks in various stages and
  • e) a calendar of significant events.

bk-1

a) Team Pool: The team pool is a collection of names/avatars, each one a picture and name of a person whom is available for work on tasks – each team member usually has two avatars. You can work on at least one task at a time, often we have capacity to take on another task in parallel.

These avatars help teach us about how we work as a team. For example, If an attempt is made to assign an avatar to more than two tasks, or is very frequently switched between tasks before tasks are finished – the board is very likely telling us something that we need to understand and deal with. Maybe we have insufficient resources in the team, maybe we have knowledge concentrated in one person (exclusive knowledge). Regardless, we need to record this and if it continues to happen, take some corrective action to resolve it.

b) Group of Associated Columns: These columns represent phases or major milestones in our software development process. Kanban does not prescribe these columns as listed here below but suggests what might be a good starting point for our team to move forward from – the process will grow and evolve with the team.

bk-2

As an example, the following columns were used for the BBC Worldwide project in which I was involved:

Inventory – A collection of features that will have a broad level description in order that it can be understood (three-point estimation only at this stage). No further detail is required right now – remember, JIT.

Identify – Features that we’ve agreed to spend some further time on. We need to understand if the feature is date driven, what the value proposition is, how to understand if the goal is achieved or not (once it’s deployed we need to measure success), identify significant dependencies (other work/functions etc). Which release version/date is targeted etc.

Analyse and Design – During this stage we will define and refine acceptance criteria – these will be written on the card. We now need to gather sufficient requirements such that the customer will get what they want (and hopefully asked for) and we understand how to build it. We will discuss and record BDD scenarios in preparation of QA. Analysis and design will be peer reviewed with a minimum agreed number of team members whom have business, technical and architectural knowledge relevant to the feature undergoing development. If necessary, the feature will be broken down into smaller units/tasks, each consisting of work that will last (approximately) between 0.5 and 5 days. Wireframes (from the whiteboard) will be drawn, described and agreed/approved by the customer. The feature will be reviewed in terms of architecture. The general test approach will be agreed, also consider technical debt, i.e. missing tests for existing code packages. Agree documentation of the feature, it’s setup, configuration, user manual entries etc. Agree to commit to build the feature.

Develop and QA Each new class, service, technical feature must have been written, have passing tests and executable documentation where required. Automated tests will be created and triggered by schedule. Tasks will not leave this column until they have been added into ‘develop’ (a Git branch) and all tests green-ticked.

Ready to Deploy – Team agrees that they’re happy for the feature to go into the next release and a demo to the customer is conducted.

One of the key aspects of Kanban is Work In Progress (WIP) limits. Kanban tells us that it’s better not to take on too much and that we should always consider finishing things in preference to starting new ones. This is often quite a challenge for teams new to Kanban because it means instead of taking on a new task, you may have to go and pair with someone that’s already working on a task. We all know that pairing should be encouraged, this is one route towards that. Pairing work may be coding, testing, documentation, QA, infrastructure, etc. WIP limits are defined for each column, they may have different values depending on the team size, columns and our experience of what works. We can increase WIP limits but beware consequences such as blocking. Instead of increasing WIP limits, get work finished instead.

It’s often convenient to decompose a column into two parts, one to show in-progress tasks and one to show those that have been completed but are not ready to move into the next column.

c) Some streams: It’s sometimes easier to track and read the board if groups of related tasks are bunched together in a stream. If during analyse and design a feature is broken down into, e.g. 8 sub-tasks, each of these sub tasks can be grouped into a single stream so we can readily see they are related.

bk-3

d) Feature tasks: The focus of the Kanban board is the tasks that represent feature development. The aim is to pull tasks through the system (from the right-hand side) rather than push them in from the left.

Each task card contains a progress history. It’s not unusual for cards to have extensions attached to the reverse of them, design diagrams, JIRA numbers, dependencies, dates for movement along the board.

To further aid recognition, it’s quite useful to use colour coded cards such that feature work, documentation, infrastructure and bugs can be easily recognised.

It’s really important that the Kanban work board is maintained up to date, so this is something that usually takes place each morning – a stand-up. Whomever is running the stand-up will go through the tasks on the board and ask for an update from any person working on the task.

e) Project Calendar: The entire team needs to know if somebody is sick, on holiday or that a release is imminent.

How do we do maintenance?

We could consider that two classes of bug exist, those directly related to features currently being undertaken and those related to historical feature development. Bugs related to current feature development should be fixed by the feature crew/team working on it, up to and including the User Acceptance Test phase, i.e. post deployment.

The primary difference between product feature development and historical bug fixing seems to be stability of day-to-day work tasks and their duration – frequently changing priorities (intra-day) and generally, very short develop/test life-cycle. Ideally, we’d expect features to undergo development, in part or whole, during a period of days or weeks. Context switching is expensive so we want engineers to remain focused on the feature(s) that they’re working on rather than frequently switching between features and historic bugs.

Therefore, having two project boards (or two streams on a single board), one representing core feature development and one representing bugs could suit quite well. The aspects that are different in these two regards, that need separating, are the units of work, rather than team members. Therefore, it’s a sensible approach to enable team members to be available to work on both core feature development and historical bugs as work-loads and priorities dictate.

Existing Technical Debt

We need to work out a mechanism that promotes, encourages and tracks necessary refactoring in order that it’s well socialised, publicised and coordinated. We could track this as a stream on the main board or use a distinct Kanban board to track it?

Estimation

For Inventory tasks, three-point estimation will be used. Best case, worst case and expected. We’ll use this to help drive which features to take on first given an arbitrary set of features that require development for a given release. This estimation can be further refined when breaking a task down in the Analysis and Design stages. This information will help feed into a release tracking board that represents feature dependencies and release dates.

Architecture

At any stage of any feature or release, it should be possible to view an architecture diagram to use for alignment of feature development. Interesting aspects are current architecture, aspirational next step architecture and final state architecture. Of course all three of these will move over time as the product and its environment matures.

As with all other work, an architectural conjecture will be proposed and presented to the team as part of the general review process. None of this work is hidden, it will all appear as tasks on the Kanban board.

Demos

It should have been made clear from column exit criteria, but running a demo for the customer or his/her representative is a pre-requisite to a feature being added to a release.

GitFlow

At what stage does software make it into ‘develop’, release and support. The Kanban board needs to be well aligned with our release process, or vice-versa.

Following a series of probing and explorative questions, the team commenced trying to understand what it would take in order to establish engineering satisfaction for software production.

What does the team consider as adequate product engineering?

Collectively, the team have identified the following criteria as those that are essential to building a product of sufficient quality such that we are happy to give it to our customers.

The groupings are somewhat artificial, but give us an idea of the type and spread of areas for which we have collectively, accumulated important engineering criteria.

Screen Shot 2014-01-02 at 16.58.05

The Kanban Board

The physical Kanban board will be the master copy of feature development status. We need to build and place a board in an area that’s always accessible, as several team members have observed it would not be a good idea to have this located in a meeting room.

JIRA tickets will be created for the highest-level feature names but at the end of the process, i.e. when they hit the last column (I.e. UAT). Generally, we will not get bogged down in keeping JIRA up-to-date with the physical board – communication is key and that can’t be replaced by an issue tracking system so let’s not try.

From the previous examples, we will now define the columns that we’re going to use and assign exit criteria to each of them.

Board Management

Although, as a software development team collectively, we will own the process, the Kanban board needs to be actively managed along with the teams using the board. Sometimes we will require arbitration, encouragement, steering and decisions in order to operate. If an agile coach is not available, we need to identify someone to adopt the role of coach or manager.

Acknowledgements

I’d like to say a big thanks to Sabina Kamber Salamanca and Kevin Ryan for all of the knowledge sharing and guidance that led me to discover a better way to engineer software – inspirational.

Spring Framework Property Configuration

Architecting software development projects often requires a mechanism to manage configuration properties in such a way that they can be defined and overridden depending on the environment in which they’re being used. This requirement often driven by a need to use different resources at each stage of the development life-cycle, i.e. development, test and production. This article describes a scheme that allows properties to be defined and overridden in a simple way within properties @Configuration classes.

Using the following class as an example, the @PropertySource annotation triggers an attempt to load property values from two properties files. One is a simple literal classpath name entry, the second is also a classpath name entry but uses an embedded expression to allow selection of the appropriate classpath locatable file at runtime:

  1. “classpath:properties/app.properties”
  2. “classpath:properties/app-${spring.profiles.active:default}.properties”.

The @PropertySource annotation triggers property file loading in definition order – consequently “classpath:properties/app.properties” is loaded first and “classpath:properties/app-${spring.profiles.active:default}.properties” second.

package com.greendot.properties;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;
import org.springframework.context.support.PropertySourcesPlaceholderConfigurer;

/**
 * Created on behalf of GreenDot Software Ltd.
 *
 * @author matt.d.vickery@greendotsoftware.co.uk
 * @since 08/07/2013
 */
@Configuration
@PropertySource(value = {
        "classpath:properties/app.properties",
        "classpath:properties/app-${spring.profiles.active:default}.properties"
})
public class PropertiesConfiguration {

    private final Logger logger = LoggerFactory.getLogger(this.getClass());

    @Bean
    public PropertySourcesPlaceholderConfigurer getProperties() {
        return new PropertySourcesPlaceholderConfigurer();
    }
}

 

Overriding is supported using this scheme as any property values loaded from the properties file ‘app.properties’ will be overridden by any property values loaded from ‘app-${spring.profiles.active:default}.properties’ that use the same properties key.

As for expansion of the variable expression (${spring.profiles.active:default}), the variable value will be populated at runtime according to the value set for the relevant Java System Property (i.e. -Dspring.profiles.active=test). You may observe that any value can be used for this property, the following code examples use default, test and production as possible values that make sense for the problem. If no value is set, then default will be used, determined by the definition ‘..:default}.properties’. This example uses the property key ‘spring.profiles.active’ specifically in order that Spring Framework Profiles can be used through the same configuration mechanism.

Notice that the example also uses a PropertySourcesPlaceholderConfigurer @Bean, this is made available in order that @Value annotations can be used in other Spring bean classes – an example follows.

A suitable mechanism to demonstrate properties configuration is using a Unit Test, the following properties will be used to exercise the PropertiesConfiguration class.

The content of app.properties is:

1
2
3
4
mongo.db.port=27017
mongo.db.name=catalogue
mongo.db.logon=mvickery
mongo.db.password=sugar

The content of app-default.properties is:

1
mongo.db.server=localhost

The content of app-test.properties is:

1
2
3
mongo.db.server=testhost.greendotsoftware.co.uk
mongo.db.logon=tester
mongo.db.password=tpassword

The content of app-production.properties is:

1
2
3
mongo.db.server=prodhost.greendotsoftware.co.uk
mongo.db.logon=operations
mongo.db.password=opassword

 

As an example of how properties management works with this scheme, the following test class loads properties files through the @ContextConfiguration loading of the PropertiesConfiguration class we defined above. The class is then run by the JUnit class runner utility SpringJUnit4ClassRunner, this means that the test can be run with a Spring context with beans loaded from any referenced @Configuration classes, e.g. PropertiesConfiguration.class. Furthermore, the @Value annotation triggers autowiring of property values into annotated variables such as dbName, dbServer etc.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
package com.greendot.properties;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.core.env.Environment;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import static org.hamcrest.CoreMatchers.is;
import static org.springframework.test.util.MatcherAssertionErrors.assertThat;

/**
 * Created on behalf of GreenDot Software Ltd.
 *
 * @author matt.d.vickery@greendotsoftware.co.uk
 * @since 08/07/2013
 */
@ContextConfiguration(classes = {
        PropertiesConfiguration.class
})
@RunWith(SpringJUnit4ClassRunner.class)
public class PropertiesConfigurationTest {

    private static final String MONGO_DB_SERVER = "mongo.db.server";
    private static final String MONGO_DB_NAME = "mongo.db.name";
    private static final String MONGO_DB_LOGON = "mongo.db.logon";
    private static final String MONGO_DB_PASSWORD = "mongo.db.password";

    @Value("${"+MONGO_DB_NAME+"}")
    private String dbName;
    @Value("${"+MONGO_DB_SERVER+"}")
    private String dbServer;
    @Value("${"+MONGO_DB_LOGON+"}")
    private String dbLogon;
    @Value("${"+MONGO_DB_PASSWORD+"}")
    private String dbPassword;

    @Test
    public void defaultProfile() {
        assertThat(dbName, is("catalogue"));
        assertThat(dbServer, is("localhost"));
        assertThat(dbLogon, is("mvickery"));
        assertThat(dbPassword, is("sugar"));
    }

    @Test
    public void productionProfile() {

        System.setProperty("spring.profiles.active", "production");

        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext();
        context.register(PropertiesConfiguration.class);
        context.refresh();
        assertThat(getProperty(context, MONGO_DB_SERVER), is("prodhost.greendotsoftware.co.uk"));
        assertThat(getProperty(context, MONGO_DB_NAME), is("catalogue"));
        assertThat(getProperty(context, MONGO_DB_LOGON), is("operations"));
        assertThat(getProperty(context, MONGO_DB_PASSWORD), is("opassword"));
    }

    @Test
    public void testProfile() {

        System.setProperty("spring.profiles.active", "test");

        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext();
        context.register(PropertiesConfiguration.class);
        context.refresh();
        assertThat(getProperty(context, MONGO_DB_SERVER), is("testhost.greendotsoftware.co.uk"));
        assertThat(getProperty(context, MONGO_DB_NAME), is("catalogue"));
        assertThat(getProperty(context, MONGO_DB_LOGON), is("tester"));
        assertThat(getProperty(context, MONGO_DB_PASSWORD), is("tpassword"));
    }

    private String getProperty(final AnnotationConfigApplicationContext context, final String property) {
        return context.getBean(Environment.class).getProperty(property);
    }
}

 

There are three test methods in this class. The first method tests loading of ‘default‘ properties – as the value of ‘spring.profiles.properties‘ is null as runtime. We expect properties found in ‘app.properties‘ to be loaded first along with a single additional property loaded from ‘app-default.properties‘ immediately aftwards.

Spring Integration – Message Gateway Adapters

Introduction

This article is going to present a technique for handling response types from message sub-systems in a service-based Message Driven Architecture (MDA). It will provide some theory, experience of message sub-system development and some robust working code that’s running in several financial  institution production deployments.

As Spring Integration is underpinned by a well established set of Spring APIs familiar to most developers, I have chosen this as a basis for this article.

If developers only ever had to build integration flows to handle positive business responses, this article would not be necessary. However, real world development, particularly within service integration environments, results in developers having to deal not just with positive business responses but business exceptions, technical exceptions and services that become unresponsive.

Spring Integration Gateways

Spring Integration (SI) Gateways are entry points for message sub-systems. Specify a gateway using a Java interface and a dynamic proxy implementation is automatically generated by the SI framework at runtime such that it can be used by any beans in which it’s injected. Incidentally, SI gateways also offer a number of other benefits such as request & reply timeout behaviour.

Typical Message Driven Architecture (MDA) Applications

As with Java design patterns for application development, similar patterns exist for integration applications. One such common integration pattern is service-chaining – a number of services are connected in series or parallel that together perform a business function. If you’ve built and deployed services using MuleSoft’s Mule, SpringSource’s Spring Integration, FuseSource’s ServiceMix (Camel), Oracle’s Service Bus or IBMs WebSphere ESB the chances are that you’ve already built an application using chained-services. This pattern will become ever more widespread as the software industry moves away from client-server topologies towards service based architectures.

A recent engagement in which I provided architectural consultancy will be used as an example  implementation of the service-chain application pattern. The engagement required the team to build a solution that would transition financial messages (SWIFT FIN) from a raw (ISO15022) state through binding, validation and transformation (into XML) and ultimately added to a data store for further processing or business exception management.

Using EIP graph notation, the first phase service composition could be represented as follows. A document message arrives into the application domain, it’s parsed and bound to a Java object, semantically validated as a SWIFT message, transformed to XML and then stored in a datastore. As a side note, the binding, semantic validation and transformation are all performed by C24s iO product. Furthermore, the full solution sample that can be found in GitHub for this project contains dispatcher configurations in order that thread pools can be used for each message sub-system, for the sake of brevity and clarity, such details will be omitted from this article.

Using EIP graph notation, the first phase service composition could be represented as follows. A document message arrives into the application domain, it’s parsed and bound to a Java object, semantically validated as a SWIFT message, transformed to XML and then stored in a datastore. As a side note, the binding, semantic validation and transformation are all performed by C24s iO product. Furthermore, the full solution sample that can be found in GitHub for this project contains dispatcher configurations in order that thread pools can be used for each message sub-system, for the sake of brevity and clarity, such details will be omitted from this article.

Although this configuration specifies a chain of services it’s not adequate to form the bases of a robust production deployment. An exception thrown by any of the services would be thrown straight back to the entry gateway thus loosing context, i.e. which service threw the exception, any non-response code invoked by the service may result in its Java thread getting parked and null values returned by a service may cause unexpected problems or even for the entry gateway to hang; if an entry gateway is used unlike in this diagram.

Unresponsive Service Invocation

The Spring Integration specific construct for accessing a message sub-system is the gateway. Although apparently simple, the SI gateway is a powerful feature that results in generation of a dynamic proxy generated by Spring’s GatewayProxyFactoryBean. This bean can be injected into new or existing code or services as an implementation of the interface. The SI gateway also provides facilities to deal with timeouts and provides some error handling facility. A typical Spring Integration namespace XML configuration is as follows:

<int:channel id="parsing-gw-request-channel" datatype="java.lang.String">
  <int:queue capacity="${gateway.parse.queue.capacity}"/>
</int:channel>

<int:gateway id="parseGateway"
             service-interface=
                 "com.c24.solution.swift.flatten.gateway.ParseGateway"
             default-request-channel="parsing-gw-request-channel"
             default-reply-channel="parsing-gw-reply-channel"
             default-reply-timeout="${gateway.parse.timeout}"/>
<int:channel id="parsing-gw-reply-channel" 
             datatype="biz.c24.io.api.data.ComplexDataObject"/>

Evolving the design, the next phase application architecture design needs to take advantage of these gateways, the Spring Integration context can be extended quite simply by creating a Java interface for each service. Building message sub-system gateways leads us towards a design model like the following:

A small amount of additional configuration and some very simple Java interfaces means that developers can now configure request/reply timeouts on the gateway and avoid any unresponsive code – assuming (as we always should) that code written locally or supplied by 3rd parties is capable of misbehaving.

Additionally, the SI gateway allows specification of an error handling channel so that you have the opportunity to handle errors. This design I am presenting here is not going to use error channels but handle them in a different way – hopefully that will become more obvious as the design evolves.

Specific semantics and configuration examples for gateways and gateway timeouts can be found in reference material provided by SpringSource and other blogs.

A significant benefit to the integration solution has been added with the use of Spring Integration gateways. However, further improvements need to be made for the following reasons:

  1. A gateway timeout will result in a null value being returned to the calling, or outer, flow. This is a business exception condition that means that the payload that was undergoing processing needs to be pushed into a business exception management process in order that it can be progressed through to conclusion – normally dictated by the business. If a transient technical issue caused this problem, resubmission may solve the exception, otherwise further investigation will be required. Whatever the eventual outcome, context about the failure and it’s location need to be recorded and made available to the business exception management operative. The context must contain information about the message sub-system that failed to process the message and the message itself.
  2. Exceptions generated by message sub-systems can be handled in a few different ways, through the error channel inside the gateway undergoing message processing failure or by the calling flow. Again, context needs to be recorded. In this case, the technical exception needs to be added to the context, along with the gateway processing the message undergoing failure and any additional information that may be used within the business exception management process for resolution.

If transactions were involved in this flow, for example a transactional JMS message consumption flow trigger, it would be possible to rollback the transaction in order that it can be re-queued to a DLQ. However, the design in this software compensates for failures by pushing failing messages to a destination configured by the developer directly; and this may of course be a JMS queue if that’s required. This avoids transaction rollback and adds the benefit of exception context directly rather than operations staff and developers having to scour logs to locate exception details.

The Gateway Adapter

In order to all handle exceptions taking place within a message sub-system and also handle null values a Gateway Adapter class can be used. The reason that the SI gateway error channel is not used for this purpose is that it would be complex to define and would have to be done in several places. The gateway adapter allows all business exception conditions to be handled in one place and treat them in the same way. The Gateway Adapter is a custom written Spring bean that invokes the injected gateway directly and manages null and exception responses before allowing the invocation request to return to the caller (or outer flow).

The architectural design diagram evolved from the previous design phase includes a service activator backed by a Gateway Adapter bean, this calls the injected gateway (dynamic proxy) which calls the business service.

Nuts and Bolts

The design diagrams are a useful guide for making the point but maybe the more interesting part is the configuration and code itself. As with the entire solution, the outer or calling flow can be seen in the project located in GitHub, however a useful snippet to view is one of the gateway adapter namespace configurations, the gateway, the gateway adapter and the gateway service configuration. As a number of gateways exist in this application, and they all follow the same pattern, the following configuration and code will merely demonstrate one of them.

Gateway Adapter Namespace Configuration

 
<int:channel id="message-parse-channel" datatype="java.lang.String"/>
<int:chain input-channel="message-parse-channel" 
           output-channel="message-validate-channel">
    <int:service-activator ref="parseGatewayService" method="service"/>
</int:chain>

Gateway

package com.c24.solution.swift.flatten.gateway;

import org.springframework.integration.Message;

/**
 * @author Matt Vickery - matt.vickery@incept5.com
 * @since 17/05/2012
 */
public interface ParseGateway {
    public Message<?> send(Message<String> message);
}

GatewayAdapter

package com.c24.solution.swift.flatten.gateway.adapter;

import biz.c24.io.api.data.ComplexDataObject;
import com.c24.solution.swift.flatten.exception.ExceptionContext;
import com.c24.solution.swift.flatten.exception.ExceptionSubContext;
import com.c24.solution.swift.flatten.gateway.ExceptionGatewayService;
import com.c24.solution.swift.flatten.gateway.ParseGateway;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.integration.Message;
import org.springframework.util.Assert;

/**
 * @author Matt Vickery - matt.vickery@incept5.com
 * @since 17/05/2012
 */
public class ParseGatewayService extends AbstractGatewayService {

  private static final Logger LOG =
      LoggerFactory.getLogger(ParseGatewayService.class);
  private final ParseGateway parseGateway;
  private final ExceptionGatewayService exceptionGatewayService;

  public ParseGatewayService(
         final ParseGateway parseGateway,
         final ExceptionGatewayService exceptionGatewayService) {
      this.parseGateway = parseGateway;
      this.exceptionGatewayService = exceptionGatewayService;
  }

  public Message<ComplexDataObject> service(Message<String> message) {

      Message<?> response;
      try {
          LOG.debug("Entering parse gateway.");
          response = parseGateway.send(message);
      } catch (RuntimeException e) {
          LOG.error("Exception response .. {}, exception: {}", getClass(), e);
          LOG.error("Invoking ... process because: {}.", e.getCause());
          buildExceptionContextAndDispatch(
            message,
            ExceptionContext.PARSE_FAILURE,
            ExceptionSubContext.EXCEPTION_GATEWAY_RESPONSE,
            exceptionGatewayService);
          throw e;
      }

      if (response != null) {
        if (!(response.getPayload() instanceof ComplexDataObject))
            throw new IllegalStateException(INTERRUPTING_..._EXCEPTION);
      } else {
        LOG.info("Null response received ....", getClass());
        buildExceptionContextAndDispatch(
          message,
          ExceptionContext.PARSE_FAILURE,
          ExceptionSubContext.NULL_GATEWAY_RESPONSE,
          exceptionGatewayService);
        throw new GatewayAdapterException(NULL_GATEWAY_RESPONSE_CAUGHT);
      }

      Assert.state(response.getPayload() instanceof ComplexDataObject);
      return (Message<ComplexDataObject>) response;
  }
}

GatewayService

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:c24="http://schema.c24.biz/spring-integration"
       xmlns:int="http://www.springframework.org/schema/integration"
       xsi:schemaLocation="http://www.springframework.org/schema/integration
 http://www..org/schema/integration/spring-integration.xsd  http://schema.c24.biz/spring-integration
 http://schema.c24.biz/spring-integration.xsd
 http://www.springframework.org/schema/beans
 http://www.springframework.org/schema/beans/spring-beans.xsd">

    <int:channel id="parsing-gw-request-channel" 
                 datatype="java.lang.String">
        <int:queue capacity="${gateway.parse.queue.capacity}"/>
    </int:channel>
    <int:gateway
      id="parseGateway"
      service-interface="com.c24.solution.swift.flatten.gateway.ParseGateway"
      default-request-channel="parsing-gw-request-channel"
      default-reply-channel="parsing-gw-reply-channel"
      default-reply-timeout="${gateway.parse.timeout}"/>
    <int:channel id="parsing-gw-reply-channel" 
                 datatype="biz.c24.io.api.data.ComplexDataObject"/>

    <int:chain input-channel="parsing-gw-request-channel" 
               output-channel="parsing-gw-reply-channel">
        <int:poller fixed-delay="50" 
                    task-executor="binding-thread"/>
        <c24:unmarshalling-transformer
          id="c24Mt541UnmarshallingTransformer"
          source-factory-ref="textualSourceFactory"
          model-ref="mt541Model"/>
    </int:chain>

</beans>

Summary

Whatever type of message based integration system you are designing, once you get beyond the use of sample code for prototyping a technology, and let’s face it, there are millions of lines of code out there that are copied from trivial examples, you need to consider invocation in the face of technical exceptions, business exceptions & non-responsive services. Using Spring Integration Gateways to represent entry to message sub-systems means that we must be able to cope with all of these types of behaviour. The Gateway Adapter pattern allows a single, central processing location for such conditions.

The intent for the Gateway Adapter should be clear from the example configuration and code provided but can also be run and tested using the full source located on GitHub. Please leave any feedback or comments as you see fit.

Spring Integration – Input Channel Definition

Input Channels

In a pipes and filters architecture, pipes are connectors or channels. Although at first sight trivial, channels are semantically rich – they allow typing, synchronous and asynchronous input, direct and multicast notifications, send and wait (rendezvous) as well as queued input and wrapping by adapters.

To define or not to define?

Explicit specification or definition of the Spring Integration input-channel is not mandatory for SI constructs to operate. The framework will create input channels automatically if they are not explicitly defined in configuration, but what are the pros and cons of this aproach?

Take the following chain as an example:

Screen Shot 2013-09-16 at 13.58.08

The input-channel named “processing-channel” will get created automatically as it’s not explicitly defined here. In a trivial configuration such as this one there’s very little difference between including the channel explicitly or using the frameworks implicit creation. In larger configurations the extra lines taken by a dozen or more channel definitions may start to make the context appear a little cluttered. In this case it’s then possible to consider decomposing the configuration but sometimes you just can’t decompose those flows into distinct contexts, all of the constructs naturally belong together in a single context.

Channel Typing

One of the features of Spring Integration channels is that they can be strongly type matched with payload object types. It’s worth considering adopting a convention for specifying the payload type on the channel because not only does this provide strong payload typing but improved better confidence that whomever is reading the flow after its built can readily see which type of objects are traversing the flows.

Of course channels can’t always be strongly typed but in many cases they can. Here’s an example of one that can:

Screen Shot 2013-09-16 at 13.59.00

You can see that on the first line of this Spring Integration configuration the contract for operation specifies that a java.util.UUID type object is expected on the input channel. In this case the payload contained a claim-check identifier and is to be replaced with an arbitrary string for the example.

In the case where a channel is strongly typed and the contract broken an exception will be thrown, here’s an example of what you’ll see: Screen Shot 2013-09-16 at 13.59.44

In this example I changed the datatype to java.util.Map and ran a test with a payload that’s actually a java.util.UUID. That was the start of the stack trace that was generated when the exception was thrown.

It’s possible to provide a collection of message types for channel specification. Any messages entering the channel must conform to at least one of the specified types otherwise a MessageDeliveryException is generated. The following example shows a change that will prevent the exception above. The configuration now allows for two types of message payload, java.util.Map and java.util.UUID to enter the channel.

Screen Shot 2013-09-16 at 14.00.34

Spring Integration – Payload Storage via Header Enrichment

There’s often a need to temporarily store transient messages during design of Spring Integration flows – several different mechanisms are available in the toolkit.

It’s pretty straight forward to take a message, use an SI header enricher construct and place the message in the header using a SpEL expression – in fact one for the header key name and one for the payload extraction.

The following SI flow demonstrates an example of how to do just that :

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:int="http://www.springframework.org/schema/integration"
       xsi:schemaLocation="http://www.springframework.org/schema/beans 
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/integration 
       http://www.springframework.org/schema/integration/spring-integration-2.1.xsd">
  <int:gateway id="headerManagementGateway"
                 service-interface="com.l8mdv.sample.HeaderManagementGateway"/>
  <int:chain input-channel="request-message-storage-channel"
             output-channel="request-message-retrieval-channel">
    <int:header-enricher>
      <int:header name="#{T(com.l8mdv.sample.HeaderManagementGateway)
                   .REQUEST_PAYLOAD_HEADER_KEY}" expression="getPayload()"/>
    </int:header-enricher>
  </int:chain>

  <int:chain input-channel="request-message-retrieval-channel">
    <int:transformer expression="headers.get(T(com.l8mdv.sample.HeaderManagementGateway)
               .REQUEST_PAYLOAD_HEADER_KEY)"/>
  </int:chain>
</beans>

This example can be executed by implementing a gateway as follows:

package com.l8mdv.sample;

import org.springframework.integration.Message;
import org.springframework.integration.annotation.Gateway;

public interface HeaderManagementGateway {

    public static final String REQUEST_PAYLOAD_HEADER_KEY = "REQUEST_PAYLOAD_HEADER_KEY";

    @Gateway (requestChannel = "request-message-storage-channel")
    public Message<String> send(Message<String> message);
}

and then running a test such as this one:

package com.l8mdv.sample;

import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.integration.Message;
import org.springframework.integration.support.MessageBuilder;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import static com.l8mdv.sample.HeaderManagementGateway.REQUEST_PAYLOAD_HEADER_KEY;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"classpath:META-INF/spring/header-management.xml"})
public class HeaderManagementIntegrationTest {

    @Autowired
    HeaderManagementGateway headerManagementGateway;

    @Test
    public void locatePayloadInHeader() {
        String payload = "Sample test message.";
        Message<String> message = MessageBuilder.withPayload(payload).build();
        Message<String> response = headerManagementGateway.send(message);

        Assert.assertTrue(response.getHeaders().get(REQUEST_PAYLOAD_HEADER_KEY).equals(payload));
    }

    @Test
    public void locateTransformedPayload() {
        String payload = "Sample test message.";
        Message<String> message = MessageBuilder.withPayload(payload).build();
        Message<String> response = headerManagementGateway.send(message);

        Assert.assertTrue(response.getPayload().contains(payload));
    }
}

For full source code and configuration, see the header-management maven module under  https://github.com/mattvickery/l8mdv-si-samples