I just read an interesting article on creating a flexible model for data integration. The idea is to make the model flexible enough so that as new schemas are released each system can adopt on their own timeline and not all at once. As a former operations person I like that idea quite a bit. To me it creates the following benefits:

1. Each system can stay on its own release schedule, no getting stuck, because one application is in yellow or red status.
2. Deployment becomes much simpler since code for each application can be pushed individually.
3. Troubleshooting after the release is much easier. You have to look at a much smaller set of applications to find the root of a problem.
4. Allows for smaller scale release which can be completed in a much quicker timeframe and with much less risk. This allows a business to be much more responsive to changing market conditions.

Where I see the only real disadvantage is in testing. When you are dealing with multiple systems all on different release cycles, and some potentially risking not making dates it requires a larger testing effort to ensure that code is compliant with the versions of applications it will interact with and is backwards compatible. Depending on the flexibility of your testing system this could increase the number of resources and amount of hardware needed in test.

As I read the article, there are basically 3 main points the authors make.

1. Look for places where schema elements can be optional.

The point here is clear. The more elements that are optional, the easier it is to create a schema that not only can be used by all systems, but doesn’t require all systems to upgrade at once when the schema changes.

2. Design each system so that it can publish data according to a specific schema version, but consume more openly.

Basically the more open you can make each application in terms of what it can accept the easier it is to allow for schema changes. At the same time, publishing much follow the version of the schema that the application is set to use so that all boundary systems will be able to consume the data.

3. Allow systems to extend the common schema and utilize some system specific data elements.

What this allows is for an application to upgrade prior to a schema update and map application specific data to its own data model and not have to wait for a common schema upgrade. You would design your application to look and see if the required data is provided in the common schema and if not gather and store it in the application specific equivalent.

I admit that I need to think this point through a bit more. It is a very interesting concept, but I want to work through some real-life situations to determine how it would work in a production system.

Overall I found the article to be quite educational. Like many architectural articles it was pretty conceptual, which is the point of course, but I like to think in more concrete terms. What I will likely do is run through some specific situations and apply the principles to individual cases to work out how they would work and see if I can poke holes in the ideas.