Friday, June 22, 2012

Inspiring New Patterns for Integration

Integration is an age-old problem that doesn't have much  opportunity to get people excited.  It's the same old problem, even if you have new ways to solve it. Over the years, we have tended to just keep massaging old "patterns" pretending like they are new  configurations of moving data around and stuffing it into databases only to take it out again. It takes totally new concepts for the underlying integration architecture to give birth to new patterns that can renovate solution concepts, and more importantly, can enable new business uses that have not been possible before.

Data federation is now finally coming of age enough to have at least a small number of products that can deliver live virtual data federated across disparate systems.

[[Wait..."Live virtual data?" If it doesn't exist, can it ever be dead? Or even stale? If it's live can it be virtual? 
Definition: Virtual
1. Existing in the mind, especially as a product of the imagination.
2. Computer Science Created, simulated, or carried on by means of a computer or computer network: virtual conversations in a chatroom

And I like the origins, from old English and Latin words  that mean effective, excellence, virtue.]]

But I digress!  And maybe it's a good thing, because we tend to throw all these terms around with great authority, completely confusing those who just want to understand what this new breed of software does.  To add to the confusion, "virtualization" is a pervasive term in hardware/operating system jargon, meaning essentially a stand-alone computer that is simulated on another, bigger computer or in the cloud. Now that I have said that, forget it, because that has nothing to do with virtualization with respect to integration.

 For the purpose of this discussion, let's say that in integration speak, "virtual" means that there is never a copy made of the data, and that it never moves anywhere. A virtual view of data allows you to look at data on a dashboard, web page, or the like, without storing it anywhere. When the screen is refreshed, it’s gone. A "federated" virtual view is a virtual view that is consolidated and aligned data from multiple sources. "Live" means that the data comes directly from the sources without staging it in any data store or virtual database along the way. "Bi-directional federation and virtualization" means that the virtual federated data can be interacted with. For example, an end user can update or correct the data on the screen, and it is sent back to the sources as updates, still without staging en route either direction.



With this new technology,  federated data no longer must be staged in order to transform and align it from multiple sources and send it to an ephemeral display endpoint, which presents the live data to the end user ("virtually").  This does wonders for Business Intelligence and Business Analytics.

But going beyond BI, there is one product that enables these displays to take end user's  changes and updates  and pass then securely back to the sources. This is what turns dashboards into control centers. http://tinyurl.com/bqfnzw9

Imagine, though, what a mind shift this requires, and the uncertainty and fear, even, of have a portal like SharePoint, for example, that can effectively become the a single window into all the applications a business user needs. He is not just looking at analytics and drilling down for detail input.


With the ability to interact with backend systems, the business user is no longer just a dead-end in the data flow,  absorbing, and absorbing, and only interacting with the business intelligence software to see more data to absorb. Instead, with bi-directional federation and virtualization, as he sees data that needs to be updated, and draws conclusions from the information, he  can enter his updates as if he were working directly with the backend source systems.

Imagine having a portal that presents you with exactly the information you need to perform your business role. This may include information from SAP, Salesforce,  and, let's say, a scheduling system. From that one screen on your portal, you may update a customer's address, look up prices, and place an order. All of the relevant information will be sent back into the appropriate backend systems automatically. You don't have to learn how to navigate in three separate systems, and you don’t have to figure out how the information is required to be handled differently with each system. Instead, all the work happens behind the scenes, and you only work with easy-to-use aliases for the virtual consolidated data.
  
While this new paradigm inspires new patterns for integration, it also brings pessimists, nay-sayers and even realists who find worrisome points to ponder.   For example, if you have everyone in the company hitting the backend systems directly, could this bring those systems to their knees? See my white paper, Inspiring New Patterns for Integration about a complex pattern that rationalizes and reduces the hits to the backend systems.

With this new technology, there's a whole new world of ideas and opportunities for streamlining integration implementations, which is what has been consistently the largest investment and risk with virtually every IT project since the beginning of time (as we know it).