Monday, May 24, 2010

Agile Integration Software Means New Best Practices

One of the most exciting benefits of Agile Integration Software is the ability to dramatically streamline the analysis and other processes around designing and implementing integration solutions. If RAD (Rapid Application Development) is fundamental to the essence of the software, then it is natural that a new genre of best practices will emerge leveraging the agility of the software.

Enterprise Enabler® has "fuzzied" the lines between the classic steps of analysis, design, development, testing, and deployment. I'd like to share a couple of ways that it is changing the way teams develop integrated solutions.

Analysis Workshops replace Specs. "What?!!!," you say. "That's heresy! No self-respecting IT department would eliminate writing specs!" Ok, so hear me out. Everyone knows that the most time consuming, messiest, most unpleasant part of the whole project is defining the specs, especially the mapping specs. Well, not just defining them, but documenting them, running them by everyone for approval, re-working them, running them by everyone for approval (and so on), and even worse, keeping them up to date over time. Wouldn't you love for that whole exercise to be history, and yet to be able to get the exact current definition any time? With Enterprise Enabler, the specs are synonymous with the solution.

Here we go. You're in a conference room (or web meeting) with three other people, Tom, Dick, and Jane. You are the project manager or an analyst, or even a programmer. You're on the keyboard with Enterprise Enabler's user interface projected on the screen. With you, remotely or in the room, are the three business analysts that really know their systems' data and how it is used. Think how many questions can be resolved in a very short time if these key people can discuss what they require while you build it!

Pass the doughnuts around while we define the big picture of the integration we are solving. Jane is the expert on the destination application, "D," so in my book, she's the boss. After all, the destination's the only the application that cares about the data in an integration. Tom owns source "A" and Dick owns source "B." You pull up the template builder to define the metadata for the appropriate data from "A." Tom points you to the instance of the system that you will be working with and you show him the schema as he indicates what is of interest from his app. Then you do the same with Dick, and then with Jane for the destination. She has indicated what needs to be populated. You go to the mapper and everyone can see on the left all the fields from both sources and on the right, the fields to be filled. First Dick and Tom help you define the cross-application relationship by identifying the appropriate key fields, while you drag and drop them to create the virtual relationship. Now you continue to work with these experts to build and test the data mapping en situ. Tom discovers he left out a couple of tables he need, so you bring up the template and add the tables, and you're back in business. You run the map through and let Tom, Dick, and Jane discuss the nuances of whichever data are not obvious until it is correct. You're done, maybe even before the doughnuts are finished. And the programmed mapping will always match exactly what you see on the screen. When Jane changes her mind or Dick's application has updates that change the schema, you simply go in an reflect it in a couple of minutes.

Integration will become a set of well defined and productive workshops with all the right people, wherever they are. You'll only reminisce about the old days and how difficult it was to get the specs from each expert, document them, and go from one to the other, changing and updating, and then not being able to find the specs document to be able to keep it up to date. Remember when the programmer would discover a problem with the specs and would fix the code, but the change never got back to the specs? Ah... for these headaches to be history!

Friday, May 21, 2010

Coercing the Mind Shift to Comprehend the Paradigm Shift

The mind certainly is stubborn. Something about re-directing those synapses to another mindset is astoundingly challenging. Just like the synapses connect "sphere" and "bounce" to the concept of "ball," "enterprise integration" is only recognizable as "integration" if it looks like a heavy, time-consuming, difficult, expensive endeavor. So what if a cubical ball could actually bounce even better than a spherical one? Even if I see it, I can't make the "connection." Well, I don't know much about synapses, but I sure have had lots of experience trying to initiate the plasticity to reformulate the way people think about integration!

Recently we presented a bit about our technology to a handful of people who are extremely well-informed about the integration space. We planned to hit a few points that we are pretty sure can't be done with other products. We zeroed in on one specific point we thought would amaze anyone who really knows the current state of integration and what is possible and what is not. This is a capability that is a natural fall-out of our new paradigm: we showed how to create virtual relationships across totally different systems even faster than you would define a relationship across tables in a single database.

In ten seconds we showed creating a virtual relationship between Microsoft Dynamics CRM and a SQL database. In that ten seconds, all the run time components were generated behind the scenes, so we ran it. This scenario happened to be sending data to an out-of-box SharePoint 2010 web part; it could have gone to any other application, dashboard, web service, or pumped to a messaging system .

We showed how an end user, from a single SharePoint web part (screen), can see live data from multiple applications, aligned and transformed as appropriate for their usage. As if that weren't enough, we showed updating a couple of fields in SharePoint, and the backend systems were immediately updated!

No staging of the data, no copy made at all, anywhere. The data was accessed, aligned, displayed, changed, and updated back in the backend systems. Our product, Enterprise Enabler, eliminates a HUGE amount of time, manpower, and cost by eliminating the need for a staging database any time you need data merged from two or more disparate systems. The systems could be SAP, Salesforce, XML, spreadsheets… it doesn't matter. So, no one needs to design a database model that is everything to everybody. No one needs to build the database, and no one needs to maintain it.

A response came from our audience that many enterprise integration platforms can do this! I would love to do a "side-by-side" bake-off with the cited competitors, Websphere or Tibco, any day. I am extremely confident that we would probably have to go home before their years of work are done. If they even can do it.

In the end I have to write the response off to the discipline's persistent twenty-plus year track record that has forced consistent synapses dancing around in the same circle, convincing everyone that there is just no other way integration can possibly be done. If Websphere, Tibco, Informatica, and others can't do it, we'll just pretend they can!

Wednesday, May 19, 2010

Characteristics of Agile Integration Software


One of my objectives in starting this blog is to see what I can do to overcome the pervasive mindset, with respect to integration in general, that conjures up a heavy, limited framework that can only actually work with a huge amount of custom coding. It is only by stepping outside that box that we can imagine the next generation of integration software.

What should one look for in an integration product that indicates that it will actually be able to generate an agile environment? Here are a few of my thoughts.

  • First, it must be "off the shelf" or "out of box" or downloadable, whatever term you like. That means that you can install it in just a few minutes. If it takes a week or months to install, you are definitely not going to have an agile environment!
  • It must require little or no training to get started building connectivity. A data analyst should be able to design and build data mapping and transfers. This means the people who know what the result really needs to be can actually do the implementation.

  • You must be able to design, develop, test, and deploy from within a single environment. No separate effort to generate and assemble a run-time components.
  • It must support accessing, transforming, and aligning data from multiple sources live without staging in a database or interim store. Real time data access and availability is a critical aspect of an agile enterprise. Think what work it takes and what value is lost when the data must be staged before it is transferred to wherever it is going.
  • It must be end-to-end metadata driven.
  • If there is any programming required, it should be doable from within the same environment, and the code saved and versioned as part of the metadata.
  • The transformation engine should be able to operate on data in any application's native format as opposed to having to step through a central view such as XML. How can you possibly get high performance when your integration must run through multiple steps.
  • If you are looking for agility at an Enterprise level, the product must be scalable across clusters and extensible so that it becomes an environment tailored with all the special requirement you have that are needed over and over.

    I guess that's plenty for now. Over time, I imagine I'll discuss most of these points in more detail, and how they are present in our Enterprise Enabler agile integration software.

Monday, May 17, 2010

Is "Agile Integration" an Oxymoron?

Most people would agree that "Agile Integration" is an oxymoron. The heavy footprint, heavy customization, and huge amounts of interdependence across an integrated enterprise absolutely negate the possibility of Agility! But we discuss Agile Enterprises as if you could instantly have an intelligent SOA implementation that detects and adjusts to business changes, maybe even anticipating the evolution of the business. What are we talking about, anyway? All the SOA initiatives I've heard about are isolated subsets of business solutions that impose the web services environment (which takes lots of design, development, and definitely anticipation of everything you will want to do in the future). The "agility" offered is limited to a specific, known set of options.

If we want to reflect the Agility that business wants and needs in this age of merger, acquisition, and increasing regulation, we need to be able to almost instantly support the underlying infrastructure implications. Buy a company and you buy a new ERP that manages and records that business. What can you do to absorb the new business? Switch them to your SAP system? Even if that would make your company Agile, you'll be frozen in place for minimum a year or two working to get them up. That's the only way many people, companies, and consultants can think about to achieve that objective. If you buy lots of companies and perhaps sell them again, you MUST be able to very quickly get the key information aligned and the systems sharing appropriate information.

And how can you ever get the latest live information to dashboards instantly, when it comes from a multitude of sources and it needs to be transformed and aligned in order to make sense? Pulling it together via a staging database simply doesn't cut it.

Agility is essential, but it's simply not achievable with the classic approaches to integration.

Agility will only come with a complete mind shift that reinvents the essence of integration. Stone Bond Technologies thinks about integration totally differently with its Enterprise Enabler system. By automating every step along the way and generating the run-time components behind the scenes, the time to implement and maintain an integration is reduced by up to 90%. I know, "90% - you are either crazy or lying!" Well, hear me out a bit. We really do have an off-the shelf product that is the product of 100+ man years of development and is built on a new paradigm. I'll discuss the fundamentals of it in the coming entries.