Friday, March 30, 2012

Data Virtualization: No Need to Look Beyond Your Own Back Yard

How is this possible? I did a few Google searches looking for "data virtualization" and "SharePoint" together. SharePoint 2007 and 2010 were designed with the concept of data virtualization built in. How is it possible that only one of the players in this new-hype of data virtualization and federation shows up on the radar as feeding SharePoint? After all, SharePoint is already found in most medium-to-large businesses, so there's no need to look for a separate dashboard or application development platform. You probably have access to it in your company right now!

Out-of-the box, SharePoint has the capabilities to interact with data virtually. I'm not talking about the old SharePoint lists, which you can write to, store data in and change; I'm talking about the BCS External Lists, which were designed with rich features for data virtualization. Why Microsoft doesn’t promote this is beyond me. Conceptually and technically, SharePoint could become the only interface end users in your company need ; they can access data from any Saas, or on-premise application, or any data from electronic instruments in one place. Federated data aligned from multiple systems can be presented in SharePoint out-of-box web parts with live refreshes from all of the sources. SharePoint even has the capability to enforce end-user specific security at the data field level for all CRUD (Create, Read, Update, Delete) functions, using SSS (formerly SSO) or claims authorization approved live at the endpoint application.

Probably one reason that these features are not often leveraged has to do with the scary-looking requirements for the format of the metadata you have to provide and import into the Business Data Catalog. It is, actually, pretty complex, with the need to generate a big hairy XML file and provide a series of specialized web services to use it. But that's where the product companies that specialize in data federation and virtualization should all be stepping handily into the picture. Are those companies oblivious to the ubiquitous state of Microsoft's SharePoint? Stone Bond Technologies, a leader in enterprise data federation and virtualization automatically generates the metadata for SharePoint as well as auto-generating and deploying the web services that execute the bi-directional (if desired) interaction with the backend applications live, federating and transforming across multiple sources.

This approach does not require a team of ten architects, twenty programmers, and long development/implementation cycles. Maybe that's one of the disconnects: the architects and developers can't even imagine such projects taking less than a day. Just think - they could actually take a vacation once in a while.

Friday, March 23, 2012

IT and OT - Shall the Twain Never Meet?

Yesterday I learned a new term in a conversation with Gartner: “Operations Technology," or “OT.” Operations Technology, they tell us, is separate from “IT,” which has the business technology focus. Having frustratingly observed this gap in the past, I found it particularly interesting to see that finally it is being recognized, although the recognition could instead work against closing the gap, forcing categorization one way or the other.

When I first graduated in computer science and began working, I worked at Shell Oil Company, along with the other dinosaurs. There were two totally separate groups, “business computing” and “technical computing.” The business side handled the mundane COBOL programs for accounting and such as well as the infamous midnight data uploads. We on the technical side focused on all the fun stuff that impacted the actual core business of an integrated oil company. I made sure I never admitted to having any knowledge of COBOL, lest I be drawn to the dark side. My focus and passion for computer graphics (in the days when you had to write to the pixel on/off level) led me to develop graphics for 3D reservoir simulation, engineering design, and manufacturing plant simulation.

As I moved on in my career, I observed from both sides the lack of mutual respect for the activities and technology requirements on the other side of the divide. I also discovered that operations couldn’t make the best decisions without a view of the market and the impact of those decisions on the overall well-being of the company. I think the hype wave of re-engineering in the late 80’s recognized this, but the IT department usually did not touch anything on the operations side, and did not want to deal with systems that include process control, complex mathematical optimizers and simulators. So the re-engineering efforts, moving on to BPM, pretended that there was nothing beyond the head office.

It was on this OT side that I began to contemplate how to put in the hands of the engineers a tool that would allow them to get the data they needed for their various models without programming. After all, they were not programmers, and it was impossible to get the programmers to leave what the engineers called the “Ivory Tower” to help out. (That turned out to be the origin of Enterprise EnablerÒ, an Agile Integration platform that naturally bridges the gap between IT and OT).

Apart from some ad hoc custom programming to feed the likes of SAP, there has been little done to bridge the gap. In my humble opinion,this is largely due to two factors. First, the Big Consultants rarely had the depth of knowledge to delve into this type of industry-specific systems and requirements. Second, the Big Integration products were designed as either ETL or EAI, neither of which alone could handle the needs of operations. Of course, integration solutions that bridge IT and OT would likely be funded by both, and there aren’t many people who are in the position to buy that understand enough technology or care enough to make it happen.

It’s much safer to get excited about Big Data and figure out some way that BD can be important. Forget closing the gap between IT and OT. That sounds too hard. But if your competitors decide to bridge the gap, you will definitely lose the edge!