SOA is Dead; Long Live Services

Blogger: Anne Thomas Manes

Obituary: SOA

SOA met its demise on January 1, 2009, when it was wiped out by the catastrophic impact of the economic recession. SOA is survived by its offspring: mashups, BPM, SaaS, Cloud Computing, and all other architectural approaches that depend on “services”.

 

Once thought to be the savior of IT, SOA instead turned into a great failed experiment—at least for most organizations. SOA was supposed to reduce costs and increase agility on a massive scale. Except in rare situations, SOA has failed to deliver its promised benefits. After investing millions, IT systems are no better than before. In many organizations, things are worse: costs are higher, projects take longer, and systems are more fragile than ever. The people holding the purse strings have had enough. With the tight budgets of 2009, most organizations have cut funding for their SOA initiatives.

It’s time to accept reality. SOA fatigue has turned into SOA disillusionment. Business people no longer believe that SOA will deliver spectacular benefits. “SOA” has become a bad word. It must be removed from our vocabulary.

The demise of SOA is tragic for the IT industry. Organizations desperately need to make architectural improvements to their application portfolios. Service-orientation is a prerequisite for rapid integration of data and business processes; it enables situational development models, such as mashups; and it’s the foundational architecture for SaaS and cloud computing. (Imagine shifting aspects of your application portfolio to the cloud without enabling integration between on-premise and off-premise applications.) Although the word “SOA” is dead, the requirement for service-oriented architecture is stronger than ever.

But perhaps that’s the challenge: The acronym got in the way. People forgot what SOA stands for. They were too wrapped up in silly technology debates (e.g., “what’s the best ESB?” or “WS-* vs. REST”), and they missed the important stuff: architecture and services.

Successful SOA (i.e., application re-architecture) requires disruption to the status quo. SOA is not simply a matter of deploying new technology and building service interfaces to existing applications; it requires redesign of the application portfolio. And it requires a massive shift in the way IT operates. The small select group of organizations that has seen spectacular gains from SOA did so by treating it as an agent of transformation. In each of these success stories, SOA was just one aspect of the transformation effort. And here’s the secret to success: SOA needs to be part of something bigger. If it isn’t, then you need to ask yourself why you’ve been doing it.

The latest shiny new technology will not make things better. Incremental integration projects will not lead to significantly reduced costs and increased agility. If you want spectacular gains, then you need to make a spectacular commitment to change. Like Bechtel. It’s interesting that the Bechtel story doesn’t even use the term “SOA”—it just talks about services.

And that’s where we need to concentrate from this point forward: Services.

Comments

[Tim Vibbert]

I totally agree with your assessment of the state of the SOA union. Without a commitment to changing the status quo, "SOA" efforts fail. Most efforts focused on the platform stacks and not the enterprise. I've began to ask "What is your SOA motivation?", if not the business/enterprise then why do it at all.

[David Bressler]

Dave,

Great article, and spot on. In fact, what I hope happens is that people hear "SOA must die, so we can get on with, well... SOA".

I think a some of the confusion can be attributed to companies focusing on the platform to deliver services, instead of the services themselves. And, in trying to be like the Romans and get everyone to do things the same way, all the time.

I think there is beauty and opportunity in the chaos... as you point out, with situational apps, cloud computing, etc... there is a broad horizon for attacking "the business needs" and delivering hugely unlike ever before because of how easy it is becoming to bring information together and present it in standardized ways for the user.

'Nuf said.

David
Progress Software, Actional

[Scott Francis]

Anne - excellent summary of my feelings about SOA :) I've actually had conversations with IT professionals about their SOA initiatives where they had trouble defining what SOA stood for, and not understanding that one could have a service-oriented-architecture without buying a "SOA" stack of software.

Another pet-peeve is when every single integration is written as point-to-point with SOA in the middle. Instead of rationalizing the services provided by a system, each time a new client comes along, the SOA team provides a brand new interface with a very use-specific service interface and implementation (possibly even cutting-and-pasting from other implementations against similar parts of the infrastructure). This, to me, defeats the whole point of writing services - that at some point, a new application trying to tie into your services would look at what you've written and largely already have their needs met by what previous projects required from the "SOA" team...

[chuck georgo]

My sincerest apologies, but I must disagree with your assessment of the relationship between the economy and SOA. In fact, it's my feeling that a hurting economy will actually drive MORE adoption of SOA principles across an enterprise, not less. SOA is about three things--effectiveness, efficiency, and agility. If your infrastructure is meeting business needs, and you have lots of money, and the business you're in isn't volatile, then skip SOA and keep doing what you're doing.

However, if your business users are screaming for better capabilities, or your budget has been slashed (or will be slashed), or you're in a volatile line of business where business rules and needs change rapidly, then I suggest that you MUST consider moving, albeit in a measured aproach, towards knocking down the vertical application stovepipes and towards an SOA approach.

The one point I totally agree with you is that SOA is disruptive to the organization, but even more so, it will be extremely disruptive to software vendors that continue to bet the farm on large, monolithic applications--SOA will require "disassembling" of these applications into reuseable service components.

I tell my associates and clients that a true example of SOA will be the day I can call out to Corel WordPerfect's spellchecker while I'm working in a Microsft Word document...thanks..r/Chuck Georgo

[Jeff Griffith]

The problem that killed SOA is the same now as it was in 1989; no one can decide what services an application provides. Back in those days, I worked on an application development platform for a robotic system in the lab automation market. The chemists insisted that the "primative operations" provided by a particular syringe pump unit were not just "valve-to-port-1", "valve-to-port-2", "draw" and "expel", but instead were complex and context dependent combinations of these operations (e.g. "wash syringe", and "wash bottle" which assumed a bottle was connected to syringe port 1). The same problem exists now; everyone wants to specify the system in the terms on the problem they are trying to solve, instead of specifying it in terms of the data that the system uses. They do this because they know what their goal is, but not the resources available to apply to the problem, and consequently, the methods needed to apply these resources.

To explore a more typical IT example, an insurance company marketing executive defines a new type of plan to protect individuals from losses caused by theft of their IP services over wireless LANs. He goes to the IT department and asks them to create a web site for it. The insurance company's IT group gets a dozen such requests a day and they are expected to provide instant gratification. Every site request designs the site based on the problem addressed by the plan, and the IT department has to build a whole new system for each one from scratch. The tight delivery deadlines cause the IT department to look for similarities between the sites, and, after a while, they find some; the word "insured" appears in every document, along with words like "damages" and "premium", and they always appear to have the same meaning. After more searching, the IT department discovers that all such sites could be built from a system that exports blocks of data that model the company's idea of an "insured" and a "damage" and a "premium" among other things. So, just before they start building such a system, they decide to validate their idea with the business experts. That's when the feces hits the fan. The business experts immediately note that a life insurance "insured" is not the same as an automobile insurance "insured". In some states the auto "insured" is the person driving the car, so the "insured" is functionally a car, not a person. The fact that people are named at all is really just a way of identifying the car that is covered ("we insure Jeff's car, not Jeff, but we rate the policy based on Jeff's friends, his neighborhood and education because these things affect who Jeff is likely to let drive his car"). And so it goes, as each type of policy finds a way to prove that its data is unique to its problem. The depressed IT department goes back and looks at the problem again. They find that the relationship between a "policy" and its "insured" is always the same, it's the details of the "insured" that change. And when a policyholder wants to get information about his policy, they always have to find out who the "insured" party is. So, IT solves the problem by replicating source code from project to project, and every time they need a new database, they just take a backup from the last system they built an extend it.

Practices like this prevent any form of "standard insured" from being developed in the insurance industry. No insurance businessman will support this because there are nuances and implications built into the idea of an "insured" that are different for each product type. To see this, what happens when a covered person has no brain activity but is on a life support system; are they dead and does a life insurance policy have to pay, or are they alive and the medical insurance policy has to pay. In reality, life insurance policies assume you're dead when a death certificate is written, because doctors will only sign a document for people they pronounce as "dead". All of the language in a life insurance policy surrounding the word "insured" assumes this to be true. But some medical insurance policies assume you're dead when your organs can be harvested for transplantation, even without a death certificate. And all of the language surrounding their term "insured" works to limit the medical policy's liability within this context. So such a person is neither alive nor dead, and it will take a court case to decide which policy is liable. A software service could only deny all payment given these rules.

These kinds of "business context" assumptions are ultimately coded into the business rules of the systems and they make it very difficult to identify the differences between systems and the data resources the systems manage. As a result, it is difficult to decide what a "service" is and what it should do.

Software developers need to understand that a service can only exist within an operating context; business people need to understand the an address is an address every place it appears. And while software systems might be able to standardize the data of an "insured", and they may be able to standardized the relationship between a policy and its "insured", the business context will always limit how the data is used, and the service will always have to include this business context. This means that every service will either provide just data, or the service will be specific to a business context, but unless the business itself is defined to use stanard contexts, the services can never be shared between contexts.

[Dhananjay Nene]

Not sure why the trackback did not appear here, but here's a link to my post : SOA ain’t dead but it certainly is transforming : http://blog.dhananjaynene.com/2009/01/soa-aint-dead-but-it-certainly-is-transforming/

[Sergey Orlik]

Thanks for the great postmortem analysis. But it's a Part I only. Part II - discussing the right place of service-orientation.

The roots of SOA problems - people initially misunderstood what SOA stands for. It was crucial for SOA that "services" were substituted by "well-defined interfaces".

Services by the nature are about business and organization structure instead of middleware and applications. And Service-orientation is more about Enterprise Architecture way. The main role of services is glue for business architecture (including organizational and functional views) and IT architecture (from apps to network, servers and storages). There are no such things like business components - there are business services. Business don't know anything like components, but understand what are the services, providers and consumers. It's time to rethink 4+1 approach and to use services instead of artificial business components. It's an only way for IT to be aligned within Business.

SOA as is dead, almost; Long Live Service-Oriented Enterprise Architecture, in any case.

[Anne Thomas Manes]

George: I agree with you that the need for application portfolio re-architecture and service-orientation is stronger now than ever before, but [in most organizations] the business guys that hold the purse strings don't understand that, and the IT guys haven't made a compelling business case to support the claim. End result: budgets will get cut and SOA dies.

[Jeff Schneider]

Anne,
When SOA was hot, I would have said SOA was failing because people were talking about it not doing it. Many SOA programs continue to suffer from 'PowerPoint syndrome' and too much emphasis on governance (without execution).

The funny thing is that I've seen SOA move into the engineering groups and out of EA. I'll admit that I have a biased set of projects to survey, but those that I observe have moved towards service orientation in a very, very strong way.

The software developers are thinking about the services early on, often before data models or object models, or in parallel. Finally, I'm seeing SO-Analysis and SO-Design being commonplace. Even my most junior guys will present the clients with an early list of service candidates, usually in a RESTful notation.

What I would say has failed is big, centralized governance programs. From my estimates, you ran about 10+ webinars on BIG SOA GOVERNANCE. It sure would be nice if you'd follow it up in 2009 with 10+ webinars targeted at engineers on the basics of service identification, analysis and design.

SOA didn't fail - however, all attempts by analyst firms to hijack it did.

Long live practical SOA.
Jeff

[Anne Thomas Manes]

Jeff -- Interesting that you say that. Service modeling is going to be the focus of our SOA research in 2008.

My point is that the term "SOA" is dead -- requests for funding of things called "SOA" will be denied. So as I said, we need to remove "SOA" from our vocabulary.

But service-orientation is more important than ever before. So it's great news that developers are starting to embrace services -- RESTful or otherwise. I haven't seen too many people in the typical large-company development organizations adopt REST yet, but I'm ever hopeful.

[chuck georgo]

Anne: Aboslutely spot on! The gap between business and IT is a significant contributor to SOA's slow/stagnant adoption. And, as an "IT guy, I place the responsibility for this squarely on the shoulders of IT.

If instead of "SOA" had we called it "Generate more revenue and have happier customers by really explaining to the IT staff what it is you really want your IT infrastructure to do without wasting money" then the business side would have probably jumped on it...instead we came up with "service-oriented architecture"...and we're surprised and upset they aren't biting? r/Chuck Georgo ;-)

[David Bressler]

Ooops, sorry Anne. I saw this post via a tweet from Dave Linthicum, so thought it was his post! Yikes.

Nice post!

David

[Mark Griffin]

I think Jeff Schneider's comments about practical SOA are spot on. I have said and will continue to say that the technical aspects of good service design were often glossed over or ignored completely. There is still not a wealth of information on this although it is getting a little better.

We went through this same thing with EAI. The promise of the agile, loosely coupled enterprise was highly touted by vendors and analyst alike. But the meat was left out and IT shops went out and bought huge expensive platforms and then proceeded to recreate the same spaghetti they were trying to get away from.

In depth conversations are needed if effective services are ever going to become mainstream for IT organizations.

markg

[Dana Gardner]

Could be that the enterprises that dismiss SOA, nomenclature or computing shift, will be all the more ready to hand off more of their IT functions to the SaaS and cloud providers that do do SOA well and pervasively. In other words, you'll do SOA one way or another ... it's just whether it's your competency or your cloud providers.'

Smartest enterprises will do both and in the correct combination. The rest will have exorbitantly expensive IT that does not respond to change nor exploits the low-cost cloud alternatives. And that spells bankruptcy (sans bailouts).

[Juergen Auer]

> SOA needs to be part of something bigger.

Agree.

Most of the SOA-Solutions are like classical programs in the web - a static set of tables as background, static pages. Every customer has the same - the business processes has to adapt to the structure of the applications.

Terrible.

The other way is to use a system as background, which allows to create specific, individual tables. So every customer has its own set of tables, its own name conventions, labels etc. The applications can be adapted to the business processes and their changes.

And the service of implementing these changes quickly (in one, two days) is mission-critical. So a customer may start using four, five tables - and a year later new business activities are coming, so the web solution grows up to 20 or 40 tables.

[Duane Nickull]

Anne:

My response:

http://technoracle.blogspot.com/2009/01/soa-is-not-dead-but-complexity-is.html.

Duane Nickull
Sr. Technical Evangelist, Adobe Systems, Inc.
Chair - OASIS SOA RM Technical Committee

[Tobias Manthey]

Hi all,

We are still in a time, where most people assume they have a SOA when they got a web serivce up an runnning (due to the word "service" in both terms).

I would assume that 90% using the word just play buzzword bingo and have actually no clue about its meaning.

Because SOA has not even really started yet.

SOA is a prerequisite related to enterprise ESB and BPM. If SOA goes, so do ESB and BPM.

I see the evolution of "Services" as follows

1.) (Web) Services
This is a classical 3-tier architecture
2.) SOA
You break up the static link between services and client by adding some sort of serivce registry
3.) ESB
You connect all your services to an ESB. Instead of searching the service of your choice yourself you let the ESB Hub decide.
4.) BPM
On top of your ESB you add BPM to design your IT to your business processes. You are able to monitor and change your processes any time.

Web Services are some sort of settled.
SOA is still young technology, but we start to get idea what could be a SOA.
ESB? What is ESB? Nothing but buzzword so far. In the moment just all former EAI vendors relabel their old fat systems with the current buzzword word "ESB". Same for BPM.

I see that IT will be continue to evolve towards BPM. Heaven is, when our services reflect exactly the users business processes and we can change our business processes simultaneously with our IT .

Business Process and IT must be one.

Btw.

If econony affects technology it means you got the wrong consultant!!!

SaaS? Forget it. Its ASP relabeled. Better stick to the old term.

Cheers,
Tobias

[Dave Chappell]

Hi Anne,
Based on my travel schedule over the past several months, which has been about 90% of my time going to meet with customers who are actively engaged in funded SOA projects, I would say SOA is alive and well. One could argue that there are discretionary projects being put on hold due to budgeting restrictions, but that has little to do with whether they are SOA.

But hey what the heck, I'll join in -
SOA is Dead. Long live service-orientation! ....and architectural patterns and practices to ensure that an IT organization has consistency across applications that are built using service orientation....Oh and I guess we'll need a means for ensuring that those services and their associated artifacts can be stored, versioned, secured, and accessable from a variety of interface technlogies and protocols. And we can't build it all from scratch so we'll need to be able to leverage our investments in existing application assets....then we'll need to compose services together, and perhaps even orchestrate the interactions between them....We'll also need to have a working plan in place to communicate between IT and the business to ensure that the right services are being built at the right level of granularity so that applications, which leverage those services, can have the flexibility and agility to meet the increasing and ever changing demands from the business. And a way to govern that process.
Long live service-orientation! Bravo!
Dave

[andrej]

Here's my feedback: SOA is not dead, we're still in the early adapter phase (http://www.andrejkoelewijn.com/wp/2009/01/06/soa-is-not-dead-were-still-in-the-early-adaptor-fase/)

[Ronk]

Anne,

Based on the fact that you think SOA is dead, you might really want to consider updating your LinkedIn profile. :-)

Regards,

RonK

[Lee]

As the Zapthink guys keep saying "SOA is something you do, not something you buy". The big problem with big SOA projects has been companies insisting that SOA can be "purchased" and then "installed and configured". Most of the failed SOA to me looks like nothing more than big integration projects that have gone horribly wrong - They never were SOA in the first place.

Hopefully SOA isn't dead but the chronic misunderstanding in the marketplace about SOA is.

[Jim Butler]

Nice piece. I believe strongly that the over-hype of the term was actually holding back the real practice. The principles are valid and essential, but the expectations and agendas were killers. With the term and its associated distractions dimming, hopefully we can just get on with it.

This phenomenon has many parallels to what happened with AI. As I wrote in http://drjbutler.wordpress.com/2009/01/06/soa-in-good-eternal-company the term "AI" became a pariah while its byproducts are now pervasive.

[Harvey Stage]

Dear Anne,
What a boat load of regurgitated crap. What does this '...SOA requires disruption to the status quo' mean anyways?! Perhaps no one really sat you down and explained the meaning behind the term SOA. Anyone intelligent could have pointed out that SOA is not about services at all and has never been about services or the technology. That's only a 1/3 of the story. The other two components are: business processes re-structuring and operations re-structuring. Perhaps you were listening to some IT contractor out of a temp-hire agency.
I'm afraid that you've totally missed what knowledgeable practitioners had always been preaching. SOA, as you define it, was only ever limited to people who never truly understood the vision of it all and, instead, saw it as a cure for everything. So, I guess, in essence you've just discovered the concept of 'hype'. Bravo! Well done.
This is precisely why I’ve come to expect nothing of insight from any of the research vendors. You do a shallow analysis, which often is incomplete and lacks any real substantiative details, and then proclaim it as some newly discovered wisdom.

Truly disappointed,
-harvey

[automotive crm guy]

SOA and "Services" are fundamentally the same thing. The ability to expose an API via web services.

[Francis Carden]

IT and Business have been head to head ever since business became frustrated at IT for taking too long to add an extra field to their mainframe screens.

What do IT expect? SOA is about IT doing it right and IT should be doing it right whatever the acronym. However, doing it right takes too long, costs too much, risks complete failure and time has proven, can't always been done at all.

The problem goes around and around but one thing that 2009 is all about is making do with what you already have and making it work a lot better. Shareholders will expect that. If you are laying off people to save money, don't expect to be let off lightly when you lose customers. Make what you have work better by optimizing the employees (end users) desktops and buy yourself time. There are $trillions of savings to be had TODAY with iterative and agile integration products. Keep planning for the perfect architecture but don't ignore the fact you want to remain in business to see it through - eventually....

SOA is a very long journey. A journey that will see businesses go under or remain uncompetitive unless you look to other solutions to business problems whilst you take that journey. They are out there if you look.

http://blog.tmcnet.com/desktop-productivity/

Francis