Puppet: System Administration Automated

The CMDB Is A Consultant's Myth


Update: Looks like the author has wisely taken his post down.

So, based on recommendations of a friend of a friend, I've been casually reading up on ITIL, including adding a few ITIL bloggers (yes, there are ITIL bloggers) to my blogroll.

So, I come across this gem in my feeds today:

The CMDB is, first and foremost, an application system. It is not an infrastructure service (like networking), nor is it a core operating system, nor is it middleware. It is an application to be used in the fulfillment of use cases that add value to the efforts of stakeholders.

(My emphasis added.)

I've long thought that the CMDB is just a bunch of crap. What is the (usually 'the', not 'a') CMDB? Well, it stands for 'configuration management database', but as far as I can tell the term is entirely meaningless. It's basically what the whole ITIL world has concluded they don't understand, wrapped into a thing and named.

If the above is a definition of a CMDB, I'll eat my shorts.

The article is apparently a rant, explaining how everyone else is just a pretender because they aren't all super-architect/developer/sysadmin ninjas like the author is.

Ugh. This stuff just drives me nuts. If the CMDB were something real, like a web server or an LDAP service, then you could talk about the actual product, instead of a bunch of "you're not good enough" crap. Just once I'd love to see someone talk about what the CMDB is, rather than what it isn't, in plain terms. I'm not convinced it's possible, because I'm not convinced anyone knows what it is.

This just seems so much like the ERP debacle of the last 15 years or so, where very large companies all competed for who could waste the most money on a product they didn't understand. "We don't know what it is, but we're sure we need it." Just like the ERP crap, any real solution depends on stand-alone, decoupled systems working through abstract interfaces. That means no huge monolithic applications that no one understands and no one can maintain, and it means replacing existing systems, not just layering on top like drywall mud.

It doesn't hurt that this blogger actually calls his blog "ERP for IT", which should say enough right there. "We don't know what it is, but we sure know it's profitable to the consultants and vendors."

add to del.icio.us Add to Blinkslist add to furl Digg it add to ma.gnolia Stumble It! add to simpy seed the vine TailRank post to facebook

Wed, 15 Aug 2007 | Tags: , , ,


Posted by The IT Skeptic at Wed Aug 15 09:28:52 2007
I actually have a lot of time for Charles Betz's ideas.  He and I disagree on CMDB and on ERP but he's still a strong thinker.

if you want some scathing criticism of CMDB check out http://www.itskeptic.org/cmdb

Posted by Alex at Sat Aug 25 00:32:18 2007
I've been working in the configuration management space for more years than I care to count, mostly helping implement or improve solutions for companies that provide software as a service. I've been watching ITIL develop, optimistically waiting for it to bear practical and useful results. I am still waiting. 

Most engineers that actually perform the routine day-to-day configuration management duties (ask someone to narrow down what CM includes) have not penetrated the ITIL literature. Sure, some have used a bit of their company's budget to spend the thousand-odd dollars to  purchase the ITIL reading material but a rare few have really penetrated any of its text. I am not talking about the enterprise architect types, that work several layers up, often in the office of the CTO. I am referring to the engineers, at the sharp pointy end of CM: creating builds, pushing software releases, making system and network changes, etc.
These are the usually some of the busiest guys in IT and don't have time nor patience for learning hard to apply academic theory. At what point does ITIL become relevant to them? Does it get so baked into interoperating toolsets they are unknowing practitioners? Do management policies get re-engineered to the point that everyone's activity is aligned along ITIL dictates?

People that implement change today, need ready and accessible solutions that can be adopted within timeframes the size of weeks not years. As it stands now, ITIL does not offer tangible value to the guys on the ground.

Posted by Charles T. Betz at Fri Aug 31 04:48:54 2007
Glad to see you're now allowing comments. Let me break down an essential CMDB kernel for you: it would include tables for Server, Application, Database and Database, all with M:M relationships, and all subtypes of a master Configuration Item table; there'd be tables for Change and Incident, and  Changes and Incidents could be related M:M to any Configuration Item. All Configuration Items would have owners.

6 physical primary entities and 5 intersection entities - hardly a monolithic ERP system. The business purpose is to ensure that the impacts of a given change or incident upon a server or database are translated to the Application and the correct Application support owner is notified.

Skeptic and I continue to disagree, even on whether or not we are disagreeing. I actually have also criticized the concept of the monolithic CMDB that attempts element management as well as enterprise configuration management, and my book calls for an iterative and incremental approach to building the CMDB out. There's clearly plenty of risk. But the benefits are there - I have made the business case and gotten the money more than once. They revolve around impact analysis, cost transparency, regulatory compliance, and reducing the redundant re-surveying of complex IT environments, which has reached a crisis point in some organizations I know. At the end of the day, all that CMDB is calling for is that we have a single system of record for our internal IT data. It can even be a virtual system with cohesive sub-components interacting across well defined abstract interfaces. If you are unclear what the data scope of such a system might include, see here:

http://erp4it.typepad.com/erp4it/2005/08/a_data_architec.html

or my book. If you want a sketch of how the CMDB might interact with other major classes of internal IT systems, see here:

http://erp4it.typepad.com/erp4it/2006/04/here_is_a_simpl.html

or again my book for much more detail, including in-depth discussion of the provisioning/release problem I think you are focusing on here.

There's of course a lot more to the whole configuration management conversation, starting with the fact that there are at least three different definitions of config:

http://erp4it.typepad.com/erp4it/2007/01/a_couple_recent.html

http://erp4it.typepad.com/erp4it/2006/09/element_versus_.html

Alex - As far as ITIL realization goes, yes some of it gets baked into toolsets. Some of it turns into policies that those engineers may resent, such as "thou shalt always have a Change Ticket." If they don't study ITIL, their lives will be unhappier because they don't understand why the restrictions are being imposed (and trust me, they have less and less choice in the matter...)

Other people may find in ITIL a lifesaver when they reach the mythical "frog in the boiling water" point where heroics have finally failed and they need structure.

Some of what you touch on depends on people's career goals and outlooks. Do you want to stay "busy" being a hero? Or do you want to have a live as well? Systematic, repeatable processes look better and better to me, the older I get.

The fundamental value of ITIL is its platform independence, which may make it seem irrelevant to people who live and die by the specifics of their platforms. But in 30 years when we are all on optical-interconnected quantum computers, we will still have Incident, Problem, and Change processes and will still need some sort of representation of the interdependencies structuring our operational reality.

-ctb

Posted by Charles T. Betz at Fri Aug 31 15:42:22 2007
Funny. I just spent some time looking over Puppet and its underlying engine Facter, and Facter seems to be a discovery engine doing a lot of things that one can purchase a more expensive CMDB to do. Interesting. In its role as a Unix sysadmin tool with its own domain specific language, it is doing what I call element configuration management. It has at least checksum-based drift management.

Looks like good stuff, even if the (clearly talented) lead architect thinks I am a joker. Wish I'd had it when I was doing Unix sysadmin work.

Lot of the links on the site don't seem to be working.

It would be in the interest of Puppet Labs to take the whole CMDB concept very seriously; it's not a consultant's pipe dream, but is the face of your competition. I hold no stock in CMDB vendors and would love to see good open source alternatives; there is a CMDB open source effort at http://www.cmdb.info/pd1/html/ that might be profitable for you to partner with.

-ctb

Posted by Luke at Fri Aug 31 17:12:37 2007
Skeptic:  Thanks, I had not seen that site.

Alex: I completely agree.  ITIL does not at all seem tactical, something you can do right now, without having a 9 month planning session or something

Charles:  The problem with your definitions of CMDB are that it isn't really something you can use and touch, they're still largely just vagueries.  If you and I each set out to implement something that fit your definition, we would likely end up with something completely different.

If you've got a clean model of a real system, it should be a model you can explain to multiple groups and have them implement essentially equivalent systems.  Instead I think we have a description of a database that's as vague as the description of the term 'configuration management' itself.  I still can't get a clean definition of that, at least, not one that more than about 3 people agree on.

And I'm pretty skeptical that ITIL is a lifesaver when people have already reached the 'frog in boiling water' point, because they'll still need to spend a year figuring out what ITIL means to their company and how to turn these vague descriptions into tactical actions.

What someone in that situation really needs is a competent, action-oriented person who can come in and make pain go away very quickly.  Obviously they can't just do silly short-term actions, but they also can't afford to spend a year or more designing and implementing some kind of enterprise-ready something-or-other.

And that's my real problem with ITIL -- it's a very top-down, Fortune 500-first approach to the problem, and I don't think any tools developed for the Fortune 500 have every trickled down to the rest of the world.  Quite the opposite, really -- the small companies develop and hone both tools and practice, and it eventually trickles up to the Fortune 500.

That's where the ERP analogy comes in -- the problem is exacerbated in the enterprise, but it's present everywhere, and if we can solve it well in small organizations then it will soon scale up well to larger organizations.

As to my projects, certainly Facter can be used to do a poor man's inventory system, and when you use PuppetShow to store your configurations, you'll have a fine-grained inventory system, including all the resources that you're managing across all of your network.  The next iteration of PuppetShow will also include fine-grained reporting so you'll have per-resource status information on all nodes.

I think this would probably do a good job of meeting the definition of a CMDB, but it's not enterprisey, and it's something people will be able to download and set up in about a day's time for free.

Even this isn't the main point of Puppet, though -- it's a great side benefit, but the main point is that people who need to get work done today can use Puppet to fantastically simplify their lives, right now.  That's why it's growing like gangbusters, and why there are already at least 6 people making a living doing consulting and development on Puppet.

I'd certainly appreciate having any dead links pointed out to me.

As to the OSS CMDB you pointed out, I can't seem to find much of a product, just a pointer to a VM with a bunch of separate software installed.  That's not manageable for most sites, because they have their own OS requirements, and even then the applications need to be upgraded and managed, so just a downloadable VM isn't sufficient.  It needs to fit in with the company's other management tools, basically.

Posted by Charles T. Betz at Fri Aug 31 18:11:23 2007
Well, my focus is the largest of the large; my organization has a $1.3bn central IT spend and probably $2bn total. However there are some smaller scale interpretations of ITIL that might be of interest to you. I think that the core glossary and process architecture is applicable to any IT shop that has reached the tipping point, when their talented people can't keep it all straight in their heads any more. Adding one more person at that point, however action oriented, does not solve the problem and in many cases can make it worse. Fred Brooks' Mythical Man Month is required reading on this question - insights he gained decades ago on the IBM OS/360 project that still hold water today. Scale is a problem in IT.

In terms of actionable guidance - I don't get a lot of money for my book so please don't think that my motives are purely financial in suggesting you read it. The bits and pieces you are reacting to are part of a much larger whole that starts with a process architecture. It's not about the CMDB; it's about the Incident and Change processes (among many others) and the less formalized general knowledge management required by large IT shops.

It is true that implementations of what I've suggested might vary - that is not a problem to me, as I am working at the conceptual and logical architecture level. This is standard approach to systems analysis and design; I could easily translate what I've proposed here into a physical design but that part of the work is less interesting to me - someone would have to pay me to do it.

You may not understand that, along with the IT Skeptic, I have actually been a critic of much of ITIL's dogma. The debates have a long history.

-Charlie

Posted by Charles T. Betz at Fri Aug 31 18:29:10 2007
PS. Framing what you are doing in terms of ITIL support and enablement, rather than reacting negatively to ITIL, can only help the success of your project, which I think is a good one. Why dismiss an avenue that might lead to someone seeking out your software?

Please consider that.

-ctb

Posted by Luke at Fri Aug 31 18:40:49 2007
The Mythical Man Month doesn't apply in this case because we're talking about the ongoing IT support, not some development project with specific goals.  Anyone brought in would focus on specific ways to let IT accomplish more with less effort, which is entirely different from bringing another developer or something onto a development project.

As to reframing Puppet around ITIL, I think doing so might help me sell better to the Fortune 500, but I don't have much interest in doing so.  Sure they're profitable but they're also extremely high maintenance with an equally high sales cost, and they tend to distort design and process in a way I think is unhealthy.  There are very large shops that don't think in the Enterprise way, in that they're more interested in really competent people building highly efficient solutions, but they are few and far between, from what I can tell.

If I reframe around ITIL, I think it will put off my regular customers, who don't know or care what ITIL is and don't want to read a book on process just so they can get their work done.

I think that's the main disconnect between us here -- my customers want tactical solutions that they can get up and running in a week or so, saving them effort ASAP.  I don't see how ITIL helps them get any closer to that, nor how it helps me develop software that does so.

Posted by Charles T. Betz at Fri Aug 31 19:03:47 2007
PS. Framing what you are doing in terms of ITIL support and enablement, rather than reacting negatively to ITIL, can only help the success of your project, which I think is a good one. Why dismiss an avenue that might lead to someone seeking out your software?

Please consider that.

-ctb

Posted by Charles T. Betz at Fri Aug 31 19:09:09 2007
The issues inherent in large development projects are analogous to those experienced in operating large and complex environments; I've got a background in both software engineering and data center operations and have seen the similarities. Throwing in more people and relying on heroics fails in both cases.

You don't need to re-brand or fundamentally re-position your product. Just a minor mention that it may be of assistance in solving some of the ITIL CM challenges, and an openness so that if someone asks you if you can help with an ITIL initiative, you don't dismiss it out of hand. There are smaller shops working with ITIL.

FWIW. 

-ctb

Posted by Charles T. Betz at Fri Aug 31 19:21:19 2007
The issues inherent in large development projects are analogous to those experienced in operating large and complex environments; I've got a background in both software engineering and data center operations and have seen the similarities. Throwing in more people and relying on heroics fails in both cases.

You don't need to re-brand or fundamentally re-position your product. Just a minor mention that it may be of assistance in solving some of the ITIL CM challenges, and an openness so that if someone asks you if you can help with an ITIL initiative, you don't dismiss it out of hand. There are smaller shops working with ITIL.

FWIW. 

-ctb

Posted by Luke at Fri Aug 31 23:39:35 2007
I definitely didn't say throw in people and rely on heroics -- the point was to bring in a completely different methodology with significantly better tools.

The problem with mentioning ITIL is that I'd have to spend time learning about it and how people could actually use Puppet with it, and I'm not convinced it's worth the effort.  I mean, sure, it could theoretically help, sure, but unless I know how, I'm just talking.

It's not the idea of ITIL I'm against, BTW, it's talking about it as though it were more than a vague process, or as though it's going to save anyone.

Posted by Shamgar at Wed Nov 28 04:22:28 2007
So, I did some thinking after we talked at LISA a couple weeks ago and decided to more forward with giving puppet a try.  So I've been reading through various things here - and tonight that was your blog.

I'm sure it probably seems silly, but this post is very reflective of why I kept coming back to puppet as an option despite my reservations.  I've been in the Config Management BoFs for the last few years at LISA, not really talking just listening.  And yours have consistently been the only ones that seemed really focused on the work at hand.  Theory is important but in general it reminded me of something I couldn't put my finger on.

Now I know what it was.  I spent several years in a very large company.  One with disparate systems that worked well enough together for 100 years.  (Not the same ones obviously, but the company had been in business that long - and longer.)  The Payroll system was written in COBOL and their programmers were all retiring.  So they decided a new one.

This snowballed into a full ERP rollout across the entire worldwide organization in all of its locations across all systems.  They refocused the entire IT staff on it, and hired a literal army of consultants.  Millions of dollars a month were being spent.  I left after about 6 months of this.  A few months later, everyone involved was fired and the old systems were put back in place.  (A happy ending in my opinion - it could've been worse.)

I'm not saying there's nothing of any value in the concepts primarily under discussion, but I greatly appreciated this post for helping me focus on that one elusive aspect that kept drawing me back to your work.

Posted by Luke Kanies at Wed Nov 28 22:17:50 2007
Shamgar,

Thanks for the feedback.  I'll try to keep the useful, no-nonsense posts coming.

Name:


E-mail:


URL:


Comment: