Export and Collection -- 0.17.0 is Out
This is a post from Luke's old blog; it is saved here statically for historical purposes, as of October 2008
0.17.0 is out now, and it's got a very exciting new feature: Export and collection of objects, so that you can configure one how within the context of another host. This feature is new and has not yet been deployed as widely as the rest of Puppet, but it has a lot of exciting possibilities.
I don't have a lot of examples for this just yet, since it's such a new feature, but it's something that we've been working on almost since the inception of Puppet, and we've been talking animatedly about it for months.
The first thing to note is that this feature introduces two new dependencies, although they're only dependencies if you use the feature. You must have Rails installed, and you must have the ruby libraries for whatever database you choose. I've only tested it with Sqlite3 so far, but since all of the database support is done via ActiveRecord from Rails, I don't see why other supported databases wouldn't work. These are only server-side dependencies, though -- the client doesn't need them, and in fact will never even know if you are using this feature.
The other thing is that, for now, you have to specify that you want this
feature enabled, either by adding storeconfigs=true to the
puppetmasterd.conf file, or adding --storeconfigs to the command line
when starting puppetmasterd.
If you just add that flag but don't modify your manifest, you'll notice (in verbose mode) an extra log statement:
info: Stored configuration for hostname.domain.com in 1.86 seconds
This is the evidence of Puppetmaster storing each client's configuration in the database. You'll notice that's relatively slow (and can get slower for larger configurations); from what I can tell in talking to people in the Rails community, Sqlite3 is just not suitable for server usage. If you want to deploy this in production, I recommend you deploy it on Mysql or Postgres or something similar. You're going to want to do that anyway, just to enable multiple puppetmaster daemons to share a single database for load balancing and horizontal scaling.
Okay! All the preliminaries out of the way, here are a couple of examples of what you can do. Given that Facter already knows how to retrieve SSH keys, here's an easy way to make sure that every machine on your network has every other machine's SSH key:
class ssh {
# ... do the stuff related to making sure ssh works ...
# Now add the key, if we've got one
case $sshdsakey {
"": { # ignore empty keys
}
default: {
@sshkey { $hostname: type => dsa, key => $sshdsakey }
}
}
# Now collect everyone else's keys
sshkey <||>
}
The sshkey type manages the ssh known host keys, normally found someplace
like /etc/ssh/ssh_known_hosts.
The two critical parts of this configuration are the export operation, which
is just a normal element declaration prefixed with an @ sign, and the
collection operation, which is the statement with the <||> symbol. At
some point this statement type will support querying based on parameter
matching:
sshkey <| type == dsa |>
... or ...
host <| ip =~ /192.168.0/ |>
Puppet does not currently support operators like that, though, so it seemed more important to get this functionality out the door and add the operators later.
This configuration is interesting, and was actually really complicated to support, because each host is collecting the objects that it exports. This is probably not going to happen as often as asymmetric operations, like the following:
class ldapserver {
# ... do the stuff to make it run slapd ...
@nagiosservice { [ldap, ldaps]:
host => $hostname
}
}
class webserver {
# ... do the stuff to make it run apache ...
@nagiosservice { [http, https]:
host => $hostname
}
}
class mailserver {
# ... do the stuff to make it run smtp and imap ...
@nagiosservice { [smtp, smtps, imap, imaps]:
host => $hostname
}
}
class nagiosserver {
nagiosservice <||>
}
You can see here we've got the same basic statement types, but we're creating
nagiosservice elements, instead of sshkey. Unfortunately, there is
actually no such element type at the moment, but considering that support of
Nagios configuration generation was the prime motivator behind finally
developing this, it will be out shortly.
So, you can see here that each of our hosts that runs a service needs to be
monitored. So, it configures a nagiosservice object that has the
necessary information to perform that monitoring. The problem is that we
don't want each host to run Nagios; we want this configuration to be sent to
our server somehow.
This is where the export and collection mechanism comes into play: Each host
exports its nagiosservice objects (and it could have many of them), and
the Nagios server collects them all then turns them into its running Nagios
configuration.
Like I said in the beginning, this allows you to configure one host (in this case, the Nagios server) from within the context of another host (any of the other servers). This could be useful for doing things like configuring firewall ports from web servers, or smtp setups from the smtp server.
Because this functionality requires that all client configurations be stored in a database (well, technically, it only requires that exported configurations be stored, but that would be silly), this also makes it easy to write a Rails app that will be able to interact with that database, performing queries for host statistics (when they last connected, how long they took to run, etc.) and also for figuring out what elements are deployed to which machines. I have not yet begun this Rails app, but it should be pretty straightforward to write, since the entities are already all in place -- I just need to write the views and controllers, and I'll be done.
Enjoy!
