Pervasive Tagging

I'm writing this on the plane to the Web 2.0 conference.

I now have basic tagging working in Puppet. They're generated on the server when a configuration is provided, and they're stored with the individual objects in the configuration. For instance, here is what some of the tags look like in a short, example configuration on my OS X laptop::

file(/etc/issue):
    remotefile base nodebase tsetse puppet file /etc/issue
file(/etc/motd):
    base nodebase tsetse puppet file /etc/motd
file(/root):
    remotefile base nodebase tsetse puppet file /root
file(/tmp/screens/.):
    base nodebase tsetse puppet file /tmp/screens/.
file(/usr/local/scripts):
    remotefile base nodebase tsetse puppet file
    /usr/local/scripts
file(/var/spool/cron):
    base nodebase tsetse puppet file /var/spool/cron
symlink(/etc/resolv.conf):
    darwin nodebase tsetse puppet symlink /etc/resolv.conf

I'll break down the configuration that generated this admittedly-basic list of objects and tags.

• tsetse The laptop name
• base The base class; all nodes are members
• basenode The, um, base node; all nodes inherit from it, and it basically just loads the node's operatingsystem class and the base class.
• puppet This is the top-level collection of the entire configuration. This tag may not stay, since it would currently be on every single object.
• remotefile A simple wrapper function to encapsulate my primary method of copying files around my network.
• darwin The laptop's operating system class (based on the output of uname -s)

For each line, the last two tags are the object type and the path to the object (they could also be service names, user names, etc.). I think I should also add a tag for each of the files that mention the object, along with the respective lines from each file.

The tags associated with a specific configuration are sent with the configuration to the appropriate server, but they're also merged into a central repository, so as each node connects and generates its tag list, the central tag repository gets more comprehensive. (Each node's configuration needs to be compiled before its tag list can be generated, and the configuration requires information from the client before it can be compiled, so the nodes need to connect to get its tags.)

Possible uses

The tags are currently unused, but their mere existence has got me thinking:

• Logging

  I have been expecting that most people would just use syslog to pass logs around -- it's easy, it's pervasive, and Puppet has been developed specifically to be easily compatible. However, it would be difficult to shoehorn tags into syslog, or at least it would be annoying to get them back out; it would need to be based on pattern matching a specific line, which is never pleasant. I'm thinking instead that I could write a log server capable of receiving these log messages, and then enhancing the log messages to store the tags associated with the node that generated the message.

  Imagine being able to trivially find every log message generated by any darwin node in a given time period, or the the error messages from a specific network associated with DNS. Build a database of all of these log messages, with the fields and tags set up appropriately, slap a Rails interface on it, and you could get that pretty easily.

• Metrics

  Similarly, I have been expecting people to ship their Puppet metrics off to a specific performance app, but there probably aren't any apps out there that gracefully handle associating arbitrary tags with each metric. Build a simple server to accept the metrics that Puppet generates, and suddenly you can generate change count reports on a specific server class like 'webserver' or on how many out of sync security directives there are in the DMZ.

• Ticketing

  I know I'd like to have each of these tags stored with each ticket generated from Puppet (yes, I hope to eventually have Puppet autogenerate tickets). Query for all outstanding tickets on the web farm or any tickets related to the recent Solaris upgrdae.

• Reference

  With a web-based annotation system for the configuration, you could use tags as a kind of wiki keyword system -- selecting a tag in a log message automatically finds the definition of that tag (as a server class or functional component or node or whatever).

Tag injections

One of the thing unaddressed in the current proof-of-concept system is that servers will often want to generate their own tags. In particular, I can see adding an 'unmanaged' tag to objects that get mentioned in a configuration (e.g., as a requirement) but that are not managed, or storing the sync state as at least a local tag for an object (e.g., error, synced). Should these tags make their way to the central system, or should users at least be able to get access to those tags?

State tags

The concept of nodes injecting tags brings up the concept of maybe using tags as a means of storing state, or rather, turning the state of an object into just another tag. Again, this would be useful mostly for reports, but it could certainly help quickly find all nodes in an error state, along with the objects associated with those nodes, and this would be especially useful if it were updated live. It might make sense to just support this as a live query against the system -- it should be pretty light-weight, at least compared to, say, querying the actual configuration.

This most likely makes sense to use as either a last resort or as a verification system. If you've got a ticketing system for reporting errors, Puppet could automatically generate tickets when an object fails, and then the ticket system could automatically verify that the object is no longer in an error state before it allows the user to close the ticket.

Collaborative tagging

One of my primary design goals in Puppet is to encourage and support configuration sharing. Configuration management will never advance as a field when every organization has to create its own definition for every server class, because those definitions, like servers managed without automation, present unnecessary and often arbitrary variation. So, I know that collaboration will already be critical to the future of Puppet.

How can tagging affect that collaboration? Would it be beneficial if people published the tag lists associated with each of the objects they manage? Could seeing someone else's tags on an object provide new ideas for organization or configuration? Would people who are unwilling to share their whole configuration be willing to just share their tags?

Configuration discovery

Could tags be used to discover a node's existing configuration? Maybe provide a few important tags through autodiscovery, like operating system, host name, and network, and then use that to start collecting a larger configuration over time.

This seems pretty damn unlikely.

Tags or classes?

Bjork or Goldthwait? Sorry, couldn't resist. Just like in cfengine (although I definitely took the long way around), classes in Puppet can generally be considered as booleans, i.e., tags. There's a heckuvalot of additional semantics associated with these tags compared to, say, a Flickr tag, since adding a tag causes work to happen on the system in question, but it can still be considered just a tag.

As the configurations get both more complex and more dynamic (changing often either because it is designated to do so by users, or because it is automatically reacting to network state, time, etc.), it might make the most sense to have a static configuration in one place that does not mention nodes at all, and then almost a scratch space for the nodes, where you can dynamically pin tags on a host and watch the chaos ensue.

At the very least, if you had a configuration that was changing constantly, you would almost definitely benefit from a map that showed the tags on a given host.

Typed tags

Is it worth deemphasizing the boolean aspect of tags, and add typing instead? For instance, is it worth differentiating server class tags from component tags? Or, like so much else, does it just make sense to intelligently pick class and component names, so that it's relatively apparent?

Obviously the easy solution is to start without typed tags, but it's something to keep in mind. I intuitively think that a lot of the power of tags comes specifically from their simplicity, and adding complexity would probably tend to decrease the flexibility.

Mon, 17 Oct 2005 | Tags: puppet, tagging