Bode's Blog

There are many best practices and standards for module development that need to be identified and agreed upon before modules can be truly reusable.

One of the most urgent needs is a well defined process for configuration data organization and overrides.

Fortunately, progress has already been made towards this goal which I can build upon…

My experiment expands on the work by R.I. Pienaar on extlookup.

Tons of ideas borrowed from conversations I’ve had with Alessandro Franceschi which are outlined in his presentation for Puppetcamp

Goal:

separation of configurable data from puppet manifests:

• puppet manifests – Puppet resources, dependencies, and language elements.
• configurable data – data that could differ between runs of the same manifests.

1. facilitates well defined data override structure
   • allows users to interact with modules in a well-defined way.

   • allows modules to be used without reading/modifying the source code (encapsulation of logic through a well defined interface).
2. easier to manage and override platforms where differences can be described as data.
   • ie: debian-ish v redhat-ish
3. allows finer grain access control.
   • ie: some users (devops) have permission to change manifests, while other users (operators) have limited permissions to change configuration data related to a specific environment (ie: production/datacenter1).

Proposed solution:

well defined override precedence:

1. modules should be packaged with a data store.
   • introduce new module directory <modulename/data/>
   • default data can be shipped with modules and stored separately from configuration logic.
   • module data store should allow fact/variable based overrides (ie: %{operatingsystem} overrides default, like extlookup}) .
2. allow per site data stores that override a module’s data store.
   • introduce a single directory that can be used to override configuration.
     • </etc/puppet/site-overrides/modulename/data> – overrides should mimic module structure.

     • overrides path should be similar to module datastore
3. all data is managed as attributes for parameterized classes
   • allows all data to be overridden during class declaration.

version 2.6.x of Puppet actually brings us most of the way towards this solution with the following new features:

• extlookup

• parameterized classes

Example:

I have chosen to implement a simple motd module to demonstrate the proposed data management solution:

Components:

1. module layout:

   |-- data
   |   |-- CentOS.csv
   |   `-- default.csv
   |-- manifests
   |   |-- init.pp
   |-- templates
   |   `-- motd.erb
   `-- tests
       |-- init.pp
   

2. data override layout

   site-overrides/
   `-- motd
       `-- data
           `-- CentOS.csv
   

3. templates/motd.erb
   Templates allow an easy way to see the resulting values of variables.

   DefaultData <%= default_data %>
   MoreData <%= more_data %>
   LocalOveride <%= local_override %>
   OverrideData <%= override_data %>
   

4. manifests/init.pp
   • simple class that specifies all data as class attributes:

   • all data defaults to the value derived from extlookup
   • all data can be overriden when class is declared.

   class motd (
       $default_data = extlookup('default_data'),
       $more_data = extlookup('more_data'),
       $override_data = extlookup('override_data'),
       $local_override = extlookup('local_override')
    ){
     file { '/etc/motd':
       content => template('motd/motd.erb'),
     }
   }
   

5. tests/init.pp
   • specify lookup precedence for extlookup

   • declare class and specify highest precedence overrides.

   $extlookup_datadir = '/'
   $local_overrides = '/etc/puppet/site'
   $extlookup_precedence = [
     "${local_overrides}/%{module_name}/data/%{operatingsystem}",
     "${local_overrides}/%{module_name}/data/default",
     "%{module_data_dir}/%{operatingsystem}",
     "%{module_data_dir}/default"
     ]
   class { 'motd':
     override_data => 'override'
   }
   

precedence of data lookup

1. default to extlookup value: use specified precendence
   1. /etc/puppet/modules/motd/data/default

      default_data, default
      local_override, default
      

   2. /etc/puppet/modules/motd/data/CentOS

      more_data, centos
      override_data, centos
      

   3. /etc/puppet/site-overrides/motd/data/default
   4. /etc/puppet/site-overrides/motd/data/CentOS

      local_override, local_centos
      

2. class parameters override everything

   class { 'motd':
     override_data => 'override'
   }
   

Results

:

#>puppet apply tests/init.pp
#>cat /etc/motd
DefaultData  default
MoreData  centos
LocalOveride  local_centos
OverrideData override

Changes to Puppet:

This example required the following changes to Puppet core:

1. patch to extlookup
   • allow multiple puppet variables to be interpolated.
2. patch to puppet (Puppet::Resource::Type)
   • create magic variable in Puppet::Resource::Type that can find the current manifest’s module’s data dir – module_data_dir
   • move the code where magic variables(module_name, module_data_dir) are added to a classes scope so they will be available before parameter defaults are evaluated.

branch of code from 2.6.3rc1 at:

http://github.com/bodepd/puppet/tree/data-hack

This code requires more significant changes to puppet:

• external node classifier interface needs to support parameterized classes.

Improvements:

• puppet.conf (and environments) should support an option for the following:
  • site_override_dir – directory where local override modules are found.
    • default – /etc/puppet/site-overrides/
  • lookup_precendence – order expressing how files are looked up
    • default-%{operatingsystem}:default

Additional requirements:

Ability to resolve data from a class.

• A modules configuration needs to be readable by other modules.

When I am writing code that works together with puppet, I often want to just use Puppet’s logging facility.

I can use puppet’s logging to log to syslog with the following code.

require puppet
Puppet.parse_config
Puppet::Util::Log.level = :info
Puppet::Util::Log.newdestination(:syslog)
Puppet.warning('dude!!!')
Puppet.warning('sweet!!)'

I recently spent a little time using puppet to model EC2 deployment.

The goal is to use Puppet’s concise language and its capability as a dependency manager to model the order in which I need to install nodes into Amazon’s Elastic Compute Cloud.

the code can be found in its entirety at

http://github.com/bodepd/puppet-ec2 .

Noder Type:

The first step is to create a type that will define the interface for deployment.

The type that I have created is called “noder” since ‘node’ is already a reserved word in Puppet.

Puppet::Type.newtype(:noder) do ...

The following parameters are added to the type using the method: newparam()

• name – unique name that will represent this EC2 instance.
• user – Amazon Web services user account.
• password – password associated with the user account.
• image – is the name of the instance’s base image(AMI).
• description – description associated with this ec2 instance.
• type – Amazon term that determines the specifications for the instance.

Modeling the provider

EC2 Namevar:

There is a significant inconsistency between how puppet models resources and how EC2 works.

Puppet needs to know the namevar for a resource in order to manage it. This means that puppet needs to be able to specify an ID for an EC2 instance before that instance exists, and then be able to query the instance based on this ID.

In EC2, the unique ID for an instance can only be determined after the instance is actually created.

There is at least one way to get around this limitation of EC2 (even though its a little hackish, suggestions are welcome ) Thanks to Jesse for helping me figure this out.

I will ensure that there is a 1-1 relationship between all managed EC2 instances and a security group whose name is prefixed with PUPPET_. This allows me to use the name of this security group as my namevar.

self.prefetch

Any logic that needs to be performed once per resource provider and before the ensurable methods are called should be performed in self.prefetch.

I will use self.prefetch to pre-generate both EC2 connections as well as the state of all EC2 instances once per provider (as opposed to once per resource).

I will store this information in the following class variables:

• @ec2 – hash of EC2 user account => EC2 connection.
• @instances – hashes each instance not in the ‘terminated’ or ’shutting-down’ state as follows:
• key: name of unique security group used as the instances namevar.
• value: hash of the following:
• key: :instance_id = real ec2 instance id
• value: :state = current state of instance when self.prefetch is called.

Ensurable:

Since the noder type was specified as ensurable, I need to define the methods: ‘exists?’, ‘create’ and ‘destroy’.

Exists?:

Since I have already stored the current state of our instances in @instances, I can just query the hash for a given instances state as follows:

def exists?
  state = self.class.instance_state(@resource.value(:name))
  state == 'running' || state == 'pending'
end

Create:

if ensure =>present and exists? returns false, ie: the instance either does not exist, or is not in the running or pending state, then I will run the create method.

In the create method, I can just retrieve the ec2 connection that I need by calling

ec2 =  self.class.ec2(@resource.value(:user))

this returns the ec2 connection associated with my user parameter.

There are two things that I need to do on creation.

1. If a security group with my namvaer does not already exist, I need to create it.

ec2 = self.class.ec2(@resource.value(:user))
group = @resource.value(:name)
begin
  ec2.describe_security_groups({:group_name => group})
  rescue Exception => e
    unless self.class.instance_state(group)
      ec2.create_security_group(
      {
        :group_name => group,
        :group_description => @resource.value(:desc)
      }
  end
end

2. Once I have ensured that the group exists, I call ec2.run_instances, specifying the AMI image, security group (namevar), and type.

ec2.run_instances(
  {
    :image_id => @resource.value(:image),
    :security_group => group,
    :instance_type => @resource.value(:type),
  }
)

Destroy:

if ensure =>absent and exists? returns true, then I need to run the ‘destroy’ method.

I can use the class method instance_id to retrieve the actual EC2 instance id from existing instances.

Then I will delete both the instance as well as the group that I am using to determine its namer.

def destroy
  group = @resource.value(:name)
  ec2 =  self.class.ec2(@resource.value(:user))
  instance = self.class.instance_id(group)
  ec2.terminate_instances({:instance_id => instance})
  ec2.delete_security_group({:group_name => group})
end

Usage

some usage examples:

noder { 'blah2':
  ensure => present,
  user => 'Dannyboy',
  password => 'password',
  image => 'ami-84db39ed',
  desc => 'happy instance',
}
noder { 'blah3':
  ensure => present,
  user => 'Dannyboy',
  password => 'password',
  image => 'ami-84db39ed',
  desc => 'happy instance',
}
noder { 'default':
  ensure => present,
  user => 'Dannyboy',
  password => 'password',
  image => 'ami-84db39ed',
  desc => 'happy instance',
  require => Noder['blah3']
}
noder { 'blah':
  ensure => absent,
  user => 'Dannyboy',
  password => 'password',
  image => 'ami-84db39ed',
  desc => 'happy instance',
  require => Noder['default'],
}

Next Steps:

Now that I have the basics working, the next step is to add more features:

1. Specify image restart on refresh
2. Consider better ways to synchronize with EC2 state.
3. Add suspend, stop ensure states.
4. Switch params to properties once its supported by the ec2 ruby bindings.

Motivation:

It’s common to deploy multiple puppetmasters behind a load-balancer.  In this setup, it is necessary to specify a single trusted certificate authority so that all clients can seamlessly connect to any of the clustered puppetmasters.

Example:

This example will use two hosts:

1. puppet1 -  will serve as the CA, puppetmaster1, and puppetclient1 (alias: puppetmaster3)
2. puppet2 – will not serve as the CA, puppetmaster2, puppetclient2 (alias: puppetmaster4)

CA

The command, puppetca, run with no arguments will create a CA.

puppet1>puppetca
puppet1 ssl]# find -type f
ssl/ca/serial
ssl/ca/ca_key.pem
ssl/ca/private/ca.pass
ssl/ca/ca_crt.pem
ssl/ca/ca_crl.pem
ssl/ca/inventory.txt

Generating ssl certs

Now, use puppetca –generate to create the ssl certificates for all puppetmasters. Additional aliases can also be specified as an option.

puppet1>puppetca --generate --certdnsnames puppetmaster3:puppet puppetmaster1
puppet1>puppetca --generate --certdnsnames puppetmaster4:puppet puppetmaster2
puppet1>find ssl | grep master
ssl/private_keys/puppetmaster1.pem
ssl/private_keys/puppetmaster2.pem
ssl/ca/signed/puppetmaster1.pem
ssl/ca/signed/puppetmaster2.pem

Copy the signed certificate, private key, and certificate for the trusted CA to puppet2.

First create the necessary directories on puppet2:

puppet2>cd /var/lib/puppet/
puppet2>mkdir -p ssl/private_keys ssl/certs

Now copy the files.

puppet1>cd /var/lib/puppet/ssl
puppet1>scp private_keys/puppetmaster2.pem root@puppet2:/var/lib/puppet/ssl/private_keys/puppetmaster2.pem
puppet1>scp ca/signed/puppetmaster2.pem root@puppet2:/var/lib/puppet/ssl/certs/puppetmaster2.pem
puppet1>scp ca/ca_crt.pem root@puppet2:/var/lib/puppet/ssl/certs/ca.pem

Start puppetmasters:

Start both puppetmasters in the foreground

on puppet1

#puppet.conf
[puppetmasterd]
  certname=puppetmaster1
  ca=true

puppet1>puppetmasterd --no-daemonize --verbose --certname puppetmaster1
info: Caching certificate for ca
info: Caching certificate for puppetmaster1

now on puppet2:

#puppet.conf
[puppetmasterd]
  certname=puppetmaster2
  ca=false

puppet2>puppetmasterd --no-daemonize --verbose --certname puppetmaster2

Clients:

on both clients, configure the following:

#puppet1 puppet.conf
[puppetd]
  certname=puppetclient1
  ca_server=puppet1

#puppet2 puppet.conf
[puppetd]
  certname=puppetclient2
  ca_server=puppet1

Now, create the file /etc/puppet/manifests/site.pp.

# site.pp
notify{$servername:}

before testing, I will set up dns in /etc/hosts file.

# /etc/hosts
192.168.0.2 puppet puppetmaster2 puppetmaster4 puppetclient2
192.168.0.3 puppetmaster1 puppetmaster3 puppetclient1

now, run the following:

puppet1>puppetd --test --server puppetmaster1
puppet1>puppetd --test --server puppetmaster2
puppet1>puppetd --test --server puppetmaster3
puppet1>puppetd --test --server puppetmaster4

puppet2>puppetd --test --server puppetmaster1
puppet2>puppetd --test --server puppetmaster2
puppet2>puppetd --test --server puppetmaster3
puppet2>puppetd --test --server puppetmaster4

Thus illustrating that the clients can connect to either puppetmaster by any of their allowed dns names.

Well, after years and years of putting it off, I finally started a blog. Partially in response to the fact that I actually have stuff to talk about that people want to learn