[Mageia-sysadm] [241] Add a module to construct files from fragments.

root at mageia.org root at mageia.org
Fri Nov 12 18:42:23 CET 2010


Revision: 241
Author:   boklm
Date:     2010-11-12 18:42:22 +0100 (Fri, 12 Nov 2010)
Log Message:
-----------
Add a module to construct files from fragments.

Downloaded from https://github.com/ripienaar/puppet-concat/

Added Paths:
-----------
    puppet/modules/concat/
    puppet/modules/concat/CHANGELOG
    puppet/modules/concat/README.markdown
    puppet/modules/concat/files/
    puppet/modules/concat/files/concatfragments.sh
    puppet/modules/concat/files/null/
    puppet/modules/concat/files/null/.gitignore
    puppet/modules/concat/manifests/
    puppet/modules/concat/manifests/fragment.pp
    puppet/modules/concat/manifests/init.pp
    puppet/modules/concat/manifests/setup.pp

Added: puppet/modules/concat/CHANGELOG
===================================================================
--- puppet/modules/concat/CHANGELOG	                        (rev 0)
+++ puppet/modules/concat/CHANGELOG	2010-11-12 17:42:22 UTC (rev 241)
@@ -0,0 +1,27 @@
+KNOWN ISSUES:
+- In 0.24.8 you will see inintended notifies, if you build a file
+  in a run, the next run will also see it as changed.  This is due
+  to how 0.24.8 does the purging of unhandled files, this is improved
+  in 0.25.x and we cannot work around it in our code.
+
+CHANGELOG:
+- 2010/02/19 - initial release
+- 2010/03/12 - add support for 0.24.8 and newer
+             - make the location of sort configurable
+             - add the ability to add shell comment based warnings to
+               top of files
+             - add the ablity to create empty files
+- 2010/04/05 - fix parsing of WARN and change code style to match rest
+               of the code
+             - Better and safer boolean handling for warn and force
+             - Don't use hard coded paths in the shell script, set PATH
+               top of the script
+             - Use file{} to copy the result and make all fragments owned
+               by root.  This means we can chnage the ownership/group of the
+               resulting file at any time.
+             - You can specify ensure => "/some/other/file" in concat::fragment
+               to include the contents of a symlink into the final file.
+- 2010/04/16 - Add more cleaning of the fragment name - removing / from the $name
+- 2010/05/22 - Improve documentation and show the use of ensure =>
+- 2010/07/14 - Add support for setting the filebucket behavior of files
+- 2010/10/04 - Make the warning message configurable

Added: puppet/modules/concat/README.markdown
===================================================================
--- puppet/modules/concat/README.markdown	                        (rev 0)
+++ puppet/modules/concat/README.markdown	2010-11-12 17:42:22 UTC (rev 241)
@@ -0,0 +1,103 @@
+What is it?
+===========
+
+A Puppet module that can construct files from fragments.
+
+Please see the comments in the various .pp files for details
+as well as posts on my blog at http://www.devco.net/
+
+Released under the Apache 2.0 licence
+
+Usage:
+------
+
+If you wanted a /etc/motd file that listed all the major modules
+on the machine.  And that would be maintained automatically even
+if you just remove the include lines for other modules you could
+use code like below, a sample /etc/motd would be:
+
+<pre>
+Puppet modules on this server:
+
+    -- Apache
+    -- MySQL
+</pre>
+
+Local sysadmins can also append to the file by just editing /etc/motd.local
+their changes will be incorporated into the puppet managed motd.
+
+<pre>
+# class to setup basic motd, include on all nodes
+class motd {
+   include concat::setup
+   $motd = "/etc/motd"
+
+   concat{$motd,
+      owner => root,
+      group => root,
+      mode  => 644
+   }
+
+   concat::fragment{"motd_header":
+      target => $motd,
+      content => "\nPuppet modules on this server:\n\n",
+      order   => 01,
+   }
+
+   # local users on the machine can append to motd by just creating
+   # /etc/motd.local
+   concat::fragment{"motd_local":
+      target => $motd,
+      ensure  => "/etc/motd.local",
+      order   => 15
+   }
+}
+
+# used by other modules to register themselves in the motd
+define motd::register($content="", $order=10) {
+   if $content == "" {
+      $body = $name
+   } else {
+      $body = $content
+   }
+
+   concat::fragment{"motd_fragment_$name":
+      target  => "/etc/motd",
+      content => "    -- $body\n"
+   }
+}
+
+# a sample apache module
+class apache {
+   include apache::install, apache::config, apache::service
+
+   motd::register{"Apache": }
+}
+</pre>
+
+Known Issues:
+-------------
+* In 0.24.8 you will see inintended notifies, if you build a file
+  in a run, the next run will also see it as changed.  This is due
+  to how 0.24.8 does the purging of unhandled files, this is improved
+  in 0.25.x and we cannot work around it in our code.
+
+Contributors:
+-------------
+**Paul Elliot**   
+
+ * Provided 0.24.8 support, shell warnings and empty file creation support.
+
+**Chad Netzer**
+
+ * Various patches to improve safety of file operations
+ * Symlink support
+
+**David Schmitt**
+
+ * Patch to remove hard coded paths relying on OS path
+ * Patch to use file{} to copy the resulting file to the final destination.  This means Puppet client will show diffs and that hopefully we can change file ownerships now
+
+Contact:
+--------
+You can contact me on rip at devco.net or follow my blog at http://www.devco.net I am also on twitter as ripienaar

Added: puppet/modules/concat/files/concatfragments.sh
===================================================================
--- puppet/modules/concat/files/concatfragments.sh	                        (rev 0)
+++ puppet/modules/concat/files/concatfragments.sh	2010-11-12 17:42:22 UTC (rev 241)
@@ -0,0 +1,123 @@
+#!/bin/bash
+
+# Script to concat files to a config file.
+#
+# Given a directory like this:
+# /path/to/conf.d
+# |-- fragments
+# |   |-- 00_named.conf
+# |   |-- 10_domain.net
+# |   `-- zz_footer
+#
+# The script supports a test option that will build the concat file to a temp location and 
+# use /usr/bin/cmp to verify if it should be run or not.  This would result in the concat happening
+# twice on each run but gives you the option to have an unless option in your execs to inhibit rebuilds.
+# 
+# Without the test option and the unless combo your services that depend on the final file would end up 
+# restarting on each run, or in other manifest models some changes might get missed.
+#
+# OPTIONS:
+#  -o	The file to create from the sources
+#  -d	The directory where the fragments are kept
+#  -t	Test to find out if a build is needed, basically concats the files to a temp
+#       location and compare with what's in the final location, return codes are designed
+#       for use with unless on an exec resource
+#  -w   Add a shell style comment at the top of the created file to warn users that it 
+#       is generated by puppet
+#  -f   Enables the creation of empty output files when no fragments are found
+#  -n	Sort the output numerically rather than the default alpha sort
+#
+# the command: 
+#
+#   concatfragments.sh -o /path/to/conffile.cfg -d /path/to/conf.d
+#
+# creates /path/to/conf.d/fragments.concat and copies the resulting 
+# file to /path/to/conffile.cfg.  The files will be sorted alphabetically
+# pass the -n switch to sort numerically.
+# 
+# The script does error checking on the various dirs and files to make
+# sure things don't fail.
+
+OUTFILE=""
+WORKDIR=""
+TEST=""
+FORCE=""
+WARN=""
+SORTARG="-z"
+
+PATH=/sbin:/usr/sbin:/bin:/usr/bin
+
+while getopts "o:s:d:tnw:f" options; do
+	case $options in
+		o ) OUTFILE=$OPTARG;;
+		d ) WORKDIR=$OPTARG;;
+		n ) SORTARG="-zn";;
+		w ) WARNMSG="$OPTARG";;
+		f ) FORCE="true";;
+		t ) TEST="true";;
+		* ) echo "Specify output file with -o and fragments directory with -d"
+		    exit 1;;
+	esac
+done
+
+# do we have -o?
+if [ x${OUTFILE} = "x" ]; then
+	echo "Please specify an output file with -o"
+	exit 1
+fi
+
+# do we have -d?
+if [ x${WORKDIR} = "x" ]; then
+	echo "Please fragments directory with -d"
+	exit 1
+fi
+
+# can we write to -o?
+if [ -a ${OUTFILE} ]; then
+	if [ ! -w ${OUTFILE} ]; then
+		echo "Cannot write to ${OUTFILE}"
+		exit 1
+	fi
+else
+	if [ ! -w `dirname ${OUTFILE}` ]; then
+		echo "Cannot write to `dirname ${OUTFILE}` to create ${OUTFILE}"
+		exit 1
+	fi
+fi
+
+# do we have a fragments subdir inside the work dir?
+if [ ! -d "${WORKDIR}/fragments" ]  && [ ! -x "${WORKDIR}/fragments" ]; then
+	echo "Cannot access the fragments directory"
+	exit 1
+fi
+
+# are there actually any fragments?
+if [ ! "$(ls -A ${WORKDIR}/fragments)" ]; then
+	if [ x${FORCE} = "x" ]; then
+		echo "The fragments directory is empty, cowardly refusing to make empty config files"
+		exit 1
+	fi
+fi
+
+cd ${WORKDIR}
+
+if [ x${WARNMSG} = "x" ]; then
+	: > "fragments.concat"
+else
+	echo -e "$WARNMSG" > "fragments.concat"
+fi
+
+# find all the files in the fragments directory, sort them numerically and concat to fragments.concat in the working dir
+find fragments/ -type f -follow -print0 |sort ${SORTARG}|xargs -0 cat >>"fragments.concat"
+
+if [ x${TEST} = "x" ]; then
+	# This is a real run, copy the file to outfile
+	cp fragments.concat ${OUTFILE}
+	RETVAL=$?
+else
+	# Just compare the result to outfile to help the exec decide
+	cmp ${OUTFILE} fragments.concat
+	RETVAL=$?
+fi
+
+exit $RETVAL


Property changes on: puppet/modules/concat/files/concatfragments.sh
___________________________________________________________________
Added: svn:executable
   + *
Added: svn:eol-style
   + native

Added: puppet/modules/concat/files/null/.gitignore
===================================================================
Added: puppet/modules/concat/manifests/fragment.pp
===================================================================
--- puppet/modules/concat/manifests/fragment.pp	                        (rev 0)
+++ puppet/modules/concat/manifests/fragment.pp	2010-11-12 17:42:22 UTC (rev 241)
@@ -0,0 +1,51 @@
+# Puts a file fragment into a directory previous setup using concat
+# 
+# OPTIONS:
+#   - target    The file that these fragments belong to
+#   - content   If present puts the content into the file
+#   - source    If content was not specified, use the source
+#   - order     By default all files gets a 10_ prefix in the directory
+#               you can set it to anything else using this to influence the
+#               order of the content in the file
+#   - ensure    Present/Absent or destination to a file to include another file
+#   - mode      Mode for the file
+#   - owner     Owner of the file
+#   - group     Owner of the file
+#   - backup    Controls the filebucketing behavior of the final file and
+#               see File type reference for its use.  Defaults to 'puppet'
+define concat::fragment($target, $content='', $source='', $order=10, $ensure = "present", $mode = 0644, $owner = root, $group = root, $backup = "puppet") {
+    $safe_name = regsubst($name, '/', '_', 'G')
+    $safe_target_name = regsubst($target, '/', '_', 'G')
+    $concatdir = $concat::setup::concatdir
+    $fragdir = "${concatdir}/${safe_target_name}"
+
+    # if content is passed, use that, else if source is passed use that
+    # if neither passed, but $ensure is in symlink form, make a symlink
+    case $content {
+        "": {
+                case $source {
+                        "": {
+                                case $ensure {
+                                    "", "absent", "present", "file", "directory": {
+                                        crit("No content, source or symlink specified")
+                                    }
+                                }
+                            }
+                   default: { File{ source => $source } }
+                }
+            }
+        default: { File{ content => $content } }
+    }
+
+    file{"${fragdir}/fragments/${order}_${safe_name}":
+        mode   => $mode,
+        owner  => $owner,
+        group  => $group,
+        ensure => $ensure,
+        backup => $backup,
+        alias  => "concat_fragment_${name}",
+        notify => Exec["concat_${target}"]
+    }
+}
+
+# vi:tabstop=4:expandtab:ai

Added: puppet/modules/concat/manifests/init.pp
===================================================================
--- puppet/modules/concat/manifests/init.pp	                        (rev 0)
+++ puppet/modules/concat/manifests/init.pp	2010-11-12 17:42:22 UTC (rev 241)
@@ -0,0 +1,164 @@
+# A system to construct files using fragments from other files or templates.
+#
+# This requires at least puppet 0.25 to work correctly as we use some 
+# enhancements in recursive directory management and regular expressions
+# to do the work here.
+#
+# USAGE:
+# The basic use case is as below:
+#
+# concat{"/etc/named.conf": 
+#    notify => Service["named"]
+# }
+#
+# concat::fragment{"foo.com_config":
+#    target  => "/etc/named.conf",
+#    order   => 10,
+#    content => template("named_conf_zone.erb")
+# }
+#
+# # add a fragment not managed by puppet so local users 
+# # can add content to managed file
+# concat::fragment{"foo.com_user_config":
+#    target  => "/etc/named.conf",
+#    order   => 12,
+#    ensure  => "/etc/named.conf.local"
+# }
+#
+# This will use the template named_conf_zone.erb to build a single 
+# bit of config up and put it into the fragments dir.  The file
+# will have an number prefix of 10, you can use the order option
+# to control that and thus control the order the final file gets built in.
+#
+# SETUP:
+# The class concat::setup defines a variable $concatdir - you should set this
+# to a directory where you want all the temporary files and fragments to be
+# stored.  Avoid placing this somewhere like /tmp since you should never
+# delete files here, puppet will manage them.
+#
+# There's some regular expression magic to figure out the puppet version but
+# if you're on an older 0.24 version just set $puppetversion = 24
+#
+# Before you can use any of the concat features you should include the 
+# class concat::setup somewhere on your node first.
+#
+# DETAIL:
+# We use a helper shell script called concatfragments.sh that gets placed
+# in /usr/local/bin to do the concatenation.  While this might seem more 
+# complex than some of the one-liner alternatives you might find on the net
+# we do a lot of error checking and safety checks in the script to avoid 
+# problems that might be caused by complex escaping errors etc.
+# 
+# LICENSE:
+# Apache Version 2
+#
+# LATEST:
+# http://github.com/ripienaar/puppet-concat/
+#
+# CONTACT:
+# R.I.Pienaar <rip at devco.net> 
+# Volcane on freenode
+# @ripienaar on twitter
+# www.devco.net
+
+
+# Sets up so that you can use fragments to build a final config file, 
+#
+# OPTIONS:
+#  - mode       The mode of the final file
+#  - owner      Who will own the file
+#  - group      Who will own the file
+#  - force      Enables creating empty files if no fragments are present
+#  - warn       Adds a normal shell style comment top of the file indicating
+#               that it is built by puppet
+#  - backup     Controls the filebucketing behavior of the final file and
+#               see File type reference for its use.  Defaults to 'puppet'
+#
+# ACTIONS:
+#  - Creates fragment directories if it didn't exist already
+#  - Executes the concatfragments.sh script to build the final file, this script will create
+#    directory/fragments.concat.   Execution happens only when:
+#    * The directory changes 
+#    * fragments.concat != final destination, this means rebuilds will happen whenever 
+#      someone changes or deletes the final file.  Checking is done using /usr/bin/cmp.
+#    * The Exec gets notified by something else - like the concat::fragment define
+#  - Copies the file over to the final destination using a file resource
+#
+# ALIASES:
+#  - The exec can notified using Exec["concat_/path/to/file"] or Exec["concat_/path/to/directory"]
+#  - The final file can be referened as File["/path/to/file"] or File["concat_/path/to/file"]  
+define concat($mode = 0644, $owner = "root", $group = "root", $warn = "false", $force = "false", $backup = "puppet") {
+    $safe_name   = regsubst($name, '/', '_', 'G')
+    $concatdir   = $concat::setup::concatdir
+    $version     = $concat::setup::majorversion
+    $fragdir     = "${concatdir}/${safe_name}"
+    $concat_name = "fragments.concat.out"
+    $default_warn_message = '# This file is managed by Puppet. DO NOT EDIT.'
+
+    case $warn {
+        'true',true,yes,on:   { $warnmsg = "$default_warn_message" }
+        'false',false,no,off: { $warnmsg = "" }
+        default:              { $warnmsg = "$warn" }
+    }
+
+    $warnmsg_escaped = regsubst($warnmsg, "'", "'\\\\''", 'G')
+    $warnflag = $warnmsg_escaped ? {
+        ''      => '',
+        default => "-w '$warnmsg_escaped'"
+    }
+
+    case $force {
+        'true',true,yes,on: { $forceflag = "-f" }
+        'false',false,no,off: { $forceflag = "" }
+        default: { fail("Improper 'force' value given to concat: $force") }
+    }
+
+    File{
+        owner  => root,
+        group  => root,
+        mode   => $mode,
+        backup => $backup
+    }
+
+    file{$fragdir:
+            ensure   => directory;
+
+         "${fragdir}/fragments":
+            ensure   => directory,
+            recurse  => true,
+            purge    => true,
+            force    => true,
+            ignore   => [".svn", ".git", ".gitignore"],
+            source   => $version ? {
+                            24      => "puppet:///concat/null",
+                            default => undef,
+                        },
+            notify   => Exec["concat_${name}"];
+
+         "${fragdir}/fragments.concat":
+            ensure   => present;
+
+         "${fragdir}/${concat_name}":
+            ensure   => present;
+
+         $name:
+            source   => "${fragdir}/${concat_name}",
+            owner    => $owner,
+            group    => $group,
+            checksum => md5,
+            mode     => $mode,
+            ensure   => present,
+            alias    => "concat_${name}";
+    }
+
+    exec{"concat_${name}":
+        user      => root,
+        group     => root,
+        notify    => File[$name],
+        subscribe => File[$fragdir],
+        alias     => "concat_${fragdir}",
+        require   => [ File["/usr/local/bin/concatfragments.sh"], File[$fragdir], File["${fragdir}/fragments"], File["${fragdir}/fragments.concat"] ],
+        unless    => "/usr/local/bin/concatfragments.sh -o ${fragdir}/${concat_name} -d ${fragdir} -t ${warnflag} ${forceflag}",
+        command   => "/usr/local/bin/concatfragments.sh -o ${fragdir}/${concat_name} -d ${fragdir} ${warnflag} ${forceflag}",
+    }
+}

Added: puppet/modules/concat/manifests/setup.pp
===================================================================
--- puppet/modules/concat/manifests/setup.pp	                        (rev 0)
+++ puppet/modules/concat/manifests/setup.pp	2010-11-12 17:42:22 UTC (rev 241)
@@ -0,0 +1,36 @@
+# Sets up the concat system.
+#
+# $concatdir should point to a place where you wish the fragments to
+# live. This should not be somewhere like /tmp since ideally these files
+# should not be deleted ever, puppet should always manage them
+#
+# $puppetversion should be either 24 or 25 to enable a 24 compatible
+# mode, in 24 mode you might see phantom notifies this is a side effect
+# of the method we use to clear the fragments directory.
+# 
+# The regular expression below will try to figure out your puppet version
+# but this code will only work in 0.24.8 and newer.
+#
+# It also copies out the concatfragments.sh file to /usr/local/bin
+class concat::setup {
+    $concatdir = "/var/lib/puppet/concat"
+    $majorversion = regsubst($puppetversion, '^[0-9]+[.]([0-9]+)[.][0-9]+$', '\1')
+
+    file{"/usr/local/bin/concatfragments.sh": 
+            owner  => root,
+            group  => root,
+            mode   => 755,
+            source => $majorversion ? {
+                        24      => "puppet:///concat/concatfragments.sh",
+                        default => "puppet:///modules/concat/concatfragments.sh"
+                      };
+
+         $concatdir: 
+            ensure => directory,
+            owner  => root,
+            group  => root,
+            mode   => 755;
+    }
+}
+
+# vi:tabstop=4:expandtab:ai
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/mageia-sysadm/attachments/20101112/c7579815/attachment-0001.html>


More information about the Mageia-sysadm mailing list