In my environment, we rely heavily upon Puppet to do large-scale automation and updating of our various systems. One of the tasks that we do on an infrequent basis is to modify grub.conf across many (or even all) systems to apply the same types of changes.
In Puppet, there are several ways you can do this.
- Create a template that gets pushed to all systems. The downside to this is that your systems may have differing grub.conf configurations due to things like differing kernel versions, differing hardware requirements that need to be noted on the kernel command line, or even things as simple as differing console settings.
- Create an exec that invokes a tool to edit the grub.conf. The downside to this is that you then have to build in a lot of support to do things like verify that your changes are syntactically correct, making sure you’re not over (or under) editing the file, and so on. An example of this would be execing a sed script.
- Use augeas with Puppet. Augeas is a tool for editing files that can be defined as a tree of data. Think of it as XPath for non-xml files.
Now, augeas isn’t necessarily easy to work with at first. There are a several tutorials on the net that get into the nitty-gritty. For the purposes of this discussion, I’ll walk you through a few real world examples of what I was doing with augeas and why these things were important to me.
Making CentOS6/RHEL6 boot more verbose
By default, CentOS6/RHEL6 systems enable two options on the kernel command line to make the boot very pretty and hide all the useful information that occurs during boot time. These options are rhgb and quiet. We want these removed from our kernel command lines on all systems. When you enable these two options, you will see at boot time is a blue screen with a moving status bar at the bottom that is supposed to convey progress information.
The puppet code:
class grub::no_rhgb {
# disable rhgb and quiet, so we get verbose booting of the kernel.
augeas { 'grub.conf/no_rhgb':
incl => '/boot/grub/grub.conf',
lens => 'grub.lns',
changes => [
'rm title[*]/kernel/rhgb',
'rm title[*]/kernel/quiet',
],
}
}
This defines a class creates an augeas resource named grub.conf/no_rhgb that edits the grub.conf using the right augeas lens.
The specific augeas commands are
rm title[*]/kernel/rhgb rm title[*]/kernel/quiet
This tells augeas that we want to remove each value multiple times, for each node in the /boot/grub/grub.conf/title/kernel tree. title[*] tells augeas to match every entry that has a valid title when searching for rhgb or quiet.
When this runs, you should end up with a grub.conf that has all references to these two options removed.
Overriding the Intel Idle driver settings
We run compute clusters that need a very specific range of performance. Ordinarily, you set this in power settings in the BIOS. It will be set to something like Max Power or Maximum Performance. The exact name is vendor specific. Under the hood, what this is doing is to limit the range of c-states that the processor can power down to. As you increase the c-state, the processor will do things like go into an idle mode, reduce CPU frequency, and will scale it up and down depending on the work load. In a compute cluster, this can have a measurable impact on overall performance of the cluster because this scaling up and down takes time. Microseconds matter.
When we put the BIOS into this Maximum Performance mode, we’re running the processor in c-state zero (0).
One of the things we discovered when moving to CentOS6/RHEL6 is that the Intel Idle driver in the kernel will automatically override BIOS setting and default you to c-state one (1). That means there’s a little power savings, but still enough to throw performance off a few percentage points. We want this stuff completely disabled and understand that the trade-off is we pay for the extra power used.
So, to disable this, we need to set a kernel command line option, intel_idle.max_cstate.
The puppet code:
class grub::no_intel_idle {
# intel_idle only exists in Centos 6 and higher.
if $operatingsystemrelease > 5 {
# this disables the intel_idle driver on the linux kernel. It
# sets this property across all instances of the kernel line in
# grub.conf.
augeas { 'grub.conf/no_intel_idle':
incl => '/boot/grub/grub.conf',
lens => 'grub.lns',
changes => [
'setm title[*]/kernel intel_idle.max_cstate 0',
],
}
} else {
fail("Attempting to disable intel_idle driver on $::osfamily or $::operatingsystemrelease which isn't supported.")
}
}
This defines a class creates an augeas resource named grub.conf/no_intel_idle that edits the grub.conf using the right augeas lens. Additionally, we want to make sure this only occurs on versions of CentOS/RHEL that are after 5 because the Intel Idle driver only exists in newer kernels.
The specific augeas command is this:
setm title[*]/kernel intel_idle.max_cstate 0
This tells augeas that we want to set this value multiple times, for each node in the /boot/grub/grub.conf/title/kernel tree. title[*] tells augeas to match every entry that has a valid title.
When this runs, it ends up looking like this:
title CentOS (2.6.32-358.11.1.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-358.11.1.el6.x86_64 ro root=/dev/mapper/rootvg-root_lv rd_NO_LUKS rd_LVM_LV=rootvg/swap_lv LANG=en_US.UTF-8 rd_LVM_LV=rootvg/root_lv rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM printk.time=1 intel_idle.max_cstate=0
initrd /initramfs-2.6.32-358.11.1.el6.x86_64.img
or, more specifically, it adds intel_idle.max_cstate=0 to the kernel line. If there were multiple title entries, each one would have this added.
Don’t forget to reboot to put this into effect. It doesn’t seem like you can do this at run time.
Make the kernel print timestamps for dmesg
One of the frustrating things about the kernel is that it will occasionally drop messages that are viewable by dmesg. Unfortunately, by default, there is no sense of time within the buffer that dmesg looks at. That means there are no timestamps associated with the output. In order to do this, you need to set a particular kernel option called printk.time on the kernel command line.
The puppet code:
class grub::print_kernel_timestamp {
# make kernel messages sent to the dmesg ring buffer have a timestamp
# so it's easier to tell when a message is old.
augeas { 'grub.conf/printktime':
incl => '/boot/grub/grub.conf',
lens => 'grub.lns',
changes => [
'setm title[*]/kernel printk.time 1'
],
}
}
This defines a class creates an augeas resource named grub.conf/printktime that edits the grub.conf using the right augeas lens.
The specific augeas command is this:
setm title[*]/kernel printk.time 1
This tells augeas that we want to set this value multiple times, for each node in the /boot/grub/grub.conf/title/kernel tree. title[*] tells augeas to match every entry that has a valid title.
When this runs, it ends up looking like this:
title CentOS (2.6.32-358.11.1.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-358.11.1.el6.x86_64 ro root=/dev/mapper/rootvg-root_lv rd_NO_LUKS rd_LVM_LV=rootvg/swap_lv LANG=en_US.UTF-8 rd_LVM_LV=rootvg/root_lv rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM printk.time=1
initrd /initramfs-2.6.32-358.11.1.el6.x86_64.img
or, more specifically, it adds printk.time=1 to the kernel line. If there were multiple title entries, each one would have this added.
One reboot later, and you end up with dmesg output that has a timestamp showing the number of seconds since system boot. You can translate accordingly.
[4326926.463088] nf_conntrack: table full, dropping packet. [6040776.862813] NOHZ: local_softirq_pending 100 [6068508.975044] netlink: 12 bytes leftover after parsing attributes. [30559653.578533] NOHZ: local_softirq_pending 100 [31231171.579993] NOHZ: local_softirq_pending 100 [32078213.799715] NOHZ: local_softirq_pending 100
Disabling swap retroactively
Every system we have has been set up with swap at some point in the past. This is always done during our kickstart procedure. Over time, we’ve come to find that there is a small subclass of systems where we absolutely don’t want it running. We’ve already disabled it via swapoff -a and modified /etc/fstab, but as removed the logical volume holding swap, we discovered that the systems would fail to boot.
We determined that the problem is occurring because dracut looks for certain keywords on the kernel command line to pre-activate logical volumes. For these particular systems, we still had rd_LVM_LV=rootvg/swap_lv listed, so the boot would fail because dracut couldn’t find the logical volume. Pretty obvious because we deleted it.
For these systems, we want to retroactively remove swap completely, so we need to make sure that it’s removed from the grub.conf configuration.
The puppet code:
class grub::no_swap {
augeas { 'grub.conf/no_swap':
incl => '/boot/grub/grub.conf',
lens => 'grub.lns',
changes => [
'rm title[*]/kernel/rd_LVM_LV[.=~regexp('rootvg/swap_lv')]'
],
}
}
This defines a class creates an augeas resource named grub.conf/no_swap that edits the grub.conf using the right augeas lens.
The specific augeas command is this:
rm title[*]/kernel/rd_LVM_LV[.=~regexp('rootvg/swap_lv')]
This tells augeas that we want to remove this value for each node in the /boot/grub/grub.conf/title/kernel tree that matches the key rd_LVM_LV. title[*] tells augeas to match every entry that has a valid title. The sticking part is that there can be multiple instances of rd_LVM_LV, so we need to give some extra hints on which specific one we want, so we use a regular expression to match the value of the key.
When this runs, you should end up with a grub.conf that has all references to this option removed.
In summary
Augeas is a pretty powerful tool. We’ve figured out ways to use it in our environment to make small, but important and targeted changes to fix issues that we’ve come across without having to resort to bad exec hacks (and thus, maintaining bad scripts) or to templates that might not be customizable enough without having a lot of puppet-specific code in place to deal with it.
