Adventures in the Linux VM Subsystem - File Cache Demonstration #1

Linux VM Subsystem - File Cache Demonstration #1

I was at work today and came across a performance problem with an internal Java application. Long story short, someone suggested that because the filesystem cache was full we had run out of memory and should drop caches. I suggested that this was a bad idea and said that Linux handled this itself and we need not do that. 

But rather than just take my word for it, or the word of people like Mel Gorman, here is a demonstration that I wrote up. I had to spin up a VM in AWS and install perl and CPAN and Linux::MemInfo. After that I wrote up the code which is available at https://github.com/ndietsch/linux-system-programming. 

As you can see from the screencast, I created a perl program that creates a 10MB file of zeros and then reads this into a noop loop. With each iteration, you can see the free memory drop by 10MB and the active file cache increase by 10MB. Once the system hits about 9-10MB free, it starts moving the cache from active to inactive and then evicting as necessary. This proves that Linux can handle the cache ejections by itself. 

The next installments of this will show how anonymous memory increasing will also shrink the cache. But for now, here is the screencast ... enjoy.

I have included the terminal session below, but also a tar file of scriptreplay files if you want the interactive experience

[ec2-user@ip-172-31-37-189 linux-systems-programming]$ cat intro; sleep 30
* This screencast is to demonstrate the cache eviction feature of the Linux Virtual Memory Management subsystem. 
* In this example, our Perl code creates a 10MB file with DD and then reads it into a loop. 
* The loop outputs three figures each time
	- MemFree
	- Active (File) which makes up the active part of the file cache
	- Inactive (File) which makes up the inactive part of the file cache
* As you can see from the /proc/meminfo output, we start with a relatively high amount of free memory on our 1GB system and a low amount of filesystem cache
* With each loop, the free memory decreases by about 10MB and the Active file cache increases by about 10MB. 
* Once we hit around 9-11MB free on the system, the VM subsystem automatically moves the cache from active to inactive where it is then evicted using the various algorithms
* Please note that this is using Amazon Linux with Kernel 4.1.17-22.30.amzn1.x86_64 which will differ slightly from the internal Linux systems at ${work} but it demonstrates the point none the less.
* The really cool bit is at the end when we delete all the files

[ec2-user@ip-172-31-37-189 linux-systems-programming]$ cat /proc/meminfo 
MemTotal:        1019444 kB
MemFree:          934832 kB
MemAvailable:     969936 kB
Buffers:            1024 kB
Cached:            16828 kB
SwapCached:            0 kB
Active:            22168 kB
Inactive:           9376 kB
Active(anon):      13700 kB
Inactive(anon):       52 kB
Active(file):       8468 kB
Inactive(file):     9324 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         13696 kB
Mapped:             5484 kB
Shmem:                60 kB
Slab:              41288 kB
SReclaimable:      31880 kB
SUnreclaim:         9408 kB
KernelStack:        1408 kB
PageTables:         2356 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      509720 kB
Committed_AS:      62876 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        2488 kB
VmallocChunk:   34359731159 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       22528 kB
DirectMap2M:     1026048 kB
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ cat ActiveFileCache.pl 
use strict;
use Linux::MemInfo;
my %mem;
my $count;
# Starting values
%mem = get_mem_info;
printf("%-20s\t%s\n","Active(file)",$mem{"Active(file)"});
printf("%-20s\t%s\n","Active(anon)",$mem{"Active(anon)"});

while() {
	sleep 1;
	$count++;
	# Create a 10MB file full of zeros and then read it to fill the Active(file) cache from /proc/meminfo
	my $retval=system("/bin/dd if=/dev/zero of=/tmp/file.${count} count=20480 > /dev/null 2>&1");
	open(my $fh, "){}

	# Get memory statistics after creating and reading a 5MB file, watch Active(file) grow until it is 5MB
	%mem = get_mem_info;
	my $time = localtime;
	printf("%s\n",$time);
	printf("%-20s\t%s\n","Active(file)",$mem{"Active(file)"});
	printf("%-20s\t%s\n","Inactive(file)",$mem{"Inactive(file)"});
	printf("%-20s\t%s\n\n","MemFree",$mem{"MemFree"});
}
	

[ec2-user@ip-172-31-37-189 linux-systems-programming]$ perl ActiveFileCache.pl 
Active(file)        	8508
Active(anon)        	14332
Wed Mar  2 06:07:54 2016
Active(file)        	18780
Inactive(file)      	9284
MemFree             	923476

Wed Mar  2 06:07:55 2016
Active(file)        	29000
Inactive(file)      	9268
MemFree             	913068

Wed Mar  2 06:07:56 2016
Active(file)        	39276
Inactive(file)      	9272
MemFree             	902536

Wed Mar  2 06:07:57 2016
Active(file)        	49504
Inactive(file)      	9272
MemFree             	891980

Wed Mar  2 06:07:58 2016
Active(file)        	59720
Inactive(file)      	9272
MemFree             	881572

Wed Mar  2 06:07:59 2016
Active(file)        	69984
Inactive(file)      	9272
MemFree             	871016

Wed Mar  2 06:08:00 2016
Active(file)        	80244
Inactive(file)      	9272
MemFree             	860360

Wed Mar  2 06:08:01 2016
Active(file)        	90448
Inactive(file)      	9272
MemFree             	849952

Wed Mar  2 06:08:02 2016
Active(file)        	100712
Inactive(file)      	9272
MemFree             	839520

Wed Mar  2 06:08:03 2016
Active(file)        	110964
Inactive(file)      	9272
MemFree             	828988

Wed Mar  2 06:08:04 2016
Active(file)        	121208
Inactive(file)      	9272
MemFree             	818548

Wed Mar  2 06:08:05 2016
Active(file)        	131428
Inactive(file)      	9272
MemFree             	808012

Wed Mar  2 06:08:06 2016
Active(file)        	141692
Inactive(file)      	9272
MemFree             	797480

Wed Mar  2 06:08:07 2016
Active(file)        	151932
Inactive(file)      	9272
MemFree             	787020

Wed Mar  2 06:08:08 2016
Active(file)        	162160
Inactive(file)      	9272
MemFree             	776484

Wed Mar  2 06:08:09 2016
Active(file)        	172400
Inactive(file)      	9272
MemFree             	765924

Wed Mar  2 06:08:10 2016
Active(file)        	182624
Inactive(file)      	9272
MemFree             	755392

Wed Mar  2 06:08:12 2016
Active(file)        	192888
Inactive(file)      	9272
MemFree             	744960

Wed Mar  2 06:08:13 2016
Active(file)        	203128
Inactive(file)      	9272
MemFree             	734404

Wed Mar  2 06:08:14 2016
Active(file)        	213380
Inactive(file)      	9272
MemFree             	723996

Wed Mar  2 06:08:15 2016
Active(file)        	223620
Inactive(file)      	9272
MemFree             	713440

Wed Mar  2 06:08:16 2016
Active(file)        	233880
Inactive(file)      	9272
MemFree             	702908

Wed Mar  2 06:08:17 2016
Active(file)        	244108
Inactive(file)      	9272
MemFree             	692476

Wed Mar  2 06:08:18 2016
Active(file)        	254360
Inactive(file)      	9272
MemFree             	681940

Wed Mar  2 06:08:19 2016
Active(file)        	264576
Inactive(file)      	9272
MemFree             	671284

Wed Mar  2 06:08:20 2016
Active(file)        	274840
Inactive(file)      	9272
MemFree             	660884

Wed Mar  2 06:08:21 2016
Active(file)        	285052
Inactive(file)      	9272
MemFree             	650352

Wed Mar  2 06:08:22 2016
Active(file)        	295316
Inactive(file)      	9272
MemFree             	639792

Wed Mar  2 06:08:23 2016
Active(file)        	305532
Inactive(file)      	9272
MemFree             	629384

Wed Mar  2 06:08:24 2016
Active(file)        	318796
Inactive(file)      	7160
MemFree             	617968

Wed Mar  2 06:08:25 2016
Active(file)        	329052
Inactive(file)      	7152
MemFree             	607448

Wed Mar  2 06:08:26 2016
Active(file)        	339288
Inactive(file)      	7152
MemFree             	597016

Wed Mar  2 06:08:27 2016
Active(file)        	349528
Inactive(file)      	7152
MemFree             	586460

Wed Mar  2 06:08:28 2016
Active(file)        	359780
Inactive(file)      	7152
MemFree             	576036

Wed Mar  2 06:08:29 2016
Active(file)        	370020
Inactive(file)      	7152
MemFree             	565536

Wed Mar  2 06:08:30 2016
Active(file)        	380272
Inactive(file)      	7152
MemFree             	555004

Wed Mar  2 06:08:31 2016
Active(file)        	390508
Inactive(file)      	7152
MemFree             	544448

Wed Mar  2 06:08:32 2016
Active(file)        	400724
Inactive(file)      	7152
MemFree             	533916

Wed Mar  2 06:08:33 2016
Active(file)        	410988
Inactive(file)      	7152
MemFree             	523360

Wed Mar  2 06:08:34 2016
Active(file)        	421228
Inactive(file)      	7156
MemFree             	512928

Wed Mar  2 06:08:35 2016
Active(file)        	431480
Inactive(file)      	7156
MemFree             	502396

Wed Mar  2 06:08:36 2016
Active(file)        	441716
Inactive(file)      	7156
MemFree             	491840

Wed Mar  2 06:08:37 2016
Active(file)        	451932
Inactive(file)      	7156
MemFree             	481432

Wed Mar  2 06:08:38 2016
Active(file)        	462200
Inactive(file)      	7156
MemFree             	470904

Wed Mar  2 06:08:39 2016
Active(file)        	472424
Inactive(file)      	7156
MemFree             	460432

Wed Mar  2 06:08:40 2016
Active(file)        	482688
Inactive(file)      	7156
MemFree             	449892

Wed Mar  2 06:08:41 2016
Active(file)        	492936
Inactive(file)      	7156
MemFree             	439476

Wed Mar  2 06:08:42 2016
Active(file)        	503176
Inactive(file)      	7156
MemFree             	428936

Wed Mar  2 06:08:43 2016
Active(file)        	513416
Inactive(file)      	7156
MemFree             	418380

Wed Mar  2 06:08:44 2016
Active(file)        	523656
Inactive(file)      	7156
MemFree             	407856

Wed Mar  2 06:08:46 2016
Active(file)        	533896
Inactive(file)      	7156
MemFree             	397440

Wed Mar  2 06:08:47 2016
Active(file)        	544144
Inactive(file)      	7156
MemFree             	386900

Wed Mar  2 06:08:48 2016
Active(file)        	554384
Inactive(file)      	7156
MemFree             	376484

Wed Mar  2 06:08:49 2016
Active(file)        	564624
Inactive(file)      	7156
MemFree             	365944

Wed Mar  2 06:08:50 2016
Active(file)        	574864
Inactive(file)      	7156
MemFree             	355280

Wed Mar  2 06:08:51 2016
Active(file)        	585108
Inactive(file)      	7164
MemFree             	344864

Wed Mar  2 06:08:52 2016
Active(file)        	595356
Inactive(file)      	7164
MemFree             	334324

Wed Mar  2 06:08:53 2016
Active(file)        	605596
Inactive(file)      	7164
MemFree             	323908

Wed Mar  2 06:08:54 2016
Active(file)        	615836
Inactive(file)      	7164
MemFree             	313368

Wed Mar  2 06:08:55 2016
Active(file)        	626076
Inactive(file)      	7164
MemFree             	302828

Wed Mar  2 06:08:56 2016
Active(file)        	636324
Inactive(file)      	7156
MemFree             	292396

Wed Mar  2 06:08:57 2016
Active(file)        	646572
Inactive(file)      	7156
MemFree             	281840

Wed Mar  2 06:08:58 2016
Active(file)        	656812
Inactive(file)      	7156
MemFree             	271268

Wed Mar  2 06:08:59 2016
Active(file)        	667052
Inactive(file)      	7156
MemFree             	260712

Wed Mar  2 06:09:00 2016
Active(file)        	677292
Inactive(file)      	7156
MemFree             	250140

Wed Mar  2 06:09:01 2016
Active(file)        	687532
Inactive(file)      	7156
MemFree             	239708

Wed Mar  2 06:09:02 2016
Active(file)        	697780
Inactive(file)      	7156
MemFree             	229136

Wed Mar  2 06:09:03 2016
Active(file)        	708020
Inactive(file)      	7156
MemFree             	218704

Wed Mar  2 06:09:04 2016
Active(file)        	718260
Inactive(file)      	7156
MemFree             	208132

Wed Mar  2 06:09:05 2016
Active(file)        	728500
Inactive(file)      	7156
MemFree             	197700

Wed Mar  2 06:09:06 2016
Active(file)        	738752
Inactive(file)      	7156
MemFree             	187128

Wed Mar  2 06:09:07 2016
Active(file)        	749000
Inactive(file)      	7156
MemFree             	176448

Wed Mar  2 06:09:08 2016
Active(file)        	759240
Inactive(file)      	7156
MemFree             	166000

Wed Mar  2 06:09:09 2016
Active(file)        	769480
Inactive(file)      	7156
MemFree             	155444

Wed Mar  2 06:09:10 2016
Active(file)        	779720
Inactive(file)      	7156
MemFree             	145012

Wed Mar  2 06:09:11 2016
Active(file)        	789952
Inactive(file)      	7160
MemFree             	134440

Wed Mar  2 06:09:12 2016
Active(file)        	800208
Inactive(file)      	7160
MemFree             	123884

Wed Mar  2 06:09:13 2016
Active(file)        	810448
Inactive(file)      	7160
MemFree             	113436

Wed Mar  2 06:09:14 2016
Active(file)        	820688
Inactive(file)      	7160
MemFree             	102880

Wed Mar  2 06:09:15 2016
Active(file)        	830928
Inactive(file)      	7160
MemFree             	92308

Wed Mar  2 06:09:16 2016
Active(file)        	841180
Inactive(file)      	7160
MemFree             	81752

Wed Mar  2 06:09:17 2016
Active(file)        	851428
Inactive(file)      	7160
MemFree             	71180

Wed Mar  2 06:09:18 2016
Active(file)        	861668
Inactive(file)      	7160
MemFree             	60748

Wed Mar  2 06:09:20 2016
Active(file)        	871908
Inactive(file)      	7160
MemFree             	50176

Wed Mar  2 06:09:21 2016
Active(file)        	882148
Inactive(file)      	7160
MemFree             	39744

Wed Mar  2 06:09:22 2016
Active(file)        	892388
Inactive(file)      	7160
MemFree             	29172

Wed Mar  2 06:09:23 2016
Active(file)        	902636
Inactive(file)      	7160
MemFree             	18616

^C
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ echo "Once we hit aboit 9MB free, the system will start moving the file data fromt he active cache to the inactive cache. From there it will evict the data, ensuring Linux always has enough free memory for OS and applications"
Once we hit aboit 9MB free, the system will start moving the file data fromt he active cache to the inactive cache. From there it will evict the data, ensuring Linux always has enough free memory for OS and applications
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ perl ActiveFileCache.pl 
Active(file)        	902680
Active(anon)        	14344
Wed Mar  2 06:10:13 2016
Active(file)        	902676
Inactive(file)      	7164
MemFree             	18476

Wed Mar  2 06:10:14 2016
Active(file)        	902688
Inactive(file)      	7164
MemFree             	18480

Wed Mar  2 06:10:15 2016
Active(file)        	902652
Inactive(file)      	7164
MemFree             	18480

Wed Mar  2 06:10:16 2016
Active(file)        	902676
Inactive(file)      	7164
MemFree             	18480

Wed Mar  2 06:10:17 2016
Active(file)        	902688
Inactive(file)      	7164
MemFree             	18480

Wed Mar  2 06:10:18 2016
Active(file)        	902688
Inactive(file)      	7164
MemFree             	18504

Wed Mar  2 06:10:20 2016
Active(file)        	902684
Inactive(file)      	7164
MemFree             	18500

Wed Mar  2 06:10:21 2016
Active(file)        	902696
Inactive(file)      	7164
MemFree             	18500

Wed Mar  2 06:10:22 2016
Active(file)        	902708
Inactive(file)      	7164
MemFree             	18500

Wed Mar  2 06:10:23 2016
Active(file)        	902696
Inactive(file)      	7164
MemFree             	18476

Wed Mar  2 06:10:24 2016
Active(file)        	902684
Inactive(file)      	7164
MemFree             	18500

Wed Mar  2 06:10:25 2016
Active(file)        	902716
Inactive(file)      	7164
MemFree             	18476

Wed Mar  2 06:10:26 2016
Active(file)        	902716
Inactive(file)      	7164
MemFree             	18488

Wed Mar  2 06:10:27 2016
Active(file)        	902680
Inactive(file)      	7168
MemFree             	18484

Wed Mar  2 06:10:28 2016
Active(file)        	902692
Inactive(file)      	7168
MemFree             	18480

Wed Mar  2 06:10:29 2016
Active(file)        	902728
Inactive(file)      	7168
MemFree             	18380

Wed Mar  2 06:10:30 2016
Active(file)        	902724
Inactive(file)      	7168
MemFree             	18480

Wed Mar  2 06:10:31 2016
Active(file)        	902712
Inactive(file)      	7168
MemFree             	18380

Wed Mar  2 06:10:32 2016
Active(file)        	902712
Inactive(file)      	7168
MemFree             	18516

Wed Mar  2 06:10:33 2016
Active(file)        	902724
Inactive(file)      	7168
MemFree             	18376

Wed Mar  2 06:10:34 2016
Active(file)        	902736
Inactive(file)      	7168
MemFree             	18400

Wed Mar  2 06:10:35 2016
Active(file)        	902732
Inactive(file)      	7168
MemFree             	18500

Wed Mar  2 06:10:36 2016
Active(file)        	902744
Inactive(file)      	7168
MemFree             	18376

Wed Mar  2 06:10:37 2016
Active(file)        	902708
Inactive(file)      	7168
MemFree             	18488

Wed Mar  2 06:10:38 2016
Active(file)        	902720
Inactive(file)      	7168
MemFree             	18360

Wed Mar  2 06:10:39 2016
Active(file)        	902744
Inactive(file)      	7168
MemFree             	18484

Wed Mar  2 06:10:40 2016
Active(file)        	902716
Inactive(file)      	7168
MemFree             	18484

^C
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  2.2G  5.5G  29% /
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ rm /tmp/file*
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ perl ActiveFileCache.pl 
Active(file)        	11884
Active(anon)        	14368
Wed Mar  2 06:10:52 2016
Active(file)        	22096
Inactive(file)      	7260
MemFree             	921944

Wed Mar  2 06:10:53 2016
Active(file)        	32324
Inactive(file)      	7260
MemFree             	911552

Wed Mar  2 06:10:54 2016
Active(file)        	42584
Inactive(file)      	7260
MemFree             	901100

Wed Mar  2 06:10:55 2016
Active(file)        	52812
Inactive(file)      	7260
MemFree             	890580

Wed Mar  2 06:10:56 2016
Active(file)        	63076
Inactive(file)      	7260
MemFree             	880032

Wed Mar  2 06:10:57 2016
Active(file)        	73316
Inactive(file)      	7260
MemFree             	869600

Wed Mar  2 06:10:58 2016
Active(file)        	83568
Inactive(file)      	7260
MemFree             	859068

Wed Mar  2 06:10:59 2016
Active(file)        	93796
Inactive(file)      	7260
MemFree             	848512

Wed Mar  2 06:11:00 2016
Active(file)        	104056
Inactive(file)      	7260
MemFree             	838052

Wed Mar  2 06:11:02 2016
Active(file)        	114284
Inactive(file)      	7260
MemFree             	827452

Wed Mar  2 06:11:03 2016
Active(file)        	124512
Inactive(file)      	7260
MemFree             	816908

Wed Mar  2 06:11:04 2016
Active(file)        	134740
Inactive(file)      	7260
MemFree             	806500

Wed Mar  2 06:11:05 2016
Active(file)        	145004
Inactive(file)      	7260
MemFree             	795944

Wed Mar  2 06:11:06 2016
Active(file)        	155228
Inactive(file)      	7260
MemFree             	785412

Wed Mar  2 06:11:07 2016
Active(file)        	165492
Inactive(file)      	7260
MemFree             	774980

Wed Mar  2 06:11:08 2016
Active(file)        	175744
Inactive(file)      	7260
MemFree             	764424

Wed Mar  2 06:11:09 2016
Active(file)        	185960
Inactive(file)      	7260
MemFree             	754016

Wed Mar  2 06:11:10 2016
Active(file)        	196224
Inactive(file)      	7264
MemFree             	743464

Wed Mar  2 06:11:11 2016
Active(file)        	206484
Inactive(file)      	7264
MemFree             	733044

Wed Mar  2 06:11:12 2016
Active(file)        	216700
Inactive(file)      	7264
MemFree             	722484

Wed Mar  2 06:11:13 2016
Active(file)        	226964
Inactive(file)      	7264
MemFree             	711960

Wed Mar  2 06:11:14 2016
Active(file)        	237192
Inactive(file)      	7264
MemFree             	701528

Wed Mar  2 06:11:15 2016
Active(file)        	247432
Inactive(file)      	7264
MemFree             	690972

Wed Mar  2 06:11:16 2016
Active(file)        	257692
Inactive(file)      	7264
MemFree             	680440

Wed Mar  2 06:11:17 2016
Active(file)        	267920
Inactive(file)      	7264
MemFree             	670008

Wed Mar  2 06:11:18 2016
Active(file)        	278148
Inactive(file)      	7264
MemFree             	659352

Wed Mar  2 06:11:19 2016
Active(file)        	288412
Inactive(file)      	7264
MemFree             	648920

Wed Mar  2 06:11:20 2016
Active(file)        	298652
Inactive(file)      	7264
MemFree             	638364

Wed Mar  2 06:11:21 2016
Active(file)        	308900
Inactive(file)      	7264
MemFree             	627840

Wed Mar  2 06:11:22 2016
Active(file)        	319152
Inactive(file)      	7264
MemFree             	617424

Wed Mar  2 06:11:23 2016
Active(file)        	329356
Inactive(file)      	7264
MemFree             	606888

Wed Mar  2 06:11:24 2016
Active(file)        	339620
Inactive(file)      	7264
MemFree             	596452

Wed Mar  2 06:11:25 2016
Active(file)        	349872
Inactive(file)      	7264
MemFree             	585920

Wed Mar  2 06:11:26 2016
Active(file)        	360108
Inactive(file)      	7264
MemFree             	575364

Wed Mar  2 06:11:27 2016
Active(file)        	370348
Inactive(file)      	7264
MemFree             	564932

Wed Mar  2 06:11:28 2016
Active(file)        	380572
Inactive(file)      	7264
MemFree             	554400

Wed Mar  2 06:11:29 2016
Active(file)        	390828
Inactive(file)      	7264
MemFree             	543956

Wed Mar  2 06:11:30 2016
Active(file)        	401080
Inactive(file)      	7264
MemFree             	533432

Wed Mar  2 06:11:31 2016
Active(file)        	411316
Inactive(file)      	7264
MemFree             	522900

Wed Mar  2 06:11:32 2016
Active(file)        	421556
Inactive(file)      	7264
MemFree             	512316

Wed Mar  2 06:11:33 2016
Active(file)        	431808
Inactive(file)      	7264
MemFree             	501760

Wed Mar  2 06:11:34 2016
Active(file)        	442048
Inactive(file)      	7264
MemFree             	491328

Wed Mar  2 06:11:36 2016
Active(file)        	452300
Inactive(file)      	7264
MemFree             	480796

Wed Mar  2 06:11:37 2016
Active(file)        	462536
Inactive(file)      	7264
MemFree             	470364

Wed Mar  2 06:11:38 2016
Active(file)        	472776
Inactive(file)      	7264
MemFree             	459792

Wed Mar  2 06:11:39 2016
Active(file)        	483028
Inactive(file)      	7264
MemFree             	449364

Wed Mar  2 06:11:40 2016
Active(file)        	493256
Inactive(file)      	7264
MemFree             	438824

Wed Mar  2 06:11:41 2016
Active(file)        	503496
Inactive(file)      	7264
MemFree             	428284

Wed Mar  2 06:11:42 2016
Active(file)        	513744
Inactive(file)      	7264
MemFree             	417868

Wed Mar  2 06:11:43 2016
Active(file)        	523984
Inactive(file)      	7264
MemFree             	407328

Wed Mar  2 06:11:44 2016
Active(file)        	534236
Inactive(file)      	7264
MemFree             	396788

Wed Mar  2 06:11:45 2016
Active(file)        	544476
Inactive(file)      	7264
MemFree             	386248

Wed Mar  2 06:11:46 2016
Active(file)        	554716
Inactive(file)      	7264
MemFree             	375708

Wed Mar  2 06:11:47 2016
Active(file)        	564964
Inactive(file)      	7268
MemFree             	365292

Wed Mar  2 06:11:48 2016
Active(file)        	575204
Inactive(file)      	7268
MemFree             	354752

Wed Mar  2 06:11:49 2016
Active(file)        	585444
Inactive(file)      	7268
MemFree             	344336

Wed Mar  2 06:11:50 2016
Active(file)        	595684
Inactive(file)      	7268
MemFree             	333796

Wed Mar  2 06:11:51 2016
Active(file)        	605924
Inactive(file)      	7268
MemFree             	323256

Wed Mar  2 06:11:52 2016
Active(file)        	616172
Inactive(file)      	7268
MemFree             	312840

Wed Mar  2 06:11:53 2016
Active(file)        	626412
Inactive(file)      	7268
MemFree             	302176

Wed Mar  2 06:11:54 2016
Active(file)        	636664
Inactive(file)      	7268
MemFree             	291636

Wed Mar  2 06:11:55 2016
Active(file)        	646904
Inactive(file)      	7268
MemFree             	281204

Wed Mar  2 06:11:56 2016
Active(file)        	657144
Inactive(file)      	7268
MemFree             	270680

Wed Mar  2 06:11:57 2016
Active(file)        	667392
Inactive(file)      	7268
MemFree             	260264

Wed Mar  2 06:11:58 2016
Active(file)        	677632
Inactive(file)      	7268
MemFree             	249600

Wed Mar  2 06:11:59 2016
Active(file)        	687872
Inactive(file)      	7268
MemFree             	239168

Wed Mar  2 06:12:00 2016
Active(file)        	698112
Inactive(file)      	7268
MemFree             	228768

Wed Mar  2 06:12:01 2016
Active(file)        	708352
Inactive(file)      	7268
MemFree             	218212

Wed Mar  2 06:12:02 2016
Active(file)        	718584
Inactive(file)      	7268
MemFree             	207640

Wed Mar  2 06:12:03 2016
Active(file)        	728840
Inactive(file)      	7268
MemFree             	197208

Wed Mar  2 06:12:04 2016
Active(file)        	739080
Inactive(file)      	7268
MemFree             	186512

Wed Mar  2 06:12:05 2016
Active(file)        	749320
Inactive(file)      	7268
MemFree             	176080

Wed Mar  2 06:12:06 2016
Active(file)        	759560
Inactive(file)      	7268
MemFree             	165508

Wed Mar  2 06:12:07 2016
Active(file)        	769808
Inactive(file)      	7268
MemFree             	154952

Wed Mar  2 06:12:08 2016
Active(file)        	780048
Inactive(file)      	7268
MemFree             	144504

Wed Mar  2 06:12:10 2016
Active(file)        	790300
Inactive(file)      	7268
MemFree             	133948

Wed Mar  2 06:12:11 2016
Active(file)        	800540
Inactive(file)      	7268
MemFree             	123500

Wed Mar  2 06:12:12 2016
Active(file)        	810780
Inactive(file)      	7268
MemFree             	112944

Wed Mar  2 06:12:13 2016
Active(file)        	821028
Inactive(file)      	7268
MemFree             	102264

Wed Mar  2 06:12:14 2016
Active(file)        	831268
Inactive(file)      	7268
MemFree             	91816

Wed Mar  2 06:12:15 2016
Active(file)        	841508
Inactive(file)      	7268
MemFree             	81260

Wed Mar  2 06:12:16 2016
Active(file)        	851748
Inactive(file)      	7268
MemFree             	70812

Wed Mar  2 06:12:17 2016
Active(file)        	861988
Inactive(file)      	7268
MemFree             	60256

Wed Mar  2 06:12:18 2016
Active(file)        	872236
Inactive(file)      	7268
MemFree             	49684

Wed Mar  2 06:12:19 2016
Active(file)        	882476
Inactive(file)      	7268
MemFree             	39252

Wed Mar  2 06:12:20 2016
Active(file)        	892716
Inactive(file)      	7268
MemFree             	28680

Wed Mar  2 06:12:21 2016
Active(file)        	902956
Inactive(file)      	7268
MemFree             	18248

Wed Mar  2 06:12:22 2016
Active(file)        	809036
Inactive(file)      	107020
MemFree             	11916

Wed Mar  2 06:12:23 2016
Active(file)        	771108
Inactive(file)      	148232
MemFree             	9036

Wed Mar  2 06:12:24 2016
Active(file)        	729564
Inactive(file)      	188620
MemFree             	10492

Wed Mar  2 06:12:25 2016
Active(file)        	706860
Inactive(file)      	212456
MemFree             	9736

Wed Mar  2 06:12:26 2016
Active(file)        	685596
Inactive(file)      	233960
MemFree             	9868

Wed Mar  2 06:12:27 2016
Active(file)        	670300
Inactive(file)      	250432
MemFree             	8976

Wed Mar  2 06:12:28 2016
Active(file)        	653200
Inactive(file)      	267188
MemFree             	9580

Wed Mar  2 06:12:29 2016
Active(file)        	639076
Inactive(file)      	281304
MemFree             	9920

Wed Mar  2 06:12:30 2016
Active(file)        	625416
Inactive(file)      	294404
MemFree             	10756

Wed Mar  2 06:12:31 2016
Active(file)        	616904
Inactive(file)      	304192
MemFree             	9872

Wed Mar  2 06:12:32 2016
Active(file)        	608672
Inactive(file)      	313408
MemFree             	9124

Wed Mar  2 06:12:33 2016
Active(file)        	599704
Inactive(file)      	322564
MemFree             	9344

Wed Mar  2 06:12:34 2016
Active(file)        	591000
Inactive(file)      	331176
MemFree             	9680

Wed Mar  2 06:12:35 2016
Active(file)        	583512
Inactive(file)      	338832
MemFree             	9792

Wed Mar  2 06:12:36 2016
Active(file)        	576248
Inactive(file)      	346064
MemFree             	10104

Wed Mar  2 06:12:37 2016
Active(file)        	569240
Inactive(file)      	352844
MemFree             	10492

Wed Mar  2 06:12:38 2016
Active(file)        	564560
Inactive(file)      	358484
MemFree             	9840

Wed Mar  2 06:12:39 2016
Active(file)        	557864
Inactive(file)      	364604
MemFree             	10688

Wed Mar  2 06:12:40 2016
Active(file)        	555524
Inactive(file)      	368920
MemFree             	9044

Wed Mar  2 06:12:41 2016
Active(file)        	551240
Inactive(file)      	373692
MemFree             	8808

Wed Mar  2 06:12:43 2016
Active(file)        	542976
Inactive(file)      	379496
MemFree             	11272

Wed Mar  2 06:12:44 2016
Active(file)        	540932
Inactive(file)      	383164
MemFree             	10048

Wed Mar  2 06:12:45 2016
Active(file)        	537024
Inactive(file)      	387208
MemFree             	10144

Wed Mar  2 06:12:46 2016
Active(file)        	535280
Inactive(file)      	390520
MemFree             	8856

Wed Mar  2 06:12:47 2016
Active(file)        	531516
Inactive(file)      	394240
MemFree             	8960

Wed Mar  2 06:12:48 2016
Active(file)        	524960
Inactive(file)      	398484
MemFree             	11532

Wed Mar  2 06:12:49 2016
Active(file)        	523316
Inactive(file)      	401304
MemFree             	10668

Wed Mar  2 06:12:50 2016
Active(file)        	521752
Inactive(file)      	404024
MemFree             	9624

Wed Mar  2 06:12:51 2016
Active(file)        	518004
Inactive(file)      	407112
MemFree             	10488

Wed Mar  2 06:12:52 2016
Active(file)        	514676
Inactive(file)      	409964
MemFree             	11120

Wed Mar  2 06:12:53 2016
Active(file)        	511444
Inactive(file)      	412676
MemFree             	11672

Wed Mar  2 06:12:54 2016
Active(file)        	512100
Inactive(file)      	414544
MemFree             	9564

Wed Mar  2 06:12:55 2016
Active(file)        	510424
Inactive(file)      	416732
MemFree             	9240

Wed Mar  2 06:12:56 2016
Active(file)        	508612
Inactive(file)      	418876
MemFree             	9128

Wed Mar  2 06:12:57 2016
Active(file)        	506692
Inactive(file)      	420960
MemFree             	9116

Wed Mar  2 06:12:58 2016
Active(file)        	504668
Inactive(file)      	422952
MemFree             	9196

Wed Mar  2 06:12:59 2016
Active(file)        	500872
Inactive(file)      	425144
MemFree             	11028

Wed Mar  2 06:13:00 2016
Active(file)        	499932
Inactive(file)      	426840
MemFree             	10512

Wed Mar  2 06:13:01 2016
Active(file)        	499008
Inactive(file)      	428456
MemFree             	9884

Wed Mar  2 06:13:02 2016
Active(file)        	498400
Inactive(file)      	430000
MemFree             	8980

Wed Mar  2 06:13:03 2016
Active(file)        	494780
Inactive(file)      	431852
MemFree             	10976

Wed Mar  2 06:13:04 2016
Active(file)        	495772
Inactive(file)      	433004
MemFree             	8948

Wed Mar  2 06:13:05 2016
Active(file)        	493212
Inactive(file)      	434544
MemFree             	10160

Wed Mar  2 06:13:06 2016
Active(file)        	492452
Inactive(file)      	435816
MemFree             	9900

Wed Mar  2 06:13:07 2016
Active(file)        	491728
Inactive(file)      	437040
MemFree             	9624

Wed Mar  2 06:13:08 2016
Active(file)        	490696
Inactive(file)      	438264
MemFree             	9360

^C
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ echo "At this point the systems tabilises around 490MB Active and about 430MB inactive"
At this point the systems tabilises around 490MB Active and about 430MB inactive
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ echo "Now have a look at /proc/meminfo before and after I delete the files that we just created"
Now have a look at /proc/meminfo before and after I delete the files that we just created
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ cat /proc/meminfo
MemTotal:        1019444 kB
MemFree:           10420 kB
MemAvailable:     969860 kB
Buffers:             308 kB
Cached:           928908 kB
SwapCached:            0 kB
Active:           504448 kB
Inactive:         438452 kB
Active(anon):      13708 kB
Inactive(anon):       52 kB
Active(file):     490740 kB
Inactive(file):   438400 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                16 kB
Writeback:             0 kB
AnonPages:         13700 kB
Mapped:             4392 kB
Shmem:                60 kB
Slab:              54276 kB
SReclaimable:      44868 kB
SUnreclaim:         9408 kB
KernelStack:        1408 kB
PageTables:         2356 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      509720 kB
Committed_AS:      62876 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        2488 kB
VmallocChunk:   34359731159 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       22528 kB
DirectMap2M:     1026048 kB
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ du -sh /tmp/file*
10M	/tmp/file.1
10M	/tmp/file.10
10M	/tmp/file.100
10M	/tmp/file.101
10M	/tmp/file.102
10M	/tmp/file.103
10M	/tmp/file.104
10M	/tmp/file.105
10M	/tmp/file.106
10M	/tmp/file.107
10M	/tmp/file.108
10M	/tmp/file.109
10M	/tmp/file.11
10M	/tmp/file.110
10M	/tmp/file.111
10M	/tmp/file.112
10M	/tmp/file.113
10M	/tmp/file.114
10M	/tmp/file.115
10M	/tmp/file.116
10M	/tmp/file.117
10M	/tmp/file.118
10M	/tmp/file.119
10M	/tmp/file.12
10M	/tmp/file.120
10M	/tmp/file.121
10M	/tmp/file.122
10M	/tmp/file.123
10M	/tmp/file.124
10M	/tmp/file.125
10M	/tmp/file.126
10M	/tmp/file.127
10M	/tmp/file.128
10M	/tmp/file.129
10M	/tmp/file.13
10M	/tmp/file.130
10M	/tmp/file.131
10M	/tmp/file.132
10M	/tmp/file.133
10M	/tmp/file.14
10M	/tmp/file.15
10M	/tmp/file.16
10M	/tmp/file.17
10M	/tmp/file.18
10M	/tmp/file.19
10M	/tmp/file.2
10M	/tmp/file.20
10M	/tmp/file.21
10M	/tmp/file.22
10M	/tmp/file.23
10M	/tmp/file.24
10M	/tmp/file.25
10M	/tmp/file.26
10M	/tmp/file.27
10M	/tmp/file.28
10M	/tmp/file.29
10M	/tmp/file.3
10M	/tmp/file.30
10M	/tmp/file.31
10M	/tmp/file.32
10M	/tmp/file.33
10M	/tmp/file.34
10M	/tmp/file.35
10M	/tmp/file.36
10M	/tmp/file.37
10M	/tmp/file.38
10M	/tmp/file.39
10M	/tmp/file.4
10M	/tmp/file.40
10M	/tmp/file.41
10M	/tmp/file.42
10M	/tmp/file.43
10M	/tmp/file.44
10M	/tmp/file.45
10M	/tmp/file.46
10M	/tmp/file.47
10M	/tmp/file.48
10M	/tmp/file.49
10M	/tmp/file.5
10M	/tmp/file.50
10M	/tmp/file.51
10M	/tmp/file.52
10M	/tmp/file.53
10M	/tmp/file.54
10M	/tmp/file.55
10M	/tmp/file.56
10M	/tmp/file.57
10M	/tmp/file.58
10M	/tmp/file.59
10M	/tmp/file.6
10M	/tmp/file.60
10M	/tmp/file.61
10M	/tmp/file.62
10M	/tmp/file.63
10M	/tmp/file.64
10M	/tmp/file.65
10M	/tmp/file.66
10M	/tmp/file.67
10M	/tmp/file.68
10M	/tmp/file.69
10M	/tmp/file.7
10M	/tmp/file.70
10M	/tmp/file.71
10M	/tmp/file.72
10M	/tmp/file.73
10M	/tmp/file.74
10M	/tmp/file.75
10M	/tmp/file.76
10M	/tmp/file.77
10M	/tmp/file.78
10M	/tmp/file.79
10M	/tmp/file.8
10M	/tmp/file.80
10M	/tmp/file.81
10M	/tmp/file.82
10M	/tmp/file.83
10M	/tmp/file.84
10M	/tmp/file.85
10M	/tmp/file.86
10M	/tmp/file.87
10M	/tmp/file.88
10M	/tmp/file.89
10M	/tmp/file.9
10M	/tmp/file.90
10M	/tmp/file.91
10M	/tmp/file.92
10M	/tmp/file.93
10M	/tmp/file.94
10M	/tmp/file.95
10M	/tmp/file.96
10M	/tmp/file.97
10M	/tmp/file.98
10M	/tmp/file.99
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ rm /tmp/file*
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ cat /proc/meminfo
MemTotal:        1019444 kB
MemFree:          957192 kB
MemAvailable:     971072 kB
Buffers:             448 kB
Cached:             6792 kB
SwapCached:            0 kB
Active:            20508 kB
Inactive:            424 kB
Active(anon):      13708 kB
Inactive(anon):       52 kB
Active(file):       6800 kB
Inactive(file):      372 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               120 kB
Writeback:             0 kB
AnonPages:         13700 kB
Mapped:             4392 kB
Shmem:                60 kB
Slab:              29412 kB
SReclaimable:      20004 kB
SUnreclaim:         9408 kB
KernelStack:        1408 kB
PageTables:         2356 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      509720 kB
Committed_AS:      62876 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        2488 kB
VmallocChunk:   34359731159 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       22528 kB
DirectMap2M:     1026048 kB
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ echo "As you can see, the memory has freed up and the cache has disappeared"
As you can see, the memory has freed up and the cache has disappeared
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ echo "This ladies and gentlemen has been the first installment in a series of Linux VM Subsystem experiments"
This ladies and gentlemen has been the first installment in a series of Linux VM Subsystem experiments
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ echo "The next installment will involve the interaction of the filesystem cache and anonymous memory increases"
The next installment will involve the interaction of the filesystem cache and anonymous memory increases
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ echo "Good night, back at work in 6 hours"
Good night, back at work in 6 hours
[ec2-user@ip-172-31-37-189 linux-systems-programming]$ exit

Testing ntp server connectivity

I came across an issue today where I needed to test ntp connectivity. Unfortunately my telnet and netcat tests were not working as expected, even with the UDP option.

I found the ntpdate -q command which when a server is specified will query the NTP source. Unfortunately, according to the man page, this command will be removed.

Just blogging this so I can come back to it later.

Postinstall scripts are not configuration management

This post is a reaction to reviewing a puppet manifest at ${WORK} that installs a package and then has an exec resource run a script that modifies a configuration file for the package. Something in me just screams that this sort of approach is wrong, but I was looking for a way to explain why. Rather than just rant and rave at length which I am known to do on occasion, I wanted to give a succinct set of rules the team could follow. Fortunately, people aren't robots and so they usually want to understand the reasons behind those rules. This post is a means of me fleshing out those reasons.

It should be noted that the author of said manifest is a junior member of staff learning Puppet and so this is not a critique of their work by any means. I wanted to take the time to explain to them and provide guidance to the rest of the team about how various components should be handled.

First of all, let's start with what this person did right

• They fully automated the installation of this agent using Puppet
• They deployed the binaries and scripts using a package

Based on that alone, this person is already heading in the right way and is leagues ahead of a large percentage of system administrators. If I asked "How would you install X on Y systems?" in an interview and that was their answer, they would be hired based purely on their potential.Furthermore, given that the manual way to install X was to install the package and run the configuration script, it is understandable why the person who wrote the manifest would do it that way. 

The issue is that using scripting to modify configuration files or settings has a number of problems with it

• Once you "shell out" of Puppet to perform an action or create configuration, Puppet has no way of knowing what was done.
• RPM also has the same issue, especially if you are using the script to modify the state of files delivered by the package. RPM does have the concept of a %config macro to help with this problem, but ultimately once you use a script to change the contents of a file from its original state, RPM verify will start reporting errors. 
• What ever the script is doing, this should be handled explicitly within puppet. In the example, a configuration file was modified to use specify that a particular user should be used to execute the agent. This should have been delivered as an ERB template with the user as a variable. Other examples would be enabling a service etc. 
• Scripts, either external to the package or things like preinstall or postinstall scripts are rarely idempotent  and unless coded specifically to detect the state of what they are modifying will generally result in different results. An example of this would be adding a line to xinetd.conf for a service, in order to make the script idempotent, you would first need to check if the line exists and if it does not, then you can safely add it. 

General rules

Based on the above, what general rules can be stated about packages and configuration?

Note: These general rules assume that you have a configuration management tool

• All binaries and scripts should be packaged. If it won't change once you install it on the system, then it belongs in a package. 
• All configuration dependencies should be expressed in Puppet (users, services etc)
• All configuration files should be delivered as ERB templates as part of the puppet module and any modifications should be made either by including Puppet variables or using system facts for any host-specific configuration. The collary to this is that configuration files should not be packaged.
• Packages should contain no pre or post install scripts, any configuration requirements should be expressed within puppet. 

This thinking lines up with how the IPS packaging system works in Solaris 11. This is described in Stephen Hahn's paper called pkg(5): a no scripting zone. While the logic of IPS is sound, there is an implicit reliance on another tool (Puppet, Chef etc) to handle the configuration. There is a hack to use SMF to launch a script to configure the package, but that just seems out of place and awkward. The flaw in the logic that Sun had and Oracle inherited was that Solaris had no native configuration management tools to handle this and still do not.

Benefits of this approach

• If your packages only contain binaries and scripts that do not change, then the package verification checks such as Red Hat's RPM Verify or Solaris' pkgchk(1m), come back with no errors as the expected contents of the package match what is actually on the system. 
• There are no conflicts between your package management systems and configuration management systems. If RPMs modify files delivered by packages or packages modify files controlled by Puppet, then a discrepancy will arise. Because the package management scripting is one shot only based on installation time, Puppet will overwrite the configuration with what it expects. 
• It is very clear what requirements a particular piece of software has. For example, if a piece of software needs a user defined, this should be clear from the manifest. Should the user need to change, this can be easily handled within puppet. 

Other considerations

One problem with this approach is that you have information about a particular component across two different area; the packages and the puppet code. It is important that both are kept under source control, preferably within the same repository because there is a dependency between them. These items should cross reference each other as well in the source repository. 

The other downside is if you have to manage systems that are under puppet control and systems which are not. If you have to manage both types of systems, then I think you need to allow configuration files to be delivered in packages and modified in scripts so that you legacy systems will continue to work. For your newer systems, you should also express that configuration within Puppet and live with the expectation that puppet will redo the work of the package. 

Closing Thoughts

This was a general introduction to how I believe configuration files should be handled. What I would really like to see is a series of patterns that can be used to explain these sorts of concepts. The Limoncelli, Hogan and Chalup book The Practice of System and Network Administration provides a good overview, but could be updated as the second version is 5 years old today as I write this. 

Virtuous cycle of devops: Standardisation - The implied link

Joe Kinsella (@joekinsella) from sonian wrote an excellent article called "Virtuous Cycle of Devops" which succinctly summed up the the benefits of adopting a more agile approach to managing systems and the applications which run on them. The article is spot on the money and sums up really well what I have been trying to achieve for the last 2 years. You should go and read the article, it's very short and well worth it, but the main point was that there are a number of benefits from this approach which flow into each other creating a cycle of continual improvement.

Source: http://www.hightechinthehub.com/2012/04/virtuous-cycle-of-devops/

Each of the benefits described in Joe's article relies on the assumption that you are building on a standardised and consistent base. Providing a standardised configuration at the server level allows you to deliver flexibility at higher layers in the stack because the basic layers are configured the same way and can be treated as a single piece. Standardisation allows you to deliver value added flexibility where innovation is most beneficial to the business; at the application layer. This concept is summed up really well in an article on the SkyDingo blog called DEVOPS: FLEXIBLE CONFIGURATION MANAGEMENT? NOT SO FAST! where the authors claim that by limiting flexibility at the infrastructure level (gratuitous flexibility) that you increase flexibility at a application level (value added flexibility).

Sysadmins: This is not just for developers, having standardised and consistent configurations across your fleet allows you to respond with agility to changing requirements at the infrastructure layer because you know that all the servers are configured the same way and that any action taken should respond uniformally (Murphy will throw you some edge cases from time to time though based on things outside of your control, that's why you need solid automated tests).

There are numerous examples where this sort of approach is applicable, but the main one of benefit to system administrators is the application of security or bug fixes. If your environment is standardised, it becomes a relatively simple exercise to test a new fix in a lab environment and then roll that out to your fleet. On the other hand, if you do not have a standardised environment, the roll out of any fix becomes a configuration by configuration (or worse server by server) exercise. Knowing how your servers will behave to a new configuration requirement is the difference between being able to patch hundreds of servers at a time or handling them one by one. 

Ask yourself: If a zero day patch was released tomorrow for SSH how would you handle it? If the answer is roll it out by hand, you already lost. These are the sorts of things that differentiate small scale thinking about individual systems from large scale thinking about an infrastructure ecosystem.  

In startups these days, working with a standard configuration is a basic assumption and through the use of tools like Puppet, Chef and Cfengine is becoming more and more mainstream. If you look at some of the names of companies sending people to puppetconf and the upcoming chefconf, these ideas are catching on in very large enterprises and this is a very different space with very different requirements.

In green field environments, it is relatively easy to control configuration drift if you built the environment correctly from the start using configuration management practices. In legacy environments, it is not that easy. You have existing servers, built by different people over a number of years, new operating system releases come out leaving the older ones behind and technical debt piles up if left unchecked. Pulling that all together is really difficult and as we see the adoption of configuration management tools in large enterprises, this is something that should spoken about more openly at conferences.   This is not a solved problem, not by a long shot.

I know of one investment bank (not my current employer, but I'd love to work there) that rebuilds its global infrastructure every night to ensure absolute consistency and avoid configuration drift. While I personally think that is a little over the top, to achieve that level of control over and confidence in your infrastructure is really the pinnacle of system administration, regardless of whether you are a startup or a bank that has been around for 200 years. 

If you are not using a configuration management system, pick one use it and get on to more interesting things like adding value for your business. 

How to not get bitten twice (or OODA loop in action)

Over the last two years, I have had the pleasure of working with one of the best admin teams I have ever worked with and here is a simple example of why.

You've discovered an issue, let's say an NFS performance problem as an example. You determine through some digging in /proc/net/rpc/nfsd (explanation of contents here) that you have too few NFS threads configured (all NFS threads were busy and IO was stalling) and this has happened a large number of times since boot. What do you do?

Note: Most of this actually happened, but some of it is what I would like to have happened (doco and build not updated yet)

• Scan the fleet for other servers with the same issue
• Create changes to fix the issue and advise your customers
• Develop a custom SNMP extension that outputs a 1-minute rolling value of the stalls instead of the absolute value
• Plug monitoring into your tool of choice (Zenoss, Zabbix, OpenView, Patrol etc)
• Set an alert threshold to generate events in case the problem ever returns (I love the etsy engineering quote "If it moves, we track it. Sometimes we’ll draw a graph of something that isn’t moving yet, just in case it decides to make a run for it" http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/)
• Update your internal wiki so that when the alert is generated again, the on-call guy knows what to do. 
• Fix your build (hopefully using configuration management) so you won't have the same problem in the future

There are a couple of things that are important for this to work properly

• You need to be able to understand the problem, having people around who have deep understanding of technologies like NFS (we use it a lot) and where the problems can occur is really useful. You can have the best tools in the world, but you need the ability to use them and that requires understanding. 
• You need to be able to move quickly, this problem and the fix were discovered in Australia, the SNMP extension and graphing was developed in London and the alerting, documentation and build updates were configured in the US; Now that is a seamless handover process if ever I saw one. The Australians will have gone from leaving the office with a problem and a fix at 5PM and when they came in the next day at 9AM, the monitoring and alerting was plugged in already so they could assess the state of the problem (Observe, Orient, Decide, Act and back to Observe again ... just in case it makes a run for it). 
• You need to be able to get it out there, developing a fix is great, but you need to be able to get it out there quickly. The number of hosts for this was limited, but had we needed to get it out to the whole fleet quickly, something like Puppet with custom resource providers for SNMP extensions 
Most of all, you need people with the right mindset and that is why I love working with the team.

Icing on the cake

If you have not read The Practice of System and Network Administration then you should. It is a fantastic book about how to be a sysadmin, not necessarily technology A or B, but rather promoting the right mindset and providing an overview of the required knowledge areas. One of the things I like about it is that it sets out recommended practices and then provides the icing on the cake section. I am all about cake and icing, both figuratively and literally.

While I would like to think that the scenario I described above is a well implemented best practice, alas it is not and people who have such an approach are few and far between. At any rate, if I were to improve the above scenario, instead of alerting and waking someone up (this is not necessarily worth that), I would like to see the system automatically scale up the number of NFS threads and drop a message in the logs saying it did just that. If Linux handled this automatically, that would be great and I will log an RFE with our vendor to do just that. 

In the interim here's something cool: Facebook have implemented a self-healing system call FBAR (FaceBook Auto Remediation - https://www.facebook.com/notes/facebook-engineering/making-facebook-self-healing/10150275248698920) that automatically responds to such issues with automatic fixes, only escalating to a human if necessary. Now if only I could figure out how to increase the number of threads without a restart ... off to google. 

Linux Long Term Support and why enterprises think they need it

Disclosure: I have worked in various enterprises (banking and telcos) for the last 10 years or so as a UNIX Admin, the last 5 in a Bank.

The guys at the Food Fight Show recently had a discussion called "Distro Dancing" where they discussed their opinions on various Linux distros, which one they used first and which ones they like now and why they find them useful. It was interesting to see how people's needs developed over time with the development of Linux as a "play thing" to Linux as an OS that powers many of today's data centres and how that impacts people's requirements. 

The main gist was that if you are using tools like chef or puppet, then you end up caring less about the distros because all of the functionality for configuration is abstracted behind the respective configuration tools and you are not tied to a particular distro's administrative interfaces. While I agree with this at a high-level there are some added extras that particular distros bring to the table such as their associated software repositories. The ability to "yum install" or "apt-get install" a particular application or library without the need for customized repos and packaging etc is quite useful. 

John Vincent (@lusis) made a very insightful series of comments around the side effects of distros such as RHEL and Debian/Ubuntu and their Long Term Support implementations. Long story short, the Red Hat guys lock their software major versions down about three years before the software is actually released. This means that by the time the software is ready for general availability, it is already behind the state of the art in terms of updated packages. The example given was RHEL locking the system version of Python at 2.4 for their system tools where 2.7 is current and 2.6 still very popular and the impact that this has on people wanting a more up to date version of Python to work with. 

The question that I had for myself was is why do enterprises require operating systems that are Long Term Supported? By that I am not only referring to the length of the support contract although that does play a part, but rather some of the restrictions around major version lockdowns etc. Here are some that came off the top of my head (in no particular order)

• Enterprises use software from major software producers (Oracle, Sybase, Weblogic etc.). Version lockdowns allow those software producers to release software that will work with a known OS configuration.  By keeping their versions of the underlying system libraries (glibc etc) stable, the software vendors have less to worry about when it comes to certifying releases of their products. This is a feedback loop where the demand for software compounds the demand for locked down OS releases. Long story short, if you want one of the big DB players, you will end up on Red Hat or Solaris. 
• Enterprises have usually been around for a while and have collected some serious technical debt. You can argue whether or not this is a valid argument, but it exists and it is reality for many admins. Developers move on, projects are abandoned, but unless someone is keeping a very keen eye on their software inventory, that code is still running on a OS somewhere and its support details can be somewhat sketchy. Because of this technical debt, people can become very tied to specific server configurations and any attempt to "mess with" said configurations is met with fear and trembling. Keeping the binary compatibility guarantees of say a Red Hat or Solaris, means that these legacy apps can keep on running without intervention.
• Enterprises do not like surprises. One of the requirements that a lot of enterprises have is that of regular patching. Were patching to introduce major version changes to parts of the system, then the anxiety associated with patching would be much higher. Imagine if Apache were to suddenly deprecate the use of single file httpd.conf configurations and force everyone over to httpd.d type configurations, this would break multitudes of applications and stop any progress of keeping systems up to date. In an ironic twist, locking down versions of system software actually helps to keep them current (at least in terms of security fixes) as there is higher confidence in the patching process. 

There are probably many more reasons that enterprises need Long Term Supported OS'; However I am more interested in what admins can do to avoid getting locked into certain OS' or specific server configurations in the first place to allow easier moves to more up to date releases. 

In summary

• STOP DOING THINGS BY HAND and get a configuration management tool
• Use said configuration management tool for defining your applications' requirements
• Test your applications' portability to flush out hidden dependencies
• Support all of the above with proper policies around life cycle management and configuration management. 

1. STOP DOING THINGS BY HAND. Seriously, 1999 wants its administration techniques back. With tools like chef or puppet available for free or with support contracts, there is no excuse to be hand crafting configurations on servers anymore. That's all very nice to say, but what are the consequences of hand crafting a server?

• There is no reasonable way to replicate the environment, this means that when you need to move from one OS version to another for support reasons, the process to do so is based on the administrator's ability to document (or remember if it is the same person) the steps required to configure the server and install the application. 
• Hand crafting a server limits your ability to create production-like development or test environments or staging environments for moving your applications to a new server. 
• People become very attached to their hand crafted servers because their hand crafted servers were built specifically to run their application. When this happens, they become complacent about maintaining the documentation and configuration of the servers because they will always be there right? Wrong! You will eventually need to upgrade the hardware or the OS soon enough and that "one-off hack adding a symlink to X" will comeback and bite you when you move over to the new machine.

Side note: I remember one particularly experienced application administrator  telling me to always remember the three P's of things that can go wrong in a migration; Passwords, Profiles and Permissions. This has stayed with me and influenced much of my thinking around configuration management. 

2. Explicitly define *all* of your specific dependencies. If your application specifically requires a particular version of a library or a user particular user defined or a particular directory permission set then this should be explicitly called out. This can be done in documentation, but we all know that documentation can suffer from bit rot just like software; It becomes outdated or downright inaccurate over the time. The best way to enforce these dependencies is using a configuration management system like chef or puppet because not only are these set at install time, they are also actively maintained as part of the run time configuration for the server so that if a permission is changed by hand, it will be changed back.  Defining all of your configuration explicitly in a configuration management tool allows you to recreate the application environment (think the three Ps) on a new system without worrying that particular pieces are missing. 

Side note: This does force people to work within the confines of the configuration management tool, but that is more of an organizational issue than a technical one.

3. Keep your applications portable. I made a previous attempt at discussing this, but basically if your application can be *easily* moved between servers, then the odds are high that you do not have underlying undocumented system dependencies and this will allow for an easy migration. If your application runs on a single server and it was deemed important enough to have a DR backup, then you should be moving the application between servers on a semi-regular basis to ensure portability. This could be done during official DR testing or more often if it is associated with events like regular patching/maintenance etc. If your application is run across multiple machines, then adding new machines to the cluster of machines will also test the configuration and ensure that there are no hidden system dependencies.

4. Don't let things become stale in the first place. If you allow configurations to drift or OS releases to age in your environment, the harder it will be to move off them. The people who originally installed the servers and the applications may no longer work for the company, finding support for the operating system may become much harder (Note: RHEL 4 and Solaris 8 both hit end of life recently) and the inevitable fear and trembling will set in. "You can't patch server X, no one knows how it works!". The only way to avoid this is to have strong policies in place around OS support (and subsequent upgrades) and configuration management. If people know that they will have to move their application in X years (defining X is an exercise for the reader) then they will be less complacent about sloppy configuration management practices, but for this to work properly they need to the right tools to record and maintain the configuration.

Am I saying that if you do all of those things, your enterprise will move from RHEL 5 to Fedora overnight? No, but it will make the move from RHEL 5 to RHEL 6 a lot easier than it otherwise would have been. If people have confidence in your ability to move from known configuration to known configuration, then maybe there would be a more relaxed attitude to say moving from RHEL to a distro that is slightly more up to date, but for that to happen you have to put in the hard yards first of collecting and maintaining all of that configuration out there.

There is a second option to relying on system defined requirements and that is to bundle all application requirements beyond basic system libraries into an application filesystem that is maintained by the developers. Example: If you application require zlib-X then bundle it with your application. This works very well for maintaining independence from the underlying server OS, but places a very high burden on the application support teams because they need to track and update versions of software as it is released for patches. It is much easier to allow the OS vendor to track and maintain this software, however the cost comes at the expense of application isolation from OS changes. I personally do not recommend this as developers should be spending their time developing rather than tracking and maintaining dependencies.

Devops Days Austin - Day Two

Day two in Austin was quite useful there were talks from Etsy on how they handle security in a devops manner, NI also talked about how they setup a SaaS team to handle new deployments in the cloud.

The open spaces were still my favourite part with lots of discussions about centralised logging, rugged devops and devops in a large IT arena. I met people with the same sorts of problems that I have (lots of legacy stuff) and people who do not (green fields in the cloud type startups).

All in all an excellent conference.