In my continued usage with SparkleFormation
I’m really growing to appreciate the convenience of having ruby available when
composing templates. Things that would be challenging in a simplistic
serialization format or lead to unmanageable duplication become easily solvable
thanks to the powerful combination of an actual programming language, which json
and yml are not, and the simplicity of the SparkleFormation DSL.
At it’s simplest you can mirror the structure of any cloudformation json (and
finally have comments inline!) but you quickly discover more advanced use cases.
SparkleFormation registries
Registries are described as:
“lightweight dynamics that are useful for storing items that may be used in multiple locations. “
A dynamic in SparkleFormation is just a reusable block of code that
generates some section of the template and can be very flexible. This is in
comparison to components which are meant as single use items to insert a
static block of code.
Registries and dynamics are similar in that they are just building blocks for
composing the template elements for any declarative orchestration API such as
CloudFormation or Azure Resource Manager, well any arbitrary json data structure
really. Registries should be simpler and more static than a full fledged
dynamic, useful for things like a list of AWS instance sizes as in the
documentation:
or even something a little more involved like shared AWS::CloudFormation::Init
data.
A more dynamic registry for AWS cfn-init
Recently the need arose to provide a shared registry for the necessary init
commands to bootstrap a new ec2 instance with chef-client. Since we deal with
both Windows and linux nodes in ec2 it would be a bit annoying to have to
declare different registries like
After a brief discussion on freenode irc #sparkleformation,
@luckymike pointed me in the direction of a
technique he discovered that took advantage of the
cfn-init configsets
feature and the fact that registries can take args, just like any other
SparkleFormation dynamic.
Essentially, cloudformation init will look for an array of config names and run
them in order. This can be used in ruby to create an empty array called
default and then conditionally append to the array based on parameters passed
to the registry dynamic.
SfnRegistry.register(:chef_client)do|_config={}|metadata('AWS::CloudFormation::Init')do_camel_keys_set(:auto_disable)# take advantage of fact that cfn-init runs the `default`# configsets http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-init.htmlconfigSetsdo|sets|if_config.fetch(:platform)=='windows'set.default+=["windows_config"]elseset.default+=["default_config"]endend# Windows init configwindows_configdo# windows specific elements hereenddefault_configdo# generic elements hereend
Clearly some of the code has been left out. With a fully fleshed out dynamic
using this approach it can then be used in a template by passing :platform as
a configuration element:
registry!(:chef_client,:platform=>'windows')
The intent of this reads much clearer in the template code and everything needed
for the :chef_client registry is contained in the same block of code.
A simple example perhaps but this is exactly the type of thing that leads to
better code reuse and eases management of complex definitions over the lifespan
of any infrastructure as code workflow.
Sure, in a perfect world I could just fire up pry and do this kind of junk:
~/s/sparkle_formation❯❯❯pry[1]pry(main)>require'aws-sdk-core'=>true[2]pry(main)>ec2=::Aws::EC2::Client.new=>#<Aws::EC2::Client>[3]pry(main)>ec2.what?# I have no idea.
But we live in the real world of hostile amis that don’t love you. You’re on your own.
Find it yourself and edit that file.
(but at least we have the comfort of .tuesday? and .saturday? methods in ::Aws::EC2::Client, right?)
Now, let’s try that update again.
~/s/sparkleformation-workshops ❯❯❯ be sfn update training-vpc --file sparkleformation/example.rb
(lots of cloudformation output)
But this looks good, right?
2015-12-12 03:56:35 UTC ExampleEc2Instance CREATE_IN_PROGRESS Received SUCCESS signal with UniqueId i-1dc9f7ad
2015-12-12 03:56:38 UTC ExampleEc2Instance CREATE_COMPLETE
yea! maybe there’s an instance somewhere in mah clouds now? ok, I guess I’ll let this finish…
2015-12-12 04:00:06 UTC InternetGatewayAttachment DELETE_FAILED Network vpc-62950a06 has some mapped public address(es). Please unmap
those public address(es) before detaching the gateway.
2015-12-12 04:00:13 UTC InternetGateway DELETE_IN_PROGRESS
2015-12-12 04:01:03 UTC PublicUsEast1aSubnet DELETE_FAILED The subnet 'subnet-bb0be8e3' has dependencies and cannot be deleted.
2015-12-12 04:01:06 UTC Vpc DELETE_IN_PROGRESS
huh?! wait, what!!? why am I deleting this vpc and testing instances? I didn’t ask for that but guess I’ll wait and see what happens.
2015-12-12 04:04:24 UTC Vpc DELETE_FAILED The vpc 'vpc-62950a06' has dependencies and cannot be delete
Is that good or bad? I didn’t ask you to delete, but maybe you are trying to save me from myself.
I give up, it’s time for bed.
bundle exec destroy training-vpc
Footnote:
In all seriousness, none of this is ever easy and all software is terrible. The alternatives are far, far worse and you should just be using SparkleFormation
Update: 2015-12-14
The actual source of my issue here was not SparkleFormation. I was
trying to create a new stack but the update command is intended to
update an existing stack (Surprise!).
What worked perfectly as advertised was to actually use:
This automatically takes the stack outputs from the previously created
vpc, training-vpc and uses it as inputs for the new example-stack
created from the template example.rb
Recently I had a need to create a bootable sd card containing OpenBSD.
Since my day to day machine is a MacBook Air with no cdrom drive this
initially seemed difficult. It turned out to be much easier than I
expected thanks to
VirtualBox raw disk access.
Raw hard disk access can be dangerous, mainly because there’s a risk
you use the wrong device and blow away things you actually need, so be
careful and make sure you know you’re using the right physical device.
The disk has to be connected but not mounted and the Mac tries to
repeatedly automount the sdcard, if you encounter an error that may be
the case. The command to unmount is simply:
diskutil unmountDisk /dev/disk6
A raw disk vmdk needs to be created which can then be attached to the
VM to give it direct access to the disk. With Mavericks the disk is
automatically owned by root each time it’s mounted but in order to
create the raw disk vmdk your user needs to have full access to the
disk. In my case the device was /dev/disk6 so it just required
adjusting permissions:
sudo chown sme /dev/disk6
It’s not enough to just chown the resulting file, since it’s basically
just a pointer to the raw disk the user VirtualBox runs as needs full
device access.
Next to create the raw disk vmdk (the magic command):
This creates a tiny file, sdcard.vmdk, that can be attached to a
VirtualBox VM to give it full access to /dev/disk6. Next I created a
new OpenBSD VM, mounting the OpenBSD 5.4 iso and attached the raw disk
vmdk as the only storage. This is done by selecting ‘Choose existing
disk’ during the VM creation disk selection phase.
From here I was able to do a standard
OpenBSD install and the VM saw
the raw disk as the only available HD. After finishing the install I
was able to ‘option’ boot to select the sdcard as the startup disk and
run OpenBSD from the sdcard on my MacBook Air.
I genuinely dislike email, yet it’s a necessary part of working and
communicating these days. I suppose there is occasional value found in
the community of certain mailing lists, but generally the way email is
used today is simply more of a distraction or interruption.
A while back Steve Losh described how to setup Mutt
the way he likes.
It was an interesting read because he was detailed, technical and
clearly prefers very customizable tools. I agree with him, and
although I’ve long since abandonded Mutt I was motivated to describe
my version of powerful, customizable, terminal based email management.
This isn’t intended to say that what I use is inherently better than
the setup Steve described, it’s simply what works for me. Much of the
setup is similar. I used Mutt for many years before giving
Sup a try, and recently settling on the
configuration I describe here. Sup is still actively developed and a
great standalone mail client.
I do use Gmail and while the Gmail web interface has a great set of
keybindings for most everything, they aren’t customizable so it’s a
whole new set of bindings to learn. That’s not a terrible thing, but I
spend most of my day in a text editor where the bindings (and
subsequent muscle memory) I know are very different than those
available in the Gmail web interface.
Several times in the past I’ve searched for a fast, configurable way
to use emacs for accessing my email. For one reason or another the
existing options were
lacking in my opinion and didn’t fit my needs. However, a little over
2 years ago I stumbled across Notmuch,
which is billed as a: {“fast, global search and tag based email reader
to use within your text editor or in a terminal”}. This sounded like
exactly what I was looking for.
It’s essentially a minimal interface to a Xapian index of your mail so
powerful searching is easy, but
not limited.
I also primarily use OS X so the basic design is very similar to what
Steve mentions in his
Overview,
the difference is that I’ve removed Mutt from the picture and
substitute the notmuch
emacs integration.
Offlineimap and msmtp setup is essentially the same and I have two
accounts configured, one personal and one for work both of which are
google accounts.
Setup
If you use OS X, it’s easily available using homebrew:
brew install notmuch
And from emacs, assuming you have
melpa setup, notmuch is only an
M-x package-install away.
This gist has the settings I use in my emacs load-path to setup custom
identities for my personal and work email accounts and configure how
notmuch works.
There’s plenty of additional customization available depending on how
deep the rabbit hole you choose to go. This setup has worked well for
me up to this point and still allows me to use the gmail web interface
in a pinch.
I spend a decent amount of time in org-mode, using Emacs + iTerm2 on a
Macbook Air. I fixed a recent annoyance I encountered when using Emacs
org-mode to track time for a project.
Org-mode has a handy
time tracking feature
and as the manual points out both C-S-<up/down>S-M-<up/down> can
be used to call various functions for adjusting recorded time. In
particular I wanted to use org-timestamp-down to add some time for a
CLOCK item I had forgotten to start before a Skype call.
Obviously though, I hadn’t used S-M-down previously because when I
did instead of adjusting the time, I only got the raw escape code [1;10B.
I tried adjusting the settings in iTerm2 to no avail, so after a bit
of a meandering search down the emacs rabbit hole I found a solution
that worked well for me.