Aws

I opened issue #265 against my own extension after a load test told me something I didn’t want to hear. Ten workers, seven hours, and the sync rate had fallen by a factor of ten: throughput workers down from 0.10 iterations per second to 0.01, write-stress workers (one model, continuous writes) failing every single iteration once they passed number two hundred. The extension is a DynamoDB backend for swamp’s datastore layer, and the datastore layer treats storage as a path-addressable file tree: every model’s data lives at data/{model-type}/{model-id}/{data-name}/{version}/raw, with a metadata file alongside it. Sync follows a fixed sequence: acquire a lock, pull changed paths from remote, do the work locally, push the dirty paths back, commit an index update, release the lock. Whatever backend serves this sequence, S3, DynamoDB, or something else, sees whole-file transfers. It never sees an individual field write. ...

The drift-state extension makes zero AWS API calls. It reads data that other models already produced, diffs it against a stored baseline, and writes the result as typed versioned state. The “drift detector” is a composition function over existing observations. This matters because drift detection is usually sold as a feature of a specific tool. CloudFormation has drift detection. Terraform has plan. AWS Config has compliance evaluations. Each one instruments its own slice and reports on what it manages. Nothing composes across them. ...

I wrote recently about the difference between easy and simple.1 That post was philosophical. This one is not. This is what it looks like in practice. I have been building an AWS operations toolkit using Swamp. The toolkit investigates outages. It runs daily health checks. It composes nine separate extensions into two workflows that gather data from CloudWatch, X-Ray, EC2, Lambda, and load balancers, then produce an actionable report. This is the machine that builds the machine. Not the code. The system that produces the code and makes sense of the output. ...

Growing into SparkleFormation In my continued usage with SparkleFormation I’m really growing to appreciate the convenience of having ruby available when composing templates. Things that would be challenging in a simplistic serialization format or lead to unmanageable duplication become easily solvable thanks to the powerful combination of an actual programming language, which json and yml are not, and the simplicity of the SparkleFormation DSL. At it’s simplest you can mirror the structure of any cloudformation json (and finally have comments inline!) but you quickly discover more advanced use cases. ...

A very brief whirlwind tour of sparkles or: How I learned to stop writing terrible serialization formats directly and love the dsl An imaginary story of the beginner exploring some infrastructure tooling. Getting started Set env variables. Maybe create an init.sh while practicing. export AWS_ACCESS_KEY_ID="your key" export AWS_SECRET_ACCESS_KEY="your sekret" export AWS_REGION="us-east-1" # because YOLO export NESTED_BUCKET="s3://mah_bukkit" load that stuff into my current shell source ./init.sh Create a config .sfn file aready exists, thanks @luckymike. no need to create one. ...