Datastore

I opened issue #265 against my own extension after a load test told me something I didn’t want to hear. Ten workers, seven hours, and the sync rate had fallen by a factor of ten: throughput workers down from 0.10 iterations per second to 0.01, write-stress workers (one model, continuous writes) failing every single iteration once they passed number two hundred. The extension is a DynamoDB backend for swamp’s datastore layer, and the datastore layer treats storage as a path-addressable file tree: every model’s data lives at data/{model-type}/{model-id}/{data-name}/{version}/raw, with a metadata file alongside it. Sync follows a fixed sequence: acquire a lock, pull changed paths from remote, do the work locally, push the dirty paths back, commit an index update, release the lock. Whatever backend serves this sequence, S3, DynamoDB, or something else, sees whole-file transfers. It never sees an individual field write. ...

By default, swamp stores everything in local SQLite. This works until you close your laptop, sit down at your desktop, and discover that your agent has no memory of what happened an hour ago. The models are there (they live in YAML), but the runtime data, the execution history, the versioned outputs from every method run, all of it is trapped on whichever machine produced it. If you work on one machine, this is fine. If you work on two or three, it is a problem. Your agent starts cold every time you switch. The voice profile you refined yesterday on your laptop does not exist on your desktop. The health check workflow that ran at 03:00 on your VM has results you cannot see from anywhere else. You are running the same models against the same code, but the data layer is fragmented across physical locations. ...