ash-spawn - spawn an agent VM
Creates or updates ash VM state, renders a virtle manifest, and starts virtle.
Plain spawn starts the VM as a background user systemd unit and returns. The VM keeps running until stopped with ash stop.
--attach starts the VM in the foreground and opens SSH. Without --keep, the VM stops when the attached session exits. Interactive --kernel-serial=console requires this foreground mode and cannot be combined with --keep.
--attach --keep starts the VM as a background unit, then attaches over SSH. The VM keeps running after SSH exits.
--ephemeral is only valid with --attach. It removes the VM state directory after the foreground attached session exits.
Only foreground spawns with --attach (without --keep) emit ssh.ready_socket in virtle.toml, so virtle waits for the guest's sshd to come up before proceeding (default 2 minutes). Background spawns omit ready_socket and do not gate launch on SSH. --ssh-ready-timeout DURATION (e.g. 90s or 2m) sets virtle's VIRTLE_SSH_READY_TIMEOUT for that launch.
Background spawns use systemd-run --user to start virtle as a transient unit named ash-NAME.service. ash stop NAME stops that unit.
After starting a background VM, ash prints the unit name and an ash logs -f NAME hint for following its logs.
A new VM is always evaluated before spawn writes ash-state.toml and virtle.toml and launches virtle. Both files live in the VM state directory. Virtle's own runtime state and control sockets use its nested virtle_state directory.
An existing named VM reuses its saved ash-state.toml and virtle.toml without Nix evaluation. Pass --eval to re-evaluate the selected NixOS configuration, refresh its generated state and manifest, and then launch it.
For a new VM, --flake is required and --eval is implied. For an existing VM spawned with --eval, omitting --flake, --override-input, or --space reuses the corresponding saved values; explicitly passing them replaces the saved selection.
--nix-store-strategy and --nix-store-image-size-mib override the [global.nix_store] defaults during evaluation. --persist-image-size-mib overrides [global.persist].image_size_mib. Explicit overrides are saved in ash-state.toml and reused by later evaluated spawns.
When the config contains an enabled [portal] section with global=false, spawn adds a dedicated agent-portal-host vsock process to the virtle manifest. Virtle starts and stops it with the VM, deriving a unique unprivileged Portal port from the allocated guest CID.
With portal.global=true, Ash requires transport=vsock and uses the configured vsock_cid and vsock_port without starting the host. The user must run agent-portal-host separately.
Both modes install /etc/profile.d/ash-agent-portal.sh for POSIX shells and ~/.local/share/nushell/vendor/autoload/ash-agent-portal.nu for Nushell through QEMU Guest Agent. The guest must already contain the Portal wrappers.
Use ash regenerate NAME to re-render virtle.toml later from saved ash-state.toml without launching the VM. Regeneration updates the manifest for a future launch; it does not reconfigure an already running VM.
--flake expects FLAKE#HOST and is required when creating a new VM. ash evaluates nixosConfigurations.HOST from that flake and uses it for the guest kernel, initrd, kernel params, system toplevel, a Nix store registration dump, host-side ssh, and host-side systemd-ssh-proxy paths.
Path-like flake references and override input references are saved in ash-state.toml as resolved absolute paths so ash regenerate NAME works from any current directory.
The selected NixOS configuration must expose normal NixOS system attributes such as config.system.build.kernel, config.system.build.initialRamdisk, config.system.build.toplevel, config.boot.kernelParams, pkgs.openssh, and config.systemd.package.
By default ash evaluates config.services.getty.autologinUser for the guest SSH user, then validates that users.users.USER exists. --user overrides the evaluated value.
The config defaults to XDG_CONFIG_HOME/ASH_NAME/config.toml or ~/.config/ASH_NAME/config.toml. ASH_NAME defaults to ash, and --config overrides every default. Each [spaces.NAME] table may define rw_mounts, ro_mounts, and files arrays, plus an extends array naming other spaces. Extended spaces are evaluated recursively before the extending space. Unknown spaces and inheritance cycles are errors.
The [global] table may set memory to the VM memory in MiB; the default is 4096. `ash spawn --memory` overrides it per VM, accepting an MiB count or an M/G suffix (e.g. 8G). Set kitty = true to use kitten ssh by default for spawn and attach sessions. network_bridge and qemu_bridge_helper configure the host bridge used for VM networking.
Each mount or file is HOST_PATH or HOST_PATH:GUEST_PATH. Host ~ resolves against the host user's home; guest ~ resolves against the guest SSH user's home. If GUEST_PATH is omitted, the original host path string is reused as the guest path. Absolute paths are also accepted. Missing host paths are skipped with a warning. Duplicate resources are removed after parsing and path expansion.
Regular files selected through a space are embedded as Virtle write_files entries with their source permission mode and overwrite enabled. Guest-home destinations are owned by the guest SSH user; other destinations keep Virtle's default ownership.
Spaces selected with --space stage their configured directories beneath the consolidated shares-ro or shares-rw tree, then bind those paths into the guest. New VMs have no selected spaces by default; existing named VMs reuse their saved selection.
--mount-cwd also adds the current host directory as a workspace/cwd mount for the guest.
The resolved kernel, initrd, NixOS toplevel, and closure-info output are protected by GC roots in the VM state directory. The roots remain while the VM state exists and are removed with that state, including after an ephemeral session.
Guest preparation is done by ash through virtle guest-exec. Ash imports the selected closure registration into the guest Nix database, then mounts workspace/space targets. Local-overlay guests skip the import because their readonly lower store already contains the registration. Foreground attached spawns use the generated SSH wrapper for the same preparation.
Runtime mounts are managed later with ash mount, ash umount, ash mount-space, and ash umount-space. Their resolved paths, modes, and manual or space ownership claims are saved in ash-state.toml and reconciled by later starts and resumes.
Set [global.nix_store].strategy to shared or image. shared is the default. image_size_mib configures the image strategy's capacity in MiB and defaults to 16384. [global.persist].image_size_mib defaults to 16384 and sizes the persist image. --nix-store-strategy, --nix-store-image-size-mib, and --persist-image-size-mib override these defaults for one VM and save the choice in its ash-state.toml.
The shared strategy stages the host /nix/store at shares/ro/system/nix-store. Both store strategies expose exactly two directory shares, shares-ro and shares-rw. --shares-ro-socket may select an existing daemon socket; --ro-store-socket remains a compatibility alias.
The image strategy creates nix-store.img as a private ext4 filesystem labeled nix-store and copies the selected closure plus its registration file into it through libext2fs. It does not expose the host /nix/store. The guest must mount the label at /nix with neededForBoot enabled; Ash initializes the Nix database through QGA after boot.
Ash keeps one read-only, closure-sized base image for each NixOS toplevel and registration output under $XDG_CACHE_HOME/$ASH_NAME/nix-store-images, falling back to ~/.cache/$ASH_NAME/nix-store-images. ASH_NAME defaults to ash. Writable capacity is not part of the cache identity, so VMs with different image_size_mib values reuse the same base.
For a new VM image, Ash clones the cached base with cp --reflink=auto --sparse=always and grows the writable clone with resize2fs to image_size_mib. Reflink-capable filesystems initially share blocks with the base; other filesystems receive a sparse copy. If image_size_mib is smaller than the closure-sized base, creation fails and reports the minimum size.
Cache entries are disposable. A missing, malformed, mismatched, or incorrectly sized cache entry is rebuilt from the selected closure. Existing VM images are updated in place rather than replaced because they may contain guest-added store paths.
Increasing image_size_mib grows a stopped VM's filesystem automatically. If the selected closure changes, Ash retains existing paths and imports only missing immutable store paths. A failed import can be retried, optionally after increasing the image size. Shrinking the filesystem still requires ash rebuild-db and discards guest-added store paths.
Every generated virtle.toml includes exactly two virtiofs directory mounts, shares-ro and shares-rw, plus the persistent ext4 image at persist.img. The image Nix store strategy additionally attaches nix-store.img, labeled nix-store. The manifest enables KVM, so the host is expected to provide /dev/kvm.
Host state uses this directory layout. Conditional entries appear only when their strategy or mount feature is used:
<state-base>/ e.g. ~/.local/state/ash
|-- <name>/ per-VM state
| |-- ash-state.toml
| |-- virtle.toml
| |-- persist.img
| |-- nix-store.img # image strategy
| `-- virtle_state/ # sockets and runtime files
`-- mounts/
`-- <name>/shares/ host mount staging
|-- ro/ -> shares-ro
| |-- system/
| | |-- nix-store/ # shared strategy
| | `-- guest-store-state/ # shared strategy
| `-- mounts/
| |-- spaces/<tag>/
| `-- hotmounts/<id>/
`-- rw/ -> shares-rw
|-- system/
| |-- guest-store-state/ # shared strategy
| |-- guest-store-upper/ # shared strategy
| `-- guest-store-work/ # shared strategy
`-- mounts/
|-- workspace/
|-- cwd/ # with --mount-cwd
|-- spaces/<tag>/
`-- hotmounts/<id>/The guest mounts shares-ro and shares-rw at /run/ash/shares/ro and /run/ash/shares/rw, then Ash bind-mounts individual staged children at their requested guest destinations.
The workspace lives at shares/rw/mounts/workspace and is bound to the guest workspace path. It is not capped like a disk image; usable size is bounded by host storage.
Runtime directories are staged below shares/{ro,rw}/mounts/hotmounts and can be mounted into a running guest without regenerating the manifest.
With the shared strategy, the shares mounts expose VM-state directories at /run/ash/shares/ro and /run/ash/shares/rw. Before launch, Ash loads the resolved NixOS closure registration into shares/ro/system/guest-store-state so a guest local-overlay store can use it as readonly lower-store metadata. The rw share provides guest-store-state, guest-store-upper, and guest-store-work for host-backed OverlayFS upper layers. Configure the local-overlay store's writable state to use guest-store-state so its metadata is reset with the upper layer. When subordinate host UID/GID ranges are available, Ash maps guest identities one-to-one so dedicated build users keep distinct ownership; otherwise it falls back to squashing identities to the host user.
When --mount-cwd is used, ash stages the current host directory at shares/rw/mounts/cwd and binds it to /mnt/cwd in the guest.
Ash creates or reuses id_ed25519 in the VM state directory and installs its public key through QGA before an attached SSH session. Foreground launches perform this in the generated SSH setup wrapper; background attaches perform the equivalent guest-exec installation before running ssh.
Pass --kitty to spawn to use kitten ssh instead of ssh for attached spawn sessions and save that choice in ash-state.toml for later regenerated launches. Set global.kitty = true in the Ash config to make Kitty the default without saving a per-VM override.
Pass --waypipe to wrap the attached SSH session with Waypipe. The guest must provide waypipe and xwayland-satellite in PATH, and the host must be running a Wayland compositor. --waypipe and --kitty may be combined.
Waypipe runs with --no-gpu because Ash does not expose a guest GPU. It forwards host compositor protocols and should only be used with trusted guest applications.
This requires the guest to have QEMU Guest Agent support and the guest user/home path expected by the generated manifest.
Ash passes ash.mdns-host=<dns-label> and ash.mdns-mac=<stable-mac> on the guest kernel command line. A compatible guest can use these values to publish <dns-label>.ash.local; the agent NixOS configuration uses Avahi. Invalid DNS-label characters are normalized with a digest suffix. The host must have .local mDNS resolution enabled, and multicast UDP 5353 must pass over the VM bridge.
The guest should run QEMU Guest Agent. For NixOS guests, enable services.qemuGuest.enable.
Attached flows wait for virtle SSH readiness. The guest must write the token SSH-READY to /dev/virtio-ports/virtle.ready after sshd is reachable.
ash-side SSH autoprovisioning assumes the guest SSH user's writable primary group is users. It creates or updates authorized_keys and applies OpenSSH-compatible ownership and permissions.
ash spawn --name work -f ../my-nix#agentash spawn --name work -f ../my-nix#agent --attach --keepGenerated from ash Cmdliner manpage metadata.