Deep Review: 20260412-171333-pr-295

		k8s:           k8s,
		cancel:        watchCancel,
		done:          make(chan struct{}),
		reconcileChan: reconcileChan,
	}

	// Capture a Docker-relative timestamp before fullSync begins. The
	// run() goroutine passes this to the event stream as the Since
	// filter, so any mutation that happens while fullSync is snapshotting
	// state is replayed once the stream opens. Without this, the window
	// between the List-based snapshot and the event subscription is a
	// blind spot during VM startup.
	//
	// Use the Docker daemon's own clock (Info.SystemTime) rather than
	// time.Now() on the host. The daemon runs inside the Lima VM and
	// evaluates Since against its own clock; any host/guest skew would
	// silently drop events inside the skew window. Fall back to a

The comment claims "Docker-relative timestamp" but time.Now() is the host clock. The Docker server (inside the Lima VM) interprets the since query parameter against its own clock. If the host clock is ahead of the guest clock by N seconds, events generated in the window [now - N, start of event stream] are filtered out by the server and never reach the watcher — silently lost. With Lima/Vz the skew is normally millisecond-scale so practical risk is low, but the failure mode is silent rather than detectable.

Two cheap fixes:

// Option A: bias the timestamp into the past so any realistic skew is covered.
// fullSync is idempotent, so duplicate replay is safe.
since := strconv.FormatInt(time.Now().Add(-2*time.Minute).Unix(), 10)

// Option B: query the Docker server's view of "now" before fullSync.
if info, err := cli.Info(ctx); err == nil && info.SystemTime != "" {
    if t, perr := time.Parse(time.RFC3339Nano, info.SystemTime); perr == nil {
        since = strconv.FormatInt(t.Unix(), 10)
    }
}

At minimum, fix the comment to say "host wall clock" and acknowledge the clock-skew assumption.

	// always set to "unknown" on creation — meaning the engine mirrors
	// Docker container state without expressing intent. The user can later
	// set it to "running" or "created" to control the Docker container;
	// the reconciler ignores "unknown".
	//
	// Deliberately omit ForceOwnership on the create apply: if a user
	// patch to spec.state landed in the window between the Get above
	// and this Apply, ForceOwnership would silently clobber it. Without
	// the flag, the Apply succeeds on a fresh resource (no other owner
	// for spec.state yet) and fails loudly on a concurrent conflict
	// (the Get-NotFound observation was racy).
	var existing containersv1alpha1.Container
	err := w.k8s.Get(ctx, client.ObjectKey{Name: inspect.ID, Namespace: apiNamespace}, &existing)
	if apierrors.IsNotFound(err) {
		applyConfig := containersv1alpha1apply.Container(inspect.ID, apiNamespace).
			WithFinalizers(mirrorFinalizer).
			WithSpec(containersv1alpha1apply.ContainerSpec().WithState(containersv1alpha1.ContainerStatusUnknown))
		if err := w.k8s.Apply(ctx, applyConfig, client.FieldOwner(controllerName)); err != nil {
			return fmt.Errorf("failed to create container %s: %w", inspect.ID, err)
		}
	} else if err != nil {
		return fmt.Errorf("failed to get container %s: %w", inspect.ID, err)

The mirror finalizer is set only on the IsNotFound (creation) branch. On any subsequent sync of the same Container, the code falls through to the status-only Apply at sync_containers.go:176 without re-asserting the finalizer. If the user removes the finalizer via kubectl edit, the next reconcile won't restore it, and a subsequent user delete bypasses the engine's Docker-side cleanup entirely — the K8s mirror is GC'd instantly while the Docker container survives.

	if t, err := time.Parse(time.RFC3339Nano, inspect.State.FinishedAt); err == nil && !t.IsZero() {
		statusApply.WithFinishedAt(metav1.NewTime(t))
	}

	// Port bindings.
	var applyPorts []*containersv1alpha1apply.ContainerPortApplyConfiguration
	if inspect.NetworkSettings != nil {
		for portName, ports := range inspect.NetworkSettings.Ports {
			var bindings []*containersv1alpha1apply.ContainerPortBindingApplyConfiguration
			for _, port := range ports {
				bindings = append(bindings, containersv1alpha1apply.ContainerPortBinding().

sync_images.go:112-133 and sync_volumes.go:128-130 reassert the finalizer on every sync because their applies are unconditional. The Container path is the only outlier because of its special "spec-only-on-create" gate.

	var errs []error

	if len(inspect.RepoTags) > 0 {
		for _, tag := range inspect.RepoTags {
			// Deterministic name from image ID + tag hash (same as mock controller).
			name := fmt.Sprintf("%s-%x",
				sanitizeKubernetesObjectName(inspect.ID),
				sha256.Sum256([]byte(tag)))
			names = append(names, name)

			if err := w.applyImage(ctx,
				containersv1alpha1apply.Image(name, apiNamespace).
					WithFinalizers(mirrorFinalizer),
				imageStatusFromInspect(inspect).
					WithRepoTag(tag).
					WithNamespace(containerNamespace),
			); err != nil {
				errs = append(errs, err)
			}
		}
	} else {
		// Dangling image (no tags). status.namespace is a required
		// field on the CRD, so set it here too even though it carries
		// no additional information beyond what the tagged branch sets.
		name := sanitizeKubernetesObjectName(inspect.ID)
		names = append(names, name)
		if err := w.applyImage(ctx,
			containersv1alpha1apply.Image(name, apiNamespace).
				WithFinalizers(mirrorFinalizer),
			imageStatusFromInspect(inspect).
				WithNamespace(containerNamespace),
		); err != nil {

}

// applyVolume creates or updates a `Volume` mirror from a Docker volume.
func (w *dockerWatcher) applyVolume(ctx context.Context, vol mobyvolume.Volume) error {
	mirrorName := volumeMirrorName(vol.Name)

	applyConfig := containersv1alpha1apply.Volume(mirrorName, apiNamespace).
		WithFinalizers(mirrorFinalizer)

	err := w.k8s.Apply(ctx, applyConfig,
		client.ForceOwnership, client.FieldOwner(controllerName))
	if err != nil {
		return fmt.Errorf("failed to apply volume %s: %w", mirrorName, err)

Fix: split finalizer reassertion into a separate Apply that always runs:

finalizerOnly := containersv1alpha1apply.Container(inspect.ID, apiNamespace).
    WithFinalizers(mirrorFinalizer)
if err := w.k8s.Apply(ctx, finalizerOnly,
    client.ForceOwnership, client.FieldOwner(controllerName)); err != nil {
    return fmt.Errorf("failed to assert container finalizer %s: %w", inspect.ID, err)
}

With SSA, applying only WithFinalizers preserves the user-owned spec.state field.

	if err != nil {
		return err
	}
	r.watcher = w
	return nil
}

// stopWatcher stops the Docker watcher goroutine and waits for it to finish.
func (r *EngineReconciler) stopWatcher() {
	r.watcherMu.Lock()
	w := r.watcher
	r.watcher = nil
	r.watcherMu.Unlock()

	if w != nil {
		w.stop()
	}
}

// setEngineCondition updates the ContainerEngineReady condition on the
// App resource. The App controller also writes App.Status.Conditions
// (to mirror LimaVM conditions) and controller-runtime does not
// serialize reconciles across controllers, so a naive Update can race

The AppReconciler at pkg/controllers/app/app/controllers/app_controller.go:247 already calls r.Status().Update(ctx, &app) to mirror LimaVM conditions onto App.Status.Conditions. The EngineReconciler now also calls Status().Update on the same object to set ContainerEngineReady. Both reconcilers run in the same manager and are not serialized across controllers (controller-runtime serializes within a controller, not across them). When both have a stale local copy and both issue a Status().Update, the loser gets a 409 conflict.

			ObservedGeneration: app.Generation,
			LastTransitionTime: cond.LastTransitionTime,
		}) || statusChanged
	}
	if statusChanged {
		if err := r.Status().Update(ctx, &app); err != nil {
			log.Error(err, "Unable to update App status")
			return ctrl.Result{}, err
		}
	}

In practice conflicts will be infrequent because meta.SetStatusCondition short-circuits no-op updates and the App reconciler reconciles less often. But the race is real.

Fix (either of two patterns already used elsewhere in this file):

• Use server-side apply with client.Apply + a distinct FieldOwner for the engine condition — field-level merge avoids resource-version conflict entirely.
• Wrap the Status().Update in retry.RetryOnConflict plus a re-Get, mirroring removeMirrorResource at docker_watcher.go:271-283 and deleteAllOfType at engine_reconciler.go:262-274.

		return w.removeImagesByID(ctx, msg.Actor.ID)

	default:
		return nil
	}
}

// handleVolumeEvent processes volume events.
func (w *dockerWatcher) handleVolumeEvent(ctx context.Context, msg events.Message) error {
	log := logf.FromContext(ctx).WithName("docker-watcher")

	switch msg.Action {
	case events.ActionCreate:
		log.V(1).Info("Volume created", "name", msg.Actor.ID)
		return w.syncVolume(ctx, msg.Actor.ID)

	case events.ActionDestroy:
		log.V(1).Info("Volume destroyed", "name", msg.Actor.ID)
		return w.removeMirrorResource(ctx, &containersv1alpha1.Volume{},
			volumeMirrorName(msg.Actor.ID))

	default:
		return nil

		errs = append(errs, fmt.Errorf("failed to delete Volumes: %w", err))
	}
	if err := r.deleteAllOfType(ctx, &containersv1alpha1.ImageList{}); err != nil {
		errs = append(errs, fmt.Errorf("failed to delete Images: %w", err))
	}
	if err := r.deleteAllOfType(ctx, &containersv1alpha1.ContainerNamespaceList{}); err != nil {
		errs = append(errs, fmt.Errorf("failed to delete ContainerNamespaces: %w", err))
	}
	return errors.Join(errs...)
}

// deleteAllOfType lists resources, strips finalizers, and deletes them.
// Finalizer removal uses retry.RetryOnConflict so a stale cache or a
// concurrent Update does not require the caller to requeue. Per-item
// errors are collected so one stuck object does not block the rest of
// the list.
func (r *EngineReconciler) deleteAllOfType(ctx context.Context, list client.ObjectList) error {
	if err := r.List(ctx, list, client.InNamespace(apiNamespace)); err != nil {
		return err
	}

	items, err := meta.ExtractList(list)
	if err != nil {

// neither — or the watcher will open a different socket from the one
// Lima is port-forwarding to, and mirroring will silently diverge per
// instance. Until then, multi-instance BATS runs cannot share the
// same host and the engine.bats test has a parallel TODO at line
// 12-14.
var DockerSocket = sync.OnceValue(func() string {
	home, err := os.UserHomeDir()
	if err != nil {
		panic(fmt.Errorf("could not get home directory: %w", err))
	}
	dir := filepath.Join(home, ".rd2")
	_ = os.MkdirAll(dir, 0o700)
	return filepath.Join(dir, "docker.sock")
})

Two concerns:

1. Hardcoded directory: On any instance where RDD_INSTANCE != "2" (e.g., bats-app-controller, a developer's dev instance), this creates ~/.rd2/ instead of instance.ShortDir(). The TODO at instance.go:124-133 already acknowledges the hardcode, but the side effect of creating the directory early means every caller that reads this value (even from code paths that never use the socket) litters the user's home. Lima itself creates the directory when port-forwarding the socket, so the MkdirAll here is likely redundant.
2. Swallowed error: _ = os.MkdirAll(...) ignores any failure (permission, ENOSPC). If the directory cannot be created, the function still returns a path to a non-existent directory, and the later dial fails with an error unrelated to the real cause.

var LimaHome = sync.OnceValue(func() string {
	return filepath.Join(ShortDir(), "lima")
})

// DockerSocket returns the path to the Docker socket for this instance
// (e.g., ~/.rd2/docker.sock). This is the host-side socket that Lima
// port-forwards from the guest's /var/run/docker.sock.
//
// TODO: use ShortDir() once the Lima template derives the socket path
// from the instance suffix instead of hardcoding ".rd2". The two
// hardcodes (here and in pkg/controllers/app/app/lima-template.yaml)
// must move together — either both parameterized on RDD_INSTANCE or
// neither — or the watcher will open a different socket from the one
// Lima is port-forwarding to, and mirroring will silently diverge per
// instance. Until then, multi-instance BATS runs cannot share the
// same host and the engine.bats test has a parallel TODO at line
// 12-14.
var DockerSocket = sync.OnceValue(func() string {
	home, err := os.UserHomeDir()
	if err != nil {

Fix (minimal): drop the MkdirAll entirely and let Lima own the directory.

-    dir := filepath.Join(home, ".rd2")
-    _ = os.MkdirAll(dir, 0o700)
-    return filepath.Join(dir, "docker.sock")
+    return filepath.Join(home, ".rd2", "docker.sock")

If Lima does not pre-create it, propagate the error instead of ignoring it.

		if watcherRunning {
			log.Info("Stopping Docker watcher",
				"running", running, "engine", app.Spec.ContainerEngine.Name)
			r.stopWatcher()
		}
		terminalReason := "Stopped"
		terminalStatus := metav1.ConditionFalse
		terminalMessage := "Container engine stopped"
		if running && !engineIsDocker {
			// NotApplicable is reported as Status=True so that
			// `rdd set running=true containerEngine.name=containerd`
			// stops waiting on ContainerEngineReady — even though the
			// engine controller is not mirroring anything in this
			// backend. UI consumers that expect Container/Image/Volume
			// resources should gate on the Reason as well, not on
			// Status alone. The condition will be renamed when
			// containerd mirroring lands.
			terminalReason = "NotApplicable"

Setting ContainerEngineReady=True/NotApplicable for the containerd backend makes rdd set running=true complete immediately (matching the BATS test at engine.bats:365-389), but it tells any other consumer of ContainerEngineReady=True ("the engine is ready") something misleading: there is no mirroring at all, and any UI relying on Container/Image/Volume resources will see nothing. Status=True with Reason=NotApplicable conflicts with the condition name.

    refute_output
}

# --- containerd backend ---

@test "containerd backend reports ContainerEngineReady=NotApplicable and skips mirroring" {
    # Stop first so there is no stale True/Connected from moby to
    # satisfy the CLI wait below before the engine reconciler has run.
    rdd set running=false

    # Start with containerd. rdd set waits for ContainerEngineReady=True,
    # which the engine reconciler satisfies immediately with reason
    # NotApplicable because engine mirroring only supports the moby
    # backend.
    rdd set containerEngine.name=containerd running=true

    run -0 rdd ctl get app app \
        -o jsonpath='{.status.conditions[?(@.type=="ContainerEngineReady")].reason}'
    assert_output "NotApplicable"

    # No mirror resources should exist in containerd mode.
    run -0 rdd ctl get containers --namespace="${NAMESPACE}" --output=name
    refute_output
    run -0 rdd ctl get images --namespace="${NAMESPACE}" --output=name
    refute_output
    run -0 rdd ctl get volumes --namespace="${NAMESPACE}" --output=name
    refute_output
    run -0 rdd ctl get containernamespaces --namespace="${NAMESPACE}" --output=name
    refute_output
}

Two cleaner alternatives:

• Rename the condition to ContainerEngineMirrored so True/NotApplicable doesn't conflict with the type name.
• Use Status=Unknown/NotApplicable — still allows rdd set to stop waiting but doesn't claim active readiness.

Option two is cheaper and keeps the condition name descriptive; the CLI wait at cmd/rdd/set.go:464-466 would need to accept Unknown as a terminal state.

	ctx, cancel := context.WithTimeout(ctx, timeout)
	defer cancel()

	if runningVal == "true" {
		logrus.Info("Waiting for container engine to be ready")
		return watchCondition(ctx, config, func(obj *unstructured.Unstructured) bool {
			return conditionStatus(obj, "ContainerEngineReady") == metav1.ConditionTrue
		})
	}
	logrus.Info("Waiting for the VM to stop")
	return watchCondition(ctx, config, func(obj *unstructured.Unstructured) bool {
		// Running=False means the VM has stopped. Running absent means
		// the VM was never started (nothing to wait for).

		current := meta.FindStatusCondition(app.Status.Conditions, conditionContainerEngineReady)
		alreadyClean := !watcherDied && current != nil && current.Reason == terminalReason && current.Status == terminalStatus
		if !alreadyClean {
			if err := r.cleanupMirrorResources(ctx); err != nil {
				log.Error(err, "Failed to clean up mirror resources")
				return ctrl.Result{}, err
			}
		}
		return ctrl.Result{}, r.setEngineCondition(ctx, &app, terminalStatus, terminalReason, terminalMessage)
	}

	if !watcherRunning {
		log.Info("App is running, starting Docker watcher")
		if err := r.startWatcher(ctx); err != nil {
			log.Error(err, "Failed to start Docker watcher")

The _ = discards any error from setEngineCondition. If the status update fails (e.g., a 409 conflict against a concurrent App reconciler update — see S3), the App keeps reporting its stale value (often True/Connected from the previous startup) until the next reconcile. The reconciler requeues on the original startWatcher error so the state eventually converges, but the API-server-visible condition is wrong in the interim.

     log.Error(err, "Failed to start Docker watcher")
-    _ = r.setEngineCondition(ctx, &app, metav1.ConditionFalse, "ConnectFailed", err.Error())
+    if condErr := r.setEngineCondition(ctx, &app, metav1.ConditionFalse, "ConnectFailed", err.Error()); condErr != nil {
+        log.Error(condErr, "Failed to update ContainerEngineReady=ConnectFailed")
+    }
     return ctrl.Result{}, err

		if removeFinalizer(c, mirrorFinalizer) {
			// NotFound is benign: a concurrent Docker destroy event may
			// already have stripped the finalizer and deleted the
			// mirror between our List and Update.
			if err := client.IgnoreNotFound(r.Update(ctx, c)); err != nil {
				errs = append(errs, fmt.Errorf("failed to remove finalizer from Container %s: %w", c.Name, err))
			}
		}
	}
	return errors.Join(errs...)
}

func (r *EngineReconciler) processImageFinalizers(ctx context.Context, w *dockerWatcher) error {
	var images containersv1alpha1.ImageList
	if err := r.List(ctx, &images, client.InNamespace(apiNamespace)); err != nil {
		return err
	}
	var errs []error
	for i := range images.Items {
		img := &images.Items[i]
		if img.DeletionTimestamp == nil || !hasFinalizer(img, mirrorFinalizer) {
			continue
		}
		if err := w.deleteImage(ctx, img); err != nil {
			errs = append(errs, fmt.Errorf("failed to delete image %s from Docker: %w", img.Name, err))
			continue
		}
		if removeFinalizer(img, mirrorFinalizer) {
			if err := client.IgnoreNotFound(r.Update(ctx, img)); err != nil {
				errs = append(errs, fmt.Errorf("failed to remove finalizer from Image %s: %w", img.Name, err))
			}
		}
	}

If a Volume mirror is created through the K8s API by something other than the engine controller — or in a startup race window where the mirror has the finalizer but the engine has not yet populated its status — Status.Name will be empty. deleteVolume then calls Docker's VolumeRemove(""), which errors out, and the finalizer is never stripped. The Volume mirror is permanently stuck in Terminating.

Fix: short-circuit the finalizer strip when there is no Docker name to forward to.

 if v.DeletionTimestamp == nil || !hasFinalizer(v, mirrorFinalizer) {
     continue
 }
+if v.Status.Name == "" {
+    if removeFinalizer(v, mirrorFinalizer) {
+        if err := client.IgnoreNotFound(r.Update(ctx, v)); err != nil {
+            errs = append(errs, fmt.Errorf("failed to remove finalizer from Volume %s: %w", v.Name, err))
+        }
+    }
+    continue
+}
 if err := w.deleteVolume(ctx, v.Status.Name); err != nil {

	reconcileChan chan<- event.GenericEvent
}

// newDockerWatcher creates a Docker client, performs a full sync, and starts
// the event stream watcher goroutine.
func newDockerWatcher(ctx context.Context, k8s client.Client, reconcileChan chan<- event.GenericEvent) (*dockerWatcher, error) {
	socketPath := instance.DockerSocket()
	cli, err := dockerclient.New(
		dockerclient.WithHost("unix://" + socketPath),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to create Docker client: %w", err)
	}

	// Verify the connection by pinging Docker.
	pingCtx, pingCancel := context.WithTimeout(ctx, 5*time.Second)
	defer pingCancel()

Without dockerclient.WithAPIVersionNegotiation(), the client uses its compiled-in default API version. If the Docker engine inside the Lima VM is newer or older than the client's default, calls will fail with cryptic version-mismatch errors. RDD controls the Lima image today so this is mostly theoretical, but a user who sshes into the VM and runs zypper up, or a future Lima image bump, could trigger it unexpectedly.

 cli, err := dockerclient.New(
     dockerclient.WithHost("unix://" + socketPath),
+    dockerclient.WithAPIVersionNegotiation(),
 )

Verify the exact option name in moby/moby/client@v0.4.0 — moby has historically used WithAPIVersionNegotiation and NegotiateAPIVersion interchangeably across versions.

# Check whether a cmdline belongs to the current RDD_INSTANCE. The
# instance name appears in any path derived from it (~/.rd<instance>/,
# rancher-desktop-<instance>/, ...) and in sh wrapper argv as
# `RDD_INSTANCE=<instance>`. Anchor on those exact markers rather than
# matching any substring, so a short instance like "bats" does not
# match every "bats-*" sibling target's processes and trigger cross-
# target SIGKILL cleanup.
matches_our_instance() {
    case "$1" in
        *"RDD_INSTANCE=${instance}"*) return 0 ;;
        *"/.rd${instance}/"*) return 0 ;;
        *"rancher-desktop-${instance}/"*) return 0 ;;
        *"rancher-desktop-${instance} "*) return 0 ;;
    esac

The helper is used by our_leaked_pids / sigquit_our_go_leaks / sigkill_our_leaks at lines 306-365 to scope SIGQUIT and SIGKILL cleanup to the current RDD_INSTANCE. A bare substring match is broader than the comment at bats-with-timeout.sh:68-70 promises: RDD_INSTANCE=bats would match every bats-* instance name in bats/Makefile:53-83, so the cleanup path can target sibling control planes rather than its own. The destructive scoping is the whole point of this helper — subtle over-matching here risks killing a parallel run's daemons.

        ps -o command= -p "$1" 2>/dev/null || true
    fi
}

# Check whether a cmdline belongs to the current RDD_INSTANCE. The
# instance name appears in any path derived from it (~/.rd<instance>/,
# rancher-desktop-<instance>/, ...) and in sh wrapper argv as
# `RDD_INSTANCE=<instance>`. Anchor on those exact markers rather than
# matching any substring, so a short instance like "bats" does not
# match every "bats-*" sibling target's processes and trigger cross-
# target SIGKILL cleanup.
matches_our_instance() {
    case "$1" in

        echo
        dump_api_state

        echo
        echo "=== End of support bundle (${context}) ==="
    } >>"${bundle_file}" 2>&1
}

# Enumerate leaked process PIDs belonging to the current RDD_INSTANCE.
# `go_only=1` restricts to Go binaries (for SIGQUIT goroutine dumps);
# otherwise includes qemu/limactl drivers too.
our_leaked_pids() {
    local go_only=$1
    local pid comm
    while read -r pid comm; do
        if [ "$go_only" = 1 ]; then
            case "$comm" in
                rdd|rdd.exe|*-controller|*-controller.exe|lima-guestagent|hostagent) ;;
                *) continue ;;
            esac
        else
            case "$comm" in
                rdd|rdd.exe|*-controller|*-controller.exe|lima-guestagent|hostagent|qemu*|limactl*) ;;
                *) continue ;;
            esac
        fi
        if matches_our_instance "$(cmdline_of "$pid")"; then
            echo "$pid"
        fi
    done < <(ps -axo pid=,ucomm= 2>/dev/null || ps -eo pid=,ucomm= 2>/dev/null || true)
}

# SIGQUIT Go processes belonging to our RDD_INSTANCE so their goroutine
# stacks land in the preserved stderr logs. No-op if nothing is leaked.
sigquit_our_go_leaks() {
    local pid
    local leaked
    leaked=$(our_leaked_pids 1)
    if [ -z "$leaked" ]; then
        return
    fi
    {
        echo
        echo "=== SIGQUIT -> leaked Go processes (RDD_INSTANCE=${instance}) ==="
        for pid in $leaked; do
            echo "SIGQUIT pid=${pid} pgid=$(pgid_of "$pid") cmdline=$(cmdline_of "$pid")"
            kill -QUIT "$pid" 2>/dev/null || true
        done
    } >>"${bundle_file}" 2>&1
    # Let Go runtimes flush goroutine dumps to stderr before subsequent
    # steps terminate them.
    sleep 1
}

# SIGKILL any process still running under our RDD_INSTANCE so the CI
# runner is clean for later steps. No-op if nothing is leaked.
sigkill_our_leaks() {
    local pid
    local leaked
    leaked=$(our_leaked_pids 0)
    if [ -z "$leaked" ]; then
        return
    fi
    {
        echo
        echo "=== SIGKILL -> leaked processes (RDD_INSTANCE=${instance}) ==="
        for pid in $leaked; do
            echo "SIGKILL pid=${pid} pgid=$(pgid_of "$pid") cmdline=$(cmdline_of "$pid")"
            kill -KILL "$pid" 2>/dev/null || true
        done

Fix: anchor the match on actual instance markers that appear in paths and environment assignments.

 matches_our_instance() {
     case "$1" in
-        *"${instance}"*) return 0 ;;
+        *"RDD_INSTANCE=${instance}"*|*"/.rd${instance}/"*|*"rancher-desktop-${instance}"*) return 0 ;;
     esac
     return 1
 }

}

// cleanupMirrorResources removes every mirror resource the engine
// controller owns. Errors are collected across all four kinds with
// errors.Join so one stuck resource type does not block the rest — the
// remaining types are still swept and the caller requeues on the
// combined error, retrying only what actually failed.
func (r *EngineReconciler) cleanupMirrorResources(ctx context.Context) error {
	log := logf.FromContext(ctx)
	log.Info("Cleaning up all mirror resources")

	var errs []error
	if err := r.deleteAllOfType(ctx, &containersv1alpha1.ContainerList{}); err != nil {
		errs = append(errs, fmt.Errorf("failed to delete Containers: %w", err))
	}
	if err := r.deleteAllOfType(ctx, &containersv1alpha1.VolumeList{}); err != nil {
		errs = append(errs, fmt.Errorf("failed to delete Volumes: %w", err))
	}
	if err := r.deleteAllOfType(ctx, &containersv1alpha1.ImageList{}); err != nil {
		errs = append(errs, fmt.Errorf("failed to delete Images: %w", err))
	}
	if err := r.deleteAllOfType(ctx, &containersv1alpha1.ContainerNamespaceList{}); err != nil {
		errs = append(errs, fmt.Errorf("failed to delete ContainerNamespaces: %w", err))
	}
	return errors.Join(errs...)
}

// deleteAllOfType lists resources, strips finalizers, and deletes them.
// Finalizer removal uses retry.RetryOnConflict so a stale cache or a
// concurrent Update does not require the caller to requeue. Per-item
// errors are collected so one stuck object does not block the rest of
// the list.
func (r *EngineReconciler) deleteAllOfType(ctx context.Context, list client.ObjectList) error {
	if err := r.List(ctx, list, client.InNamespace(apiNamespace)); err != nil {
		return err
	}

	items, err := meta.ExtractList(list)
	if err != nil {
		return err
	}

	var errs []error
	for _, item := range items {
		obj := item.(client.Object)
		key := client.ObjectKeyFromObject(obj)
		// meta.ExtractList strips the TypeMeta from each item (the API
		// server only populates GVK on the top-level list), so
		// obj.GetObjectKind() is empty here. Format the concrete Go
		// type name with %T instead.
		retryErr := retry.RetryOnConflict(retry.DefaultRetry, func() error {

Items extracted from a List response via meta.ExtractList have empty TypeMeta — the K8s API server only populates GVK on the top-level list object, not its items. Error messages log as failed to remove finalizer from /volume-name instead of Volume/volume-name, making operator debugging harder. Cosmetic (doesn't affect behavior) — downgraded to suggestion.

-        errs = append(errs, fmt.Errorf("failed to remove finalizer from %s/%s: %w",
-            kind, obj.GetName(), retryErr))
+        errs = append(errs, fmt.Errorf("failed to remove finalizer from %T %s: %w",
+            obj, obj.GetName(), retryErr))

Each syncAll* collects per-item errors with errors.Join, fullSync joins across all four sync types, and newDockerWatcher fails the whole startup on any non-nil error. A single permanently-broken inspect (engine-side corruption, rare race) prevents every other healthy resource from being mirrored, and the ContainerEngineReady=ConnectFailed condition stays sticky until the problem resource disappears.

Consider logging individual inspect errors and continuing the sync rather than failing the entire startup on any per-item error. The stale-prune step already runs regardless and will eventually clean up mirrors for items that stay broken.

# bats-with-timeout.sh captures a support bundle (process tree, Go
# goroutine dumps via SIGQUIT, open fds) before killing the target so
# we can diagnose CI hangs post-mortem. Use `=` so $@ resolves to the
# current target at recipe time.
#
# NOTE: setting BATS_TIMEOUT=0 disables the wrapper entirely, which
# also disables the support-bundle capture and leaked-process cleanup.
# If you are debugging a hang locally and need both the long runtime
# and the diagnostic capture, set BATS_TIMEOUT to a large value
# (e.g. 86400) instead of 0.
BATS_TIMEOUT ?= 1800

When a developer runs make bats-app BATS_TIMEOUT=0 to disable the timeout (e.g., for debugging a hang), the wrapper is not invoked at all — no support-bundle capture, no leaked-process cleanup. Document this in the Makefile comment, or split the wrapper so timeout=0 still runs it with the timeout disabled.