agent-desktop-ffi

Direct C-ABI access to every PlatformAdapter operation. Build the cdylib with the workspace's release-ffi profile:

cargo build --profile release-ffi -p agent-desktop-ffi

The output is target/release-ffi/libagentdesktopffi.dylib (.so on Linux, .dll on Windows) plus a committed C header at crates/ffi/include/agent_desktop.h.

A Python ctypes smoke harness lives at tests/ffi-python/smoke.py and serves as a worked end-to-end example covering the ABI handshake, struct size validation, ad_version, and the snapshot pipeline leg. See tests/ffi-python/README.md for usage.

Four reference topics, loaded as needed:

• [ownership.md](references/ownership.md) — who allocates / who frees,

for every *mut T the FFI hands back to the caller.

• [error-handling.md](references/error-handling.md) — errno-style

last-error contract, enum validation, panic boundary.

• [threading.md](references/threading.md) — host-thread contract,

cross-process mutation serialization, AXIsProcessTrusted inheritance, and adapter-bound native handles.

• [build-and-link.md](references/build-and-link.md) — ABI handshake,

struct size validation, minimal C and Python examples, observe-act workflow, and prebuilt archive locations.

Observe-act workflow (canonical path)

ad_init(AD_ABI_VERSION_MAJOR)                    // verify header ↔ dylib match
adapter = ad_adapter_create_with_session("s1")   // or ad_adapter_create()
rc = ad_snapshot(adapter, "Finder", 0, 10, false, false, &json_out)
// parse json_out: locate snapshot-qualified refs in data.tree
ad_free_string(json_out)
// build action:
AdAction act = {0}; act.kind = AD_ACTION_KIND_CLICK;
rc = ad_execute_by_ref(adapter, "@s8f3k2p9:e5", NULL, &act, 0, &result_out)
ad_free_string(result_out)
ad_adapter_destroy(adapter)

adsnapshot returns a {version, ok, command, data} JSON envelope identical to the CLI output. The data.tree field contains snapshot-qualified ref IDs for interactive elements. Pass a qualified ref, or a legacy bare ref plus its explicit snapshotid, to adexecuteby_ref to drive the pipeline (RefStore load → strict resolution → actionability preflight → dispatch).

Core constraints

• ABI handshake. Call adinit(ADABIVERSIONMAJOR) once after loading the

dylib. A mismatch between the compiled-in constant and the loaded dylib returns ADRESULTERRINVALIDARGS — abort rather than proceed. You can also read the raw dylib major via adabiversion() for diagnostic display. New ad_* symbols and new error codes are additive (no bump required); removed or layout-changed symbols increment the major.

• Session adapters. adadaptercreatewithsession("session-id") associates

the adapter with a session namespace for refmap persistence — the same as CLI --session <id>. A null snapshotid is valid only for a qualified ref; legacy bare @eN refs require an explicit snapshot ID. Session IDs: 1–64 chars, ASCII alphanumeric / - / . Invalid IDs return null (check adlasterror_*).

• Structured session trace (no ABI change). File-based JSONL tracing activates

only when the session has a manifest with trace: on from session start (CLI) or equivalent on-disk setup. adadaptercreatewithsession alone does not create trace files. When tracing is active, commandcontext()-backed commands append to one segment per OS process under ~/.agent-desktop/sessions/<id>/trace/<pid>-<procTs>.jsonl. A long-lived host reuses the same segment filename for all calls in that process. For unstructured diagnostics regardless of session manifest, use adsetlogcallback (below).

• Threading and mutation leases. Adapter entrypoints may be called from any

host thread. Native handles remain bound to their creating adapter and thread. Desktop mutations acquire the same canonical cross-process interaction lease as the CLI; reads carry finite deadlines without taking the mutation lock. See [threading.md](references/threading.md) for the Apple documentation basis and the read/read, read/mutation, and mutation/mutation ordering matrix.

• Release profile. cargo build --release produces panic = "abort" —

any Rust panic inside an extern "C" fn will SIGABRT the host. Use --profile release-ffi to get the correct panic = "unwind" profile. CI enforces this.

• Last-error lifetime. Pointers returned by adlasterror_* remain valid

across any number of subsequent successful FFI calls on the same thread. Only the next failing call rotates them. Cache the pointer once, read it as many times as you need.

• adlasterrordetails. A fourth accessor, adlasterrordetails(),

returns a borrowed JSON string carrying structured details (e.g. the actionability check report on ACTIONFAILED, candidate summaries on AMBIGUOUSTARGET). The details may contain element names, values, and window titles from the user's screen — treat as sensitive diagnostics and avoid routing to shared log surfaces.

• Handle release. Every adresolveelementexact / adfind_exact result must be

released with adfreehandle(adapter, &handle) on the same adapter that produced it, before that adapter is destroyed. On macOS this balances the internal CFRetain; on Windows/Linux the call is a no-op but safe to issue. adfreehandle zeroes handle.ptr so a follow-up call is a safe no-op.

• Primary ref-action path. adexecuteby_ref is the recommended entrypoint

for the observe-act loop: it loads the RefStore, looks up the ref in the refmap (STALEREF on miss), runs strict element re-identification (STALEREF / AMBIGUOUSTARGET), runs the live actionability preflight, then dispatches. TypeText and PressKey default to focusfallback policy (matching CLI type/press-key); all other actions default to headless. Pass ADPOLICYKIND_HEADED (2) to opt in to cursor-based fallbacks.

• Generation-safe direct APIs. Legacy AdRefEntry and

AdWindowInfo layouts remain available for binary compatibility, but they do not carry process-generation evidence and direct targeting functions fail closed. Use AdExactRefEntry, AdExactWindowInfo, adlistwindowsexact, and the *exact targeting symbols. Likewise, adlistsurfaces_exact preserves SurfaceInfo.id; the legacy surface list is an observation-only projection that omits it.

• Display discovery. Call adlistdisplays before using

AdScreenshotTarget.screenindex. List order is the screenshot index order; inspect each AdDisplayInfo for its stable display ID, bounds, primary flag, and scale, then release the handle with addisplaylistfree.

• Low-level action paths. adexecuteaction (headless, no preflight) and

adexecuteactionwithpolicy are raw escape hatches for callers holding a live AdNativeHandle from adresolveelementexact / adfind_exact. Use them when you need to bypass the ref-action pipeline.

• Ref-action preflight. adexecuteby_ref and

adexecuterefactionexactwithpolicy both resolve the element strictly and run the live actionability preflight (visible, stable, enabled, supported action, policy, editable) before dispatching — a disabled or unsupported target fails before any platform call. On ADRESULTERRACTIONFAILED, the structured check report is available as JSON via adlasterror_details().

• Action result steps. AdActionResult.steps mirrors the CLI steps array

for activation-chain actions. Each entry has label and outcome strings and is owned by the result; release with adfreeaction_result(&out).

• Tracing / log callback. Two tracing surfaces coexist:

1. Structured file trace — same JSONL contract as CLI --trace, gated by a trace: on session manifest. Segments include event, ts_ms, seq, and redacted fields. Requires session start (or equivalent manifest on disk) before creating the adapter; plain session-id adapters write nothing to disk.

2. adsetlogcallback(cb) — installs a tracing subscriber layer that delivers events as JSON to your callback. cb receives an int32t level (1=ERROR … 5=TRACE) and a const char *msg valid only for the duration of the call. Pass NULL to unregister. The layer is installed on the first non-null call; if a foreign global subscriber already owns the process at that point, the install fails with ADRESULTERR_INTERNAL and no events are ever delivered. Sensitive field values (password, token, text, …) are replaced with {"redacted":true} before formatting. A panicking callback is caught and silently discarded. The callback may fire from threads other than the registering thread, and may still fire briefly after a NULL unregister — keep the callback and any data it captures valid for the process lifetime.

• Wait. ad_wait(adapter, args, &out) runs the full CLI wait command

(element-appear, window-appear, text-appear, menu-open/close, notification, element predicates). Zero-initialize AdWaitArgs, set the fields you need, and validate the struct size against ADWAITARGSSIZE / adwaitargssize() before calling. The output is a {version, ok, command, data} JSON envelope freed with adfreestring. adwait blocks the calling thread up to timeoutms ms — ensure the adapter is not destroyed from another thread while it is running.

• Text input privacy. On macOS, focus-fallback or headed text insertion may

briefly use the clipboard for non-ASCII text. For sensitive text, prefer ADACTIONKINDSETVALUE with ADPOLICYKIND_HEADLESS when the target supports settable values.

• Enum discriminants. Every #[repr(i32)] enum field is validated at the C

boundary — invalid discriminants return ADRESULTERRINVALIDARGS instead of undefined behavior.

• ABI stability. The major version in ADABIVERSION_MAJOR increments on any

breaking change (removed symbol, incompatible layout). Additive changes (new symbols, new error codes) do not bump it. Before 1.0, pin the exact version of libagentdesktopffi you link against.

• adgettreeexact vs adsnapshot. adgettree_exact returns a raw flat BFS tree

without @e refs, no refmap persistence, and no JSON envelope — use it for custom traversal or UI inspection. For observe-act agents that drive actions via adexecutebyref, always start with adsnapshot.