Skip to content

PCM capture ownership

ESPURR_AUDIO (Espurr cat ears → Audio capture, diagnostics and sound effects) is a build-time/menuconfig switch, on by default only for v2. Disabling it omits capture/level tasks, receivers, recording and spectral processing. Source status is unsupported; ordinary lighting and BX motion/SFLP remain available. Sound bands keeps its stable effect ID but its compiled audio requirement is unavailable.

Independent ESPURR_AUDIO_PDM and ESPURR_AUDIO_CONTACT sub-options select stereo air capture/spectral features and BX contact capture respectively. Contact needs the BX motion driver; disabling it leaves that driver's ordinary motion/SFLP path available. PDM-only retains Sound bands without TDM or ESP-SR. Contact-only omits PDM setup and FFT processing. Both off use the unsupported stubs with no capture/ level tasks. The 8 MB v2-8mb preset defaults to PDM-only and no PSRAM; levels and spectral processing work without PSRAM, while diagnostic recording is unavailable. No production VAD/recognition implementation exists to enable; future speech processing must remain independent of PDM reactivity and flash-layout selection.

Raw stereo PCM and per-channel levels remain separate. Current Sound bands averages stereo powers and does not estimate direction. A future left/right loudness reaction or calibrated direction estimator can consume the same stereo input without SR; library/ISA-accelerated direction finding is deferred to a later task.

V2 starts a core-0 capture owner before the IMU service. Air/contact requests are independent and both default off. Its 8192-byte internal task stack is permanent; queues and I2S allocations exist only while a source is enabled and permitted. The idle task waits for notification, with a 100 ms recovery poll. Legacy builds use unsupported-source/no-op coordination stubs; v2 with IMU disabled retains air capture and reports contact unsupported. Optional startup/allocation failure does not reject render/input health or prevent ordinary lighting/control.

The internal request/snapshot interfaces are in audio.h. A priority-3 core-0 level worker is the sole PCM queue consumer, with a permanent 4096-byte internal stack and a 10 ms, at-least-one-tick wait. Each iteration drains at most four blocks per source. It computes one 10 ms block's AC RMS (mean removed), raw absolute peak, signed mean and clipping mask. Integer variance preserves tiny AC signals on large DC offsets; standard sqrt finishes the scalar statistic. This needs no FFT/DSP library. Spectral features use managed ESP-DSP below; VAD selection remains pending worn recordings. Do not treat this meter as a voice classifier.

Both phone apps expose Settings → Audio diagnostics only when the typed source records exist. A confirmed, exclusive audio.capture command requests air, contact or both for 60 seconds; a new command can restart the lease. Render owns this demand, its session generation and expiry. Repeated command IDs do not renew it. Disconnect, radio-mode departure, OTA begin and shutdown clear it; NVS/ordinary flash holds pause receivers without renewing it. A rejected unsupported-source request changes neither source nor the existing lease. The service's atomic mask API validates both sources before mutation. Capture remains off at boot, has no NVS setting and is unavailable if the level consumer failed to start; typed status reports that allocation state and apps disable capture with restart guidance. Future effect demand must join the render-owned demand calculation rather than independently overwrite diagnostic requests.

The optional per-source 64-byte wire records contain bounded health/levels, never PCM. Level validity requires matching capture generation, ready/running/requested state and both observations within 100,000 µs. Invalid, stale, paused, settling and restarted data lose their level payload; a failed block cannot retain an earlier valid reading. A failed receiver stop remains failed/running even after demand or contact permission is revoked; an off request does not establish that capture stopped. Measurements are signed-16 PCM counts, with AC RMS displayed relative to 32768 full scale, not calibrated SPL. The gap flag describes the current measured block; loss counters provide history. Apps show observations from their last update rather than sample-rate telemetry.

Both phones can explicitly record a five-second clip from an already requested source, download it, then save/share the WAV and a JSON provenance sidecar. Level capture alone never records a clip, and firmware never saves audio to flash. Sound bands uses the air features below; useful contact VAD remains work. Capture has not been tested on hardware; useful VAD, acoustic quality and physical timing remain unverified.

Air spectral features

The same sole worker now computes six air-microphone bands with managed ESP-DSP 1.8.2, using its Hann window, FFT2R and bit reversal. A 512-frame/nominal-32-ms window advances by each 160-frame/10-ms block. There is no new task, PCM consumer or FFT implementation. Contact X/Y/Z remains level/recording data; no axis/filter choice or air-trained speech model is silently applied to contact audio.

Each channel's full-window mean is removed before windowing. Independently transformed stereo powers are averaged before taking the square root, preventing opposite-phase microphones from cancelling. Values are band RMS relative to 32768 signed-16 PCM counts, normalised by the Hann window power. Positive bins count twice, Nyquist once, DC is omitted. Nominal bands are 80–250, 250–500, 500–1000, 1000–2000, 2000–4000 and 4000–8000 Hz, with exclusive upper edges except Nyquist. Actual bin centres are 31.25 Hz apart; these are coarse visual features, not brick-wall audio filters, SPL, speech probabilities, beat or wearer detectors.

Four contiguous blocks are required for the first window. Gaps, skipped offsets, receiver generations, masked/settling/invalid blocks, observation regressions and counter overflow discard history. Each contributing block must still be fresh; a fresh final block cannot hide an old part of a window. Clipping covers the exact retained samples. audio_features_snapshot() copies only compact provenance, quality and six bands, then gates them against requested/running/READY source state, matching generation, the entire window and receiver observation within 100 ms. Invalid/paused/ stale data has zero spectral payload and valid=false, distinct from valid silence.

An aligned internal workspace is allocated lazily only while air capture is requested and READY, and freed outside the publication lock on off/pause/failure. Its 8272 bytes hold history/window/FFT scratch instead of using the task stack. ESP-DSP also owns its small FFT-table resources; the S3 uses the IDF ROM twiddle pointer and requests a 960-byte bit-reversal table before allocator overhead. The PCM worker alone owns global FFT2R initialisation/deinitialisation. An unavailable workspace/table leaves levels, recording and both receivers usable; retries are bounded to once per second while READY, reset when the source stops or leaves READY. No DSP workspace is allocated at boot or on legacy/audio-omitted profiles. Existing generated SDK choices are retained.

Render-owned Sound bands demand and explicit diagnostic leases share the receiver through their union. Selecting the effect with normal lighting enabled starts air capture without a phone; stopping/expiring diagnostics leaves effect capture running. Off, another effect, flashlight, diagnostics mode, OTA and shutdown remove effect demand. Ordinary flash holds pause capture without discarding that demand. Recording still requires an explicit diagnostic lease, even if the effect already uses air PCM.

Render snapshots compact qualified features once per draw; the renderer never reads PCM or performs FFT/I/O. Optional audio.features.status exposes workspace, validity, source/effect demand and six pre-gain bands; device.features exposes compiled support. Both phones show health/fallback with separate Sound gain and advanced band details; the browser keeps only support/health and the descriptor-driven gain editor. See the catalogue and wire contract. Physical S3 SIMD numerical/timing checks, CPU/stack/heap measurements with audio, LED output and radios, and useful contact VAD still need device/worn evidence.

Bounded diagnostic recorder

audio_record.h stages one temporary clip from either source, using the existing sole PCM consumer. It adds no task or queue consumer. The render owner publishes the current authenticated session, revoking a previous clip when that ownership changes, on mode departure, OTA begin or shutdown. The transport checks the live authenticated ledger before and after every bounded read; the recorder's generation comparison is an additional ownership check. Ordinary capture requests do not start a recording, and recording does not create capture demand: the selected source must already be requested by the diagnostic lease.

Requests cover 100..5000 ms in whole 10 ms blocks. The worker allocates exactly one buffer from MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT, with no internal-RAM fallback. The maximum contact clip is 480,044 bytes (five seconds, three axes, WAV header); air uses two channels. A new request invalidates the previous completed clip before the worker frees/replaces it, so two clip buffers are never retained together. If PSRAM or the recorder mutex/buffer is unavailable, the level meter and ordinary capture remain usable. No clip buffer exists at boot; on PSRAM builds only the small recorder mutex is allocated with level-worker startup.

audio_clip.c writes ordinary signed-16 little-endian PCM WAV at the nominal 16 kHz setting: air channel 0/1 and contact X/Y/Z. These are diagnostic channels, not speaker positions or calibrated acoustic measurements. Raw samples are preserved, including DC and saturation; a per-channel clipping mask remains in metadata. Source, receiver generation, first frame offset and first/last host observation times stay outside the WAV header in the exported JSON metadata. Observation time is not qualified acoustic acquisition time, and separate clips do not establish air/contact synchronisation or physical mic identity.

The first valid, fresh block must arrive within one second of the request. Queued observations older than the request and initial decoder settling are skipped. A gap before that chosen first block is outside the clip; subsequent gap flags, non-consecutive offsets, receiver-generation changes, invalid blocks or regressed observations fail the clip. Future/older-than-100-ms observations are rejected. Completion has a nominal-duration-plus-250-ms deadline from the first accepted block. Pause, requested-off or source failure aborts an incomplete clip. Failed partial data cannot be read, is not zero-filled and is released by the worker.

Only a complete immutable clip can be read, at arbitrary offsets in chunks of at most 256 bytes. Reads hold a short mutex during that bounded PSRAM copy; allocation, release, PCM copying and receiver operations are outside it. Session revocation removes the published pointer before the worker releases memory; repeated reads of an old session/clip ID fail. A completed clip remains in volatile RAM until explicit clear/replacement/session departure. There is no routine flash logging or background recording.

The exclusive momentary audio.record command selects clear/cancel (0), air (1) or contact (2); reads always return zero. Public requests use five seconds and require at least 6250 ms on the existing session's selected capture lease. A new recording does not silently start/renew capture. audio.record.status is a strict optional 64-byte/type-18 publication; unsupported recorder allocation leaves levels usable.

Download uses operation 0x80 on the existing authenticated espurr/catalogue transport, dispatched outside the catalogue loader/cache. GATT layout is unchanged, including for older Apple clients which reject unfamiliar characteristics. Each reply carries at most 256 WAV bytes; the phone validates session/clip/offset/count, the final WAV header/length and provenance. Cancel stops between complete encrypted exchanges without replaying ciphertext or exporting partial data. Disconnect clears the phone's in-memory download. Explicit Save/Share first stages private phone files so a native picker can finish after normal background connection release. Watch and browser remain compact controls with no PCM workflow. See the wire contract.

A tiny fixed WAV header needs no codec framework; native C fixtures plus Python's independent standard-library WAV reader verify byte order, channel interleaving, length and exact samples. VAD evaluation remains separate; this writer does not introduce an audio-processing algorithm.

audio_rx.c uses IDF 6.1's I2S channel driver, not a separate codec framework:

  • I2S0 receives stereo PDM as 16-bit PCM at 16 kHz, with 128x downsampling and the board's shared clock/data pins. Physical mic L/R identity needs checking.
  • I2S1 generates 16 kHz WCLK and nominal 2.048 MHz BCLK for eight 16-bit slots. The initial framing is PCM-long, active-high 16-clock WCLK, one-bit delay and rising-edge reception. MCLK multiple 512 explicitly gives divider 4; IDF would otherwise adjust the default divider 2. MCLK is not routed to a pin.
  • Both receivers use four DMA descriptors of 160 frames (10 ms), explicitly at interrupt priority 1. Internal DMA payload is 2560 bytes per TDM descriptor or 640 per PDM descriptor: 12,800 bytes combined, before descriptors/driver overhead. No application PCM copy or processing occurs in the ISR; it counts queue overruns.

One priority-4 core-0 task owns both receivers' open/start/read/stop/close calls. The receiver context and queue must be internal RAM; the API rejects external memory for them. On S3, a decoded block is 992 bytes, the four-block queue is 3984 bytes and a receiver context is 24 bytes. Queues are allocated on source startup and released after stop, pause or recovery; pointer removal and consumer access share only a short lock, with I2S work outside it. Separate history/workspace can later use PSRAM. No capture pool is statically instantiated and legacy needs no PSRAM. Start/restart advances a configuration generation. Read requests one full DMA block with a 20 ms IDF timeout. Partial/failed reads or an observed queue overrun reject the block and require stop/start; IDF resets the RX queue on enable. Disable/delete failure retains the handle for owner retry. Never lose that handle or reset/reconfigure BX while its receiver is still running.

The BX owner revokes capture permission and waits for an explicit stop-epoch acknowledgement before every full reset, including changes in TDM demand. Only successful sensor initialisation grants contact permission. FIFO-only overflow recovery does not reset or stop contact capture; full I/O/watchdog recovery revokes it. A missed/failed stop acknowledgement prohibits sensor reset. Sensor failure does not stop independent air capture.

Render-owned NVS saves, external flash holds, OTA lifetime and shutdown acquire nested audio pause holds. An acknowledgement comes from the capture owner after in-flight starts/reads finish and both receivers stop, never from an earlier published running=false. A failed acquisition rolls its hold back and rejects the flash operation. Only the last release permits restart; new generations and fresh queues prevent pre-pause PCM escaping. Deadlines are 250 ms with RTOS tick quantisation; the 100 Hz profiles always yield at least one tick while waiting.

The component uses 32-bit C11 atomics. The N16R8 SDK enables IDF's S32C1I/PSRAM workaround, so the compiler's global lock-free macro is not a suitable assertion. Contexts stay internal; the pinned libc dispatches those operations to native S32C1I helpers. Current v2 libc helper placement is IRAM. The current flash policy stops capture, so it does not require continuous cache-off I2S operation. Any future continuous path must configure/verify the driver's IRAM path; the callback's own annotation alone does not establish lossless capture.

audio_pcm_decode() accepts exactly one wire block, including unaligned bytes. Air PCM remains stereo. Contact slots 0/1/2 map Z/Y/X to native X/Y/Z; other slots are ignored because the BX drives them in high impedance. BX setup enables settling masking, and the decoder invalidates blocks containing 0x8000 on any active contact axis. It also discards the first 107 contact frames conservatively; the whole first block is invalid. Positive/negative saturation is reported per channel. valid means this decoding/settling contract passed, not that the data have been qualified as wearer speech or that PDM startup is acoustically settled.

Each block carries a generation, frame offset, host observation time and gap flag. Observation time is deliberately not labelled acoustic acquisition time. Receiver backlog, actual clock rate, cross-stream drift and interrupt/capture latency still need measurement before synchronising streams or claiming onset accuracy.

The four-block single-producer/single-consumer queue never waits for a consumer. A full queue drops the new block, increments its drop counter and flags the next accepted block as a gap. Consumers must drain promptly and reject stale observations; renderers will consume compact features, not this queue or PCM. Reset/init is only legal while producer and consumer are both quiescent. Generations must not be mixed across receiver restarts. There is no routine BLE/browser PCM telemetry or flash logging.

The BX adapter's optional TDM initialisation uses the pinned ST driver while the accelerometer is powered down. It retains all axes, the separate 120 Hz UI/FIFO configuration and optional SFLP setup, then selects high-performance TDM mode. Native register tests cover intended state and every transaction failure. The IMU owner selects TDM from requested contact demand after the stop acknowledgement. Actual simultaneous clocks, FIFO timing/SFLP behaviour and mounted audio remain hardware acceptance work.

References: BX AN5845 section 7, BX driver provenance, and IDF I2S. The exact local pinned IDF headers and implementation were inspected for framing, divider adjustment, DMA queue/partial-read semantics and libc atomic placement. Frequency configuration does not prove the BX's accuracy/jitter requirement; qualify BCLK/WCLK duty, edge/delay/slot order and active/tri-state data on the bench.

Contact VAD evaluation

Further VAD development/evaluation is pending hardware and worn recordings, per Andi's 2026-10-31 decision. Keep the existing capture/export/replay/scoring tools; do not select or integrate a detector without those inputs. MLC remains optional.

The isolated replay tool pins ESP-SR 2.6.0 and builds its standalone WebRTC VAD against IDF 6.1. It prepares a private build from a contact WAV/JSON export, retaining X/Y/Z independently and testing every supported aggressiveness mode. It records raw classifications and target timing, local heap minima and stack margin when run. There is no selected production axis, mode, contact filter or claimed wearer detector. Worn data and combined-workload measurements still gate adoption in the sole PCM worker.

The offline scorer validates one complete replay against the exact prepared clip and explicit whole-clip wearer-speech annotations. It reports block errors, interval misses, false-positive runs, nominal block-completion delay and recorded costs without selecting an axis/mode or declaring acceptance. Already-active detections cannot masquerade as fast onset. See the replay workflow for label semantics, wrapper limitations and the distinction from live acoustic latency.

Normal firmware does not depend on ESP-SR or allocate its VAD state. The replay image has no live capture/control path and is rejected by normal ESPurr OTA image family checks. Its source/dependency pins, input validation and deployment boundary are documented with the tool; no device is contacted during preparation/build.