Operating note4 min read
A year of chasing a pocket AI hardware companion
From serial ping tests in June 2025 to OpenCharm on a Waveshare ESP32-S3 AMOLED board, and how streaming cut voice latency from 11.8 seconds to under 3.
Looking back through my folders, I apparently spent the best part of a year orbiting the exact same hardware itch: a tiny screen on my desk that I can talk to, that talks back, and that runs against my own tools instead of somebody else’s locked-down cloud.
It definitely didn’t start out looking like OpenCharm. It took five separate projects on progressively nicer ESP32 boards before the whole thing finally clicked.
1. Just getting bytes back and forth (test-esp-serial-round-trip)
In June 2025, I started about as basic as you can get: a PlatformIO firmware on a cheap ESP32 DevKit and a Python script measuring serial round-trip times, including relaying packets across two ESP32s.
Nothing glamorous, no screen, no audio. Just getting comfortable with flashing boards, measuring actual timings instead of guessing, and making sure a microcontroller doing a loop didn’t mysteriously stall on me.
2. Finding the right board (esp-radio)
The real turning point was picking up a Waveshare ESP32-S3-Touch-AMOLED-1.8 V2. For roughly thirty quid you get an ESP32-S3 with 16 MB of flash, 8 MB of PSRAM, a crisp 368×448 AMOLED screen (CO5300 driver), capacitive touch, an ES8311 audio codec with onboard mic and speaker, an IMU, and battery power management.
Before trying to wire AI into it, I built a standalone pocket internet radio (esp-radio) in ESP-IDF. Sixteen MP3 streams over Wi-Fi, decoded on-chip with micro-mp3, ICY artist/title metadata, a touch volume slider, and a swipe-down control centre.
That project paid for itself ten times over. It forced me to figure out the V2 board support package, the ES8311 codec init sequence, PSRAM buffers, and how not to starve the speaker while Wi-Fi is busy.
3. Adding a voice loop (ai-esp and grokbot-esp32)
Once audio and touch worked reliably, I tried two different shapes of AI companion:
grokbot-esp32was a desk companion with an animated face, quick-ask chips (What’s next today?, PR status?), a local web UI over mDNS (grokbot.local), and webhook-driven replies pushed back to the board over HTTP.ai-espwent full voice: streaming 16 kbps Opus frames over a private Tailscale gateway for remote wake-word detection ("Hey Hermes"), plus hold-BOOT to talk, local Piper TTS, and proactive announcements so a cron job could tell the board to speak up on its own.
Both worked, and both taught me the same painful lesson: waiting 12 seconds for a tiny face to answer you ruins the magic completely.
In ai-esp, the device waited for the agent to finish generating the entire text reply, synthesised the whole WAV file on the host, downloaded the entire file into PSRAM so network jitter wouldn’t glitch I²S playback, and only then started speaking.
It was super clean and glitch-free. It also felt like talking to a walkie-talkie connected to Mars.
4. Putting it all together in OpenCharm
When Meta showed off their Muse Charm concept at Connect 2026 — a pocket-sized voice companion with an animated plush character — I realised I already had 80% of the hardware stack sitting on my desk.
OpenCharm is my open-source take on it, built to talk to your own self-hosted agent rather than a closed cloud. Hold the side button (or the screen), talk, let go, and it listens, shows what tool it’s running ("Checking Fastmail…"), and talks back with its mouth moving in sync with the audio. Tilt the board and its eyes follow using the QMI8658 IMU. Shake it and it gets dizzy.
Getting it to feel alive came down to two big fixes:
Splitting rendering and networking across both cores
On an ESP32-S3, if you try to push a 368×448 16-bit AMOLED frame over QSPI on the same core that’s juggling Wi-Fi TLS and I²S audio, either the face stutters or the audio clicks.
OpenCharm splits the work cleanly across the two cores, keeps sprite assets in the 8 MB PSRAM, and pushes display updates over QSPI DMA so the face stays around 30 fps while voice audio plays smoothly underneath.
Cutting latency from ~11.8 s to ~2.8 s
This was the single biggest upgrade in the whole year (you can run the side-by-side comparison at the top of this post).
Instead of spawning a fresh CLI process and waiting for the entire answer and full audio file:
- The bridge keeps a warm streaming API connection to the agent.
- As tokens arrive, it slices the response at sentence boundaries.
- Sentence one goes straight into TTS and begins streaming audio chunks back to the ESP32 over the open socket.
- The ESP32 starts feeding I²S as soon as a tiny jitter buffer fills, while sentences two, three, and four are still being generated and synthesised behind it.
Going from almost twelve seconds of silence down to under three seconds completely changes how often you actually bother picking the thing up. At 12 seconds it’s a neat demo you show a friend once. At 2.8 seconds you genuinely hold the button and ask it when your next meeting is while making coffee.
The whole firmware and bridge are open source at github.com/henrydennis/opencharm if you want to flash a Waveshare board and hook it up to your own agent.