Jules ShermanHealthcare product design & research since 2012
Digital product · 2026

BCI Emotion Capture

Turning decoded speech into images, for people whose words already reach a screen but whose expression does not

Try it

Visual Communication Space

Choose an emotion, or switch to Paint and describe something. Set how strong it should be and whether it looks like a photograph or a painting, watch the picture form, then pick who it goes to.

The opportunity

Expanding the expressive options for communication

A person with ALS or a brainstem stroke still knows exactly what they want to say. The words are formed in the brain, but the muscles needed to speak them can no longer respond.

Implanted brain-computer interfaces are beginning to decode that intention and display it as text. That is an extraordinary achievement because it gives people their words back. But communication is more than words. It can also be expressed through art.

This concept uses the same decoded brain signal to create an image from words the person is trying to speak. The image replaces the words, offering a second channel for expressing an emotion, wish, atmosphere, or memory that text alone may not fully convey.

About this project

How someone with a BCI implant uses it

  1. 01The input

    Attempted speech, nothing else

    No typing, tapping or pointing. The person would attempt to speak exactly as they already do to produce text, and the system would read the same decoded words but deliver them as imagery.

  2. 02Two modes

    Emotion, or Paint

    Emotion turns a feeling word into color and light. Paint builds whatever they describe, so they can show something they are feeling or imagining instead of describing it with text.

  3. 03Two dials

    How strong, and how literal

    Intensity (for the Emotion feature) is how loud the picture is. Realism (for the Paint feature) decides whether it looks like a photograph or a painting. To change it, keep your description in the box, move the slider, and press Paint Picture again.

  4. 04The output

    A wall, or a phone

    To send an image, the user would pick a person's name from the dropdown address book and press Send. The picture can show up on the computer screen attached to their wheelchair, or be sent by text to someone's phone.

Design decisions

Why it is built this way

01

Sometimes words are not enough

Communicating through an implanted BCI can be slow, and sometimes what a person wishes to express may be better shared with an image.

That is why the interface begins with six commonly used emotion words on a single screen, and many more could be added. A person can communicate an immediate feeling with one selection if they choose.

The Paint feature is available for moments when a simple word is not enough. It takes more time, but gives the person a richer way to express a more complex emotion, memory, or experience when that extra effort feels worthwhile.

02

Two dials, not a settings panel

Intensity and realism are the only two things that change what the image looks like. When each attempt costs the person energy, keeping the interface simple is the best decision.

Transparency and trust

Staying in charge

  • A system that speaks on someone's behalf is only worth
    building if the person stays in charge of what it says about them.

  • Nothing is set in stone

    The user can make an image again, and again, until it evokes what they actually want to express. Nothing is fixed until they choose to send it.

References

Inspired by the literature

  1. Ikegawa Y, Fukuma R, Sugano H, Oshino S, Tani N, Tamura K, Iimura Y, Suzuki H, Yamamoto S, Fujita Y, Nishimoto S, Kishima H, Yanagisawa T.

    Text and image generation from intracranial electroencephalography using an embedding space for text and images

    Journal of Neural Engineering, 2024 May 20; 21(3). doi:10.1088/1741-2552/ad417a · PMID 38648781

  2. van Leeuwen JEP, Boomgaard J, Bzdok D, Crutch SJ, Warren JD.

    More Than Meets the Eye: Art Engages the Social Brain

    Frontiers in Neuroscience, 2022; 16:738865. doi:10.3389/fnins.2022.738865

Colophon

How the prototype was made

The prototype runs in a browser. The emotion visuals are drawn frame by frame in code, and Paint sends the description out to an image model and brings the picture back.

Frontend
HTML5, JavaScript and Canvas for the emotion visuals
Backend
Netlify serverless functions
Image generation
Replicate API with Stable Diffusion XL
Deployed at
bci-emotion.netlify.app
Code
github.com/juless26/BCI-Emotions

The thread

A woman seated in a power wheelchair in a sunlit living room, wearing a floral EEG sensor cap labeled Cadence. She looks toward a tablet on an articulated arm showing the Cadence interface: a message field reading Your message will appear here, a row of predicted words, and an on-screen keyboard.
Digital Product

Cadence

Helping people communicate with an EEG brain-to-computer interface

A man using a power wheelchair sits at a dining table across from his daughter, a small sensor at his temple. A translucent panel floats between them showing meters for Your energy and Whose words, her line "I don’t know if I should take it. It’s so far.", and the reply Take the job. You’re ready for this., with Accept, Edit and Retry. A small speaker sits on the table between them.
Digital Product

Whose Words?

At what point does your speech get lost when saying it requires more of the machine to do the talking?