Can the HoloGaze liberate S2Ced individuals from their facilitators?
Over the past couple of years, S2C proponent Vikram Jaswal, Professor of Psychology at the University of Virginia and the father of an S2C user, and Diwakar Krishnamurthy, Professor of Electrical and Computer Engineering at the University of Calgary and the father of an RPM user have co-authored several papers on the development of virtual tools that enable S2C users to virtually select virtual letters rather than pointing to letters on letterboards. Like Jaswal’s other papers, therefore, these papers are worth reviewing in detail. In my last three posts, I discussed Alabood et al. (2024), a study involving a virtual letterboard, or HoloBoard, and Alabood et al. (2025), a study involving the use of virtual letter cubes to spell words.
In my next two posts, I’ll discuss two additional studies in Jaswal’s 2023-2025 virtual reality oeuvre that are relevant to us here at facilitatedcommunication.org: Nazari et al. (2024), a study involving the “HoloGaze,” and Shahidi et al. (2023), a study involving the “HoloLens.” What makes these studies relevant is that, like all the other Jaswal studies we’ve reviewed:
They involve language and literacy (some of Jaswal et al.’s other studies instead involve visual games)
They appear to show levels of language comprehension that are generally not found in minimally speaking autism, particularly in those with significant motor impairments (see Chen et al., 2024).
(Minimal speakers with autism and significant motor impairments being representative, per Jaswal et al., of the population that their studies are targeting.)
In this post, I’ll focus on the HoloGaze study (Nazari et al., 2024), a paper in which Krishnamurthy and Jaswal are listed, respectively, as the second and third authors. This paper focuses on a virtual reality technology, complete with headsets, that allows users to select letters and spell words by shifting their gaze to different letters from a virtual display. Users can select a particular letter either by sustaining their gaze on that letter, or by pushing a button while looking at it. The HoloGaze study differs from the HoloBoard and LetterBox studies in one important way: for the most part, it doesn’t have its non-speaking autistic participants do anything other than select letters and spell words—as opposed to tasks that require actual comprehension of language. (It’s worth noting that a significant proportion of autistic individuals are hyperlexic and able to spell words that they don’t understand).
Like the other two papers, the HoloGaze paper opens with the same unsubstantiated/false claims about minimal speakers with autism, beginning with the purported attentional, sensory, and motor issues:
[N}onspeaking autistic people have significant attentional, sensory, and motor challenges.., making it difficult for them to write by hand or type in conventional ways. For example, many nonspeaking autistic people have difficulty with the fine motor control needed to use a pen or pencil.
As in the LetterBox paper, their authors’ sources for these claims are (1) an entire book attributed to someone who has been subjected to facilitated communication (Higashida, 2013), and (2) a meta-analysis of 41 studies on motor coordination in autism, none of which included the skills involved in pointing to letters on a letterboard (Fournier, 2010). To give Jaswal et al. the benefit of the doubt, when they mention typing in “conventional ways,” they may mean ten-finger typing. Ten-finger typing, as compared with typing with an index finger, does involve significant motor coordination. But, to the extent that the authors’ goal here is to motivate an examination of eye-gaze technology, they haven’t managed to explain why, for minimal speakers with autism, eye pointing would be superior to index-finger pointing.
Furthermore, the authors make no mention of the one specific motor issue, sometimes alleged in non-speaking autism (see Handley, 2022), that might be relevant here: ocular apraxia—or the ability to move one’s eyes in a desired direction. Ocular apraxia would have been relevant to their HoloGaze study because it reportedly involved a training phase in which non-speaking autistics learned how to select letters via eye gaze. If there’s no eye gaze impairment in autism, and if the non-speakers already know how to pick out letters on demand, then why do the participants in this study require anything more than a brief explanation in how to use the system?
Beyond their claims about attentional, sensory, and motor challenges in autism, the authors claim, once again citing the memoir attributed to Higashida (Higashida, 2013), that the tendency of non-speaking autistics to have trouble sitting still while being prompted by their facilitators to point to letters on the letterboard is the result of “regulatory” issues:
They may also be in constant motion (which seems to serve a regulatory function... making training to use a keyboard while remaining seated difficult [)].
Left out of this discussion is the more likely possibility: boredom with a task that, due to those significant comprehension problems in non-speaking autism that are particularly severe in those with significant motor impairments (see Chen et al., 2024), these individuals probably find meaningless.
Next, the authors claim that:
Some nonspeaking autistic people have learned to communicate by typing, which has allowed them to graduate from college and to write award-winning poetry.
Their sources are a National Public Radio piece about RPM user Elizabeth Bonker’s valedictory speech at her graduation from Rollins College (in which she stood at the podium while a pre-recorded speech, attributed to her, was broadcast), and an autobiographical piece attributed to David James Savarese, better known as Deej.
Despite the purported communicative successes of those who “communicate by typing,” the authors note that
the process by which they learned to type was lengthy and expensive, and often requires the ongoing support of another person.
The most likely explanation for these hurdles is that those subjected to “communication by typing,” aka Spelling to Communicate (S2C), depend on facilitator prompts and cues to determine which letters to select; for Jaswal et al., these hurdles are instead a reason to develop virtual reality technologies like the HoloGaze.
Importantly, however, the HoloGaze
allows a caregiver to join an AR session to train an autistic individual in gaze-based interactions as appropriate.
(“Caregiver” here is used in this paper to denote the facilitator, or what S2C-proponents call a “Communication and Regulations Partner.)
Those familiar with evidence-based AAC devices might wonder, given that there already exist AAC devices that allow eye-gaze selections or “eye typing,” what the point is of this new tool. But the authors, acknowledging that such technologies already exist, claim that VR tools offer “unique additional benefits.” They cite:
The “wider context” in which these devices can be used: “e.g., not just at a special education classroom but also for personal use cases.”
“mobility”
The “3-dimensional environment shared between educators and students,” which “can facilitate the training process for those who require extensive training.”
These strike me as pretty weak arguments. Standard AAC devices are mobile and can be used in a broad set of contexts. And why is “extensive training” necessary? Believing, as Jaswal does, that his non-speaking participants already know how to identify letters and point to them to spell words, and that non-speaking autistic individuals, like people in general, look at the letters that they point to (Jaswal, 2020), it’s unclear why these participants would need “extensive training” in using selecting virtual letters via the HoloGaze.
So who are these participants? In all the other studies we’ve reviewed so far, Jaswal et al.’s participants are explicitly described as individuals who point to letters on letterboards with the support of communication partners (the hallmarks of Spelling to Communicate and Rapid Prompting Method). This study differs: its inclusion criteria do not mention communication partners; only “experience in communicating using a physical letterboard.” Participants therefore could have included those who communicate independently: without someone hovering, holding up the letterboard, prompting, or otherwise cueing them and controlling their letter selections. And yet the identities of every single person who is thanked in the paper’s acknowledgments suggest otherwise. In order of mention, they are: a neuropsychologist who believes in S2C and is doing her own experiments on S2C-ed individuals, an RPM user, the mother of an S2C user, an S2 practitioner, another S2C practitioner, and S2C promoter and practitioner Elizabeth Vosseller.
Let’s turn, now, to the actual experiment. After an initial “tolerance and calibration” phase, participants underwent a “training phase” in which they first learned to select flashing tiles and then flashing letters from arrays of tiles/letters in which only the target item flashed. If they selected the flashing item, it turned green and was surrounded by a “bounding box” that indicated “successful eye gaze engagement.” Also providing cues were the “caregivers”:
The person assisting the user could also observe the tile’s colour (because they were also wearing a device), enabling them to provide verbal prompts to guide the user’s attention if necessary.
If the participants really had the language comprehension skills that Jaswal et al. regularly attribute to them, why couldn’t they just be told, verbally, how the system worked? All they needed to know, in order to use the system correctly, was this: “Direct your eyes to the flashing letter. Then either look at it for one second, or push this button while you look at it.” One has to wonder whether, at some level of consciousness or sub-consciousness, Jaswal suspected that his participants didn’t have the kinds of comprehension skills that he has long attributed to the broader non-speaking population to which they belong.
The flashing letters and the facilitators’ “attentional prompts and cues” continued into the “Assisted Spelling” part of the testing phase, where the participants had to spell actual words by selecting letters from a virtual letterboard. The researchers dictated six three-letter words (JET, DRY, EVE, FAN, GUM, RUG, IVY), and the target letters flashed one by one in order as they were selected. Besides the flashing, prompting, and facilitator cueing, participants received one additional cue:
To increase their visual load gradually, participants did not see the full letterboard at the beginning of the testing phase. Instead, only the letters in the first word were presented. After the first word (and after all subsequent words), the additional letters required to spell the next word were added.
To justify this, the authors cite feedback that was almost certainly generated by S2C:
This design was suggested by our nonspeaking autistic consultant to reduce visual clutter initially as the participant learned the affordances of a new interface.
This feedback, if generated via S2C, was almost certainly authored by the facilitators/CRPs rather than by the non-speakers themselves.
Following the “Assisted Spelling Phase” was the “Unassisted Spelling Phase.” This involved six four-letter words (ARCH, BALL, DUCK, EARL, FALL, GIFT, HOPE), without the target letter flashing. It’s unclear whether the facilitators were still allowed to prompt and cue, but in any case it’s a lot harder to detect and cue people’s eye gaze than their extended index fingers.
Curiously, there was a fair amount of attrition at each stage, from the “tolerance and calibration” phase to the “training” phases to the “testing” phases:
Twelve of the 13 participants who tolerated the device attempted the testing phases that involved spelling... Half of those who tried the testing phases (6 of 12) completed both the phase where the letters flashed in sequence ("assisted") and the phase where the letters did not flash in sequence ("unassisted").
In other words, less than half the participants made it through the whole study—short though it was. And yet, the authors professed to be impressed:
This is a remarkable number given this was their first experience using eye gaze interactions using a head-mounted AR device.
As for the actual results of those who completed at least one of the testing phases, the author report two factors: correct interactions per minute (which presumably means “correct letter selections per minute”) and error rate. Mean interactions per minute decreased over time from 13.49 to 10.53, which the author claim reflects increased complexity (more letters to choose from; the shift from assisted to unassisted spelling; the shift from three-letter words to four-letter words). Thus, the unassisted spelling averaged between 5 and 6 seconds per correct letter—a surprisingly low rate for anyone who actually knows how to spell the given words.
Meanwhile, the error rate, “surprisingly” according to the authors, “improved from 0.42 (range: 0.07 - 0.79) to 0.39 (range: 0.08 - 0.63).” In other words, participants only selected the correct letters, on average, about 3/5 of the time. And, assuming the top and bottom scorers (across the 6 participants) were consistent from start to finish, their results ranged, in the case of the top scorers, from a slight decrease in near-complete success (from a 0.07 error rate to a slightly worse 0.08 error rate) to, in the case of the bottom scorers, a slight improvement in rather low scores (from a 0.79 error rate to a 0.63 error rate, or from selecting the correct letter only about 1/5 times to selecting it just under 1/3 of the time).
While the official training only involved spelling three and four-letter words, some participants, “if they had time and interest,” were asked to engage in one activity that actually required comprehension: “answer[ing] five questions with one word answers on the virtual board.” However, the authors tell us:
This data is not reported here because these tasks were completed by only a subset of participants and because of space limitations.
We can only wonder what the excluded data would have suggested to their readers about language and literacy skills in non-speaking autism.
BLOG POSTS IN THIS SERIES
Can Jaswal’s “HoloBoards” substitute for letterboards? Part 1
Can Jaswal’s “HoloBoards” substitute for letterboards? Part II
Can Jaswal’s “LetterBoxes” substitute for letterboards?
REFERENCES
Alabood, L., Nazari, A., Dow, T., Alabood, S., Jaswal, V.K., Krishnamurthy, D. Grab-and-Release Spelling in XR: A Feasibility Study for Nonspeaking Autistic People Using Video-Passthrough Devices. DIS '25: Proceedings of the 2025 ACM Designing Interactive Systems Conference. Pages 81 – 102 https://doi.org/10.1145/3715336.3735719
Alabood, L., Dow, T., Feeley, K. B., Jaswal, V.K., Krishnamurthy, D. From Letterboards to Holograms: Advancing Assistive Technology for Nonspeaking Autistic Individuals with the HoloBoard. CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems Article No.: 71, Pages 1 - 18 https://doi.org/10.1145/3613904.3642626
Chen, Y., Siles, B., & Tager-Flusberg, H. (2024). Receptive language and receptive-expressive discrepancy in minimally verbal autistic children and adolescents. Autism research : official journal of the International Society for Autism Research, 17(2), 381–394. https://doi.org/10.1002/aur.3079
Fournier, K. A., Hass, C. J., Naik, S. K., Lodha, N., & Cauraugh, J. H. (2010). Motor coordination in autism spectrum disorders: a synthesis and meta-analysis. Journal of autism and developmental disorders, 40(10), 1227–1240. https://doi.org/10.1007/s10803-010-0981-3
Handley, J. B., & Handley, J. (2021). Underestimated: An autism miracle. Skyhorse
Higashida, N. (2013). The reason I jump: The inner voice of a thirteen-year-old boy with autism. Knopf Canada.
Jaswal, V. K., Wayne, A., & Golino, H. (2020). Eye-tracking reveals agency in assisted autistic communication. Scientific reports, 10(1), 7882. https://doi.org/10.1038/s41598-020-64553-9
Nazari, A., Krishnamurthy, D., Jaswal, V. K., Rathbun, M. K., & Alabood, L. (2024). Evaluating Gaze Interactions within AR for Nonspeaking Autistic Users. 1–11. https://doi.org/10.1145/3641825.3687743
Shahidi, A., Alabood, L., Kaufman, K. M., Jaswal, V. K., Krishnamurthy, D., & Wang, M. (2023). AR-based educational software for nonspeaking autistic people – A feasibility study. 2023 IEEE International Symposium on Mixed and Augmented Reality
