Can Jaswal’s “HoloBoards” substitute for letterboards? Part II
This post picks up where I left off in my discussion of a recent paper co-authored by Vikram Jaswal, Professor of Psychology at the University of Virginia and the father of an S2C user, and Diwakar Krishnamurthy, Professor of Electrical and Computer Engineering at the University of Calgary and the father of an RPM user. This paper, Alabood et al. 2024 (Jaswal and Krishnamurthy are listed, respectively, as its fourth and fifth authors), discusses the development and preliminary study of a virtual letterboard, or what the authors call a “hologram” or “HoloBoard.” Here, instead of having a facilitator, or what the authors call a “Communication and Regulation Partner” or “CRP,” hover next to them and hold up the letterboard in their faces, S2C users don a virtual reality headset that projects a virtual letterboard, or “HoloBoard,” in front of them and follows them around wherever they turn their heads. Purportedly, this is an improvement on physical S2C and gives users more autonomy and privacy.
In my previous post, I left off with the author’s conviction that the challenges that autistic individuals have in generalizing a skill learned in one environment to other environments, together with the environmental differences between pointing to letters on physical letterboards vs. holographic ones, means that S2Ced individuals need to go through explicit training in order to transfer the skill of pointing to letters on a letterboard to pointing to letters on HoloBoards. In my previous post, I questioned this assumption, in part, by noting that the generalization difficulties in autism don’t include difficulty spelling words in one environment vs. another, and that:
hyperlexic autistic kids have been observed spelling words in all sorts of contexts, and without any explicit instruction: from refrigerator letters to sidewalk chalk, to letters they form out of playdough.
Given the design of the HoloBoard, there’s good reason to think that transitioning from typing on a letterboard to typing on a HoloBoard is much simpler than transitioning from refrigerator letters to playdough. As I noted in my earlier post, the researchers meticulously replicated in virtual space the physical letterboards the kids are used to, down to the precise arrangement of the letters. Indeed, given that the HoloBoards so closely replicated the physical letterboards, it’s hardly surprising that the training, in fact, went quickly—surprised though the authors profess to be about this.
The training involved going back and forth from a physical letterboard to a virtual one, first selecting an individual letter (as requested by the researcher), and then spelling an entire word first with visual cues (the target letters would pulse), then without them. Choosing names with which they assumed the participants would be familiar—CAT, DOG, FISH, TIGER, TURTLE—the authors report that “16 of the 23 participants completed the training module with their average training time being less than 10 minutes.”
And they find all this impressive, even given the extremely slow but improving typing speed:
Although the speed with which they spelled on the virtual letterboard was, on average, about two thirds as fast as on the physical letterboard, that most participants got faster at using the virtual letterboard across phases is impressive given that they were adjusting to a new training environment and were asked to do increasingly demanding spelling tasks.
They found it, in addition, reassuring that “a number of” their participants “explained,” via S2C-generated output, “that with further practice they ‘could get really good at it.’”
Feedback provided via S2C, of course, is more likely to be coming from the CRPs than from the participants themselves. And this leads to an alternative account for why the participants’ typing on the HoloBoard was slower than their typing on the letterboard: they had to get used to a different set of facilitator cues (more on that below). With further practice, they might indeed “get really good at it.”
This S2C-generated comment isn’t the only instance in which the participants “provided feedback” to the researchers. Arguably, the biggest moment for participant feedback was before the study actually began, when prospective participants decided whether to give consent to participate in it. This particular study, on one hand, might seem short and innocuous: all it had participants do was point to letters on physical and virtual letter arrays to spell a small set of short words. However, inasmuch as the comprehension deficits in non-speaking autism (see my previous post) mean that letter pointing in S2C is relatively meaningless for those subjected to it, the same goes for the letter pointing activities in the experiment.
Furthermore, there are two key ways in which interacting with the HoloBoard is potentially more aversive than interacting with a CRP. First, participants had to wear a headset the entire time; second, they had a virtual letterboard constantly hovering in their lines of sight, whichever way they turned. Many people might find this uncomfortable, particularly those with the sensory issues that often accompany autism. Indeed, the authors report that “3 [participants] could not tolerate the device long enough to engage with the application, likely due to their sensory sensitivities;” that “[t]hree participants found the experience to be overstimulating but managed to complete a few phases by taking short breaks,” and that one “indicated” partway through the trials “that they were too tired to continue.” Another person for whom the HoloBoard didn’t work out, curiously, was someone whose caregiver indicated that they interacted with the physical letterboard via peripheral vision. Since the HoloBoard is programmed to remain in front of the user’s face, it doesn’t lend itself to peripheral viewing—as unlikely as it is anyone can distinguish letters through peripheral vision (see Janyce’s post on this subject).
So how did participants give consent for this possibly aversive study? Through S2C-generated output: i.e., output that may well have been totally under the control of their CRPs. Given all the other things that Institutional Review Boards (IRBs) get worked up about when deciding whether to approve clinical research, it’s ironic that FCed consent procedures don’t raise red flags. It would seem that, for all their concerns about safeguarding people with disabilities, the programs that certify IRB members are ignorant of some of the greatest threats to some of the most vulnerable of such people.
Participants also purportedly provided feedback to the researchers about the design of the HoloBoard: “HoloBoard was a result of more than a year of consultations with nonspeakers” (and with their CRPs and others). Specific suggestions attributed to the participants included “interest in the visuals and sound effects,” with one participant purportedly spelling "what I most liked was the sounds that went when pressing and clicking [the buttons]." Reporting such inputs allows the researchers to check off that box that so many in disability studies fields now prioritize, i.e., making sure your research is “inclusive” or ”participatory;” in other words, mindful of the “nothing about us without us” mantra of the disability rights movement. One of the perverse effects of this new priority is that it’s incentivized those who do research on minimally-speaking autism to include non-speaking autistics in the only way that seems (to the more deluded or denialist of these researchers, anyway) to work: namely, through FC/RPM/S2Ced output. And what this means, more broadly, for any research on non-speaking autistics is that a specific subset of non-speakers, namely those subjected to FC/RPM/S2C, will be the ones recruited to provide “feedback” on everything from experimental design to future directions for research—feedback that is most likely coming, ironically, from their non-autistic, non-clinically-trained CRPs.
Even more perversely, some of the feedback that the participants in this study purportedly provided could be used to motivate the subjecting of growing numbers of minimally-speaking autistics to virtual letterboards:
It was “cool because [their] CRP doesn’t have to be with [them]” but "make[s] it harder for [the CRP] to prompt and regulate [them].”
"[I]t felt amazing to be independent. I loved how easy the letters were to access. I am thinking it will get easier with practice."
"I think it was nice to not have to get a CRP [to hold the board], that I was the only one who could use it."
"I liked many things: in particular, I like the inclusion and the independence from mom"
[It could] “improve the education standard for. . . nonspeakers.”
“[T]he headset on made the distractions seem to really disappear. . . it made me greatly focused, having my visual field more restricted."
All this output, generated as it was through letter selections (it’s unclear how much of it was generated through held-up letterboards and how much of it through interactions with the HoloBoard), may well have been controlled by the CRPs.
Let’s turn, now, to the findings.
At first glance, they might seem impressive. To begin with, the HoloBoard environment eliminated one major opportunity for CRP influence. In RPM/S2C, it’s up to the CRPs to determine when a letter selection has been made: they decide which letters to call out and transcribe, which allows them to ignore certain letter selections and to call out different letters than those that were actually selected (something often evident in videos of S2C). In the HoloBoard environment, it’s the HoloBoard that reads out and transcribes the letters.
Another way in which CRPs regularly influence letter selection is through board movements, including pulling the board away and then repositioning it. But the researchers report that, though the HoloBoard environment allowed them to do these things, the CRPs in the experiment generally chose not to:
We observed only one CRP who initially moved the virtual letterboard to make room for the physical letterboard. Other CRPs simply left the virtual letterboard in the same place even when the participant was touching a letter on the physical board.
In addition, of the 16 (out of 23) participants who completed the study, the majority performed well in the testing phase, where they had to spell words that were spoken out loud to them, and which were mostly not words they had trained on in the study’s training phase. The authors report that:
“73% of participants were able to complete Phase 5 with success rates of 100%, while 10% scored between 63% to 71% in that phase. The total average Phase 5 success rate for those who completed that phase is 95.8%.”
“when participants were allowed to continue interacting with the virtual letterboard after the formal session, 14 participants spelled lengthy sentences, sent emails via the application, or offered their feedback on our system solely using the virtual letterboard.”
14 participants “quickly learned to self-correct typos using the backspace and space bar.”
“many participants were able to perform optional, more advanced tasks such as providing independent full sentence feedback on our system using solely the virtual letterboard” (despite the fact that they couldn’t select letters independently with the physical letterboard)
5 participants were able to use HoloBoard in “solo mode” in which the CRPs removed their headsets and couldn’t interact with the virtual environment. In solo mode, they “spelled short answers independently... when asked to do so (e.g., spell your name, home city, parents’ names, favorite movie, age).”
But on closer inspection, these results are not as impressive as they first appear. First of all, in the “multiplayer mode” that was used during the training and test phases, the CRPs had access to the virtual reality and could “see the virtual letterboard” and “gauge what letters the nonspeaker is attempting to interact with.” Based on what they saw, the CRPs could, and did, deploy what the authors call “verbal prompts” and “cues,” and this extended into the testing phase. As examples of prompts, the authors give "resist the mental loop," "scan the board," or "find your next letter”; as examples of cues, they give “repeating the word that needed to be spelled, verbally guiding the participant to the location of the next letter, or spelling out the next letter.” Such prompts and cues occur regularly in S2C and often suffice, on their own, to influence letter selection—even if they aren’t as explicit as prompts that provide verbal guidance to a letter’s location or that spell out the next letter.
Indeed, there are numerous published videos of FC/RPM/S2C where the typing is done on a stationary device, with the S2C merely providing occasional verbal prompts and cues, along with body language cues like moving their torso, head, or hand in the direction of the next letter, or giving a subtle non-verbal signal when a wandering index finger has arrived at the correct letter (see, e.g., here). Some of these published videos come from the Telepathy Tapes podcast, where the non-speaker types out a number or word that only the CRP was shown, such that the only other explanation, other than CRP control through subtle verbal and/or gestural cues, is telepathy (see Janyce’s analysis of this one). Telepathy is a topic that Jaswal has thus far avoided discussing. Nor do he and his co-authors discuss the possibility that the CRPs might be providing gestural cues in addition to verbal ones to their participants. There is nothing in the HoloBoard environment to rule this out.
Instead, the authors simply stress that there was less prompting and cueing in the test phase.
The number of cues dropped to 5 in phase 5. Participants needed prompts for 11.5% of interactions(a total of 27 interactions)... In the testing phase (Phase 5), we observed only 7 prompts in total.
Now let’s look at what the test phase required participants to do. As the authors explain:
[P]articipants were “required to spell five words (their first name, PIG, DUCK, CAMEL, DONKEY, spoken aloud by the researcher), one at a time, solely on the virtual letterboard and without visual cues or mirroring. In this phase, we expected the words they were asked to spell would be familiar, but only one (DUCK) had appeared in a previous phase.
One thing that’s striking here is just how short these verbal items are compared to the length of most S2C-generated messages. The authors have suggested that the reason for this relates to the purported cognitive demands of learning to use the HoloBoard, and that “communicating [on the HoloBoard] in a more generative matter” (which presumably includes longer sentences) might be something that would be achieved with “further training.” But assuming that the participants’ literacy skills are genuinely intact, you’d think, once they learn to type any letters on the board, longer, “more generative” sentences would come along automatically for the ride.
Second, some of the words/letter sequences were familiar. Two of the five short words the participants were asked to spell were their first name and the word DUCK, which had also appeared in the training. Given that the participants “had an average of 5.26 years (range: 2.25 to 13.33 years) of using the letterboard,” some of the other words, as well as some of the letter sequences (particularly “c-k”) may have been familiar as well.
As an aside, it’s quite telling that, despite the participants’ multi-year experience with S2C, the “CRPs reported that none of our participants could reliably spell on a physical letterboard unless it is held by a CRP.” This, along with everything else, suggests they were highly dependent on facilitator cues.
Another important consideration is the difference between spelling and understanding. Participants were only asked to spell words; not to use them in meaningful ways. Nor did the verbal prompts to spell specific words require much in the way of comprehension: with the exception of the prompt to spell one’s first name, these were simple dictation exercises in which the word to be spelled was spoken out loud. As for the prompt for the participant’s name, this common question may have been a highly familiar prompt with a highly conditioned response.
Taken together—the simple familiar words, the minimal comprehension demands, and the opportunities for the kinds of CRP cueing that can completely control letter selection (cf. the Telepathy Tapes), the fact that “73% of participants were able to complete [the test phase] with success rates of 100%, while 10% scored between 63% to 71% in that phase” isn’t convincing evidence for the HoloBoard, as deployed in this experiment, as an authentic way for non-speaking autistic individuals to communicate.
The more seemingly convincing HoloBoard interactions only occurred after the actual experiment was over: namely, the 14 participants who “spelled lengthy sentences, sent emails via the application, or offered their feedback on our system solely using the virtual letterboard” and “were able to perform optional, more advanced tasks such as providing independent full sentence feedback on our system using solely the virtual letterboard.” Since these post-experimental claims are anecdotal, they only carry so much weight—particular as the authors don’t provide any details about either the message content or their communicative context, or what the CRPs may have been providing by way of verbal prompts, cues, and gestures.
The only time the participants’ interactions with the HoloBoard were definitively unprompted was when they occurred on “solo mode” (where the CRPs removed their headsets and couldn’t interact with the virtual environment). Only five participants used the letterboard in solo mode, and here all the authors report is that they “spelled short answers independently... when asked to do so (e.g., spell your name, home city, parents’ names, favorite movie, age).” Given
the many years of practice that these participants with S2C (5.26 years, ranging from 2.25 to 13.33 years)
the likelihood that these were highly familiar-sounding questions (even if not fully understood) to which the participants had frequently spelled out answers
the well-attested rote memorization skills in autism
the significant comprehension deficits that researchers have found in non-speaking autism (see my previous post)
...it’s much more likely that these results reflect memorized letter patterns rather than intentional, independent communication—as well as the CRPs’ notions of what their clients’ favorite movies were, as opposed to their clients’ actual cinematic preferences. If the authors had really wanted to establish whether the HoloBoard users were authentically communicating their own messages, they could have conducted message-passing tests. Both in eschewing message-passing tests, and in presenting results that reflect memorized letter patterns rather than intentional, independent communication, Jaswal et al.’s paper resembles his two previous papers on letter pointing by non-speaking individuals with autism (Jaswal et al., 2020; Jaswal et al. 2024).
But surely this is not Jaswal’s last paper, and the future directions the authors propose for this latest technology, based both on their questionable results and the problematic, S2C-generated feedback they obtained from their participants, are alarming. Motivated by the fact that the CRPs didn’t need to hold the HoloBoard, which suggests to the authors that it “may have affordances that facilitate more independent typing,” the authors seem to be suggesting a full-scale replacement of traditional S2C with HoloBoard-based communication that seems to combine the worst elements of each:
“Head tracking could be exploited to ensure the virtual letterboard remains in the nonspeaker’s field of view even when they move.”
CRPs will continue to be present, “dedicat[ing] their focus to other aspects of supporting a user, such as promoting attention and regulation,” even when the HoloBoard user might wish to “engage in private conversations with a third person.” (They propose to explore ways that “would allow a CRP to support their nonspeaker while allowing the nonspeaker” during these private conversations.)
Or possibly CRPs would be replaced by a virtual CRP: “a personalized virtual CRP within the virtual environment. The virtual CRP would emulate the behaviour and appearance of a user’s human CRP to provide attentional and regulatory support.” But the virtual CRP may essentially replicate the sorts of message-controlling cues done by the real-world CRP: “Machine Learning (ML) techniques could be used to train the virtual CRP based on observations from a user’s real-world interactions with their human CRP.” While the authors don’t mention that the “virtual CRPSs” might learn to mimic the human CRPs’ prompts and board movements (part of how CRPs unwittingly control letter selections), in an article about this paper in IEEE Spectrum, a magazine published by the Institute of Electrical and Electronics Engineers, Jaswal et al. write that “This virtual assistant [which now has a name: “ViC”]... can demonstrate motor movements as a user is learning to spell with the HoloBoard, and also offers verbal prompts and encouragement during a training session.”
Finally, in what are even more powerful venues for message control, “[l]arge Language Models (LLMs) could be integrated to reduce the effort needed to communicate thereby reducing user fatigue. For example, such a system would allow the user to produce elaborate responses by providing just a succinct prompt to an LLM.”
Many of us predicted these last two items would next be on Jaswal’s agenda. In other words:
Machine learning that allows S2C’s message-controlling prompts and cues to be taken over by machines and safely hidden away within their obscure, machine-learned, neural networks from FC critics and others concerned about the communication rights of autistic non-speakers.
LLMs that elaborate the short messages authored by actual or virtual CRPs into messages that are even more filled with predictable blather and bromides, and even more removed from what minimal speakers actually want to communicate, than FC/RPM/S2C-generated output is.
REFERENCES
Alabood, L., Dow, T., Feeley, K. B., Jaswal, V.K., Krishnamurthy, D. From Letterboards to Holograms: Advancing Assistive Technology for Nonspeaking Autistic Individuals with the HoloBoard. CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems Article No.: 71, Pages 1 - 18 https://doi.org/10.1145/3613904.3642626
Jaswal, V. K., Wayne, A., & Golino, H. (2020). Eye-tracking reveals agency in assisted autistic communication. Scientific reports, 10(1), 7882. https://doi.org/10.1038/s41598-020-64553-9
Jaswal, V. K., Lampi, A. J., & Stockwell, K. M. (2024). Literacy in nonspeaking autistic people. Autism : the international journal of research and practice, 28(10), 2503–2514. https://doi.org/10.1177/13623613241230709
