- how we built Avaz

The research behind Avaz

Tablets like the iPad are revolutionizing the way children with special needs communicate. To tap the true power of these devices, we need relentless invention: in technology, in special education and in therapy.

Avaz, the speech therapy app for autism, is the culmination of four years of research, involving more than 20 speech therapists and educators, 25 schools, and 500 children across two countries. The innovations behind Avaz have received widespread acclaim and recognition, including being named to MIT’s TR35 list of transformative innovators in 2011 .

Here are a series of short monographs that describe some of Avaz’s research under-pinnings. Please contact us if you need any more information.

Language development versus incentivized communication

There have been two broad approaches to AAC over the last several years. Incentivized communication is a system of communicating using stock sentences or phrases, and its objective is to simplify and support activities of daily living. For example, communicating hunger, thirst, tiredness or illness; indicating preference in food or clothing; and requesting assistance to go to a particular place: these are all quotidian activities that everyone performs. For a non-verbal child, especially one that may be dependent on the assistance of caregivers for performing these activities, it is necessary to communicate efficiently and unambiguously these necessities. The incentive for successful communication of such messages is very direct: it results in increased comfort or well-being.

Language development, on the other hand, has completely different aims. It assumes, at its core, that the primary purpose of communication is to enable and facilitate interaction between individuals. In other words, the primary purpose of communication is to have conversations. These conversations can have different functions – satisfying curiosity, establishing relationships, building an identity – and also for performing activities of daily living.

Because language development, in some sense, subsumes incentivized communication, it is now widely accepted in the speech and language therapy community that any AAC system should promote language development first. Only facilitating incentivized communication is widely seen as being too limiting, because it does not permit the re-use and generalization of words and concepts, that are, in some sense, key to the concept of language.

Language development happens through the acquisition of a core vocabulary, complemented by the development of a peripheral vocabulary. A core vocabulary is a basic set of words which enable a communicator to stitch together other words to convey specific meaning. These are often the first words that a child acquires while learning their first language. For example, the words ‘hi’ and ‘bye’ are learnt in the context of responding to the presence of people. The verbs ‘go’, ‘look’, ‘like’ and ‘help’ are learnt in the context of interacting with the world around them. More abstract words like ‘stop’, ‘no’, ‘that’, ‘more’ and ‘all gone’ are used to convey specific preferences and states. Core words could also be associated with specific actions; for example, a set of core words associated with food may include ‘eat’, ‘drink’, ‘some’, ‘more’, ‘all done’ and ‘yummy’.

The power of a core vocabulary is that it provides combinatorial power to language. If a child is able to manipulate a small set of words capably, the same skills can be generalized to manipulating a larger set. A core-words-centric approach builds this manipulative power first before expanding a vocabulary.

This is the approach taken in Avaz, an effective autism ipad app. The vocabulary in Avaz is organized into three grades: ‘Getting started’, ‘Basic’ and ‘Advanced’. In each of these grades, the first screen shows a set of core words, and a set of categories that build a peripheral vocabulary around different contexts. Thus, a child can construct complete sentences at each level, thereby harnessing the infinite potential of language.

While language development is the primary and central motive of Avaz as an AAC tool, we also recognize the importance of incentivized communication in a special needs environment. It is hardly fair to expect a child to be able to learn the meaning of each word in a sentence like ‘I want to go to the bathroom’ before they are provided a way to express the idea. The compromise is achieved in Avaz through the use of a ‘Quick’ vocabulary. At any stage, a child may access, through a ‘Quick’ button, a set of phrases and sentences that are relevant to their daily lives.

In a broad sense, therefore, Avaz facilitates both language development and communication for the purpose of daily living.

Communication challenges in children with autism

Communication and language developmental delays characterize autism. Therefore, AAC is particularly important for children with autism, but the priorities for an autism-focused AAC system must be different from systems meant for other disabilities.

One observation that has defined the AAC approach to autism over the last several years is the discovery that children with autism have a visual retention and recollection capability that is far higher than their verbal capability. So the standard process for language development in children with autism has been, in some sense, to ‘short-circuit’ the verbal pathway through the visual pathway; in other words, facilitating picture-based communication. The idea is to build generalizable concepts, such as words, word order, and association of sentences with the real world, through pictures. This is also the broad idea behind Avaz.

There are, however, a few important things that must be kept in mind while designing a picture AAC system for language development in children with autism. The first is minimizing distraction. It is all too common for children with autism to fixate on the wrong elements of a picture; for example, if a photograph of a yard includes a dog in it, it is likely to confuse the child about what ‘single concept’ the photograph conveys. This negates the principle of using the picture to replace, or reinforce, a word.

The approach taken by Avaz (and this is also adopted by many other apps) is to use stylized images, or symbols. Avaz uses the Symbolstix picture library, which is rapidly becoming a standard in picture-based communication.

Sometimes, though, it is very helpful to ‘literalize’ pictures for a child with autism (without, however, introducing distraction). A child may be able to retain and recall the word ‘father’ much better if the picture is a real photograph of the child’s father specifically, instead of a generalized photograph of a father. This is the main reason that Avaz leverages the capability of an iPad to take and store photographs, which can be used instead of symbols as appropriate.

A major complexity of language is the concept of choice. When constructing a sentence, people without autism are able to hunt their vocabulary for the right word, considering and discarding several alternatives, in just a moment. For a child with autism, however, the action of choosing one word out of many, when they are all represented with pictures, can be a challenge. This is why, for example, systems like PECS have very well-defined phases by which pictures are introduced to children in a distraction-free manner: starting with a single picture, moving on to two or three, and at each step reinforcing generalized language acquisition skills. Avaz supports systems like PECS by allowing a therapist to configure the screen to be very minimal to start with – just one large picture, without even a message box – and progressively moving to two, three, four, eight and fifteen pictures per screen. This approach has demonstrably built vocabulary and language skills in children with autism.

One further feature required by children with autism is reinforcement. A child with autism benefits from explicit, multi-sensory confirmation of basic actions such as scanning, choice and selection, because they help imprint a picture-word association very strongly in a child’s mind. This is why we built in some of Avaz’s most autism-friendly features: a zoom-in animation reinforcement on selection, speaking out every word that is selected, and moving it to a message box where it is accessible even after it has been selected. Selection of a picture (and associating it with a word) in Avaz is, therefore, a tri-sensory activity: touch, sight and sound.

Motivating communication: a therapist’s role

No AAC system available today can replace the role of a therapist or skilled educator: the role of apps like Avaz is to augment, not supplement, a specialist. When designing Avaz, we asked ourselves: how could we help a therapist make a session with a child with autism a lot more effective with Avaz?

The major factor driving speech therapy sessions in children with autism is motivation. A conversation with a child with autism cannot revolve around a fixed curriculum, or what the therapist decides to talk about. It must, instead, conform to what the child wants to say.

Of course, we have no way of knowing in advance what a child would want to talk about on a particular day. And so, with any AAC system, the therapist wants to customize it quickly, efficiently, and effectively. That was our primary objective with Avaz, the autism ipad app.

Consider, for example, that a therapist is accompanying a child to lunch. The child wants to talk about the food they will be eating today, which is spaghetti, and tries to find it in the vocabulary hierarchy. But horror of horrors! – the vocabulary does not have a picture of spaghetti in it.

If the therapist is able to add a picture of spaghetti within seconds, the conversation can continue unimpeded and without lost momentum. It may seem hard to believe, but for a therapist to add a new word to the vocabulary in Avaz would take under 15 seconds!

Another important consideration for a therapist is the ability to quickly get started with an AAC tool for a specific child; the setup time should be as small as possible. This is tricky, because depending on the child’s cognitive level, age and disability, a suitable vocabulary has to be first chosen. The act of choosing the vocabulary (by a therapist) and using the vocabulary (by the child) are substantially different; however, both are equally important.

The solution we’ve selected in Avaz for this dilemma is to provide three graded vocabularies for language development, called “Getting Started”, “Basic” and “Advanced”. These three vocabularies, along with one grade called “Quick” for incentivized communication, are present as folders in the root screen. A therapist could try each of these vocabularies to assess which suits the child best; then, the therapist can select that vocabulary to be the ‘home screen’, so that the child will not be distracted by the other vocabularies. The therapist could even add new vocabularies to the root level, and select one of those for the child.

Another therapist-friendly feature that Avaz supports is the ability to password-protect the Settings and Customization screens, so that a child is restricted (if necessary) to using the device, and not reconfiguring it.

Transitioning to text: a key step in mainstreaming

Whether we like it or not, the world around us mostly communicates verbally; reading, writing and speaking are the mainstream channels of academic and professional communication.

That’s why, though researchers recognize and promote the use of pictures in communication for children with autism, they simultaneously also recognize the importance of learning and using the alphabet.

There are a few well-established methodologies that allow children with complex communication needs to be gently introduced to text. For example, the development of sight reading. A child learning to read, who has persistent difficulties understanding the abstractions of alphabets and spelling, may instead choose to skip over the construction of words, and directly learn to recognize the shape of these words instead. That is to say, the child associates a word with a literal picture, as well as a kind of stylized picture which is, in reality, the alphabetical representation of the word.

How can an AAC tool help in sight reading? One feature that Avaz has is the ability to gradually adjust the relative sizes of pictures and the captions below them. As a child becomes more and more familiar with pictures, and is able to use them effectively in language, a therapist or teacher could de-emphasize the picture and emphasize the way the corresponding word is written, by adjusting the caption size.

A child who is exposed to a variety of high-tech communication aids is also likely to have been exposed to a computer, and is probably familiar with the use of a keyboard. For children who are at this level of linguistic ability, the need for a keyboard-based, spelling-based AAC tool becomes important. It is important that the keyboard presented through a touch-screen to a child familiar with a real keyboard, have a strong resemblance to the physical object, in order to avoid distraction. This means including numbers, special keys like ‘Enter’ and ‘Backspace’, and a consistent look and feel to the keys.

Spelling is often quoted as the biggest problem that children with complex communication needs encounter when using text to communicate. It has been proven that assisted spelling, in the form of word prediction, significantly reduces the ‘barrier to entry’ for children who are transitioning from pictures to text. This is because they are able to gradualize the shift from a ‘recognition-based’ system (that is used for word creation in picture mode) to a ‘construction-based’ system (that is used when spelling out a word through alphabets). When a strong prediction system is incorporated into an AAC aid, the child can start construction by spelling the first one or two letters – and then select the word from a list, instead of remembering the six or seven further letters that must be sequenced in order to spell out the entire word.

Avaz, the speech therapy app for autism, goes one step further in making this transition even smoother, by removing the requirement that the child achieve high proficiency in sight reading before beginning with text. This is by marrying together prediction and pictures: in addition to showing a child the spelt-out word form of a predicted word, the prediction bar in Avaz also shows a picture of the word (in many cases). This allows the process of recognition to proceed through the familiar pathways of picture recognition, instead of the newly-forming pathways of sight reading, in the child’s brain.