Ideas for Assessing the Benefit of StreamerTM Automated Captioning
How do we know when an accommodation really benefits the student?
The student:
- Keeps up with classroom pace of learning so it is more similar to class peers
- Is able to perform better on tasks such as following directions without hesitation
- When quizzed on content of a lecture, is able to provide more answers or more accurate answers
- Seems more relaxed about what is going on in the classroom; less tense, angry, confused about missing information
- Requires fewer instances of teacher help or redirection
The most effective medium for student access needs to be determined.
We need to identify the student’s access to verbal teacher information.Access to verbal instruction can also be considered listening comprehension ability in a group setting. Although listening comprehension tests/subtests are available, they may not provide as relevant information as if you determine how well the student is comprehending information typical of his class content.
Functional Assessment of Benefit of Assistive Devices
An Informal Evaluation of Auditory Comprehension of Information with and without Accommodations
Consider completing the JUSTIFICATION SUMMARY – BODY OF EVIDENCE FOR CONSIDERATION OF ASSISTIVE TECHNOLOGY which is a one-page Body of Evidence that can be used to justify Speech-to-Text software. You will likely be familiar with the tests called for on this form. The student needs to be a good reader (reading within one year of grade level with good comprehension and fluency). Low auditory discrimination scores and/or a low functional listening (FLE) score support a recommendation for Interact captioning as an accommodation. The LIFE-R can include input as to how much a current FM system is really benefiting the student.  The Listening Comprehension Data Collection (with and without assistive technology) will likely be your main item for evaluating a trial of the software. The combination of all the scores on the form should provide a picture as to whether the student is obtaining ‘reasonable’ access to verbal instruction – which following a successful trial – can be used to justify a purchase of the software and hardware.
Additional information or other strategies are as follows:
- In a 1:1 setting
The DHH teacher, or someone else who works 1:1 with the student would select one of the student’s text books (i.e., social studies) and read a paragraph. Then ask the student questions about the content of the paragraph, similar to the kinds of questions asked in a class discussion or on a simple quiz. This would be done with and without the Interact speech-to-text translation available, going back and forth between the two conditions a number of times.
The Listening Comprehension Test Adolescent (grades 6-12) is norm-referenced so it is appropriate to use to estimate how a student is performing when listening alone. Unfortunately, the nature of a norm-referenced test is that it really can’t be used in a pre- and post-test fashion as there is only one set of items and the student will possibly be able to remember and do better the second time just through experience.
Another approach would be to use materials like those found here, or passages from the student’s text book comprised of material that has not been covered yet. The teacher would read the paragraph out loud. Control for noise, distance and availability of speechreading cues so that the situation simulates the classroom setting. The teacher could then (1) ask specific questions about the content, or (2) ask the student to recall as many points from the passage after they heard it and then after they heard it and had captioning available (do not allow the student to reference the captions to help with answers). The work ahead for the teacher would be either to develop questions or to count the number of possible salient points, potentially by reading a passage themselves and then jotting down the key points they remember. In an ideal world there would be results from a number of ‘average’ classmates performing these same tasks, or possibly a whole class activity that requires this type of recall of all students for a few passages. This process could be done with say 3 paragraphs alternating use without and then with the captioning available. This exercise will provide the student with active practice for using the text and also give the DHH teacher an estimate of level of benefit. The teacher would have the data for correct answers to questions or improvements in recall of salient points. By the end of this process the student should be able to verbalize if/how/how much benefit is received from the captioning (i.e., confidence in recall, effort required). This technique should be applicable to any student who would potentially be considered a candidate.
- Classroom setting
Obtaining reliable data in the actual classroom is more difficult.
- One way would be for the teacher to do 10 question quizzes after lectures to see the level of comprehension of the material for all students, including the one with hearing loss. Do one or more quizzes without the speech-to-text translation and then the same number when the student with hearing loss has the speech-to-text captions available. This method will not be a popular choice because of the workload on the teacher, unless the teacher is already in the habit of routinely giving quizzes.
- An alternative would be for someone (i.e., teacher of the deaf/hard of hearing) to observe the student in the classroom and take your own detailed notes on the content presented. At a break in the class flow, pull him aside and quiz him on the content. There will need to be an opportunity to quiz teacher-presented content both with and without the speech-to-text translation.  It may be prudent to also ask 2-3 classmates these questions individually to determine not only the level of apparent benefit from having the speech-to-text translation available, but how the student’s ability to process this instruction compares to typical class peers.
- Another idea, although less reliable, would be to have a blank sticky note on the student’s desk with two lines down the center and (+) and (?) and (-)  at the top of each third of the paper. As the instructor asks the class questions the student would hash mark if he thinks he would be able to answer the question if called upon: (+) for yes, (?)  for partially/maybe,(-) for no. It is the student’s honor system that would make this data collection valid. He would have a sticky and record in those classes that he would likely be using the speech-to-text translation. It would be advisable to have him collect this information for a couple days without the speech-to-text translation available and then a couple of days with the text available. Of course if he is strongly biased toward or against using the speech-to-text system this bias will likely be reflected in the data collected.
- In terms of usage, after the student has had the speech-to-text translation available for a number of days and seems comfortable, he could be observed in the classroom setting to estimate and answer to the question: How much is he really using the speech-to-text translation? An observer could observe him and count how many times he refers to the text and for how many seconds each time. This may be important information to consider when analyzing the benefit of the level of improvement observed in the student’s ability to answer questions about presented information.
I hope these ideas help those of you who are trying Interact with a student.  Please provide feedback or great ideas of your own to  karen@success4kidswhl.com
Questions about Interact? Contact Mike Massine, Streamer Support Consultant at streamer-support@success4kidswhl.com or contact 888-963-8991 x4