Assessing comprehension of health-related texts in non-native and native English speakers

Assessing comprehension of health-related texts in non-native and native English speakers

Khushboo Anup Agarwal

13/09/2022

Background — Health written materials are more often complex to comprehend if they mismatch the reading ability of people in the target audience. We need to consider how to make text accessible, by considering individual differences that affect comprehension of written health materials. Surprisingly, there are very few studies that indicate how non-native English speakers and native English speakers differ in comprehension of written health texts. Methods — A total of 557 participants were studied in the present study. In the study, participants were asked to respond to multiple-choice questions that were designed to examine understanding of 25 health texts with different text properties. Each participant responded to tests measuring individual differences in demographics, reading strategy, vocabulary, and health literacy. Findings — Using mixed effects logistic regression analysis, we found that non-native English speakers and native English speakers have different accuracy of responses for written health texts. Effects of vocabulary skills and text readability were significant. These effects were different for different language groups. Native speakers of English with higher scores on vocabulary were more likely to make correct responses to written health texts. Native speakers of English were more likely to make correct responses to written health texts as text readability increased. Conclusion — In future, experimental studies should look at the effects of training to improve vocabulary on reading comprehension for different language groups. Alongside consider sources of variances due to individual differences and text properties for different language groups.

reading comprehension, health literacy, individual differences, language groups.

Design
We conducted experimental research on factors that influence the response to written health information, aiming to answer the research question:
RQ.1 How does the reader’s attributes such as age, vocabulary skills, health literacy, reading strategy skills, along with text features, interact to predict the comprehension of health information in written texts for native language English speaker and non-native English speaker?

We conducted the study to test the hypotheses:
1. Comprehension will be better for people with higher scores on reading skill, vocabulary and health literacy. Comprehension will be lower as age increases.
2. Comprehension will be better for responses to texts which are higher on measures of readability, cohesion, word frequency, referential cohesion, and passive sentences.
3. There will be differences between native and non-native speakers. Comprehension will be better for native speakers. The effect of age, reading strategy, vocabulary skills, and health literacy will be different for different language groups. The effect of cohesion, readability, word frequency, referential cohesion, and passive sentences will be different for different language groups.

Ethical approval. The data collection plan and study design were reviewed and approved by a member of the Psychology Department Research Ethics Committee.
Pre-registration. The study has not been pre-registered.

Participants
Participants were recruited using primarily opportunity and snowball sampling. Participants were invited using social media such as Facebook, Instagram, and WhatsApp. We aimed at recruiting Bilingual/Multilingual Indian Residents (18+) who have access to the internet. We collected 201 responses, but only 112 participants were included in our analysis due to incomplete forms by other respondents. Our criterion for including participant data in analyses was that they had to complete 80 percent or more of the survey. We had 112 responses, but we did not test any of the respondents who were aged 100. We had three respondents who were aged 100 removed from our data set, leaving us with 109 observations. To enable a comparison between native and non-native speakers of English, we combined the data on responses from Indian and Chinese non-native speakers with data on responses from native speakers of English collected previously by supervisor Rob Davies. Thus, we had a large sample size of 557 participants for analysis. We did our final analysis on 557 participants with minimum age of 18 and maximum age of 81. Average age range in sample was 28, skewing towards younger population. The sample consisted of 392 females, 160 males, 1 non-binary, and 4 prefer not to say. There were 273 participants who spoke English as their first language and 284 participants who spoke English as their second language.
All participants were debriefed, and steps were taken to ensure confidentiality and anonymity.

Materials
We collected information on attributes of participants and linguistic properties of texts to see its influences on accuracy of responses made by participants to questions related health information. To measure participants attributes, we assessed demographic details, and participant’s vocabulary knowledge, health literacy, and reading strategy. Health texts differed in their linguistic properties, as measured by word frequency, readability (Flesch score and grade level), number of passive voice sentences, cohesion, and referential cohesion (Coh-Metrix).
Vocabulary knowledge.
The Shipley Vocabulary Test (Shipley et al., 2009) was used to test participants' vocabulary knowledge as it predicts 39-45% of variance in reading comprehension (Landi, 2010). The test includes questions in a multi choice question format, with incorrect and correct answers. Each question contains a word followed by four options—one of which is the correct meaning of the word. The higher the points, the higher the level of vocabulary.
Health Literacy.
The Health Literacy Vocabulary Assessment (HLVA) developed by Ratajczak (2020), adapted for online presentation by Chadwick (2020) was used to test participants’ health literacy. The adapted version of the HLVA contains 16 multiple-choice word items. The test consists of multiple-choice questions with incorrect and correct answers. Each question contains a word followed by four options. The participant must select the correct meaning of that word. High scores on HLVA indicate high health literacy vocabulary.
Reading strategy.
To determine participants’ motivation for reading and understanding reading strategies, we used Calloway’s (2019) third sub-test: Desire for Understanding and Reading Regulation Strategies. The items have been developed to measure the extent to which readers are willing to expend cognitive effort to understand a written text (Van den Broek et al., 2001). A higher score on this measure predicts better comprehension (Calloway, 2019).
Demographics.
We collected participants’ demographic characteristics: gender (coded: Male, Female, non-binary, prefer not to say); education (coded: Secondary, Further, Higher); and ethnicity (coded: White, Black, Asian, Mixed, Other); age; native language.
Health information stimulus text sampling.
Comprehension passages are selected based on previous research paper by Davies and colleagues (in prep.) In total there are 25 comprehension passages. However, reading 25 passages in one sitting could lead to fatigue in the reader. Therefore, we created 5 sets of 5 comprehension passages. Each set contained 5 passages, which were randomly given to participants. The comprehension passages were then followed by questions in a multiple-choice question format. The response to each question is either right or wrong, which indicates whether the reader understands the passage. The questions have been constructed in ways to ensure that questions probed for the most important information in each text, such as who the information was relevant to, who was involved in diagnostic or treatment procedures, and the risks and benefits of different options. The questions were constructed in a manner that could not be answered by matching or referring to the text but required text-level and interpretation-level comprehension processing to correctly choose answer options (Kintsch, 1994).

Lancaster University

Excel spreadsheets - .csv
R Script - .r

Agarwal2022

Huzaifah Adam, Coco, Alex Myroshnychenko

Open

This work is based on Kintsch, W. (1994). Text Comprehension, Memory, and Learning. American Psychologist, 10.
White, S., Happé, F.M., Hill, E., & Frith, U. (2009). Revisiting the Strange Stories: Revealing
McNamara, D., & Magliano, J. (2009). Chapter 9 Toward a Comprehensive Model of Comprehension. Psychology of Learning and Motivation, 51, 297–384. https://doi.org/10.1016/S0079-7421(09)51009-2
O’reilly, T., & Mcnamara, D. S. (2007). Reversing the Reverse Cohesion Effect: Good Texts Can Be Better for Strategic, High-Knowledge Readers. Discourse Processes, 43(2), 121–152. https://doi.org/10.1080/01638530709336895

English

Data

Developmental, Other