Mary Rudner, Department of Behavioural Sciences and Learning, Linköping University

2008-2013

The project aims to explore how congenital deafness and experience of sign language influence linguistic and non-linguistic working memory processing in the visuospatial domain, and thus test and develop the ELU model (Rönnberg, Rudner, Foo & Lunner, 2008). ELU stands for Ease of Language Understanding and the model describes how working memory supports multimodal communication. There are three parts to the project. Study 1 examines how visuospatial working memory capacity is influenced by sign language use and deafness. Study 2 examines how processing phonologically relevant characteristics influences visuospatial working memory capacity and Study 3 examines whether linguistic and non-linguistic visuospatial skills are predicted by linguistic and non-linguistic visuospatial working memory capacity.

The project is based on the use of two different sign languages: Swedish Sign Language (SSL) and British Sign Language (BSL), which have relatively little overlap in terms of lexicon. This means that there are enough BSL signs that are unknown to SSL users and enough SSL characters that are unknown to BSL users for it to be possible to create an experimental stimulus material with signs which for sign language users have 1) both semantic and phonological content (familiar signs), and 2) phonological but not semantic content (unfamiliar signs). A third material comprises made-up signs which look like signs to the uninitiated but whose composition does not follow linguistic rules. A fourth material consists of recordings of a person catching a ball, where there are hand and arm movements but no linguistic content. These different materials are used as stimuli in different working memory tests.

In addition to these tests, a battery of independent cognitive tests is also included in the study. These tests are administered to 1) deaf persons who have been using sign language since birth, 2) hearing persons who have been using sign language since birth, and 3) persons who are hearing and do not know any sign language. This allows us to distinguish the effects of sign language knowledge and deafness in our results.

The purpose of the project has not changed but we have been able to make some additions. Among other things, we have investigated how reading comprehension and imitation abilities are related to linguistic and non-linguistic working memory in the visuospatial domain and the effect of aging on working memory.

The project's three main findings and a discussion of these

1. Both congenital deafness and experience of sign language have an effect on the way in which sign-based stimuli are processed in working memory. When deaf sign language users perform a working memory task in their own language, ie, British Sign Language, they are less sensitive to high working memory load than hearing sign language users. For both deaf and hearing sign language users, working memory capacity is greater for familiar than unfamiliar signs.

2. Somewhat surprisingly, phonological encoding does not seem to play a prominent role for sign language users (either deaf or hearing) in working memory processing of signs. In other words, neither of these groups performed better when stimuli had a legal phonological structure (unfamiliar signs) compared to when stimuli lacked phonological structure (made-up signs). This has implications for the ELU model (Rönnberg, Rudner, Foo & Lunner, 2008) that predicts phonology-based working memory processing across language modalities.

3. We found no relationship between phonological ability and performance on working memory tasks. However, for the deaf participants, spatial skills were associated with performance on the working memory task under certain conditions. This was true only when the task required spatial processing (matching sign location), but for all three types of sign-based material. This association was, however, negative. Ie deaf people with better spatial ability performed worse on the spatial working memory task with sign-based material. This suggests that a working memory task that requires explicit spatial processing interferes with the natural spatial organization of working memory for sign language that we have found support for in previous studies.

New research questions generated by the project

The project has generated the question: What are the neural substrates underlying linguistic and non-linguistic visuo-spatial working memory function? This question is being examined in an ongoing functional Magnetic Resonance Imaging (fMRI) study. Another ongoing fMRI study is investigating the neural networks supporting perceptual processing of stimulus material similar to that used the present project. Furthermore, a project is in progress designed to further investigate the link between reading comprehension, cognition and sign language in deaf sign language users. Finally, the project has generated the question: How is visual communication, such as sign language affected by visual noise? This is the subject of a new proposal to be submitted to RJ in 2013.

The project's two main publications, and a discussion of these

1. Orfanidou, E., Rudner, M., Cardin, V., Capek, CM, Woll, B. & Rönnberg, J. (in prep.) Deafness and sign language experience contribute separately to efficiency of processing language-like stimuli in visuospatial working memory.
2. Rudner, M., Orfanidou, E., Capek, C.M., Cardin, V., Woll, B. & Rönnberg, J. (in prep.) Working memory task demands interact with the richness of encoding of dynamic visual items.

These two articles report the project's three main results. We plan to submit them to Cognition and the Journal of Experimental Psychology: Learning, Memory, and Cognition in February. The delay in the writing process is due to the first author of article 1 having moved to a lectureship at the University of Crete and currently having a large teaching burden. A total of five journal articles will be generated directly by the project (see starred in the appendix). An article related to the project is to be published in Nature Communications in February 2013 (open access). I am writing a further article related to the project.

Other types of communication of project results, such as seminars, lectures, links to websites

The project results have been communicated at national and international conferences and through lectures for various professional groups (see appendix).
Link to website: www.ibl.liu.se/medarbetare/rudner-mary

Publications

Tidskriftsartiklar (* indikerar direkt koppling till projektet)

*Andin, J., Orfanidou, E., Cardin, V., Capek, C.M., Woll, B., Rönnberg, J. & Rudner, M. (under review). Identical text-based presentation and hand-written response for deaf signers and hearing non-signers cancel out short-term memory differences.

Cardin, V., Orfanidou, E., Rönnberg, J., Capek, C.M., Rudner, M. & Woll, B. (in press). Dissociating cognitive and sensory neural plasticity in human superior temporal cortex.  Nature Communications.

*Orfanidou, E., Rudner, M., Cardin, V., Capek, C.M., Woll, B. & Rönnberg, J. (in prep.) Linguistic and non-linguistic visuospatial working memory: effects of sign language experience and deafness.

*Rudner, M., Orfanidou, E., Capek, C.M., Cardin, V., Woll, B.  & Rönnberg, J. (in prep.) Linguistic and non-linguistic working memory in the visuospatial domain.

*Rudner, M., Orfanidou, E., Cardin, V.,  Capek, C., Woll, B. & Rönnberg, J. (in prep.). Reading ability in adult deaf native signers is positively associated with their ability to judge the grammatically of their native sign language.

*Rudner, M., Orfanidou, E., Cardin, V., Capek, C., Woll, B. & Rönnberg, J. (in prep.). Relative effects of aging on cognition for communication in deaf signing and hearing non-signing adults.

Rudner, M., Orfanidou, E., Kästner, K., Cardin, V., Woll, B., Capek, C.M. & Rönnberg, J. (in prep.). Neural substrates of sign language processing differ partially between Swedish and British signers.

Editorial
Campbell, R., Rudner, M. & Rönnberg, J. (2009). Cognitive Hearing Science: the view from hearing impairment and deafness. Scandinavian Journal of Psychology, 50, 367-369.

Bokkapitel

Rönnberg, J., Rudner, M. & Foo, C. (2010). The cognitive neuroscience of signed language: applications to a working memory system for sign and speech. In L. Bäckman and L. Nyberg (eds.), Memory, aging and the brain: A Festschrift in honour of Lars-Göran Nilsson. London: Psychology Press, pp265-286.

Vetenskapliga konferensbidrag

Rudner, M., Orfanidou, E., Kästner, K., Cardin, V., Woll, B., Capek, C.M. & Rönnberg, J. (2013). Neural substrates of sign language processing differ partially between Swedish and British signers. Theoretical Issues in Sign Language Research (TISLR) Conference 11. University College London, 10th - 13th July 2013.

Rudner, M., Orfanidou, E., Capek, C., Cardin, V., Woll, B. & Rönnberg, J. (2012). Access to linguistic structure enhances visuospatial working memory capacity. Nineteenth Annual Cognitive Neuroscience Society Meeting, March 31 - April 3, 2012, Chicago, Illinois

Rudner, M. (2012). Visuospatial Working Memory – Linguistic and Non-linguistic. HEAD Seminar, Linköping, 14 March 2012.

Rudner, M. (2012). Effects of aging on cognition for communication in deaf sign language users. Invited seminar, Center for Dementia Research (CEDER), Department of Medical and Health Sciences Linköping university, 15 May 2012, Linköping.

Rudner, M. (2012). Remembering sign language: cognitive aging in deaf signers. Invited workshop contribution, ”A Life with Dementia: Personhood,  Citizenship & Identity”, Linköping University 17-19 October 2012

Orfanidou, E., Cardin, V., Hamzah, H.H., Capek, C.M., Woll, B., Rudner, M., Rönnberg, J. (2011). Working memory and visuospatial cognition in deaf signers and hearing non-signers. First International Conference on Cognitive Hearing Science for Communication, June 19-22 2011, Linköping, Sweden.

Orfanidou, E., Kästner, L., Capek, C., Cardin, V., Woll, B., Rudner, M. & Rönnberg, J. (2011). Semantic and phonological processing in the visuospatial domain: Evidence from Swedish Sign Language. Eighteenth Annual Cognitive Neuroscience Society Meeting, San Francisco, CA, April 2-5, 2011

Rudner, M., Orfanidou, E., Capek, C., Andin, J., Karlsson, T., Dahlström, Ö., Kästner, L., Cardin, V., Fransson, P., Ingvar, M., Johnsrude, J., Woll, B. & Rönnberg, J. (2011). Sign Language phonology and its role in neurocognition. First International Conference on Cognitive Hearing Science for Communication, June 19-22, 2011 Linköping, Sweden.

Rudner, M., Orfanidou, E., Cardin, V., Capek, C., Woll, B. & Rönnberg, J. (2011). Relative effects of aging on cognition for communication in deaf signing and hearing non-signing adults. Aging and Speech Communication, Bloomington, Indiana, October 10-12, 2011

Rönnberg, J. & Rudner, M. & Zekveld, A.A. (2010). Cognitive hearing science: Ease of Language Understanding in old age. Hearing Care for Adults 2009 - The Challenge of Aging. Proceedings of the Second International Adult Conference, 67-73.

Andin, J., Rönnberg, J & Rudner, M. (2010). Language modality specific effects on simple spans in deaf signers and hearing non-signers. Second Meeting of the Federation of the European Societies of Neuropsychology (ESN), 2010-09-22 - 2010-09-24, Amsterdam, The Netherlands.

Andin, J., Rönnberg, J & Rudner, M. (2010). Phonological similarity and sensory memory traces modulate span size in deaf signers and hearing non-signers. HEAD Graduate School 3rd Summer Workshop, 14-15 June 2010, Kolmården, Sweden

Rudner, M. (2010). Cognition for communication - hearing impairment and deafness. Seminar at Department of Psychology, Stockholm University, 30 November 2010

Rönnberg, J. & Rudner, M., (2009). Cognitive hearing science: Ease of Language Understanding in old age. Hearing Care for Adults 2009 - The Challenge of Aging. November 16-18, 2009, Chicago Illinois, USA.

Rudner, M., Andin, J. & Rönnberg, J. (2009). Differences in temporal and spatial processing mechanisms in working memory for signed and spoken language. The 11th European congress of Psychology, Oslo, Norway, 7-10 July 2009

Populärvetenskapliga föreläsningar

Rudner,M. (2013). Arbetsminne och tvåspråkighet på tecken och tal, Landstingets tolkcentral, 11 January 2013

Rudner, M (2012). Cognition, Communication and Sign Language. Deaf Team in Östergötland Training Day, 19 April 2012.

Rudner, M. (2011). How does my cognitive ability influence my hearing and communication. Swedish Association of Hard of Hearing People (HRF) 90th anniversary celebration, Stockholm, 6-7 May 2011

Rudner, M. (2008). Working memory for sign and speech. Invited lecture. Annual Psychiatry Meeting, Odigos/Mogård, May 12-13, 2008.

Rudner, M. (2008). Time and space in working memory for sign and speech. Invited lecture. Research seminar within the project "Better interpretation for people with deafblindness", Örebro, November 6, 2008