Integrating visual and semantic cues with language learning

Girl taking part in an eye tracker study

This work package focused on how children use semantic cues such as animacy, causality and intentionality to develop linguistic concepts and knowledge.

1. Infants’ visual preferences and their relation to later language learning

Although we know quite a lot about infants’ understanding of intentions and causality, there is a gap in our knowledge of how perceptual and cognitive biases and abilities in infancy link to later language comprehension and production. For example, how does what infants prefer to look at relate to the kinds of sentence structures they find easiest to learn? How does the animacy of the different participants in a scene influence which parts of the scene infants attend to, and how children learn related word meanings?

In our first set of studies, we examined infants’ early visual preferences for different kinds of event participants. Successful language acquisition relies in part on infants establishing mappings between concrete event components (e.g. who is the agent or patient) and how we describe these events with language. To do this, infants must recognise the differences between agents and patients and between causal and non-causal actions. We presented English-learning 13-month-old infants with silent animations of simple causal interactions between two participants. Across three experiments we systematically varied different components of the events to assess infants’ sensitivity to changes in i) the action performed by the agent, ii) the identity of agents and patients, and iii) the animacy of agents and patients, as well as reversals of the thematic roles of agent and patient in each case. In addition, in a fourth experiment we looked at their preferences for causal vs. non-causal events involving animate and inanimate participants. Eye tracking data was used to examine changes in infants’ attention to agents and patients, and causal and non-causal scenes as the different types of events unfolded. We found that infants were sensitive to changes in various properties of the events (the action, participant identities and their animacy) as indicated by their relative focus on the agent vs. the patient of the events, but the conditions under which they showed sensitivity to each aspect varied. Thus, as children begin to learn their first words, they are already sensitive to some of the key components of event structure that they will need in order to map these events onto linguistic constructions. We are currently writing these studies up for publication, and have presented preliminary results at several conferences (see below for a poster).

One year later, we re-tested the same infants on their ability to generalise the meaning of a novel verb to new exemplars of the novel action as a function of changes in the identity of the agent and patient, the role played by specific participants, and their animacy. In our ongoing analysis of these longitudinal data, we are investigating whether the visual preferences displayed in infancy play a role in determining children’s ability to generalise verb meaning.

Outputs:

Jackson, I. R., Parise, E., Reid, V., & Theakston, A. (2017). How is infants’ attention distributed between agent and patient in causal events? Third biennial Workshop on Infant Language Development (WILD), Bilbao, Spain. 

Jackson, I. R., Parise, E., Reid, V., & Theakston, A. (2017). Agents vs patients: How is infants’ attention distributed in two-participant causal events? Lancaster International Conference on Infant and Early Child Development (LCICD). 

Jackson, I. R., Parise, E., Reid, V., & Theakston, A. (2017). Who is doing what to whom? Investigating the dynamics of infant attention in two-participant causal events. Cognitive Development Society Biennial Conference (CDS).

Jackson, I., Theakston, A., Parise, E. & Reid, V. (2016). Agents, patients, and actions: What is encoded in 12-month-olds’ perceptions of dynamic events? Poster presented at Lancaster Conference on Infant and Child Development (LCICD 2016) 

2. The relation between animacy, intentionality and children’s developing sentence comprehension and production.

In two related PhD projects, we examined how the animacy of participants and the intentionality behind events influenced children’s comprehension and production of different types of sentence structures. Children are sensitive to participant animacy and intentionality from infancy, so many researchers have argued that children might use these cues as heuristics to understand sentences when they begin to learn language. For example, children might assume that animate participants are agents, or that participants acting intentionally are involved in causal actions which in turn correspond to particular kinds of sentence structures. However, the role of animacy and intentionality in children’s understanding of sentences and how this might change over development as children begin to rely more on purely linguistic cues is not well understood.

i. A developmental investigation of animacy and prototypical transitivity in sentence production and comprehension 

With PhD student, Leone Buckle, we focussed on how the animacy of event participants, specifically an object (theme) and its recipient, influenced children’s descriptions of transfer-type events (e.g. The man gave a present to the monkey; The girl gave the library a rabbit). 3 and 5-yr-old children (plus adult controls) played a Bingo game in which they took turns with an experimenter to describe a series of animated events, winning pictures for their Bingo card for each response. We were interested in whether children would use the same sentence structure as the experimenter (i.e. a syntactic priming effect). In particular, we wondered if this would vary depending on the animacy of the participants in the scene described by the experimenter, how typical or non-typical their sentence was, and whether these features matched or did not match with the types of participants in the scene the children were asked to describe. This would allow us to work out how much children depend on animacy information when comprehending and producing their own sentences.

We found that although all age groups showed a structural priming effect, there was some evidence that priming in the youngest children was affected by participant animacy. Moreover, all groups were influenced by whether the scenes they were asked to describe had the same animacy mappings as those produced by the experimenter. In addition, the children were influenced by the order in which animate and inanimate participants were named in the sentences, whereas adults were not. These results show that animacy information is important in determining what kinds of sentences speakers choose to produce, and that children rely on this information more than adults.

Outputs 

Buckle, L., Lieven, E. & Theakston, A. (2017). The Effects of Animacy and Syntax on Priming: A Developmental Study. Frontiers in Psychology: Language Sciences, 20 December 2017. https://www.frontiersin.org/articles/10.3389/fpsyg.2017.02246/full

Buckle, L., Theakston, A., & Lieven, E. (2017). Is structural priming in children facilitated by interactions between animacy and syntax? In Proceedings of the 39th Annual Meeting of the Cognitive Science Society (pp. 1691-1696). https://mindmodeling.org/cogsci2017/papers/0330/paper0330.pdf

Buckle, L. Lieven, E. & Theakston, A. (2017). Is    structural    priming    in    children    influenced    by    animacy‐syntax    interactions? 3rd Annual Language and Communicative Development Conference, July 7th, Lancaster, UK http://www.lucid.ac.uk/media/1781/compressed-abstract-book.pdf

Buckle, L. Lieven, E. & Theakston, A. (2017).Do animacy-syntax interactions influence structural priming? The 23rd Architectures and Mechanisms of Language Processing Conference (AMLaP), September 7-9, Lancaster, UK http://wp.lancs.ac.uk/amlap2017/files/2017/06/amlap_proceedings_10.8.2017.pdf

ii. Cross-linguistic differences in encoding of causality: how English and Japanese children learn form-meaning mapping during development 

With PhD student, Akiko Okuno, we focussed on how English and Japanese children learn the conditions under which different types of sentences are used to describe causal events in their language. Researchers have suggested that whereas English speakers typically prefer to use transitive sentences which focus on the agent to describe causal events (e.g. The man dropped the ball), speakers of Japanese show a weaker preference, instead choosing to describe the scene using an intransitive sentence which highlights the patient and defocuses the agent of the action (e.g. The ball dropped). However, little is known about when and how these preferences might develop. We investigated this question in English- and Japanese-speaking children (aged three and five years) and compared this to preferences displayed by adults. We were particularly interested in two factors suggested to influence choice of sentence structure: language and intentionality. We conducted a corpus study examining the patterns of language use in the input to English and Japanese children. In addition, we carried out two experiments where participants watched videos depicting familiar and novel causal actions, and made a best-match choice between a transitive and intransitive description.

In our corpus study, we found a stronger preference for intransitive sentences with causal verbs and more frequent argument omissions in Japanese child-directed speech than in English child-directed speech. Our experiments revealed that Japanese and English children followed a different trajectory of acquisition in their selection of transitive sentences for causal events which appeared to reflect differences in their input. In addition, the two language groups showed sensitivity to whether the event was intentional or not; both Japanese and English speakers selected fewer transitives for accidental than intentional scenes, but this pattern was stronger in Japanese speakers, and there were changes over development for the two language groups. These findings suggest that learning the sentence structures of language is interdependent with learning how the meaning of events (e.g. the degree of intentionality) is mapped onto the specific language(s) being learned and takes developmental time.

Outputs

Okuno, A., Cameron-Faulkner, T. & Theakston, A. (2019): Crosslinguistic differences in the encoding of causality: Transitivity preferences in English and Japanese children and adults. Language Learning and Development

Okuno, A., Cameron-Faulkner, T. & Theakston, A. (2015) How does the world look to you? Talk given at Child Language Symposium, Warwick, UK. 

 

Project Team: Anna Theakston (Lead), Laura Boundy, Leone Buckle (PhD student), Thea Cameron-Faulkner, Steph HargreavesIain JacksonElena Lieven, Akiko Okuno (PhD student), Eugenio Parise and Vincent Reid

(Work Package 2)