Kiose M.I., Potekhin V.O., Zubkov O.D. Experimental Methods of Exploring Multimodal Discourse: Crossmodal Alignment
DOI: https://doi.org/10.15688/jvolsu2.2024.5.12
Mariya I. Kiose
Doctor of Sciences (Philology), Chief Researcher, Centre for Socio-Cognitive Discourse Studies, Moscow State Linguistic University
Ostozhenka St, 38/1, 119034 Moscow, Russia
Leading Researcher, Laboratory of Multichannel Communication, Institute of Linguistics of the Russian Academy of Sciences
Bolshoi Kislovsky Lane, 1/1, 125009 Moscow, Russia
This email address is being protected from spambots. You need JavaScript enabled to view it.
https://orcid.org/0000-0001-7215-0604
Vadim O. Potekhin
Senior Lecturer, Diplomatic Academy of the Russian Ministry of Foreign Affairs
Ostozhenka St, 53/2, Bld. 1, 119021 Moscow, Russia
Postgraduate Student, Department of General and Comparative Linguistics, Moscow State Linguistic University
Ostozhenka St, 38/1, 119034 Moscow, Russia
This email address is being protected from spambots. You need JavaScript enabled to view it.
https://orcid.org/0009-0003-2370-5097
Oleg D. Zubkov
Postgraduate Student, Department of General and Comparative Linguistics, Moscow State Linguistic University
Ostozhenka St, 38/1, 119034 Moscow, Russia
This email address is being protected from spambots. You need JavaScript enabled to view it.
https://orcid.org/0009-0008-9670-6273
Abstract. The study advances Crossmodal Alignment Framework to explore multimodal discourse in its three formats – semiotic, communicative and perceptive – via multimodal experiment. It considers the alignment patterns obtained from two semiotic modes (text and image), transferred in two communicative modes (speech and gesture), sensed by two perception modes (visual and audial). The common research framework determines the patterns as modulated by discourse tasks. The study features the results of multimodal experiments with the participants engaged in three discourse tasks: 1) receptive, which presumes obtaining information from text and image stimuli; semiotic alignment patterns are identified indirectly via participants' gaze response; 2) productive, in which the participants communicate the information in monological format; communicative alignment patterns are identified directly via their speech and gesture; 3) receptive-productive, which presupposes the participants perceive information visually and audially; alignment patterns are identified directly via participants' gaze behavior contingent on the stimuli areas of interest and indirectly via their speech response. Data analysis allows to determine and scale the degree of crossmodal alignment to discourse tasks, which helps identify the input of each mode to solving these tasks. The research framework and obtained results contribute to further development of multimodal discourse methods.
Key words: multimodal experiment, crossmodal alignment, discourse task, information construal, semiotic mode, communicative mode, perceptive mode, alignment patterns.
Citation. Kiose M.I., Potekhin V.O., Zubkov O.D. Experimental Methods of Exploring Multimodal Discourse: Crossmodal Alignment. Vestnik Volgogradskogo gosudarstvennogo universiteta. Seriya 2. Yazykoznanie [Science Journal of Volgograd State University. Linguistics], 2024, vol. 23, no. 5, pp. 149-160. DOI: https://doi.org/10.15688/ jvolsu2.2024.5.12
Experimental Methods of Exploring Multimodal Discourse: Crossmodal Alignment by Kiose M.I., Potekhin V.O., Zubkov O.D. is licensed under CC BY 4.0
 1_Kiose_et al.pmd.pdfURL: https://l.jvolsu.com/index.php/en/component/attachments/download/3009 | 8 Downloads |