MultiCo – multimodal resources, or a holistic approach to interpersonal communication

AMU Faculty of Modern Languages and Literatures

The MultiCo multimodal corpus and the multimodal recording studio are developed at the Faculty of Modern Languages of the Adam Mickiewicz University in Poznań. The coordinator of the module is prof. Maciej Karpinski. The resources include a collection of audio and video recordings with a multilayer description of speech, gesture and body movement. The speaking styles include: conference recordings, parliamentary speeches and television conversations about sport, as well as task-oriented dialogues. Thanks to the funds from the DARIAH.Lab project, a professional studio was equipped to record multimodal data at Collegium Novum UAM, at al. Niepodległości 4 in Poznań.

MultiCo laboratory

• infrastructure for research on multimodality in communication
• multichannel sound recording with different microphones (headsets, lavalier, on stands; condenser and dynamic) or from other analogue and digital sources, with maximum sampling frequency 192 kHz and 32-bit resolution (Cubase Professional);
• processing and analysis of sounds, in particular speech, using software such as WaveLab, Cubase Professional, Praat, AnnotationPro and others;
• annotation of sound and video materials: AnnotationPro, Praat, ELAN;
• recording videos using 4k and HD cameras, also with an increased frame rate; image preview and recording also directly on the computer;
• recording and parametrization of movement using cameras with depth sensors Kinect 2 and Kinect Azure, compatible with the iPi Recorder and iPi Studio;
• video editing (Blackmagic DaVinci Resolve Studio software and Speed Editor console);
• body movement analysis (iPi BioMech system);
• IT infrastructure for the collection, storage and sharing of data and metadata: CorpusMINI.

MultiCo multimodal corpus

• A collection of audio and video recordings with annotations (including transcripts and description of selected gestural elements)
• The MultiCo corpus has three components:
  • MULTICO Persuasion – persuasive speeches, such as TEDex motivational talks, parliament speeches
  • MULTICO Task – task-related dialogues based on specially designed tasks which require both speakers to be active and engaged
  • MULTICO Sport – polylogues (discussions in TV sports programmes)
• The corpus includes about 15 hours of films, with an orthographic transcription and an additional multi-layer description, including among others phonemic transcription (with segmentation); a significant proportion of the films are also accompanied with descriptions of gestures or movement (e.g. boundaries of gesture phrases for hand gestures, head movements).

Who is it for?

• anyone whose job involves public speaking (MulticCo can help analyse the effectiveness of available rhetorical devices, and increase own persuasion skills);
• trainers in interpersonal communication courses;
• managers who seek to analyse and improve communicative processes at work;
• creators of avatars – animated pictures of persons in a user interface (e.g. a museum guide or a ticket agent at a railway station);
• producers of speech recognition and dialogue systems;
• researchers interested in spoken language and the role of facial expressions and gestures in language communication (e.g. linguists, phoneticians, psychologists, sociologists);
• institutions responsible for safety, including the police: expert opinions based on speech analysis and the analysis of the speaker’s movements;
• artists interested in speech recording, processing, synthesis and analysis;
• artists interested in movement analysis and reconstruction.

Name and contact details of the module coordinator
Associate Professor Maciej Karpiński (maciej.karpinski@amu.edu.pl)