This systematic review examines 34 research articles published from 2013 to 2024. The primary focus of the study is to explore more about the application of multimodal pedagogies in higher education, the methods and materials used to assist learners in acquiring English language skills, the English language skills acquired through the usage of multimodality, and the main results of using many modes. This systematic review employs the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) standards. It adopts a thorough search strategy across electronic databases, which include Web of Science and Scopus.

The applied disciplines of architecture and civil engineering require students to communicate multimodally, and to manipulate meaning across media and modes, such as image, writing or moving image. In their disciplinary studies for example, students must be able to transform the language of lectures and textbooks into models and diagrams. In their future workplaces, they will commonly be required to transform reports and legal documents into floor plans and digital & physical 3D models. Such multimodal literacy, however, is not typically reflected in their related subject-specific English language courses, especially in Germany, where a text-centric approach is favored. To better reflect the demands placed upon them, students in two courses of English for Architecture and Civil Engineering were tasked with creating digital, multimodal artifacts to explain a concept from either of these fields to a lay audience. The resultant artifacts used a wide variety of semiotic resources to make meaning, including a total of 26 separate architectural and civil engineering models. This is a quantity sufficiently large enough to invite closer examination, and also reflects the important role models play in the fields of architecture and civil engineering, both at university and in the workplace. This paper suggests that models of this kind exist within a system of signs, in which meaning is created in the relationships between the signs. The process of transforming one resource into another also invites the consideration of the artifacts in terms of the notion of “transduction”, to discern how meaning changes between contexts, practices and modes and to contribute to existing literature on multimodal texts in tertiary education, particularly within a language-learning context.

Introduction Recent times have witnessed an increasing number of applications of deep neural networks towards solving tasks that require superior cognitive abilities, e.g., playing Go, generating art, ChatGPT, etc. Such a dramatic progress raises the question: how generalizable are neural networks in solving problems that demand broad skills? To answer this question, we propose SMART: a Simple Multimodal Algorithmic Reasoning Task (and the associated SMART-101 dataset) for evaluating the abstraction, deduction, and generalization abilities of neural networks in solving visuo-linguistic puzzles designed specifically for children of younger age (6--8). Our dataset consists of 101 unique puzzles; each puzzle comprises a picture and a question, and their solution needs a mix of several elementary skills, including pattern recognition, algebra, and spatial reasoning, among others. To train deep neural networks, we programmatically augment each puzzle to 2,000 new instances; each instance varied in appearance, associated natural language question, and its solution. To foster research and make progress in the quest for artificial general intelligence, we are publicly releasing our SMART-101 dataset, consisting of the full set of programmatically-generated instances of 101 puzzles and their solutions. The dataset was introduced in our paper Are Deep Neural Networks SMARTer than Second Graders? by Anoop Cherian, Kuan-Chuan Peng, Suhas Lohit, Kevin A. Smith, and Joshua B. Tenenbaum, CVPR 2023 Files in the unzipped folder: ./README.md: This Markdown file ./SMART101-Data: Folder containing all the puzzle data. See below for details. ./puzzle_type_info.csv: Puzzle categorization (into 8 skill classes). Dataset Organization The dataset consists of 101 folders (numbered from 1-101); each folder corresponds to one distinct puzzle (root puzzle). There are 2000 puzzle instances programmatically created for each root puzzle, numbered from 1-2000. Every root puzzle index (in [1,101]) folder contains: (i) img/ and (ii) puzzle_<index>.csv. The folder img/ is the location where the puzzle instance images are stored, and puzzle_<index>.csv the non-image part of a puzzle. Specifically, a row of puzzle_<index>.csv is the following tuple: <id, Question, image, A, B, C, D, E, Answer>, where id is the puzzle instance id (in [1,2000]), Question is the puzzle question associated with the instance, image is the name of the image (in img/ folder) corresponding to this instance id, A, B, C, D, E are the five answer candidates, and Answer is the answer to the question. At a Glance The size of the unzipped dataset is ~12GB. The dataset consists of 101 folders (numbered from 1-101); each folder corresponds to one distinct puzzle (root puzzle). There are 2000 puzzle instances programmatically created for each root puzzle, numbered from 1-2000. Every root puzzle index (in [1,101]) folder contains: (i) img/ and (ii) puzzle_<index>.csv. The folder img/ is the location where the puzzle instance images are stored, and puzzle_<index>.csv contains the non-image part of a puzzle. Specifically, a row of puzzle_<index>.csv is the following tuple: <id, Question, image, A, B, C, D, E, Answer>, where id is the puzzle instance id (in [1,2000]), Question is the puzzle question associated with the instance, image is the name of the image (in img/ folder) corresponding to this instance id, A, B, C, D, E are the five answer candidates, and Answer is the correct answer to the question. Other Details In our paper Are Deep Neural Networks SMARTer than Second Graders?, we provide four different dataset splits for evaluation: (i) Instance Split (IS), (ii) Answer Split (AS), (iii) Puzzle Split (PS), and (iv) Few-shot Split (FS). Below, we provide the details of each split to make fair comparisons to the results reported in our paper. Puzzle Split (PS) We use the following root puzzle ids as the Train and Test sets. Split Root Puzzle Id Sets Test { 94,95, 96, 97, 98, 99, 101, 61,62, 65, 66,67, 69, 70, 71,72,73,74,75,76,77} Train {1,2,...,101} \ Test Evaluation is done on all the Test puzzles and their accuracies averaged. For the 'Test' puzzles, we use the instance indices 1701-2000 in the evaluation. Few-shot Split (FS) We randomly select k number of instances from the Test sets (that are used in the PS split above) for training in FS split (e.g., k=100). These k few-shot samples are taken from instance indices 1-1600 of the respective puzzles and evaluation is conducted on all instance ids from 1701-2000. Instance Split (IS) We split the instances under every root puzzle as: Train = 1-1600, Val = 1601-1700, Test = 1701-2000. We train the neural network models using the Train split puzzle instances from all the root puzzles together and evaluate on the Test split of all puzzles. Answer Split (AS) We find the median answer value among all the...

This dataset was created in October 2023, after the initial public release of ChatGPT vision. It contains 60 completed Test of Understanding Graphs in Kinematics 4.0 (TUG-K 4.0) surveys. The responses are sorted by item number. An analysis of the responses was published in https://doi.org/10.1103/PhysRevPhysEducRes.20.010109

v2 of the dataset takes care of some accidentally duplicated responses present in v1. Because the analysis in the above cited paper used data directly out of the chatbot, the updates in no way impact the analysis or findings in the paper.

This project on multiliteracies involved groups of deaf learners in India, Uganda, and Ghana, both in primary schools and with young adult learners. The Peer-to-Peer Deaf Multiliteracies project examined how some of the dynamics that contribute to learners’ marginalisation can be changed by involving deaf individuals in the design of new teaching approaches, and by using children and young people's lived experiences and existing multilingual-multimodal skills as the starting point for theme-based learning. The aim was for participants to develop not only English literacy, but "multiliteracies", i.e. skills in sign languages, ICT, written English, creative expression through drawing and acting, and other forms of multimodal communication. The data collection includes reports from classroom settings compiled by tutors and by research assistants, pre-and post-tests on language and literacy abilities with learners, samples from an online learning platform, and multimedia portfolios collected from learners. A total of 124 young deaf adults and 79 deaf primary school children took part in the research

The exclusion of deaf children and young adults from access to school systems in the developing world results in individuals and communities being denied quality education; this not only leads to unemployment, underemployment, low income, and a high risk of poverty, but also represents a needless waste of human talent and potential. To target this problem, this project extends work conducted under a pilot project addressing issues of literacy education with young deaf people in the Global South. Creating, implementing and evaluating our innovative intervention based on the peer teaching of English literacy through sign language-based tutoring, everyday real life texts such as job application forms, and the use of a bespoke online resource, enabled us to generate a sustainable, cost-effective and learner-directed way to foster literacy learning amongst deaf individuals. To reach further target groups and conduct more in-depth research, the present project extends our work to new groups of learners in India, Uganda, Ghana, Rwanda and Nepal, both in primary schools (ca 60 children in India, Ghana, and Uganda) and with young adult learners (ca 100 learners in interventions, plus ca 60 young adults in scoping workshops in Nepal and Rwanda). In the targeted countries, marginalisation begins in schools, since many have no resources for teaching through sign language, even though this is the only fully accessible language to a deaf child. This project intends to examine how we can change some of the dynamics that contribute to this, by involving deaf individuals in the design of new teaching approaches, and by using children and young people's everyday experiences and existing literacy practices as the basis for their learning. Participants in such a programme not only develop English literacy, but "multiliteracies", i.e. skills in sign languages, technology, written English, gesture, mouthing, and other forms of multimodal communication. Developing a multilingual toolkit is an essential element of multiliteracies. Being 'literate' in the modern world involves a complex set of practices and competencies and engagement with various modes (e.g. face-to-face, digital, remote), increasing one's abilities to act independently. Our emphases on active learning, contextualised assessments and building portfolios to document progress increases the benefit to deaf learners in terms of their on-going educational and employment capacity. Apart from the actual teaching and interventions, the research also investigates factors in existing systems of educational provisions for deaf learners and how these may systematically undermine and isolate deaf communities and their sign languages. Our analyses identify the local dynamics of cultural contexts that our programmes and future initiatives need to address and evaluate in order to be sustainable. One challenge we encountered in the pilot was the lack of trained deaf peer tutors. There is a need for investment in local capacity building and for the creation of opportunities and pathways for deaf people to obtain formal qualifications. Therefore, we develop training in literacy teaching and in research methods for all deaf project staff. We also develop and adapt appropriate assessment tools and metrics to confirm what learning has taken place and how, with both children and young adults. This includes adapting the Common European Framework of Reference for Languages (CEFR) for young deaf adult learners and the 'Language Ladder' for deaf children so that we use locally-valid test criteria. To document progress in more detail and in relation to authentic, real life literacy demands we need to create our own metrics, which we do by using portfolio based assessments that are learner-centred and closely linked to the local curricula.

ABSTRACT This paper adopts a conception of language as repertoire, conceived as an emergent set of semiotic resources that reflects life trajectories located in specific times and spaces. From this theoretical perspective, it analyzes dimensions of the configuration of the communicative repertoires of Indigenous individuals in the postcolonial contemporaneity. The empirical data under analysis was generated in a linguistic education context and consists of oral, written and multimodal registers of emerging interactions on the production and presentation of linguistic portraits. The analysis aims to highlight the pedagogical relevance of (self)representation of linguistic repertoires as a starting point for language education and as a research tool on linguistic resources, practices and ideologies.

The research data were collected for the dissertation "The impact of multimodal language teaching on learner autonomy, motivation, and free time language usage". The data were collected from a group of students in a B2 German class during Years 8 and 9 at comprehensive school (N = 14) and it comprise four questionnaires, two interviews, eight learning tasks, related outputs and feedback questionnaires, and the learners' visual narratives.