Activities: Equipment standing on the edge of technology in terms of hardware, firmware and auxiliary software (i.e. drivers) will be employed for tracking eye-movements, recording brain electrical signals and capturing bio-feedback. A series of calibration and fine-tuning activities will be undertaken in order to achieve the optimal setting adapted to the requirements of the environment. The need for real-time and synchronized access to the different input streams makes this objective particularly challenging, especially if we consider that we are dealing with highly sophisticated and specialized equipment.

Activities: Pattern recognition and tracking algorithms will be employed to give substance to the inputs received by the aforementioned objective. These algorithms will translate raw signals into meaningful control by identifying, for instance, the place on the screen where the user focuses on, or the mental command that he is currently thinking of. Our intention is to specify the novel interaction paradigms of our system (cf. Section 1.1.2) by defining control patterns that stand one level higher from what are currently the elementary action units of conventional interfaces. More specifically, we distinguish between low-level control (comparable to onMouseMove), meso-level control (comparable to onMouseClick, onMouseOver, onKeyStroke) and high-level control (comparable to select-n-out-of-m items). In defining the high-level controls we plan to analyze the most typical operations performed in the context of multimedia management and authoring tasks, so as to dismantle the operation into its constituent basic controls steps (i.e., low-level and meso-level controls) and discover if the same process can be articulated using more elaborated control patterns (e.g., high-level controls consisting of a sequence of elementary control steps). For instance, instead of having to think the command “drag-and-drop” n times, which is closer to the machine way of functioning, the user can select specific n objects (i.e., “select n-out-of-m”) and the action (i.e. “drag-and-drop”) so as to complete the same process, which is closer to the way humans plan their course of actions. In this way, we will devise a vocabulary of controls (i.e., low, meso and high) suitable for operating multimedia-related applications that will be offered to the developers (as explained in Obj.3) in the form of semantic widgets. The challenge for this objective is, on the one hand, to build robust and reliable algorithms that will be able to cope with noisy signals in real-time and, on the other hand, to combine the inputs received from heterogeneous sources to boost the recognition performance.

Activities: The semantic widgets implementing the vocabulary of controls will be wrapped as modules of MAMEM’s middleware (and accompanying SDK) that will extent the core API of current operating systems with new and more elaborated function calls. The middleware will make transparent to the developer the code necessary to interface with the core API of the operating system, while the accompanying SDK will incorporate the algorithms developed in Obj.2 for translating signals into meaningful control. In this way the developers will be offered an extended set of options for designing their interfaces to enable the intuitive and natural communication of people with disabilities.

Activities: Assess the psychological impact of a disease affecting the human’s neuromuscular system with respect to his ability and willingness to operate multimedia-related software applications. Create profiles (personas) that will be used to categorize users based on their (dis-)abilities, interaction behaviour, emotions and intentions. Establish design principles that will motivate disabled people to explore the new interaction opportunities for managing and authoring multimedia content. For instance, these principles may suggest that the user should be gradually exposed to the functionalities of the new interface allowing for enough time to familiarize and acquire the necessary self-confidence. Moreover, these principles will also incorporate tactics for motivating the social integration of disabled people (e.g. provide direct links to social sharing platforms like Facebook and YouTube). The user profiles together with the persuasive design principles will be the basis for the design and implementation of the prototype interface applications that will be used during the pilot trials.

Activities: A number of usage scenarios related to multimedia will be detailed at the level of: i) the intended objective, ii) the semantic control units (drawn from the vocabulary described in Obj.2) required to execute the scenario, iii) the expected results (intermediate and final), and iv) the expected time required for reaching the intended objective. The number and exact nature of these usage scenarios will be determined during the course of the project (some preliminary ideas are already described in Section 1.1.2) and, among others, will incorporate actions like editing shapes, arranging a block diagram, applying filters on images and changing their size and orientation, making a collage of pictures, or even making a presentation consisting of multimedia content. Finally, these usage scenarios will be used to setup and conduct a set of pilot trials with MAMEM’s end-user groups, while methods like questioners, interviews, think-aloud testing and usage monitoring logs will be employed to extract the necessary conclusions.


Activities: Undertake the activities that will be necessary to implement a lightweight version of the prototype applications that will be appropriate for use in non-controlled environments. More specifically, the activities under this objective consists in: a) deciding on a lightweight configuration of the sensors that will be easily portable and appropriate for non-controlled environments, b) designing and implementing of a series of calibration and fine-tuning processes for the adaptation of the system to the requirements of a home environment, and c) providing detailed instructions on how to setup, configure and launch the system.

Activities: We will establish a number of indicators aiming to indirectly spot any changes at the inclusion level of disabled people. Soft indicators such as an increase in the number of on-line friends, Facebook or YouTube posts will be combined with more hard indicators like the number of disabled people that have found their way into workforce (e.g. working from home as freelancers, or even participating in the creative groups of their organization by designing gift cards).