Develop an AI system that combines chatbot assistance with real-time facial and body language interpretation to enhance user intent understanding and communication. This system will act as a subtle, background agent that understands both verbal and non-verbal feedback, improving the accuracy and fairness of AI-driven interactions.

Purpose

The aim of the project is to explore and develop an intelligent system that combines chatbot interactions with real-time interpretation of facial expressions and body language. This system will assist in understanding and expressing intent, potentially guiding or interrupting the "Thinking Mode" of advanced reasoning models. The goal is to create a subtle, imperceptible supervising agent that enhances communication and decision-making processes.

Current Situation

Common practices in foundation models for this thematic area include the use of natural language processing (NLP) for chatbots and computer vision for facial expression analysis. State-of-the-art models often focus on individual tasks, such as sentiment analysis or emotion detection, but integrating these capabilities into a cohesive system that can interpret and respond to both verbal and non-verbal cues is less explored. The project aims to bridge this gap by creating a unified system that leverages both NLP and computer vision technologies.

Activities

The main activities foreseen in the course of this project include:

1. Data Collection: Gathering datasets on facial expressions, body language, and chatbot interactions.
2. Model Training: Training machine learning models to interpret facial expressions and body language.
3. Integration: Combining the trained models with a chatbot system to create a cohesive interaction platform.
4. Prototyping: Developing a prototype that demonstrates the system's ability to interpret and respond to both verbal and non-verbal cues.
5. Testing and Refinement: Conducting user testing and refining the system based on feedback.

Resources

Access to AI models and infrastructure will be crucial for executing these activities. This includes:

• Datasets: Publicly available datasets on facial expressions, body language, and chatbot interactions.
• AI Models: Pre-trained models for NLP and computer vision tasks.
• Infrastructure: Cloud computing resources for model training and deployment.
• Tools: Software tools for data preprocessing, model training, and prototyping.

Team

The expertise expected of the team members includes:

• Data Scientists: For data collection, preprocessing, and model training.
• Machine Learning Engineers: For developing and integrating AI models.
• Software Developers: For prototyping and deploying the system.
• User Experience Designers: For creating an intuitive and user-friendly interface.
• Ethicists: For ensuring the system complies with ethical norms and regulatory guidelines.

The activities planned will stimulate collaboration among team members by encouraging cross-disciplinary problem-solving and knowledge sharing.

Outputs and Outcomes

This project will promote open science by making the developed models and datasets publicly available. It will also catalyze a potential larger project based on Swiss AI by demonstrating the feasibility and benefits of integrating NLP and computer vision technologies. The outcomes of this project have the potential to enhance communication and decision-making processes, promoting fairness and inclusion in AI-driven interactions.

Geographic Relevance

The proposed activities align with the goals of the Swiss AI Initiative by leveraging advanced AI technologies to address real-world problems. The strategic importance and potential impact of this project for Swiss society, Europe, and the world include enhancing communication and decision-making processes, promoting fairness and inclusion, and advancing the field of AI-driven interactions.

Ethics and Regulatory Compliance

Ethical considerations include ensuring the privacy and consent of individuals whose data is used for training the models. Compliance with legal and regulatory guidelines, such as data protection laws, will be crucial. The team will adhere to the ethical norms and guidelines outlined in the FAQ of the Swiss {ai} Weeks to ensure the project's compliance with ethical standards and regulatory requirements.

🅰️ℹ️ Generated with MISTRAL24B

Further thoughts

The older idea of Ambient Intelligence (see Ramos et al IEEE 2008) is being renewed beyond smart devices and industry use cases, to a way of describing supervising agents that assist subtly or imperceptibly behind the scenes. A typical example is my laptop and network's intelligent firewall and anti-spam services making it possible for me to write and for you to receive this message without hassle.

I've been wondering if a Computer Vision assisted interpretation of body language (something I've seen in experiments with professional dancers, etc.) and facial expressions, could help us express our intent. Or even better, to train our dissent :) Basically, my hacky idea is to combine the use of a chatbot with interpreting facial expressions, providing rapid-fire thumbs up/down responses. Initially by padding prompts with extra inputs, long-term in order to guide or interrupt the "Thinking Mode" of the new generation of Reasoning LLMs.