Virtual Analysis Networks and Explanations for Social Sensing Analysis (VANESSA) is a collaboration between the NLP&C Lab co-PIs, Sonja Schmer-Galunder and Bonnie Dorr, and Smart Information Flow Technologies (SIFT), led by PI Dr. Jeremy Gottlieb. Funded by the Defense Advanced Research Projects Agency (DARPA), the project builds on prior research projects (MEDULLA, OBLONGATA) where high-fidelity data was collected in four 1-month-long space simulation studies (teams of 3 women and 3 men) at the HI-SEAS habitat on Mauna Loa on the Big Island of Hawai’i.

VANESSA analyzes behavioral, physiological, cognitive, and linguistic data from 24 participants (12 women, 12 men) who lived and worked in extreme, confined, and isolated environments over extended periods of time. The goal is to understand social cohesion of small teams and to provide evidence-based diagnostics and interventions based on multi-modal early signals. Such capabilities are relevant for many high-stress, high-stakes work environments such as space, military, and medical operations. The project investigates how non-intrusive, evidence-based diagnostics and interventions may be integrated into recommendation systems as a means of improving team performance in real-world, high-stakes settings.

Leveraging text, audio, and video data, VANESSA researchers conduct multi-modal feature extraction to identify behavioral patterns (e.g., expressiveness in verbal or non-verbal communication, or lack thereof) capable of indicating how social constructs like team cohesion evolve over time. In doing so, the team performs a variety of tasks (e.g., sentiment analysis, toxicity detection, emotion classification, automated speech recognition, body pose estimation, and many more) using rule-based systems, classical machine learning, and deep learning approaches. VANESSA researchers also explore software as a means of integrating the features derived from various natural language processing and computer vision tasks into a recommendation system which tracks team state changes over time.

Initially, a novel approach to retrieval-augmented generation (RAG), named MergeRAG, was evaluated. Team state and cohesion were able to be tracked and analyzed through interaction graphs: networks linking mission entities (nodes) and their relationships (edges), with MergeRAG prompts targeting different cohesion facets. To prevent data leakage, knowledge graphs were dynamically merged using temporal metadata. Hierarchical clustering was also conducted in order to create team behavioral profiles as a potential means of predicting team cohesion.

Most recently, prior developments became the foundation for the design of an application programming interface (API) for an agentic, large language model (LLM)-based system named ORCA (Orchestrated Reasoning Coding Agent), which offers three core functions: retrieving raw metrics, performing ad-hoc analysis on available data, and investigating literature sources (i.e., documents) via RAG. Engineering efforts to develop ORCA and prototype its functionality have prioritized AI safety through rigid, tested constraints and human-in-the-loop interventions, reliability through containerization and documentation, and the standardization of inputs through use of open-source tooling.

Project contributors at UFL: Sonja Schmer-Galunder (co-PI); Bonnie J. Dorr (co-PI); Norman Bukingolts (project lead through Oct 2026); Divyansh Singh (project lead through Jun 2026); Manuel Martinez Nunez (project lead through Feb 2026); Rohan Shah, Sangpil Youm, David-Benjamin Ogilvie; Phoebe Huang.