The HCI&AI project focuses on developing innovative machine learning methods for human-computer interaction (HCI) that connect extended reality (XR) technologies, large language models (LLMs), user experience evaluation, and generative models.
The project will research and develop new methods and software in four areas:
- Interactive AR simulations for training and user support
- Large language models adapted for interactive assistance and understanding of specialized texts
- User feedback mechanisms and objective evaluation of HCI system effectiveness
- Generation of synthetic data and content using advanced generative models
By integrating these areas, the project will enable new forms of education, intelligent assistive systems, and domain-specific content creation, with applications across various interdisciplinary fields. The project outcomes are expected to contribute to the scientific excellence of the research group and the broader community through publications, open-source software tools, data repositories, and the establishment of interdisciplinary collaborations. The developed technologies and collected data will have international relevance, and the HCI&AI project will facilitate further advancements in HCI and artificial intelligence.
Work packages
WP1: Augmented reality (AR) for interactive simulations
The aim of Work Package 1 is to develop an AR system that enables the training of complex tasks with virtual guidance, personalized feedback, and real-time user tracking. The system will be domain-adaptable and will automatically adjust educational content to the user’s level of expertise. Unlike existing solutions that rely on fixed scenarios, the proposed system will support dynamic adaptation and automated performance analysis.
Activity 1.1: State-of-the-art review and AR scenario design
This activity includes reviewing existing AR methods, selecting suitable training scenarios, and defining educational goals in collaboration with domain experts. The focus is on identifying domains where AR provides the highest added value, such as medical training.
Activity 1.2: Development of AR software
A prototype AR application will be implemented that precisely projects and registers 3D objects into the real environment. Relevant 3D models will be developed or integrated, and the system will support interaction through gestures and user motion tracking.
Activity 1.3: Testing and iterative improvement of the system
The prototype will be tested with a smaller group of experts, the user experience will be evaluated, and the interface, visual presentation, and technical stability will be optimized. If needed, haptic and auditory feedback mechanisms will be added.
WP2: Large language models (LLMs) for interactive knowledge and assistance
The aim of Work Package 2 is to investigate how LLMs can improve interactive knowledge systems, especially for low-resource languages such as Croatian, and how to adapt models to specialized domains using limited task-specific data. Special emphasis is placed on local deployment of models in sensitive contexts and on integration with other components of the HCI&AI ecosystem.
Activity 2.1: Investigating model adaptability across domains and languages
This activity analyses knowledge transfer and the effectiveness of adapting models to specialized vocabularies. Methods for prompt engineering and evaluation frameworks will be developed, with a particular focus on Croatian and comparisons with other languages.
Activity 2.2: Adapting LLMs for local deployment in sensitive domains
Techniques for quantization, compression, and optimization for local model execution will be investigated. Performance will be evaluated and a benchmark platform created, with students participating in experiments.
Activity 2.3: Integration of LLM components into the HCI&AI ecosystem
This activity explores the integration of voice interfaces with AR simulations and the multimodal capabilities of LLMs for processing visual data. Technical requirements and specifications for integration will be defined.
WP3: Implementation of haptic interaction in extended and virtual educational scenarios
The aim of Work Package 3 is to develop and integrate a haptic system into XR environments to provide realistic, synchronized, and pedagogically effective tactile experiences. The focus is on XR–haptics communication, integration into educational scenarios, and performance optimization.
Activity 3.1: Development of a bidirectional communication system between XR applications and the haptic device
A software bridge for real-time data exchange between the XR application and the haptic device will be developed, using available SDKs and custom solutions. The precision and reliability of the communication will be tested.
Activity 3.2: Integration of haptic feedback into initial educational XR scenarios
Haptic responses (pressure, texture, hardness) will be integrated into selected XR scenes, aligned with visual and auditory content, enabling the first validation of the haptic system in an educational context.
Activity 3.3: Optimization of haptic system performance in XR environments
Latency, stability, and user comfort will be evaluated, and software and hardware parameters will be optimized to ensure reliable and comfortable use.
Activity 3.4: Documentation and guidelines for implementing haptic modules in future XR educational content
Documentation on architecture, integration procedures, and pedagogical guidelines will be prepared, including a comparative analysis of learning with and without haptic feedback.
WP4: Generative models for synthetic visual data and content
The aim of Work Package 4 is to investigate generative models for creating synthetic visual data that enhance the training of machine learning models, especially in challenging and medical contexts. The focus is on developing targeted synthetic datasets and evaluating their practical usefulness.
Activity 4.1: Preliminary research and selection of generative models for implementation
State-of-the-art generative models (text-to-image, image-to-image) will be analysed, selected approaches will be implemented, and their applicability will be assessed on generic datasets.
Activity 4.2: Development and evaluation of targeted synthetic datasets
Synthetic images will be generated for challenging detection conditions, variations in illumination, occlusions, and artifacts. Fine-tuning will be applied where needed, and students will participate in data preparation and annotation.
Activity 4.3: Application and transfer of synthetic data generation techniques to medical contexts
Synthetic medical images (X-ray, CT/MRI, dermatoscopy) will be generated to address data scarcity and class imbalance. Methods for generating dermatoscopic and volumetric medical data will be developed and evaluated.
Research team
- prof. dr. sc. Irena Galić
- prof. dr. sc. Krešimir Nenadić
- prof. dr. sc. Časlav Livada
- doc. dr. sc. Hrvoje Leventić
- doc. dr. sc. Dragana Božić Lenard
- doc. dr. sc. Krešimir Romić
- doc. dr. sc. Ivana Hartmann Tolić
- dr. sc. Marija Habijan
- dr. sc. Marin Benčević
- Robert Šojo, mag. ing. comp.
- Juraj Perić, mag. ing. comp.
The project is funded by the European Union – Next Generation EU. However, the views and opinions expressed are solely those of the author(s) and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them.
