Abstract: Recognizing food image in the free-living environment is a highly challenging task due to wildly different types of cuisines and dish presentation styles. The fundament step towards the success of this task is to enable zero-shot prediction of any ingredients, portion sizes, calories and nutrition facts for food prepared under any methods and images taken in any environments. The conventional approach is by retrieving the recipe associated with the query food image, where the recipe provides necessary information (e.g., ingredients and their weights) for estimation of food content and amount. Nevertheless, this line of approaches is practically limited by the number of recipes available for retrieval. Recently, due to strong reasoning capability of multimodal foundation models, direct prediction of food content and amount is possible without referring to recipes. Nevertheless, these models generally suffer from hallucination and are difficult to apply for real-world applications. This talk will present the research efforts in these two directions. Particularly, a recently developed Food Large Multi-modal Model (FoodLMM), which is a recent effort to unify different tasks into a foundation model for food recognition, will be introduced. This talk will share insights regarding the training of FoodLMM and its performance compared to the conventional approaches.

Bio: Dr. Chong-Wah Ngo is a professor with School of Computing and Information Systems, Singapore Management University (SMU). Before joining SMU, he was a professor with the Department of Computer Science, City University of Hong Kong. His main research interests include multimedia search, multi-modal fusion, and video content understanding. He has researched on various issues in food computing, including ingredient recognition, open-vocabulary ingredient segmentation, food image generation from recipe, cross-domain food recognition, cross-modal and cross-lingual food image and recipe retrieval, mobile app food logging systems. He leads the research and development of FoodAI engine in SMU, an engine deployed by Singapore Health Promotion Board for citizen food logging.

Abstract: In recent years, it has been a cutting-edge research direction on leveraging the power of brain signals to help improve recommendation and search. In this talk, I will firstly share our latest research advancements in exploring the correlation between users’ experiences and brain EEG signals, emotions, and user satisfaction in short video recommendation scenario. Subsequently, I will demonstrate how user immersion, a new factor that links psychology, brain science, and information retrieval, contributes to more effective short video recommendation. Moreover, in the realm of search, I will briefly introduce Brain-Aug, a novel approach that decodes semantic information directly from brain fMRI signals of users to augment query representation. We believe there will be more valuable future directions on utilizing brain signals for sophisticated information generation, recommendation and search.

Bio: Dr. Min Zhang is a full professor in the Dept. of Computer Sci. & Tech., Tsinghua University. She is the chief director of the AI Lab. She specializes in Web search, personalized recommendation, and user modeling. She has been the Editor-in-Chief of ACM Transaction on Information Systems (TOIS) since 2020, and also serves as the General co-Chair of ACM MM’25, and PC co-Chairs of CHIIR’24, RecSys’23, CIKM’23, ICTIR’20, WSDM’17, etc. She won the “Test-of-Time” award at SIGIR’24, WSDM’22 Best Paper award, IBM Global Faculty Award, Okawa Fund, SIGIR’20 Best Full Paper Honorable Mention, etc.