{"id":10674,"date":"2021-10-15T14:12:29","date_gmt":"2021-10-15T06:12:29","guid":{"rendered":"https:\/\/cde.nus.edu.sg\/ece\/?post_type=nus-news&#038;p=10674"},"modified":"2024-07-31T16:05:20","modified_gmt":"2024-07-31T08:05:20","slug":"nus-facebook-ai-and-other-world-class-universities-collaborate-to-teach-ai-to-understand-the-world-through-our-eyes","status":"publish","type":"nus-news","link":"https:\/\/cde.nus.edu.sg\/ece\/news\/nus-facebook-ai-and-other-world-class-universities-collaborate-to-teach-ai-to-understand-the-world-through-our-eyes\/","title":{"rendered":"NUS, Facebook AI and Other World-Class Universities Collaborate to Teach AI to Understand the World Through Our Eyes"},"content":{"rendered":"<div id='gallery-1' class='gallery galleryid-10674 gallery-columns-1 gallery-size-full'><figure class='gallery-item'>\n\t\t\t<div class='gallery-icon landscape'>\n\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"1920\" height=\"1079\" src=\"https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_select.jpg\" class=\"attachment-full size-full\" alt=\"1920 Select\" aria-describedby=\"gallery-1-10675\" srcset=\"https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_select.jpg 1920w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_select-300x169.jpg 300w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_select-1024x575.jpg 1024w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_select-768x432.jpg 768w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_select-1536x863.jpg 1536w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_select-1350x759.jpg 1350w\" sizes=\"auto, (max-width: 1920px) 100vw, 1920px\" \/>\n\t\t\t<\/div>\n\t\t\t\t<figcaption class='wp-caption-text gallery-caption' id='gallery-1-10675'>\n\t\t\t\tBy collecting first-person video data in everyday scenarios, such as having a meal in a food court, the NUS team and collaborators have forged an entirely new path for building smarter video AI models to power functions like memory augmentation: AR glasses can remember where users have left their wallets and even remind them not to leave it behind.\n\t\t\t\t<\/figcaption><\/figure>\n\t\t<\/div>\n\n<p style=\"text-align: justify\">There is a marked difference between viewing and interacting with the world as a third-party spectator, and experiencing the action intimately from a first-person point of view.<\/p>\n<p style=\"text-align: justify\">This difference is similar to watching others ride a roller coaster, from the ground, as opposed to riding the roller coaster yourself \u2013 what ultimately ends up informing the experience and understanding of the roller coaster ride, is entirely disparate.<\/p>\n<p style=\"text-align: justify\">This is the current obstacle facing Artificial Intelligence (AI) and its applications for use.<\/p>\n<p style=\"text-align: justify\">To unlock the next wave of AI technology that will power future assistants and innovations for Augmented Reality (AR) and robotics, AI needs to evolve to an entirely new paradigm of egocentric (i.e. first-person) perception. This means teaching AI to understand the world through human eyes in the context of real-time motion, interaction, and multi-sensory observations.<\/p>\n<p style=\"text-align: justify\">To do so, a consortium has been formed by the National University of Singapore (NUS)\u00a0and 12 other universities around the worldto undertake an ambitious, long-term project, called\u00a0<a href=\"https:\/\/ego4d-data.org\/\" target=\"_blank\" rel=\"noopener\">Egocentric 4D Live Perception (Ego4D)<\/a>.<\/p>\n<p style=\"text-align: justify\">A team from the\u00a0<a href=\"https:\/\/cde.nus.edu.sg\/ece\/\" target=\"_blank\" rel=\"noopener\">NUS Department of Electrical and Computer Engineering<\/a>\u00a0has been actively working to collect first-person data, specifically in Singapore. This data was collected over the course of 2021, via\u00a0head-mounted cameras and AR glasses distributed to a total of 40 participants locally. This allowed the NUS team to capture their eye-level, first-person, unscripted experiences of day-to-day scenarios: routine activities such as getting a haircut, dining at a hawker centre, or going to the gym. Based on the collected video data, the NUS researchers were then able to\u00a0train AI models to understand people and their interactions by leveraging both audio and visual cues.<\/p>\n<p style=\"text-align: justify\"><strong>Assistant Professor Mike Shou<\/strong>, who leads the NUS research team, said, \u201cOver the past 10 years, we have witnessed the revolution of AI for image understanding, which is built on the foundations laid down by datasets like the ImageNet. Similarly, I believe our Ego4D dataset will lay down the necessary groundwork for egocentric video research, and spur remarkable progress in building AI models for AR and robot applications.\u201d<\/p>\n<p style=\"text-align: justify\"><strong>Improving AI technology for smart and useful assistants<\/strong><\/p>\n<p style=\"text-align: justify\">The current paradigm of computer vision (CV) has, so far, excelled at understanding what is in an image or video by learning from vast amounts of online photos and videos from a third-person view, where the camera is a spectator from afar.<\/p>\n<p style=\"text-align: justify\">Advancements in first-person, or egocentric perception, will provide the fundamental building blocks necessary to develop smarter cognitive capabilities for AI assistants in the context of the person interacting with it. Such AI assistants will prove more useful in our day-to-day lives and work. Imagine: when trying to cook a new recipe, instead of referring repeatedly to a cookbook and attempting to multi-task, simply wearing a pair of AR smart glasses can direct you to perform each specific step, as you are doing it.<\/p>\n<div id='gallery-2' class='gallery galleryid-10674 gallery-columns-1 gallery-size-full'><figure class='gallery-item'>\n\t\t\t<div class='gallery-icon landscape'>\n\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"1920\" height=\"1080\" src=\"https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_cookingcombinedforpp.jpg\" class=\"attachment-full size-full\" alt=\"1920 Cookingcombinedforpp\" aria-describedby=\"gallery-2-10676\" srcset=\"https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_cookingcombinedforpp.jpg 1920w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_cookingcombinedforpp-300x169.jpg 300w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_cookingcombinedforpp-1024x576.jpg 1024w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_cookingcombinedforpp-768x432.jpg 768w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_cookingcombinedforpp-1536x864.jpg 1536w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_cookingcombinedforpp-1350x759.jpg 1350w\" sizes=\"auto, (max-width: 1920px) 100vw, 1920px\" \/>\n\t\t\t<\/div>\n\t\t\t\t<figcaption class='wp-caption-text gallery-caption' id='gallery-2-10676'>\n\t\t\t\tExamples of video data collected by the NUS team of first and third-person perspective, as a user is preparing a meal.\n\t\t\t\t<\/figcaption><\/figure>\n\t\t<\/div>\n\n<p style=\"text-align: justify\">Asst Prof Shou explained, \u201cIn particular for Singapore and her aging population, such AI assistants can be an exceptional aid for the elderly, especially those with health conditions like dementia or Alzheimer\u2019s. A pair of AR glasses could help elderly patients remember, and memorise \u201cwhat happened and when\u201d, to answer questions like where they left their keys, or if they remembered to lock the door. AI assistants applied to healthcare robotics can also understand if a person is speaking to, or looking at the robot itself, and thus take care of multiple patients in a single ward concurrently.\u201d<\/p>\n<p style=\"text-align: justify\">This technology can be applied across a spectrum of devices, and fuel a world where physical, augmented, and virtual reality can co-exist together in a single space.<\/p>\n<p style=\"text-align: justify\"><strong>World\u2019s largest first-person video data set<\/strong><\/p>\n<p style=\"text-align: justify\">Today, the consortium is announcing the world\u2019s largest first-person video data set that\u2019s captured \u201cin the wild\u201d, featuring people going about their normal daily life. The NUS team, together with its partners, have collectively gathered more than 3,000 hours of first-person video data \u2014 which will be publicly available in November 2021 \u2014 from more than 700 research participants across nine countries.<\/p>\n<div id='gallery-3' class='gallery galleryid-10674 gallery-columns-1 gallery-size-full'><figure class='gallery-item'>\n\t\t\t<div class='gallery-icon landscape'>\n\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"1920\" height=\"1079\" src=\"https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_activitymap.png\" class=\"attachment-full size-full\" alt=\"1920 Activitymap\" aria-describedby=\"gallery-3-10677\" srcset=\"https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_activitymap.png 1920w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_activitymap-300x169.png 300w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_activitymap-1024x575.png 1024w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_activitymap-768x432.png 768w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_activitymap-1536x863.png 1536w, https:\/\/cde.nus.edu.sg\/ece\/wp-content\/uploads\/sites\/3\/2021\/10\/1920_activitymap-1350x759.png 1350w\" sizes=\"auto, (max-width: 1920px) 100vw, 1920px\" \/>\n\t\t\t<\/div>\n\t\t\t\t<figcaption class='wp-caption-text gallery-caption' id='gallery-3-10677'>\n\t\t\t\tEgo4D is a massive-scale egocentric video dataset of daily life activity spanning 73 locations worldwide. It offers greater global representation, and provides more diversity in scenes, people, and activities, which increases the applicability of AI models trained for people, across backgrounds, ethnicities, occupations, and ages. Photo credit: Ego4D Academic Consortium\n\t\t\t\t<\/figcaption><\/figure>\n\t\t<\/div>\n\n<p style=\"text-align: justify\">Progress in the nascent field of egocentric perception depends on large volumes of egocentric data of daily life activities, considering most AI systems learn from thousands, if not millions, of examples. Existing data sets do not yet have the scale, diversity, and complexity necessary to be useful in the real-world.<\/p>\n<p style=\"text-align: justify\">This first-of-its-kind video dataset captures what the camera wearer chooses to gaze at in a specific environment; what the camera wearer is doing with their hands and objects in front of them; and how the camera wearer interacts with other people from the egocentric perspective. So far, the collection features camera wearers performing hundreds of activities and interactions with hundreds of different objects. All visible faces and audible speeches in the Ego4D dataset\u2019s video footage have been consented to by the participants for public release.<\/p>\n<p style=\"text-align: justify\">Global representation is crucial for egocentric research since the egocentric visual<\/p>\n<p style=\"text-align: justify\">experience will significantly differ across cultural and geographic contexts. In particular, the NUS team\u2019s collected data can be extrapolated to represent the Southeast Asian demographic, so that AI systems developed under-the-hood can ideally recognise the nuances and needs that may look different from region to region.<\/p>\n<p style=\"text-align: justify\"><strong>Unpacking the real-world data set<\/strong><\/p>\n<p style=\"text-align: justify\">Equally important as data collection is defining the right research benchmarks, or tasks, that can be used for testing and measurement.<\/p>\n<p style=\"text-align: justify\">To provide a common objective for all the researchers involved to build fundamental research for real-world perception of visual and social contexts, the consortium has collectively developed five new, challenging benchmarks, which required rigorous annotation of the collective egocentric data set.<\/p>\n<p style=\"text-align: justify\">\u201cOur NUS research team has been focusing on two of these key benchmarks: audio-visual diarization, that is, to leverage cues of sound and sight to help AI machines identify \u2018who said what when\u2019; and the second is to train AI models to better understand the nature of social interactions. A socially intelligent AI should understand who is speaking to whom, and who is paying attention to whom at any given point of time,\u201d illustrated Professor Li Haizhou, a co-Principal Investigator also from the NUS Department of Electrical and Computer Engineering.<\/p>\n<p style=\"text-align: justify\">To date, the NUS team has created multi-modal deep learning models for:<\/p>\n<p style=\"text-align: justify\">1) Active speaker detection (detecting who is speaking, using both audio and visual signals)<\/p>\n<p style=\"text-align: justify\">2) Detecting who is speaking to whom<\/p>\n<p style=\"text-align: justify\">3) Detecting who is paying attention to whom in a given social interaction<\/p>\n<p style=\"text-align: justify\">Currently, these models are only trained on video data up to 100 hours, which is a small part of the entire Ego4D dataset. Next steps for\u00a0<a href=\"https:\/\/sites.google.com\/view\/showlab\" target=\"_blank\" rel=\"noopener\">the NUS team<\/a>\u00a0include conducting large-scale pre-training, to create one generic, strong model that is trained on the whole dataset and can learn to perform multiple tasks\u00a0jointly.<\/p>\n<p style=\"text-align: justify\"><em>In addition to Asst Prof Shou, other Principal Investigators that comprise the academic consortium include: CV Jawahar (International Institute of Information Technology, Hyderabad); David Crandall (Indiana University); Dima Damen (University of Bristol); Giovanni Maria Farinella (University of Catania); Bernard Ghanem (King Abdullah University of Science and Technology); Kris Kitani (Carnegie Mellon University, Pittsburgh and Africa); Aude Oliva and Antonio Torralba (Massachusetts Institute of Technology); Hyun Soo Park (University of Minnesota);<\/em>\u00a0<em>Jim Rehg (Georgia Institute of Technology); Yoichi Sato (University of Tokyo); Jianbo Shi (University of Pennsylvania); and Pablo Arbelaez (Universidad de los Andes).\u00a0For the complete list of benchmarks and partners involved, read more\u00a0<a href=\"https:\/\/ai.facebook.com\/blog\/teaching-ai-to-perceive-the-world-through-your-eyes\/\" target=\"_blank\" rel=\"noopener\">here<\/a>.\u00a0<\/em><\/p>\n<p>*This article is also available at <a href=\"https:\/\/news.nus.edu.sg\/nus-facebook-world-class-universities-teach-ai-to-understand-the-world-through-our-eyes\/\" target=\"_blank\" rel=\"noopener\">NUS NEWS<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>There is a marked difference between viewing and interacting with the world as a third-party spectator, and experiencing the action intimately from a first-person point of view. This difference is similar to watching others ride a roller coaster, from the ground, as opposed to riding the roller coaster yourself \u2013 what ultimately ends up informing<\/p>\n","protected":false},"author":31,"featured_media":10678,"parent":0,"menu_order":0,"template":"","meta":{"_acf_changed":false,"rs_blank_template":"","rs_page_bg_color":"","slide_template_v7":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"news_category":[37,48,49],"class_list":["post-10674","nus-news","type-nus-news","status-publish","has-post-thumbnail","hentry","news_category-ece","news_category-ece-in-the-news","news_category-research-impact"],"acf":[],"_links":{"self":[{"href":"https:\/\/cde.nus.edu.sg\/ece\/wp-json\/wp\/v2\/news\/10674","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cde.nus.edu.sg\/ece\/wp-json\/wp\/v2\/news"}],"about":[{"href":"https:\/\/cde.nus.edu.sg\/ece\/wp-json\/wp\/v2\/types\/nus-news"}],"author":[{"embeddable":true,"href":"https:\/\/cde.nus.edu.sg\/ece\/wp-json\/wp\/v2\/users\/31"}],"version-history":[{"count":1,"href":"https:\/\/cde.nus.edu.sg\/ece\/wp-json\/wp\/v2\/news\/10674\/revisions"}],"predecessor-version":[{"id":15091,"href":"https:\/\/cde.nus.edu.sg\/ece\/wp-json\/wp\/v2\/news\/10674\/revisions\/15091"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cde.nus.edu.sg\/ece\/wp-json\/wp\/v2\/media\/10678"}],"wp:attachment":[{"href":"https:\/\/cde.nus.edu.sg\/ece\/wp-json\/wp\/v2\/media?parent=10674"}],"wp:term":[{"taxonomy":"news_category","embeddable":true,"href":"https:\/\/cde.nus.edu.sg\/ece\/wp-json\/wp\/v2\/news_category?post=10674"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}