Deep Self Supervised Learning for 3D Surface Parameterization and 3D Garment Retargeting

Abstract

Metaverse has gained massive popularity these days owing to its potential in various industries and research fields such as the health sector, online social interaction, gaming and entertainment, education, eCommerce transactions, etc. Some of the challenges to be addressed in building a Metaverse, including human-computer interactions and scene understanding, rely on AI techniques for solutions. However, most of these methods rely on supervised learning that requires labor-intense annotated data and have limited generalizability. Self-supervised learning methods offer a viable alternative to overcome these limitations as well as leverage large amounts of readily available unlabeled raw data, which is often more abundant and easier to collect. These methods can learn to capture temporal or spatial relationships within data, modeling contextual information, which is valuable for several computer vision tasks. Hence, these methods can be useful in addressing the challenges of building the Metaverse. Within the Metaverse, users can interact with each other and a digital environment through avatars. Using UV parameterized maps of avatars, textures can be accurately applied, resulting in realistic skin, clothing, and other visual details. They also play a crucial role in the creation of virtual fashion and design within the Metaverse. This can also benefit the E-Commerce industry, particularly fashion and apparel, which have experienced tremendous growth in recent years. The Metaverse can provide immersive virtual environments for virtual try-on where users can explore and interact with products more realistically and engagingly than traditional online shopping. Given the above advantages, we intend to address the following two challenges in this thesis. First, we explore self-supervised data-driven methods for UV parameterization of general objects. The existing methods for surface parameterization of arbitrary 3D objects face challenges when dealing with closed surfaces and regions of extreme extrinsic curvature. Mapping a surface from 3D to 2D almost always introduces a certain amount of distortion, and the aim is to keep this distortion as low as possible. Finding optimal seams that lead to low distortion is another challenge. We present a novel framework for learning the discretization-agnostic surface parameterization of arbitrary 3D objects with closed and open surfaces. We evaluate our framework on multiple 3D objects from the publicly available dataset and report a comparison with conventional methods. Secondly, utilizing self-supervised methods, we aim to innovate a solution for a 3D virtual tryon system. The goal is to retarget real, non-parametric garment meshes over a target human body (parametric or non-parametric). This 3D virtual try-on system needs to generalize to arbitrary body shapes and poses, modeling topological differences among various categories of garments, along with realistic deformations arising out of the physical interaction with the underlying body and resolving the penetration/intersection of the garment with the underlying body. These systems need to run in real time with very less delay. We propose a self-supervised method for draping non-parametric, 3D garment meshes by first obtaining the initial alignment between the garment and the human body by establishing correspondences via Isomap Embeddings. Further, this coarse retargeting is refined by training an MLP that preserves the geometry of the garments guided by our novel losses. We propose a wrinkle generation module to obtain realistic details on the draped garments. We also contribute a new dataset of real-world reposed garments with realistic noise and topological deformations. Finally, we discuss the limitations of our work and lay down the potential solutions that can be explored. We also discuss the future directions that can be pursued based on the findings of our work. We believe this thesis advances the field of virtual try-on systems significantly while providing a learningbased solution for parameterization.