Project Name: Blend: Tron’s Ultra-Realistic Game Character 3D Face Reconstruction Engine using Monocular Depth Estimation and Hybrid CNN Deep Learning Architecture with Real-Time Post Skeletal Mesh Fusion
Project Track: AI
Team Name: Blend Research Lab
Team Member(s): @blend
HackerEarth Project Link: Click here
GitHub: Click here
Project Website: Click here
Any smart contract links: Blend is a developer tooling
Pitch Deck: blend_pitch_deck.pdf (1.7 MB)
Hey everyone, here is the Demo Video: Watch on Youtube
Project Goal:
Our team aims to deliver an exceptionally realistic and immersive blockchain gaming and metaverse experience for Web3.0 gamers on Tron by harnessing AI technology
Project Value:
This groundbreaking project uses deep learning for realistic 3D face reconstruction from a single selfie, and introduces Web3.0 gamers as game characters, transforming gaming in the metaverse. Its technical prowess in computer vision, deep learning and seamless integration with Tron’s blockchain (TRC-721 NFT, BTFS, Account Abstraction) offers a scalable solution. Blend’s user-friendly SDK and related toolset make it accessible to all, and it is specially designed for on-cloud computation and pixel streaming of game servers. This innovation brings personalization to blockchain gaming and metaverse in the Tron ecosystem, making a notable contribution to Tron by promoting ownership and wider adoption of token assets.
Impacts on Tron gaming ecosystem:
Blend addresses the current issue of limited immersiveness in blockchain games within the Tron ecosystem, which also a major challenges faced by the Web3 gaming industry. When we refer to the metaverse, our aim is to provide gamers with a sense of genuine interaction within the virtual realm to the fullest extent possible. Our research team has developed a solution that involves using a single user’s selfie image, employing advanced deep learning techniques to generate an ultra-realistic face mesh of the user. This mesh is then seamlessly integrated with the game character in real-time. In other words, through a straightforward and accessible onboarding process, we transform a real-life Web3 gamer into a highly realistic in-game character, enabling them to embark on adventures or engage with other users in the virtual world. All the 3D files and information required to render the user’s metaverse human are securely stored on the BTFS decentralized storage, utilizing Blend’s specially designed NFT passport technology to establish an identity pass for all blockchain games integrated with Blend on the Tron platform, demonstrating the concept of interoperability in the Tron gaming ecosystem.
Introducing Blend on Tron
Blend is Tron’s Ultra-Realistic Game Character 3D Face Reconstruction Engine using Monocular Depth Estimation and Hybrid CNN Deep Learning Architecture with Real-Time Post Skeletal Mesh Fusion. Our team aims to enhance and focus on delivering an exceptionally realistic and immersive gaming experience for blockchain gamers in the Tron ecosystem with the power of artificial intelligence. Combining with Tron blockchain technology like Non-Fungible Token (NFT), Bittorent Decentralised File System (BTFS), and Account Abstraction, we finally built and compiled this ultimate toolbox to define a whole new level of on-chain game development on Tron.
Here is an overview from a higher perspective to let you understand what actually Blend is trying to achieve in order to create an immersive experience for your gamers. The whole pipeline starts with the gamers providing a single selfie image, and Blend will use this single selfie image to reconstruct an ultra-realistic 3D head with high-detail textures by leveraging AI techniques. Then, Blend will merge the head on a game character skeleton built according to Blend’s standard. Now, a real-life gamer has turned into a game character that can be controlled and played in a Tron blockchain game. All of the 3D files (stored as a folder on BTFS) of the generated game character will be bound with the user’s Blend NFT Passport in the form of a TRC-721 token.
Blend Three-Stages AI Pipeline
Stage 1: Image Pre-processing Phase
The first stage is the image preprocessing phase. Instead of performing multi-image 3D face reconstruction by exploiting complementary information from different images for shape aggregation, our team designed the pipeline to utilize just a single image of the user. We know that turning a single 2D image into a 3D object is challenging due to the loss of information from different perspectives. Hence, we solve this by first feeding the selfie image into a dynamic pipeline to enhance quality, remove unnecessary illumination, and color corrections. Then, the processed selfie image will be passed into a monocular depth estimation model to generate a depth map. This map indicates the depth of each portion of the user’s face from a single camera viewpoint, providing us with depth data in 3D space. This step is crucial as not all the camera devices on phones or laptops out there have a true depth or ToF sensor. Through this phase, we are still able to perform 3D reconstruction regardless of the user device. This is an important point to focus on as users’ capturing devices should not be the obstacle for them to onboard into the metaverse on the Tron network.
Stage 2: 3D Face Reconstruction
The second stage is the 3D face reconstruction pipeline. The depth data obtained from the first stage will be passed into the pipeline together with the selfie image. Blend implements a differentiable renderer, leveraging a 3D Morphable Model fitting algorithm. This algorithm leverages the depth data to recover facial geometries which takes advantage of a powerful 3DMM basis constructed with extensive data generation and perturbation. Our 3DMM boasts significantly enhanced expressive capabilities compared to conventional models, enabling us to attain more precise facial geometry utilizing only linear basis functions. For the synthesis of reflectance properties, we employ a hybrid strategy that combines parametric fitting and deep CNN. This combined approach enables the production of high-resolution albedo and normal maps with intricate, realistic details such as hair, pores, and wrinkles.
Here are more samples we generated using Blend’s pipeline:
Stage 3: Post Skeletal Mesh Fusion
After we obtain our reconstructed ultra-realistic 3D face, we will enter our last stage, which is the post-skeletal mesh fusion. Blend had a built-in Unreal Engine-based automated mesh fusion pipeline that would merge the reconstructed face mesh with the mannequin game character skeleton which is provided by the game developer seamlessly in real-time. Once everything is done, we will now have a complete game character generated based on the user-uploaded selfie and is ready to play inside the metaverse.
It is worth mentioning that Blend is a toolbox that our team specially designed to run in real-time. Hence, it is built in a way in which the entire 3D face reconstruction deep learning pipeline can be easily hosted on an AWS virtual machine and also connect with an Unreal Engine-developed game via pixel streaming technique.
Blend’s NFT Passport Concept on Tron (TRC-721)
Our team introduced a concept on Blend, which is called NFT Passport. It is a TRC-721 token and has the connection to all your metaverse human-related 3D files. In other words, this NFT passport is your identity card in the virtual blockchain game or metaverse, which is used to render and visualize your virtual self in the virtual world on the Tron network. This idea is a paradigm shift in how you onboard Web 3.0 gamers on Tron. Instead of storing all the related files of each user on your cloud server, retrieve them once needed. Now, the user just has to connect their wallet containing their NFT passport and we can retrieve the files that are bound with the token from a decentralized storage.
This feature is included in the toolbox of Blend in order to help developers set up the entire NFT Passport minting procedure and onboard Web 3.0 users to their blockchain game. Our team built a demo using Blend and demonstrated how we implement Blend’s NFT Passport in a Tron DApp and onboard people in the space. At the same time, it shows the capabilities of the NFT Passport. Below is the entire flow of the demo application:
When the 3D files of a metaverse human are ready, they will directly be stored on the BTFS decentralized storage. Now, you need to mint your Blend NFT passport. This NFT passport binds with all the 3D files on BTFS, and the blockchain game built using Blend will be able to retrieve all the related 3D files of a gamer’s metaverse human by just using his NFT passport. So basically, Blend’s NFT passport is bringing the concept of interoperability to the Tron gaming ecosystem. Users just have to create their NFT passport once, and able to start their adventure in different games built using Blend on Tron. Interoperability is essential to enhance users’ overall blockchain gaming and metaverse experience on Tron, and Blend helps with this perfectly.
For demo purposes, our team built two different games using Unreal Engine, using just one NFT passport (TRC-721 token) to shift between two different games but using the same face of one of our team members.
Demo Game 1 
Demo Game 2
Progressing Milestones (Post-Hackathon)
Project Milestones:
Our team is actively continuing the research and development of Blend, aiming to achieve even greater heights in the post-hackathon period. Currently, we are focused on several key initiatives.
Firstly, our team is dedicated to refining the entire pipeline for reconstructing a full head mesh, complete with high-detail texture, to achieve a more seamless and lifelike blending of the game character. This process involves three additional components: the upper scalp section, the side face, and the junction between the scalp and the side face.
These newly reconstructed segments will seamlessly integrate with the existing reconstructed facial part. We have chosen to isolate the top scalp section in the reconstructed head model to concentrate on incorporating Neural Haircut, a deep-learning hairnet, to recreate and generate detailed hair strands based on the Web3 user’s real-life hairstyle.
Here is a hair reconstruction sample based on a single selfie image using the Neural Haircut pipeline:
To further enhance the capabilities of our reconstructed head model, we have implemented an additional pipeline to automatically attach a face rig to the reconstructed face mesh. By employing text-to-speech AI and lip sync technology, we can effortlessly enable our Web3 gamers to take on roles within the narrative and engage with other in-game characters, creating a personalized gaming experience.
About testing:
-
The GitHub repo consists of the Blend toolbox which is a Pytorch-based Deep Learning pipeline and built in into a demo application which you can see it in our demo video. You can download the project folder and run it locally or host it on an AWS version if computer computation is required in your case. If you’re not familiar with this, feel free to watch our demo video and you will have a clear idea on how it works.
-
The demo web application is a simple Tron DApp that our team developed for demo purpose, demonstrating how Blend can be integrated into a Tron stack developed web application (to onboard gamers) and use Blend NFT passport to start a blockchain game via pixel streaming (demonstrating Blend’s real-time processing pipeline)
-
Two demo Unreal Engine games which is developed by our team using Blend can be downloaded (version 5.2 in executable file). This is to demonstrate the post skeletal mesh fusion pipeline which Blend specially built for Unreal Engien and how smooth the game character can be control and interact in the game scenes.
