Blend: Character Engine powered by AI

Project Name: Blend: Tron’s Ultra-Realistic Game Character 3D Face Reconstruction Engine using Monocular Depth Estimation and Hybrid CNN Deep Learning Architecture with Real-Time Post Skeletal Mesh Fusion
Project Track: AI
Team Name: Blend Research Lab
Team Member(s): @blend
HackerEarth Project Link: Click here
GitHub: Click here
Project Website: Click here
Any smart contract links: Blend is a developer tooling
Pitch Deck: blend_pitch_deck.pdf (1.7 MB)

:wave: Hey everyone, here is the Demo Video: Watch on Youtube

Project Goal:
Our team aims to deliver an exceptionally realistic and immersive blockchain gaming and metaverse experience for Web3.0 gamers on Tron by harnessing AI technology

Project Value:
This groundbreaking project uses deep learning for realistic 3D face reconstruction from a single selfie, and introduces Web3.0 gamers as game characters, transforming gaming in the metaverse. Its technical prowess in computer vision, deep learning and seamless integration with Tron’s blockchain (TRC-721 NFT, BTFS, Account Abstraction) offers a scalable solution. Blend’s user-friendly SDK and related toolset make it accessible to all, and it is specially designed for on-cloud computation and pixel streaming of game servers. This innovation brings personalization to blockchain gaming and metaverse in the Tron ecosystem, making a notable contribution to Tron by promoting ownership and wider adoption of token assets.

Impacts on Tron gaming ecosystem:
Blend addresses the current issue of limited immersiveness in blockchain games within the Tron ecosystem, which also a major challenges faced by the Web3 gaming industry. When we refer to the metaverse, our aim is to provide gamers with a sense of genuine interaction within the virtual realm to the fullest extent possible. Our research team has developed a solution that involves using a single user’s selfie image, employing advanced deep learning techniques to generate an ultra-realistic face mesh of the user. This mesh is then seamlessly integrated with the game character in real-time. In other words, through a straightforward and accessible onboarding process, we transform a real-life Web3 gamer into a highly realistic in-game character, enabling them to embark on adventures or engage with other users in the virtual world. All the 3D files and information required to render the user’s metaverse human are securely stored on the BTFS decentralized storage, utilizing Blend’s specially designed NFT passport technology to establish an identity pass for all blockchain games integrated with Blend on the Tron platform, demonstrating the concept of interoperability in the Tron gaming ecosystem.

Introducing Blend on Tron

Blend is Tron’s Ultra-Realistic Game Character 3D Face Reconstruction Engine using Monocular Depth Estimation and Hybrid CNN Deep Learning Architecture with Real-Time Post Skeletal Mesh Fusion. Our team aims to enhance and focus on delivering an exceptionally realistic and immersive gaming experience for blockchain gamers in the Tron ecosystem with the power of artificial intelligence. Combining with Tron blockchain technology like Non-Fungible Token (NFT), Bittorent Decentralised File System (BTFS), and Account Abstraction, we finally built and compiled this ultimate toolbox to define a whole new level of on-chain game development on Tron.

Here is an overview from a higher perspective to let you understand what actually Blend is trying to achieve in order to create an immersive experience for your gamers. The whole pipeline starts with the gamers providing a single selfie image, and Blend will use this single selfie image to reconstruct an ultra-realistic 3D head with high-detail textures by leveraging AI techniques. Then, Blend will merge the head on a game character skeleton built according to Blend’s standard. Now, a real-life gamer has turned into a game character that can be controlled and played in a Tron blockchain game. All of the 3D files (stored as a folder on BTFS) of the generated game character will be bound with the user’s Blend NFT Passport in the form of a TRC-721 token.

Blend Three-Stages AI Pipeline

Stage 1: Image Pre-processing Phase

The first stage is the image preprocessing phase. Instead of performing multi-image 3D face reconstruction by exploiting complementary information from different images for shape aggregation, our team designed the pipeline to utilize just a single image of the user. We know that turning a single 2D image into a 3D object is challenging due to the loss of information from different perspectives. Hence, we solve this by first feeding the selfie image into a dynamic pipeline to enhance quality, remove unnecessary illumination, and color corrections. Then, the processed selfie image will be passed into a monocular depth estimation model to generate a depth map. This map indicates the depth of each portion of the user’s face from a single camera viewpoint, providing us with depth data in 3D space. This step is crucial as not all the camera devices on phones or laptops out there have a true depth or ToF sensor. Through this phase, we are still able to perform 3D reconstruction regardless of the user device. This is an important point to focus on as users’ capturing devices should not be the obstacle for them to onboard into the metaverse on the Tron network.

Stage 2: 3D Face Reconstruction

The second stage is the 3D face reconstruction pipeline. The depth data obtained from the first stage will be passed into the pipeline together with the selfie image. Blend implements a differentiable renderer, leveraging a 3D Morphable Model fitting algorithm. This algorithm leverages the depth data to recover facial geometries which takes advantage of a powerful 3DMM basis constructed with extensive data generation and perturbation. Our 3DMM boasts significantly enhanced expressive capabilities compared to conventional models, enabling us to attain more precise facial geometry utilizing only linear basis functions. For the synthesis of reflectance properties, we employ a hybrid strategy that combines parametric fitting and deep CNN. This combined approach enables the production of high-resolution albedo and normal maps with intricate, realistic details such as hair, pores, and wrinkles.

Here are more samples we generated using Blend’s pipeline:

Stage 3: Post Skeletal Mesh Fusion

After we obtain our reconstructed ultra-realistic 3D face, we will enter our last stage, which is the post-skeletal mesh fusion. Blend had a built-in Unreal Engine-based automated mesh fusion pipeline that would merge the reconstructed face mesh with the mannequin game character skeleton which is provided by the game developer seamlessly in real-time. Once everything is done, we will now have a complete game character generated based on the user-uploaded selfie and is ready to play inside the metaverse.

It is worth mentioning that Blend is a toolbox that our team specially designed to run in real-time. Hence, it is built in a way in which the entire 3D face reconstruction deep learning pipeline can be easily hosted on an AWS virtual machine and also connect with an Unreal Engine-developed game via pixel streaming technique.

Blend’s NFT Passport Concept on Tron (TRC-721)

Our team introduced a concept on Blend, which is called NFT Passport. It is a TRC-721 token and has the connection to all your metaverse human-related 3D files. In other words, this NFT passport is your identity card in the virtual blockchain game or metaverse, which is used to render and visualize your virtual self in the virtual world on the Tron network. This idea is a paradigm shift in how you onboard Web 3.0 gamers on Tron. Instead of storing all the related files of each user on your cloud server, retrieve them once needed. Now, the user just has to connect their wallet containing their NFT passport and we can retrieve the files that are bound with the token from a decentralized storage.

This feature is included in the toolbox of Blend in order to help developers set up the entire NFT Passport minting procedure and onboard Web 3.0 users to their blockchain game. Our team built a demo using Blend and demonstrated how we implement Blend’s NFT Passport in a Tron DApp and onboard people in the space. At the same time, it shows the capabilities of the NFT Passport. Below is the entire flow of the demo application:

When the 3D files of a metaverse human are ready, they will directly be stored on the BTFS decentralized storage. Now, you need to mint your Blend NFT passport. This NFT passport binds with all the 3D files on BTFS, and the blockchain game built using Blend will be able to retrieve all the related 3D files of a gamer’s metaverse human by just using his NFT passport. So basically, Blend’s NFT passport is bringing the concept of interoperability to the Tron gaming ecosystem. Users just have to create their NFT passport once, and able to start their adventure in different games built using Blend on Tron. Interoperability is essential to enhance users’ overall blockchain gaming and metaverse experience on Tron, and Blend helps with this perfectly.

For demo purposes, our team built two different games using Unreal Engine, using just one NFT passport (TRC-721 token) to shift between two different games but using the same face of one of our team members.

:wave: You can also download the demo game execution files for Unreal Engine (Version 5.2) here:

Demo Game 1 :arrow_down:

Demo Game 2 :arrow_down:

Progressing Milestones (Post-Hackathon)

Project Milestones:

Our team is actively continuing the research and development of Blend, aiming to achieve even greater heights in the post-hackathon period. Currently, we are focused on several key initiatives.

Firstly, our team is dedicated to refining the entire pipeline for reconstructing a full head mesh, complete with high-detail texture, to achieve a more seamless and lifelike blending of the game character. This process involves three additional components: the upper scalp section, the side face, and the junction between the scalp and the side face.

These newly reconstructed segments will seamlessly integrate with the existing reconstructed facial part. We have chosen to isolate the top scalp section in the reconstructed head model to concentrate on incorporating Neural Haircut, a deep-learning hairnet, to recreate and generate detailed hair strands based on the Web3 user’s real-life hairstyle.

Here is a hair reconstruction sample based on a single selfie image using the Neural Haircut pipeline:

To further enhance the capabilities of our reconstructed head model, we have implemented an additional pipeline to automatically attach a face rig to the reconstructed face mesh. By employing text-to-speech AI and lip sync technology, we can effortlessly enable our Web3 gamers to take on roles within the narrative and engage with other in-game characters, creating a personalized gaming experience.

About testing:

  1. The GitHub repo consists of the Blend toolbox which is a Pytorch-based Deep Learning pipeline and built in into a demo application which you can see it in our demo video. You can download the project folder and run it locally or host it on an AWS version if computer computation is required in your case. If you’re not familiar with this, feel free to watch our demo video and you will have a clear idea on how it works.

  2. The demo web application is a simple Tron DApp that our team developed for demo purpose, demonstrating how Blend can be integrated into a Tron stack developed web application (to onboard gamers) and use Blend NFT passport to start a blockchain game via pixel streaming (demonstrating Blend’s real-time processing pipeline)

  3. Two demo Unreal Engine games which is developed by our team using Blend can be downloaded (version 5.2 in executable file). This is to demonstrate the post skeletal mesh fusion pipeline which Blend specially built for Unreal Engien and how smooth the game character can be control and interact in the game scenes.


Welcome team to Hackatron S5, this looks amazing. I like the graphics. I am waiting for a demo video.

All the best


Oh wow, this is giving some sci-fi vibes and quite intriguing having through read through each paragraph of your offering. I see “Blend: Tron’s Ultra-Realistic Game Character 3D Face Reconstruction Engine using Monocular Depth Estimation and Hybrid CNN Deep Learning Architecture with Real-Time Post Skeletal Mesh Fusion” is a groundbreaking initiative aimed at revolutionizing the blockchain gaming and metaverse experience on Tron. The project uses AI and deep learning for 3D face reconstruction, merging the user’s face into a game character skeleton and storing the 3D files as TRC-721 tokens on BTFS.
Out of curiosity, I’ve few questions I need clarity on;

How does Blend ensure the privacy and security of users’ data, especially when using a single selfie for 3D reconstruction?

How does Blend plan to tackle potential challenges related to different facial expressions and emotions during 3D face reconstruction?


Hi @Prince-Onscolo, happy to hear that. The demo video will be open public by today or tomorrow. I will make an announcement in this forum post. Thanks for waiting.

1 Like

Hi @manfred_jr

  1. Our on-cloud computation will only retrieve and use the user’s uploaded single-view selfie once during the process and will not save this data again. The Blend’s NFT Passport, which the user mints, will be responsible for storing these 3D-related files as well as acting as a link between the user and any blockchain game developed with Blend.

  2. Our deep learning pipeline is able to predict and process situations like multi-angle faces, extreme illumination conditions, light occlusion, and extreme facial expressions in the single-view selfie image the user provided. As Blend is trained with a wide range collection of different face conditions and utilizes a modified version 3D Morphable Model.


Welcome to Tron Hackathon season 5, your project is good but there are some important questions.

What are the specific challenges of reconstructing a 3D face from a single selfie image, and how does the Blend pipeline address these challenges.

In what way does Blend’s NFT Passport concept work and what are its benefits, and what are some of the key technical challenges of integrating Blend with Unrea.

Elaborate the advantages of using a hybrid CNN deep learning architecture for 3D face reconstruction, and in what manner does Blend plan to achieve even greater heights in the post-hackathon period.

What is the accuracy of Blend’s 3D face reconstruction pipeline,and enumerate the computational requirements for running Blend in real time.

By what means does Blend ensure the security and privacy of user data,and identify some of the potential use cases for Blend beyond gaming and the metaverse.

Cool, thank you
I will be waiting

Thanks for replying, it was fun reading through your response. It’s great to learn that Blend emphasizes data privacy by only using the user’s uploaded single-view selfie once during the process and not saving it again. The concept of the NFT Passport securely storing 3D-related files and acting as a link to blockchain games is a commendable approach, the ability of Blend’s deep learning pipeline to predict and process various facial conditions like multi-angle faces, extreme illumination, light occlusion, and diverse facial expressions is impressive. It shows the robustness of the system and its potential to handle real-world scenarios. I’ve a few questions moving forward;

How does Blend ensure scalability to handle a large number of users concurrently and maintain high performance during the 3D face reconstruction process for real-time integration into games?

What are the potential monetization models for developers using Blend, and how does Blend plan to sustain its operations in the long run, ensuring continuous support and enhancements?

1 Like

Thanks @manfred_jr, happy to hear that! Hope you love our project and show your interest on us in the AI track.

For the scalability and performance when going to on cloud computation or real time processes, it really depends on the computation resources of the virtual machine which utilise by the game provider, and not us. However, our deep learning pipeline do designed for optimum performance during the computation process itself and also made to host on an AWS instances and pixel streaming optimize.

Blend itself don’t monetize, but the blockchain game provider do can benefit a lot by using Blend as the game can be level up to unpredictable level in the terms of graphics quality and immersiveness. Our research team set this project as a community open source project and welcome any talented one to join the development or research tasks. So in the long run, our work are mainly supported by fund source like hackathon (this time), grants, or any open source contributions in the form of donations from different parties (but not limited to).

1 Like

Oh there you go again with the really detailed response, thanks for replying and for providing insights into scalability and performance aspects, clearly understood and I really do appreciate.

I did miss a very important question bordering on education initiatives.

Are there plans to organize workshops, webinars, or other educational programs to educate both developers and potential users about Blend’s capabilities and how they can leverage them effectively?

That will definitely be a thing to arrange in the future. However, for this hackathon, our team will focusing on AI-related researching task and development more, and hope to speed up the progress for this project public release. But thanks for your recommendation and it will be on our list. Thanks @manfred_jr

1 Like

You’re welcome, thanks once again for replying to every inquiry, goodluck on your buidl.

:wave: Hey everyone, the demo video for Blend has finally been uploaded on YouTube ~ We spent a lot of time preparing this video to explain the concept and idea of Blend, and also demonstrate how Blend can be integrated into your next Tron blockchain game. As Blend is built with complex AI technology, our team knows that the video will be a bit lengthy, but you definitely can learn a lot from our project.

Let me know in the comment section how you think after watching it~ Enjoy!

1 Like

Hey @Prince-Onscolo @manfred_jr @Okorie~

The demo video for Blend is out~ Enjoy!

alright I will check it out

Oh wow, checking it out in a bit.

The thumbnail has this description; “Tron’s Ultra Realistic Game Character
3D Face Reconstruction Engine
using Molecular Depth Estimation and Hybrid CNN Deep Learning Architecture with Real-Time Post Skeletal Mesh Fusion” can you break this down to noob language haha

And first few seconds of the video my eyes did a typo by thinking that was "Studio Universal’ had to look again over and over to discover that was actually "Unreal":point_down:t2:

Haha @manfred_jr , it’s Unreal Engine~ Thanks for watching the demo video. The title in the thumbnail is the title of this research and development project, hope you can understand. But we do explain about what are the technical terms are in a breakdown way (if you watch through this video)

1 Like

I thought it was complex machine learning type language lol.

1 Like

Greetings @blend ! Your project seems to be missing details on:

Any smart contract links:
Project Website:
Project Milestones:

Please add them to your project. Thank you!