In this blog, we explore the exciting collaboration between We and Margarita, VR students, as we collaborate with Sound Art students to create a captivating VR game. We aim to develop an immersive experience that lasts 10-15 minutes, entirely in VR. The Sound Art students will play a crucial role by providing immersive sounds that enhance the gameplay, showcasing the incredible impact sound can have in a VR game.
For the collaboration project, Miranda and I were chosen to work with London temporary dance university students. The overall goal for our group is to animate the dancer’s dance in different forms, such as smoke, liquid, and feather. we will be filming dancers’ movements on the phone and also motion capture.
There are five people in our group:
Jason Cui (BA Virtual Reality)
Miranda (BA Virtual Reality)
Algo AU (BA LCDS)
Cynthia Cheung (BA LCDS)
Melina (BA LCDS)
PROCESS AND COMMUNICATION
For the first time, we went to LCDS and met our group members to discuss ideas and record videos. Experiencing AI Rokoko Tool and Motion capture room gives us a deeper understanding and unique ideas for doing this. here are some videos that we use as a reference for our early stage.
My Initial thought is VR can create virtual environments that simulate unique performance spaces, from grand theatres to fantastical landscapes. Dancers can perform in these virtual spaces while wearing VR headsets, transporting themselves and their audiences to immersive worlds that enhance the overall experience.
by removing physical barriers, This enables dancers to explore new forms and shapes, and interactions beyond the limitations of the physical world. it makes it more tempting for me to create this inclusive vision.
Firstly we asked them to stand in a T-pose position, the reason T-pose is used to establish a neutral position with aligned joints. This helps ensure accurate tracking and mapping of movements in motion capture. By starting with the T-pose, dancers provide a consistent reference point for the system to effectively recognize and track their body movements.
For the task distribution, Miranda is doing real-time Animation which she will be using a projector where dancers will have to perform at the same time. I will be using mainly Blender to make different element effects for three of the dancers.
I’m glad that everyone seems to enjoy the process, for the LDCS students have to dance 5 hours a day, therefore I understand that it will take a long time for them to post the videos that we require, during this period of time Miranda and I are trying to book the motion capture room, the room is hard to book cos its usually used by animation student and we are lucky that have a chance to become the first two students from VR that experience the mocap suits.
Motion Capture & Green Room Recording
Miranda and I tried the mocap suit and showed it to the entire class, we also made some animation clips for our other project, it was an enjoyable experience, here is the video of the dancers dancing in the green room starting with the T_pose.
Continuous Green Room Recording
The purpose of the piece is to show the dancer’s style in a way that morphs from solid to fiery or smoky shapes that add a more stunning visual effect and attract the audience. To achieve this capturing effect I use mainly blender and cinema 4D
We aimed to portray the movement of each element through human interpretation of our project’s characters. To accomplish this, we allowed the dancers to choose an element based on their personalities and dance styles. for example, Cynthia Cheung is very introverted and has soft dance movements, which represent the water element, I think it will have a good combination together.
For the liquid and smoke effect, I used a video from Algo, and the body needs to be in a t-pose when the recording starts. then I use the Rokoko website to turn him into an animation, and then I found a character from Maximo as Algo’s body and import it into Blender with the animation.
Experience & Rendering issues
There are a few things that I did when importing a character from Mixamo make sure to get rid of the animation on the body and have the automatic bone orientation.
Then I use the Rokoko blender plugin to target the bone and body together which I found very helpful than adjusting the bones and clearing the parent original bones from the body by myself.
The experience that I have when making the liquid effect is incredible, by increasing the revolution divisions and changing the mesh and diffusion, it looks so realistic!
then I started to put lighting, texture and render the scene.
Updates
Unfortunately, certain videos recorded by the dancers cannot be utilized on Rokoko, as the computer is not capable to convert their dance moves into animations. Consequently, we requested them to reshoot the footage. However, as previously mentioned, it is time-consuming to await the submission of their new videos. at the same time, I used the animation from Mixamo and test it.
Also, we are shocked that the rendering is taking way too long, 2000 frames nearly took me a whole day to finish.. therefore After extensive discussions, we have concluded that it would be more advantageous to conduct the rendering process at LCC where these desktops possess greater processing power compared to our laptops.
Reference & Inspiration
I like this flame dude the whole body transforms from smoke to fire.
Welcome to myblog.arts. This is your first post. Edit or delete it, then start blogging!
Research Blog – Research And development Process
Purpose
The main purpose of this project is to create a VR Music project. Based on ideas, imagination and solid research, Using MetaHuman to make 3D characters, download assets & online models to create the VR environment.
Introduction
The music that I chose for this VR movie is called Nevermind by Dennis Lloyd, Nevermind” is a track Lloyd wrote while living in Bangkok, Thailand. He moved there to focus on writing music, and met his ex-girlfriend there, where they dated for a year and a half. Lloyd states that he wrote the track at the beginning of their relationship. The track starts with the chorus, with Lloyd asking his ex if he left, would she tell her friends and would they support her through it. But then decides against it: “Baby, never mind, never mind.” I believe the song is about man who wants to be loved but is scared of leaving her or him getting left by someone but she is playing the victim. Thus he states, ‘’ your friend hold your hands’’. The first verse is Lloyd admitting that he is “ready now” for a relationship, but just asks his lover to be honest with him about being just his. After the chorus again, the second verse is same as the first one, except he is asking if his lover wants to “see his fire” rather than be his lover, and then the chorus ends the song. The song is very upbeat and dancey and has been received with extremely positive reviews.
The lyrics of the song:
What if I left and it made no sense And you tell your friends And they hold your hands Baby, nevermind, nevermind Nevermind, nevermind-What if I left and it made no sense And you tell your friends And they hold your hands Baby, nevermind, nevermind Nevermind, nevermind-Alright, I’m ready now, ready now I ain’t gonna, I ain’t gonna fall back down now Alright, I’ll take it on, take it on me Take it on me, hmm, babyAll I, I ever ask, ever ask Are you gonna, are you gonna be my lover? Tonight, and take it with, take it with me Take it with me, hmmWhat if I left and it made no sense And you tell your friends And they hold your hands Baby, nevermind, nevermind Nevermind, nevermind.
My thoughts & ideas
my ideas is to create a scene that looks like where Lloyd live in Bangkok, Thailand. Where there will be a swimming pool, house, gym and other stuff. And there will be two characters one is Lloyd and another one is his ex girlfriend. And the scene will represent the time when they just met each other and started their relationship. The scene are made base on the story behind the lyrics.
Problems
First of all I use maya to create some furniture for the interior of his house, then I apply some texture on the top of the chairs and tables, however it takes a long time and also when I import it to unity the color does not appear on the furniture, thereforeI have to redo it on unity. Also from the start I was trying to use maya to put some light inside the room which can helps to show more clearly about the interior, but when I import to unity I can not find the light, because of the errors that on the top right corner I can not process the music and the animation of the two characters.
Then when I made everything I just realised I didn’t change the system to android, and it contains a lots of errors on my console, therefore I can not show my animation and sound.
Solutions
while doing research online and finding videos about how to animate the characters, I realise there are so many easier way to model a product, some of them can even just download it online and import it to unity which saves me a lot of time and in unity I found out the app called unity asset, and there are plenty of ideas that inspired me from 3D to 2D, ground to sky, people already made it which convenience other people but some of them have to pay for the product as it’s very complex but fancy! I got a lot of inspiration from the resources, and its really helpful, by modelling and import the resources mostly from Sketchfab and unity and some products I made in maya, it solve a lot of problems for me, right now the only problems for me is to add the VR camera and delete the main camera, but in order to link on the bar headset there still a little bit more stuff need to do, and I already watched the tutorial on YouTube which I will catch up on that later on.
These pictures are my latest process on my project, and there are still quite a few stuff that I have to change, i will keep improving and updated.
