Video Transcript
Unreal Engine Virtual Production Masterclass Complete Guide from Wiring to Final Composite
Jose Chemmassery · 4,738 views 24 Sept 2026 ITALY ROME · Duration 49:10 · youtu.be/ZIgqpYxGfsA
Unreal Engine Virtual Production Masterclass: Complete Guide What you will Learn Hardware & Wiring: Overview of all required devices and how to connect them properly.Camera Calibration - How to calibrate your camera for accurate tracking.Camera Tracking - Setting up real-time tracking data in Unreal Engine5.Live Keying & Compositing- Integrating live green screen footage using the Ultimatte 12.Final Output- Achieving a professional final composite ready for broadcast or film. Download files https://drive.google.com/drive/folder... Used Devices Retracker Bliss,Bmpcc4k,blackmagic Decklink duo 2,blackmagic ultimatte 12,blackmagic sync generator,blackmagic video assist,blackmagic mini hdmi ti sd
Auto-generated YouTube captions, grouped into ~45-second blocks. Product and technical terms may be misspelled. Timestamps link to the video.
This is the powerful and most affordable virtual production system you can ever build. I'm Josh Masheri from Pantheon Virtual Production Studio. Let's get started. Gather these boxes. Wire them up. Copy these files. Set up Unreal Engine. Now you have the powerful and most affordable virtual production system to take your film making to next [music] level. Okay. Now let's wire everything up. So we will get an input to Unreal Engine to do our camera calibration and model offset. Okay. Now we are getting our video signal to Ultimate and to Unreal Engine. Now if you check the monitor you can see we have a foreground input. We are shooting in 24 frames per second. So we have to keep that frame rate both in Ultimate and Unreal [music] Engine. To do that we are using a sync
generator to gen lock our signals. So [music] everything will be perfectly synced. We are going to use our Ultimate Tool HD as a key and [music] compositor. We are not going to composite inside Unreal Engine. We are going to do it externally. Now if you check the monitor you can see we are getting the composite. But the problem here is we are also seeing the unwanted areas of the stage by using the garbage mat input of the ultimat. Now if you check the monitor you will see we are getting the composite without any unwanted areas. This setup is perfect for static [music] shots. But if you move the camera everything get messy. Our background our garbage mat is not moving accordingly. To move them accordingly, we have to use a camera [music] tracker. Let's connect the
tracker to Unreal Engine using the live link plug-in. Here you have the powerful and most affordable virtual production system to take your film making to the [music] next level. Hi, I am Jos Masheri. Before we get started, make sure that you have downloaded all these files from my website and then go to the tag to print and print out all these tags. And we are going to print this ID number one. Make sure the ID is correct. And then print this into a A3 size paper. And print the aroo marker ID number eight. Make sure it's printed to 15 cm. And we need this camera calibration checkerboard. Print this into a A3 size paper.
Take the HDMI output from your camera and connect it to the converter. And take the output and connect it to the connector one of your deck link card. If you are using a camera with the built-in SDA output, you don't want to use the converter. You can connect it directly and then open your desktop video setup. Click on these three lines. So you'll get the input and outputs of your deck card. Go to the deckling do one and click here. Go to connectors here. Select SDI1 and SDI 2. Save. Connect the tracker to your computer using the USB3 cable.
Connect your ret tracker bliss to a USB 3 port of your computer. You will have to use a USB extension cable. The cable comes with the tracker is very short. I'm using a 10 m cable for this. Attach the retractor bliss on your camera and make sure it is perfectly leveled. Now let's set up the tracker settings. Install ret tracker bliss software and the ret tracker bliss settings that comes with the tracker and activate your tracker by contacting retractor bliss. If you are having any problem with the tracker or if you have any doubts about it, you can contact the retracker bliss.
They have a video call support and take you through all the steps you need to know. So no worries. Okay. After installing the software and the ret tracker bliss settings, go to the ret tracker bliss settings and enable live link bliss options and type the render ID as shown here. Render port 50,000 camera ID 1 and enable the tag tracking. Tag tracking port 50,000. Our printed April tag size is 25. So you have to input it here. And our marker ID is one. Tag camera ID 1 01. Set it up. And in
the world post options, enable full mode, autoquit and high precision. That's all we need for Unreal Engine. Now save the settings. Now start your ret tracker bliss and make sure it's starting properly. Okay, let's use our ret tracker bliss high precision calibrator to calibrate our lens. Install the software that comes with your tracker and then download and install the NDA tool 6 to work everything properly. Now make sure your camera is turned on and it is connected to the deck link card. Open
your high precision calibrator. Go to settings, input data. In the project frame rate, I'm using 24 frames per second. So I'm going to select that. Now come down in the lens data. Here you can enter the lens model and name. I'm using a 18 mm lens. Here you can add a serial number if you need. I'm going to leave it like that. In the sensor size, you have to enter the effective sensor size of your camera. You can check it online. I'm using a Blackmagic Pocket Cinema Camera 4K. You
can see mine is 18.96 into 10 mm. I'm going to input that data here. 18.96 and 10. Here in the raw office input zoom input method, I'm going to leave it manual. And the zoom I'm using a 18 mm lens. So I'm going to input that here. Focus input method. Here you have to enter the distance from your camera to the camera calibration checkerboard. Mine is 200 cm. As you can see here, everything is in mm. So 200 cm to mm it will be 2,000.
In the iris section I'm using F4 because my lens is set to F4. So I'm going to input that here. Now let's go to the camera calibration tab. In the settings, enable coverage map. So when you click on the checkerboard, you can see the overlay. Scroll down. In the calibration board, you have to enter the corner rows and calls. You have to count where the squares are intersecting. 1 2 3 4 5 6 7 and 8. In this side it's 1 2 3 4 and five. Let's input that data. So rows is five
and our call is 8. Now you have to enter the square side length. You have to measure it with a scale and mine is 2.7. Now let's click on the checkerboard. As you can see there is an overlay created that means it is detected. Now let's move the camera and leave it bit over overlapped and click again. I'm going to fast forward it. Make sure you are getting a good coverage.
Now, let's change the checkerboard angle a bit and then let's bring it a bit closer to the camera so you'll get some close-up shots, too.
Now let's change the angle a bit again and let's bring it bit more closer. Okay, that's enough. Now go to add to lens distortion and click on add. It
takes a moment to get done. Now scroll down. Here you can see your distorted parameters. Now we can do our nodal offset. Before you do anything, start the ret tracker bliss. Minimize this. And now let's place our April tag. Go to the settings and go to live options. Here assign liveing objects. Go to the camera and select the camera one.
Go to April tag and select the bliss camera 101. And make sure your tag and your camera is detected here. If it is not detected, it's not reading the April tag. There are two reasons. It might be the lighting because it's reflecting from the April tag or it is out of focus. Now let's go to the nodal offset tab. Scroll down. Make sure you have select the same tag as we have printed. And make sure the tag ID is one. Our tag size is 25 cm. Now click on generate.
Do a final check and make sure everything is correct. Now let's click on the April tag from different distances and different angles. Make sure it is detected. And always make sure the camera one and camera 01 stays Green.
Now let's change the angle and position a Now let's change the angle and distance Okay.
I think that's enough. You can have more samples or less. Now come to the add to n set and click on add. Now you can see it snapped perfectly. If you adjust the transparency, you can see it clearly or you can compare it. Now let's move the camera to see how everything works. As you can see, there is a delay. If you go to the settings, go to input data here, you can adjust the delay. I'm going to input one. And let's see. Now,
it's a bit off again. Let's go to the settings. Let's input two. And here you can see your nodal offset. Now go to settings, export, click on export, and set the file location and save it. Okay, let's see how to wire everything up. First, let's connect the monitor output of our Ultimate to the SDI input
of our Monitor Plus recorder. Then, take the HDMI output of your camera and connect it to the converter. If you are using a camera with built-in SDI output, then you don't need to use the converter. You can connect it directly. In this case, we are using a camera with HDMI output. So, take the HDMI output and connect it to the converter. In this way we are splitting the signal into two. One we are going to connect to the foreground input of the Ultimat and the other one we are going to connect to the deckling do2 card because we need a video input to Unreal Engine to align our virtual and physical set. Here we have a sync generator to genlock our signals. Take the reference output
of the sync generator and connect into the reference input of the ultimate. In the same way, take the reference output of your sync generator and connect to the deckling do card. In this way, our signals will be gen locked. We are going to use the connector one of the deckling do card to output our garbage mat. Connect that into the GMAT input of the Ultimat. and the connector three we are going to use as our background. Take the connection and connect it to the background input of the Ultimat. Connect the tracker to USB 3 port of your computer using a USB cable. Okay, let's set up our DeckLink DO2 card inputs and outputs for this project. Go
to blackmagicdesign.com under support. Select the capture and playback here. Scroll down and download the latest desktop video version. Download it and install. After installing the software, go to your Blackmagic desktop video setup. Click on these three lines here. So, you can see your all outputs here. Now let's go to deckling do1 here. Click here. Go to connectors and select the connector SDI1 and SDI2. Save it.
Go to deck link duo2. Click here. Go to connectors and select the connector three. Save it. Go to your deckling do3. Click here. And in the connectors, leave it none. Save. Go to your deck link do4. Click here. Go to connectors and select the connector 4. Save it. In this way, we are going to output our fill and key from the one and three connectors. We are going to output our GMAT from our
one connector. And we are going to output our background from the three connector. And we are going to get a video input to our connector 4. Let's wire up the Ultimate. Take the Ethernet connection from your Ultimate and connect it to the Wi-Fi router, the same router that you are using for your Unreal Engine PC. Now, let's see how can we use our Ultimate 12HD for virtual production. Go to blackmagicdesign.com, click on support, go to Ultimate, scroll down and download the Ultimate Software control software. After installing the software, open the Ultimate software control,
go to Ultimate, Ultimate Connection and select Ultimate HD. Save. Okay. Now, let's set up our screen color for keying. Go to menu. Use the scroll wheel to navigate to the settings. Select screen reference color. I'm using a blue screen. So I am going to select blue. If you are using a green screen, you can select green. Set it up. Now let's see how can we use our Ultimat for live keying and compositing.
Here in the foreground input, we are getting the foreground input directly from our camera. Background input that is our output from our Unreal Engine scene. By using the combined mat, we can refine our mat and clicking on this icon. You can auto key. Now, let's auto key it first. By doing the auto key and clicking on the program button here, you can already see the composite is looking good. If you go to the campaign mat, you can refine it further. You can use your cleanup levels to clean up any unwanted shadows and you can use the black gloss to adjust the false transparency.
Now, if you come to the program, you can see the final composite. Here in the foreground, you can adjust the black level and the contrast of your foreground input. If you go to your black level, foreground black level master, you can see you can adjust the black level. In the same way, you can adjust the white levels. You can you can bring it brighter. or darker. In the same way, you can adjust the background. If you go to the background white level master,
you can match it in real time. And here in the matting settings, we have the garbage mat. We will see later in the tutorial. Now, let's set up the Unreal Engine for virtual production. First we have to copy all the files that we have downloaded from the website. Open your Epic game launcher. Go to library and right click on the project that you are working on. Show in folder. So it will take you to the content folder. Copy the Aruko tag magic mask and the post-process material. Copy them and paste to the content folder.
Now we have to create a new folder for our live link plug-in. Note inside the content folder. Just click outside and create a new folder. Call this plugins. Now select the plug-in version according to the engine version that you're using. I'm using the engine version 5.6. six. So, I'm going to copy this 5.6 version. Copy and paste into the plug-in folder. Now, you can close the windows. If you check the content drawer, we have successfully copied our files. But we have to restart the engine to work
everything properly. Let's do it now. Go to files. Save the current level and close. Restart the engine. Now let's bring our calibrated lens file to Unreal Engine. If you are using the same camera and lens that I'm using, you can use this pre-calibrated lens files. Otherwise, use your own lens files. Just drag and drop them into the content jar. After copying all the files, we have to turn on some plugins and change some project settings. Go to edit. In the project settings, search for alpha.
Turn on alpha output. Now search for stencil in the post-processing custom depth stencil pass enabled with stencil. Search for lumen. Turn on use hardware rateracing when available. Go to the edit again plugins. And now we can turn on some plugins. Go to installed and turn on live link bliss. Go to all plugins. In the search, search for camera and turn on camera calibration. Now search for Blackmagic.
We are using the Blackmagic deck link card. So we have to enable the Blackmagic media player. Now search for media. Scroll down and enable the media foundation media player. media framework utilities and media IO framework. Make sure these three are enabled. Search for live link and make sure you have enabled the live link plug-in. Now search for timed data monitor and turn on timed data monitor. Now restart the engine.
After enabling the plugins and changing the project settings, go to windows virtual protection and turn on live link. Turn on timed data monitor. Turn on media capture. We will use the live link to connect our tracker to Unreal Engine. We will use the time data monitor to set up our tracking delay. Media capture to output the Unreal Engine scene to Ultimat. Now it's time to set up our virtual camera. To set up our virtual camera, first we need to set up our lens file. If you go to your lens file and double
click on it, you can see the letters highlighted in yellow. That means it is not getting some inputs needed. First of all, it is not linked to a virtual camera and it is not getting its f input. F means focus iris zoom. If you come to this part, you can see there is no focus curve and iris curve. First, let's create a file to fix this problem. Close your lens file. Now right click on the content jaw. Go to live link. Create a blueprint virtual subject. Here select live link camera roll. Okay. I am going to call this 18 mm
fizz. Double click to open it and click and drag to create a new node and search for virtual here. Select update virtual subject static data. Right click on it and split the structure pin. Here we have to disable all these things and enable static data is focal length supported. Static data is aperture supported. Static data is focus distance supported. Turn on only these three.
Now click and drag again. Search for virtual. This time select virtual subject frame data. Right click on it and split the structure pin. Here we are going to make three variables. Click on variables here and call this focus. Change the boolean to float. Click on this eyelid so you can edit it. Now create another one. Call this iris. Create another one. Call this zoom.
Now click and drag to the graph and click on get focus, get iris, get zoom. Now connect the zoom to the frame data focal length, iris to the aperture, and the focus to the focus distance. Now compile everything. Click on zoom. Go to the details panel. I'm using an 18 mm lens, so I'm going to input that. Go to iris. I'm using f4. So I am going to input
that here. In the focus section here, you have to input the distance from your camera to the camera calibration checkerboard. Mine was 200 cm. Now you can compile everything and save. Okay. Now we can create our virtual camera. To create our camera, go to place actors windows, go to scene and drag and drop a cine camera actor into the scene. I am going to rename the camera as camera out. Now let's create a basic actor. Drag and drop the actor into the scene.
And I'm going to call this camera parent. Then let's attach the camera to the camera parent. Click on the camera and add to the camera parent. Now if you move the parent, you can move the camera also. Now click on the camera out. Come to the details panel and click on this add button. Search for live link and create a live link controller. Call this fist. Let's add another live link controller. Search for live link and call this
tracked camera. Click on the add button again and search for lens and add a lens component. Okay. Now we have added three components to our virtual camera. Tracked camera we are going to use to connect our tracker. We will use the fist to attach our fist blueprint to the virtual camera. And the lens we will use to assign our calibrator lens file to the virtual camera. Now let's go to our camera parent and set its position to zero. And go to the camera and set its position to zero.
After setting the camera and the parent position to zero, start the tracker. Okay. Now minimize this. Go to your live link and add source here. Live link bliss source. Add. Now you can see we have added our tracker to the engine. Now click again here and go for the add virtual subject. Here we are going to add our fist blueprint. Select your fist blueprint that is 18 mm fist for this case. Select your own and
add. Now we have added our virtual subject and the tracker to the live link. Now let's go to the camera and come to its details panel and go to tracked camera in the subject representation. Here select camera one. Now go to your f and in the subject representation select virtual. Now let's go to our lens file and select your lens file and assign to the lens file slot here.
Scroll down and here apply distortion. Scroll down again and in the film back film back. All right. Choose lens file. Now we have set up our virtual camera for tracking. Now if you go to your lens file, double click to open it. You can see we are getting our fix input now and it is connected to our virtual camera. The only problem here we are not getting the focus curve and iris curve. Let's fix it now. Go to your lens file panel here. Click on it and go to focus.
Click on this plus icon. Input focus. And I'm using an 18 mm lens. So I am going to input that here and click add. Click on the plus icon again. And here I'm going to input 1,000 as our infinity add. Now we have the focus curve. Now let's go to the iris. Click on the plus icon. I am using F4.
So I'm going to input that here. and click again. And the maximum iris of my lens is 22. So I'm going to input that. Now we have the iris curve. Now you can see we are getting all the inputs needed. Now you can save it and close. Now it's time to create a media profile to output our Unreal Engine scene to Ultimate. To do that, go to the media profile icon on the top here. Click on these three dots. Create a new empty media profile. Select the media profile
and select. Now you can save it. Here there is no proxies configured. Click on the configure now button here. And I'm going to select one out and one in. 1 one configure. Now double click on the media profile. Go to the outputs. Media outputs here. Select blackmagic media output. Expand this. Go to blackmagic and the configuration. Click on this arrow. Here,
select fill and key. I'm using 1080p, so I'm going to select 1080p and 24 frame per second. We are using a sync generator to keep everything synced. So, click on the reference external. Make sure it's matching to your camera settings, otherwise the output will not work. Now let's gen lock our system. Go to the general lock section here. Override project settings. Custom time step. Select blackmagic SDI input. Expand this. Click on genlock.
And we have to configure it. Now here select your deckling do1 1080 progressive and 24 frames per second. Apply. Now go to the output tabs here underneath the media output and invert key output. Save it and close. Now let's go to our media capture settings. In the media capture here, click on this plus icon. Scroll down. Expand this. On the camera, click the plus icon.
And here, select your camera. In the media output, click here and select our media profile that we have created. Now, make sure you have turned on and connected all the devices properly. Now, if you click on the capture button, you can see we are getting a video output. No worries if you see some weird images because our camera position is sitting zero now, not in the scene here. That's why we are getting this bad image. Now let's check our ultimate software to see we are getting our foreground and our background. If you click on the foreground you can see we are getting the foreground. If you click on the background you can see
we are getting a background input. Okay. Now let's set up everything for the final composite. I am going to choose this area of the virtual set as my background and I need my character and the physical set to be appeared here. Go to your Aruko tag and take the Aruko ID8 and place it on the floor. Bring it little up so you can see it. And I need my camera to look in this way. So, let's rotate the aroo tag to 90° like this
and place the printed aroo tag onto your stage floor. Now, we need a video input to Unreal Engine to align the physical set and the virtual set using our Aruko tags. Right click on the content drawer. Go to media and create a media bundle. Double click on the media bundle and in the media source here select blackmagic media source. Go to blackmagic and click on this arrow and check auto. Here we are going to use our deckling do4 connection to input our video to Unreal Engine. Select deckling doall 1080p
progressive and 24 frames per second. Apply and save. Now if you drag and drop the media bundle to the scene. You can see we are getting a video input. We are not going to use this for the final composite. Instead we are going to use this to align our virtual set and the physical set using our aruko tags. Now open your lens file. In the calibration steps, go to nal offset and if you come to the nal offset algorithm, select nal offset aruko markers. Make sure the calibrator is selected to aro ID8. Now if you go to the outliner, you
can see there is a camera calip media plate is created. Click on it. and come down to its details panel. Input, expand inputs, expand media source. And in the media source, select our video input. Now we will get a video input to our calibrator. If you adjust the transparency, you can see it clearly. Now if you check on the calibrator points here you can see TL stands for top left top right bottom left bottom right. We are going to click on the aruka markers in this order. Top left
top right bottom left and bottom right. Now scroll down and make sure it is detected here. And now apply to the camera parent. Now if you go to the transparency section here. Now you can see the arot again Unreal Engine and the Arukot again our physical set snapped together. In this way our physical set and our virtual set from Unreal Engine aligned perfectly. Okay. Now let's close the calibrator. And if you go to your media capture and capture it,
make sure you are getting the output. Now go to your ultimate software and you can see we are getting the foreground input, the background input and if you go to the program mode, you can see the final composite. Now let's move the camera to see how everything works. You can notice a delay between our signals. The background and the foreground is not moving accordingly. To fix that, go to our time data monitor and here in the bliss,
change your buffer size to 200 and set the tracker delay to 0.1. Now you can see everything is synced perfectly. Here in the time data monitor, you can fix the tracker delay. If you have the delay in your video feed, you have to go to the ultimate. Go to settings inputs. Here you can set your foreground input delay. Mine is five. I have already set it. Your might be different. You have to find it and finetune it. Now let's move the camera and see how it works. Now you can see our background and our foreground aligned perfectly. There is no sliding or slipping.
As you can see, everything works perfectly. But when we pan the camera, we are getting the unwanted areas of the stage. We can use a garbage mat to fix this problem. To create a garbage mat, go to your place actors window. I'm going to grab a cube. And let's scale it.
Now go to your content browser, open the magic mask folder and let's apply this material to the cube. Select the cube and come to details panel. on the material. Select the material and apply. You can see it become invisible. Now, now select your cube. I'm going to call this magic mask. Go to the details panel. Search for depth.
In the render custom depth pass, turn it on and set it value to one. Now go to your virtual camera, select it. Come down to it details panel and search for render. Here in the post-process materials, click on this plus icon. Here, choose asset reference. Now go to your post-process material folder, select this material and assign it. Now if you go to your ultimate software
mat in garbage mat garbage mat in now you can see we are getting the composite without any unwanted areas of the stage. Now let's move the camera to see how it works. Congratulations. Now you have the powerful and most affordable virtual production system you can ever build. Thank you.
Transcribed 60 blocks · Source: YouTube