Video Transcript
Virtual Production Project Tutorial from scratch with Unreal Engine and REtracker Bliss!
Greg Corson · 22,156 views 3 Oct 2022 #unrealengine #virtualproduction #RETrackerBliss · Duration 1:08:05 · youtu.be/ebuLqk85VTQ
This tutorial starts from scratch and builds a project that replicates my "Neo's apartment from the Matrix" demo. I'm using Unreal Engine 5 and the new REtracker Bliss as my camera tracker. This setup will work with other camera trackers or no camera tracking at all if you want. To buy REtracker Bliss visit https://www.retracker.co/ With Unreal Engine 5 the whole process takes about an hour, once you have some experience you can do it even quicker! Use the table of contents below to skip around. If you have an old graphics card that does not support Lumen and Nanite on UE5, this project will still work, just don't enable them in the project settings. 00:00 Intro, what we will be doing 00:26
Auto-generated YouTube captions, grouped into ~45-second blocks. Product and technical terms may be misspelled. Timestamps link to the video.
hello everyone I'm Greg Corsten and in this tutorial I'm going to start from scratch and show you how to build an Unreal Engine project that will put you in Neo's apartment from The Matrix we'll be using Unreal Engine 5 to do this and you'll also need some kind of a camera that you know how to hook up to Unreal if you're going to be shooting your set with stationary cameras that's all you need but if you'd like to be able to move your cameras around you will need some kind of camera tracking system this time I'll be using a new camera tracker called retracker Bliss you can see its website right here and this little box on top of the camera that I'm pointing to is the actual tracker you'll get a pretty good idea how retract or Bliss works during this tutorial and you can also take a look at my previous demo of
me and Neo's apartment from The Matrix if you'd like to buy retract or Bliss the first 25 viewers of my channel who placed an order before the end of October will get a 25 discount for information on how to get this discount hang around till the end of the video or I'll explain it fully or you can check the video description just so you know retract or has given me one of their trackers to try and I've been helping them to improve the quality of the product I haven't taken any money for this and this video is not sponsored by retracker okay let's get started with the tutorial first launch Unreal Engine by clicking this button in the upper right after it finishes launching you'll see a window like this one we're going to create a film video and live event project
using the blank template and I'm going to call it list tutorial I'm also going to turn on Ray tracing here if you don't think your system can handle this you don't have to turn it on but I'm trying to show the maximum quality in this demo either way if you don't use the film and video template the setup of this project is going to be a little bit more complicated I'll try to go through the extra settings you'll need towards the end of this video for now let's assume you use the template after a few moments you should be looking at an unreal editor screen a lot like this one this is a good time to add any plugins you need to your project I need to install a live link plugin to use Bliss so I'm going to do that now if you have any other plugins you need for your tracking system or for anything else you use regularly on real projects
this is a good time to go and install them as well to do that just quit unreal back in the launcher find your project list tutorial right here and click show in folder now you see the folder where the project is stored on your computer create a new folder in here call it plugins and open that up now I copy my Bliss live link plugins over to here and if I was going to install any other plugins I'd copy them too then you can close this folder and double click on Bliss tutorial to start it back up when you're back in the editor we need to set up a few things so let's go up to settings choose plugins and we need to enable a few plugins here the first thing I'm going to do is enable Bliss because that's my tracker
you may have a different plugin for your tracker there's different ones for Vive and other things it's probably going to tell you you must restart Unreal Engine for your changes to take effect but I'm going to ignore that for now because I'm going to do a number of other things that may also require me to do a restart and I only want to have to do that once because we're using the film and video template a lot of the things that we need are already turned on for us but there are a few things that we still have to do start by clicking all plugins and then type in time data Monitor and turn that on now type camera calibration find that and turn it on you may also need to turn on a plug-in for your video capture card mine's called Aja and as you can see it's already turned on so
I'm good there now you can close the plugins window go back up to settings in the editor and this time choose project settings start by scrolling down until you can see engine general settings here and click it then scroll down to you see section titled frame rate you want to click use fixed frame rate and set the frame rate to whatever it is that your camera uses I'm in the US so I'm using 29.97 now go down to time code and change it to the same number in Unreal Engine 5 they introduced Lumen and nanite a high quality geometry and lighting system that's brand new and makes things look really pretty the template we chose will have already
turned on most of these things but there are a few things we have to turn on ourselves start by scrolling down to engine rendering and click it then scroll down to you see global illumination right here so you can see a few things already have Lumen turned on but there are a couple things missing under aluminum we want to check use Ray tracing when available and you should also check raytrace Shadows now we need to restart unreal to let it apply all the changes to plugins and settings we've just done so go up and click the close box then find your project Bliss tutorial and start it back up this can take a little time because some of the shaders may have to be rebuilt when the editor finishes restarting you can close this window here and you'll be ready to make
changes now the first thing I want to do is get rid of some things that I'm not going to use and you can see that's pretty much everything except for the floor you can hit delete and say yes to all and now you've got a nice black screen with nothing in it that you can see first thing we want to create is a point light so we'll go over here choose Point light and then go down and say that it's movable when you make a light movable that means that you can use the full high quality Lumen lighting so now we can see the light and the floor now I know I told you I was going to show you how to put yourself in Neo's apartment from The Matrix but to start out with I'm going to use a lot less cluttered and more simple scene so you can see what's going on a little easier
so we'll make that now so the first thing we want to do is have our floor centered at zero zero zero the template creates it at 20 for some reason so I'm just going to change that down to zero here and you can see the floor move down a touch now I'm going to create a really basic scene so we have something to test with I'll go up here and go down to shapes pick Cube and drag a cube out into the scene and I'll go back again to shapes go to Cone and drag a cone out now you probably notice these are stuck in the floor so we need to make a few adjustments I'm going to start by making both of these things smaller because right now they're a meter tall and that's kind of big the easiest way to do
that is just click this lock next to the scale and that'll allow you to just change one of these numbers and it'll propagate everything through all three so I'm going to put in point one five that scales the cone down to about 15 centimeters and I'll select the cube and I'll do the same both of these objects now look pretty small so I'm going to use my mouse wheel and move forward to get us a little closer and then we'll make some more adjustments both these objects have their Center points in the center of the shape so to get them out of the floor I'm going to move them both up by about half their height which is 7.5
centimeters you can see the cone jump up and now the cube so now they're both sitting directly on the floor then I'll move both of them closer together so they're near the center of the floor I'll use x of 0 and a y of about 30 centimeters from the center and for the cone I'll do the same X of 0. but this time I'll make it minus 30 centimeters so when we get a little closer the spot where the pattern meets up is zero zero zero our origin and the cube and The Cone are both about 30 centimeters away from it the last thing we need on our virtual production stage is a way to market the
center of it and a way to align that center with the center of the real world stage this asset takes a little bit to build so I provided it as a separate file you can find a link to the asset file in the description of this video you'll want to go download that now and then I'll show you how you install it once you've downloaded that small file unzip it and inside you'll see a folder named a ruko tag like this then you can just quit out of unreal right click on your bliss tutorial say show in folder and you'll see the folders again where all your content lives go into the content folder and copy that aruko tag folder into it just drag it and drop it then you can close this up
and restart list tutorial now in your project you should see this a roko tag folder here in the content browser go ahead and open it up and you'll see this actual picture of sort of this funny looking image that looks like a tag go ahead and open that up as well now you should be in the blueprint editor go up here and click on the viewport tab and roll your mouse wheel to zoom in on the object on the screen should look pretty much like what you're seeing here don't worry too much about this I've tested this a few times and so far it always works for me but if for some reason your tag doesn't look like this please write me in the comments for help now go up to the blueprint editor tag for the uruko and close it
and we're going to drag one of these out into the world and use it to Mark the center of our stage now the center of the stage is going to be at zero zero zero and somehow I've managed to exactly land this tag on zero zero zero if yours doesn't you can just type zero zero zero into location and also zero out the rotations you may notice the tag is flickering like you see it here if that's happening and you don't like it you can just add a very small number to the Z of the tag like .01 and that'll stop it from flickering now it's time to hook up your camera to Unreal unfortunately this is something I can only advise you on because everybody's camera and camera card is going to be a little different than mine so I'm going to show you how I
do it with mine this is an Aja Aja card and the setup here will be very similar if you have a black magic card if you're using something like a USB video in or a webcam it's going to be potentially way different I'll try to do a separate tutorial on how to do that so here's the basic process that I use for setting my card up go up to the content folder the top level you can right click and choose new folder and I'm going to call it Aja video and I'll go into that folder and right now that's empty at this point you want to make sure that your camera is turned on and that your video capture device is definitely seeing the camera it also helps if you know what format your
camera's video is such as HD 4K 10 bit 8-bit and so on video capture cards from Aja and Blackmagic both have desktop apps that can show you the format of your camera and let you know that they're getting a good signal for other kinds of capture cards it'll be different sorry I can't be too much help there so here's the basic process I right click and I choose media and I'm sorry you can't see this it's off the bottom of the screen I'm capturing but you go down to Media bundle and click it and now you have New Media bundle and I'm going to call this Aja camera one and you see it also created this inner assets file over here go ahead and double click now on your media
bundle and you should see a screen much like this one the first thing you do is set your media Source you'll see there's sources for our Azure media Blackmagic and some other things I'm going to choose Aja and then you need to expand it and under the first one here Aja it needs you to set the video format my card's got four inputs and my cameras on HDMI one I've currently got it set to 4K UHD Progressive and it's running at 29.97 frames per second your camera may be different now go down to the video section here and expand that uh you obviously want capture video to be true whether you need 8-bit yuv or 10-bit yuv depends on your camera mine uses 10 bit
it also has a check box for is srgb that's a formatted video for most cameras you want that on usually if it's off you'll see the colors of your camera look rather odd and you can try just flipping it back and seeing if they look better with it turned on now you can go back up to the top here save this and close the window now to see if the camera is working properly I'm going to drag it out into the world I have to back up just a bit and move it and now you can see that my camera is looking into the corner of my green screen studio now you have to leave this asset in your world someplace to keep the video going but you don't really want it to be visible so I'm just going to drag it down below the floor now we need to set up the composure
composite that you're going to use to put the live video together with the CG background to do that come up to window virtual production composure compositing and this window should appear click right click in it anywhere say create new comp choose empty comp slot now right click on this say add layer element you'll get this doc and click media plate right click on here one more time add layer element and this time you want to click CG layer
this is pretty much the simplest layout you can have a media plate for foreground and a CG element for the background just to keep things organized I like to go up here in the outliner and create a new folder for composite elements and then I just move the various compositing passes and the camera into them just to keep things organized before we can set up our composite we also need to put a camera into the world for it to use so start out by going up here and selecting under cinematic cine camera actor and we also need to build a parent for this so I'm going to go up here
select under basic actor select that in the outliner and rename it to camera parent then take your cine camera actor and attach it to the camera parent by dragging and dropping it on top you can see it's off to one side now that means the two are attached click on your cine camera actor the location and rotation should all be zeroed out like this if it's not go ahead and zero them out then go back and click on camera parent it may be just about anywhere so for now I'm going to zero it out get a slightly better angle on it like this and you can grab the X and move it back grab the Z and move it up a bit switch to the rotate widget
and tilt down and you should see a view of this little scene that we've built don't worry if it's a little bit out of focus or something we'll fix that I'm going to move it back just a bit more so you can see the whole thing we'll fix that out of focus issue soon now we can finish building the composite start with the media plate click on it come down to where it says input and expand that under inputs that should already have one array element in it expand that and then expand media Source scroll down a bit and you'll find this section called media Source you want to hook this up to the output of your camera so click here scroll down and you should find your Aja camera texture just click on it
and now the media plate is working to set up the CG part of the composite first we need to identify all the objects we want to appear on the screen in this case it's pretty much all of them to set this up we want to put all the objects we want visible in a layer so to do that I got to get in a position where I can actually select the objects I want in the layer I can either do that on the screen here by clicking on them or I can do it here in the outliner sometimes it's easier just to click them in the outliner so what I want to see in my CG window is the cone the cube the floor and I also think I'll have it show you the arucou tag now go over here to window select layers
you should see the layers window appear here or possibly somewhere else on your screen right click in here and say add selected actors to new layer now all these actors that we had selected here are in a layer for us you can name that layer but you don't have to I'm just going to leave mine named layer 1 like you see here click back on the outliner window and select the CG element and you'll see a bunch of things down here scroll down a bit till you see composure and expand it find input and expand that too under capture actors hit the plus to add a group and then open up this
you want an include group meaning that the things in the layer will be the things that we render and then we want to select the layer that we just made now the CG layer is all set and you can see everything rendering in it up here to finish setting up the composite we need to tell it how we want to connect everything together that's up here in the composite you want to scroll down until you see output here select that hit the plus and on output pass I'm going to choose player viewport that means it's going to put the output up here on the screen for us it's also possible to send the output to your video capture card if your video capture card supports output the last thing is to set up a material that controls how all these passes are Blended together
right above output you should see transform compositing passes open that up and create one by placing the Plus you'll see it says compositing element material pass which is what we want so open that up now we need to actually make one so we go into here and select material and we give it a name I'm going to call it compositing material and save it then you can just double click it to open it up and this is your basic empty material go over to the details section here on the side and under material domain you want to select post process and you'll see this material node changes a bit right click in an empty
space over here to bring up a menu and type in texture and select texture sample parameter 2D from the menu then you can right click on this and choose duplicate because we need two of them finally right click over here and choose over these two Samplers need to be named based on our compositing passes the first one will be media plate one the second one will be CG element one the media Plate's going to be in the foreground so we hook it to the top hole of the over node and the CG element is going to be behind
that so we take that and put it in the background here then we take the output connected to the emissive color and we're done save everything the last thing to do to get a working composite is to set up our chroma key that's what removes this green background and replaces it with the CG stuff to do that you go click on your media plate and you can scroll down here to transform compositing passes under transform passes if it's not open open it up the first thing you'll see will be chroma keying and you want to make sure that's enabled and for now we're not going to try and fine tune it we're just going to set up the color of the chroma key so under key colors press Plus
and then use this cross and you'll get a window like this that shows your setup now just take the eyedropper and scrub it over the green screen and click accept don't worry if this doesn't look perfect for now it's good enough if you go back now to the composite you can see the chair sitting in between the cube and The Cone that we set up as I said the key is not quite right but we'll clean that up later and we'll also get this properly aligned so everything's in the right scale the last thing that needs to be done to get good virtual production results is to calibrate your camera this isn't terribly complicated but it does take a little time you can skip it
but I'd definitely recommend you don't because doing it will save you a lot of time later on you only have to do this once so you can reuse the calibration information for every project you do after this if you happen to be using Vive trackers in your setup I've got a separate tutorial that covers how to measure your camera's Distortion and nodal Point entirely using the Vive and unreal's calibrator I'll put a link up for that now you can also find it in the video description if you don't have a Vive I've done another tutorial on how to measure the nodal point of your camera and the link for that is up right now I'll also put that in the description don't worry about getting the calibrations exactly right the first time if they're not quite right you'll probably notice it as a slipping between
the real world and the CG world when you move the camera you'll only see this if you're moving the camera of course and you have a tracking system if you're planning on shooting with a stationary camera it's not so much of a problem in any case if you notice problems later on down the road you can always come back and try redoing these calibrations to get them more exact it's not a difficult process it just takes time so let's get started on this the first thing we need is what's called a Fizz blueprint which stands for Focus Iris and zoom these are the three major settings that you have on any camera lens some people have special Hardware attached to the lenses on their cameras that report these values back to Unreal in real time but most of us just decide to use our
cameras at a fixed focus and zoom setting throughout a shot so that we don't have to have this extra Hardware to make everything look right the camera unreal is using to shoot the CG needs to match the camera we're using to shoot the real world this Fizz blueprint tells unreal the settings of our current lens so it can match them with its camera to make this Fizz blueprint start out by right clicking in the content browser here go down to live link and select blueprint virtual subject It'll ask you to select a role and you want to select live link camera roll and I'm going to name this virtual fizz and double click to open it go to the variables section here and click the plus to add a variable
in this drop down select a float and name the first variable focus and click the plus again and name this variable Iris and one more time and name it Zoom next to each one of these videos is a closed eye you may want to click each one of those to open it that'll allow you to adjust any of these things back in the editor whenever you need to right click here in the event graph and choose update virtual subject and you want the static data put that up here connect it to event on initialize right click and choose split struck pin so that you can now see all the different things
again right click and choose update virtual subject but this time you want the one for Frame data this one you connect to event on update and you also want to take the Frame data and split that structure pin in the update virtual subject static data box you want to turn on is focal length supported is focus distance supported and is apture supported turn off the others leaving just these three turned on now connect the three variables that you created earlier to the right places over here on update virtual subject Frame data so we'll start with zoom and drag
it out say we want to get it and we'll attach that to Frame data focal length and we'll pull Iris out get that and attach that to Frame data aperture and again with Focus and that gets attached to Frame data Focus distance go up and press the compile button here and then you want to set up the default values for these three variables for the zoom set whatever setting you're going to use on your lens whether it's a zoom or a fixed Focus lens for mine that's 24 millimeters
and select Iris and put in the iris value and I'm going to put in F22 and for Focus I'm going to put in about a meter and a half those are good starting values we can change them based on your actual camera setup now you can close this next we need to start everything up and hook it up to live link I have retracker Bliss which requires me to run an application in Windows to talk to it so I'll double click that application now it opens up a Windows command window and after a few seconds it should connect to the Bliss and start running
and over here you can see the status of my tracker is high confidence and it's running at a very high frame rate and these are the IP addresses that it's using to send me information if you're using a bliss you may need to change these in the settings file to match whatever setup you have and I'll go up to Windows virtual production and choose live link Live Links the system unreal uses to bring tracking data into itself sort of hook up the Bliss I click on source choose live link Bliss source and over here I need to put in the proper port number which is fifty thousand that was in the Bliss app on its screen
as you can see right here now you can see the Bliss is receiving I've got camera one and the green light means that everything is good for a start click on the Bliss up here make the max number of frames to be buffered 200 and you also need to tell it how much to delay the tracking data so it'll match up with your camera's video I'll show you how to set this properly later for now I'm just going to set it to what I know my delay should be which is point point one four five of a second
then we'd have to hook up the virtual Fizz object that we made so we go in here and say add virtual subject and we choose the virtual fizz and now all that's set up and ready to go the last thing to do here is to make a preset and save it so we won't have to keep re-entering this information every time we start up so go up to the presets up here and click save as preset and I'm going to call this preset live link Bliss preset and save it to make sure that this loads up automatically every time you start unreal go to the gears up here in the live link window click them and here where it says default live link preset
choose the one we just made now this should load automatically every time that you start up this project we just have to connect it to our cine camera actor so go into the outliner click on the city camera actor and click on the add button and type in Live and choose live link controller and just to keep them straight I'm going to name these one of the first ones going to be live link controller fizz now click the plus up here again type in Live Again add another live link controller
and we're going to call this one live link control component controller Bliss so you can see the two things here one of these is going to be bringing in the data from our Fizz and the other one is going to be bringing in the live data on where the camera is to finish setting these up go to the one named Bliss under subject representation choose camera one and you should be ready to go then under fizz find subject representation and choose virtual and now all that stuff is all hooked up now that all the tracking is set up if you find your camera here in the unreal editor window and you move your real
world camera the tracking should be coming through everything won't be aligned correctly yet but you should see the camera pan and tilt when you pan and tilt the real world camera so we're almost ready to do the calibration now's a good time to stop and print out a couple of things that you're going to need to get it done one thing you need is a black and white checkerboard this doesn't have to be unusually large you can do it on a standard sheet of writing paper just print it on your inkjet or laser printer I've included a checkerboard pattern in the file you downloaded earlier to get the uruko tag information also in that file there's a PDF of an aruko tag to print when you print the uruko tag be sure to
select actual size after it's printed the black part of the uruko should be 15 centimeters square as far as the checkerboard goes you can print that any size and you don't have to use mine you can use pretty much any checkerboard pattern if you've already printed one for some other purpose it just needs to be black and white squares and it needs to have a reasonable amount of border around the edges so there's some white between the edge of the paper and the start of the first Square it can also have almost any number of squares horizontal and vertical but I'd recommend that you have at least five in each Direction ideally both the uruko and the checkerboard should be printed on matte finish paper standard printer paper should be just fine there's no need to use special photo printer paper to get good results
these have to be the right size make sure that your printed aruko tag is exactly 15 centimeters square as far as the checkerboard goes you just need to measure the size of one of the squares and measure it as exactly as you can in millimeters you should attach both of these printouts to something stiff and flat like heavy cardboard a piece of foam board some blue board insulation from a home store or a thin sheet of plywood you can attach these permanently using something like contact cement or you can tape them to the board or connect them with binder clips the main thing is that you want these pieces of paper to be held perfectly flat and don't give them any chance to bend or wrinkle now that you have your calibration
boards all printed out we just need to make a resource to represent one of them luckily this is pretty easy because all you have to do is create one up here in unreal under virtual production checkerboard and then you fill in the measurements of your checkerboard the checkerboard I'm using has five rows and seven Columns of squares that are 5.15 centimeters across because of some Oddities in the math that they use we need to subtract one from the number of rows and columns when we enter them here in unreal so for mine I need four rows and six columns and I put in my Square size of
5.15 centimeters the other thing we need to do is set the materials for the squares black and white is typical but you can set up a different contrasting color and it makes things a little easier to see so I'm going to just put in Black for one of these types of squares and red for the other one and now if I go over here into the window I can move the checkerboard to where I can see it and you can see I have a black and red checkerboard at the right sizes the last resource we need is a lens file which is where all the camera calibration information gets stored to get one of these right click here in the content drawer go down to miscellaneous and
sorry this is off my screen but about halfway down you'll see lens file with a green bar under it just click that and you'll get a lens file like this one up here I'm just going to name that based on my camera lens a Sony 24 to 70. now go up to the cine camera actor click it select live link component controller fizz scroll down a bit until you see camera roll this might be set to camera type or might not set it to live link camera controller scroll down just a bit more and you see this lens file picker
so here you just select the lens file we just made and now everything's connected up and we're ready to start calibrating double click on the lens file you made earlier I'm going to set it up for full screen so you can see it well so there's a number of fields you have to mess with but not all of them the main thing is Lens model name which I'm going to set like I did the lens file to Sony 24 to 70. serial number I don't really care so I'm just going to put in one two three four five if you have multiple lenses particularly if they're similar you might want to use the actual serial number to help you keep track of the calibrations now the next part can be a little tricky you need the sensor dimensions of your camera's image sensor
and they need to be pretty accurate the problem here is that not all cameras use their entire sensor so just grabbing the sensor Dimensions from the manual doesn't always get you there a lot of cameras have 4x3 aspect ratio sensors and only use part of the sensor when you're shooting in 16x9 which is the ratio that we use for most HD recording so this is the one place I really can't give you too much guidance you're going to have to look through the manuals of your camera and find out whether it uses the full width of the sensor when it's recording video in the case of my camera my sensor is 35.7 wide as you can see I've already entered here but it doesn't use the entire height because it's a 4x3 sensor and we're shooting 16x9 so I can get the height by dividing 35.7
by the 16 by 9 aspect ratio which is I think 1.77 something and that gives me 20.081 and that works for this camera the other thing you need is the image dimensions in terms of pixels in my case I'm shooting standard 4K video so that's 3840 by 2160 as I've entered here The Lens model should be spherical lens that's the only one and everything else you can leave pretty much at the defaults from here on I'm going to be doing a simplified version of lens calibration now what you can do if you want to is it's possible to calibrate your lens at different Focus positions Zoom positions and Iris positions to make sure everything is exactly spot on in practice we really usually don't need
to do that and to prevent this tutorial from taking forever I'm just going to calibrate it for the standard Focus zoom and Iris settings that I normally am using this should be fine for you particularly if you don't have some kind of lens encoding system to feed you changes in the lens or if you just don't change your lens setup very often you can already see here that the rough is information that we had in our Fizz object is coming through to the lens calibrator now the lens calibrator can take this data and remap it if you are using a lens encoder a setting of 24 millimeters for the zoom might come out as some other number in our case I'm going to just make it a one-to-one mapping so whatever we put in the focus Iris and zoom comes straight through into the lens calibrator as is this is pretty easy to do just go up to the lens file panel tab click on Focus click on Plus
and you'll see it's asking us to add a data point so I'm going to put in that an input value of 1 gives us an output value of one and I'm going to add one more point and I'm going to do an input value of 10 meters is an output value of 10 meters so now you can see we have a straight line curve the Fizz is currently saying we're at 1.5 meters and the output is going to be 1.5 meters and that's confirmed down here now we're going to do the same thing
with the iris okay plus I'm going to say with the input of F1 that the IRS will be F1 and one more plus sign [Music] and a mapping of say F32 we'll also output F32 and now you can see that's coming through down here you don't have to do the focal length because the calibrator is going to figure that out for you so we're done on this screen and you can put it go back to the calibration step screen now now we need to shoot a number of images
of our checkerboard that the calibrator will use to figure out the Distortion of our lens and fix it up for us so set up your checkerboard in the studio and have it well lit try to make sure there's no glare off of it or any hot spots it should be a far enough away from your camera so that it feels maybe about a third of the image at this point we need to connect our camera's video feed to the calibrator we go up and look at the unreal screen again and if you look over in the outliner you'll see a section here that's got camera calibrator in its name we want to set up this plate right here that says camera calibrator and you're going to scroll down to the point where you see media Source right here and this gets set up similar to the plate we used in our compositor we just
find the video feed which is the texture and select it and now you see it's installed here and if we go back to the camera calibrator we can see sort of an overlay that's partly the CG screen and partly what's coming through the camera here in the calibrator there's a transparency slider that controls the mix from the reel to the CG you can see it changing here for calibration it really doesn't matter where you set that we'll use it a little more later I'm just going to use it on 50 percent now all we have to do is shoot a whole bunch of images of this checkerboard for the calibrator to work with to do this you want to get good coverage that means you want the calibrator board to appear in all the corners around near the edges it pretty much every place the
lens can see it just keep this tutorial short I'm going to speed this part up and not take quite as many images as I normally would you want to do in general is take one set of images looking straight at the board like this then take a set with the board set at a 45 degree angle in One Direction and then another 45 degree angle in the opposite direction by the time you're done you'll probably have taken somewhere between 16 and maybe 30 images I'm not going to do that many just to keep this short
once you've got all your images come down to add to lens Distortion calibrator you might have to scroll down to see it and just click it you'll get a dialog like this one that gives you your error normally anything below 1.1 is fine if I'd taken more shots of the checkerboard I could have gotten this error even lower how many shots you take is up to you now you can see the camera calibration settings have been filled in here the focal length is set to 24.915 now I have a 24 millimeter lens so this is slightly different but the markings on your lenses are never quite
exactly correct so usually if this number is reasonably close it's fine these other parameters have to do with how distorted your lens are they're fairly small on my lens on yours it may be different to double check your lens calibration go up and click on the nodal offset tab select noodle offset checkerboard make sure that checkerboard is pointed at your camera calibration checkerboard actor it's only one and then click in the screen you should get a bunch of detected Corner points here then click apply to calibrator now what you're looking at is the video image overlaid on the checkerboard
that's part of your unreal scene and if you fade back and forth in transparency you should notice that the two match exactly if you go back to your unreal scene you'll see the camera and you'll see the checkerboard in roughly the same orientation that you see them in your own Studio if for some reason when you do this you don't you can't see the checkerboard it's probably because the camera is tilted in such a way that the checkerboard is blocked and or under the floor you can get around this by just clicking on the camera parent and temporarily tilting the camera up or down to it's more or less horizontal so that the checkerboard won't be blocked by anything else unfortunately the lens calibration process is a little bit tricky if you
don't do it just right you might not get good results the easiest way to tell if you're doing it right is to try doing it more than once and see if you get more or less the same results they probably won't always come up exactly the same but they should be pretty close if you need to do the lens calibration over again you can just go up to the lens file panel click on focal length and hit this trash can click on Distortion hit this trash can and click on Image Center and hit this trash can that'll reset everything so that you can do the lens calibration over from scratch my advice is not to obsess too much on getting the calibration exactly right just get a good one to start with try some shots and if you're noticing problems think about recalibrating again
the last thing that you need to do is to set the tracking offset I've talked a little bit about the tracking offset and then the nodal point or entrance people in other videos so I'm not going to go too deep into it here I'll put one of the videos that shows how to get a good estimate of your entrance people location on the screen now once you have that information what you want to do is measure from this Tracking Center on your tracker down to the nodal point on your lens and make a record of that when you have those numbers go back to the lens file panel click on nodal offset and hit the plus then we need to enter our offset here there's a place to enter location and rotation you're usually best if the tracker is pointed straight ahead so
that you don't have to do any rotation if it's centered right above the entrance people or nodal point of your lens that also makes things easier that's the way mine is right now so the only thing I really have to enter is a vertical offset of minus 17 centimeters believe it or not now we're pretty close to done the only thing we have to do is set What's called the world pose this attaches a point in the real world to a point in the virtual set that you've set up our set's pretty simple you remember it looks like this and here's the aruko tag and that's going to Mark the center of this set going back to the lens calibrator the noodle offset section I'm going to tilt my camera down and you'll see I have a
similar aruko tag sitting here on my floor so just put your Roku tag on the floor roughly where you want the center of the set to be now go to nodal offset algorithm select noodle offset a ruko make sure that the calibrator point is set like this and the calibrator is set to your arugo calibrator now we just click and you should see some detected points come up and we say apply to camera parent and all of a sudden everything jumps and we're lined up correctly you can see that uh by fading back and forth you can see the aruku tag on my floor matches the Roku tag in my virtual set if I move my camera back and forth the aruko on the floor and the uruko in
the virtual set should stay pretty well aligned you can see a little bit of slippage here but not much that slippage is usually caused by an error in the nodal point or some issue with the calibration this is something you can try and dial in in more detail later on now if you go back to the main unreal window you can see by clicking on the comp element you can see a preview of what our full comp looks like you can see the actual orico tag on the floor the chair this roll of paper and a light and the rest of this is all virtual so if I pan around you can see everything is staying in sync pretty well now the key doesn't look too pretty yet but we'll clean that up in a second if you click play now you can see this full screen you can see
these two sort of bullseyes this is a feature that for some reason is turned on by default we don't want it so we're going to turn it off so just stop this go to settings go to Project settings and then type in touch and you'll see this always show touch interface just turn that off close it up press play again you can see those are gone the last thing we need to do is clean this key up a bit so go to your media plate and you'll notice we've got these sort of green Corners going on that's because of a feature called vignetting that's where the corners of a image may
be a different brightness than the rest generally you won't have that problem so we want to turn the vignetting feature off that's down here in transform passes chroma keying and material parameters you can see there's a devignette inner and outer generally I just set those to one and zero that's get gets rid of those funny green Corners now our key looks kind of like this which is pretty close to what we want the last thing we want to do is make sure our key colors are correct so we want to scroll in chroma keying to where it says key colors we did this once before we're going to do it again click on this cross
and then just kind of scrub this around until you don't see any green leaking through and now we've got a pretty good key to work with there are a number of other parameters you can adjust to try and get this a little cleaner I'll try to go over those later in a different tutorial but for now we just want to get something basic up and running so I'm not going to delve into all the fine points to get a better look at what your final composite looks like you can come up here and choose new editor window pi and that'll bring up a window that's in the correct aspect ratio and shows everything and you can see now everything looks pretty good now if you've done this for the first time you're probably going to notice
that when you move the camera the floor is a little out of sync with the chair you can see when I do sudden movements like this the CG moves first and then the chair catches up this is because the tracking information from our sensor usually gets to Unreal before the video does so we need to adjust for that so open up your live link window click on Bliss and find the offset if you're using a different kind of tracker it'll have an offset too so mine is set to zero right now and since I we've seen that the CG is way ahead of the real world we need to slow the CG World down a little bit
so I'm going to put in point one here and then we'll go back and take a look now when I move the camera you can see things are much closer to being in sync but they're still not quite right the best way to get this set right is to just keep at it trying smaller and smaller changes till you find one that hits on exactly right and you can see now that I've just about got it there's still a bit of lag
now it's just about on the last thing to take a good look at is this thing called the time data monitor you can find that up here in Windows under virtual production and what's useful to see here is you'll see your video input in my case it's this Aja and my tracking input in that case it's a bliss and a light that says synchronized that basically means that everything is set up correctly and that you're tracking data and your video data and is showing up in a way that unreal can sync the two together if this green light wasn't on it probably means that your buffer size isn't big enough for your time correction you can try raising this buffer size
some if you don't if it doesn't work the other thing that's useful on this screen is the Gen lock bar here shows how much time is left in your frame so the more complex your CG gets the more full this bar will get and when it finally hits the end you'll start dropping frames and you won't be able to keep up the frame rate you've set the last thing I have to show you is how to import a set and I'm going to import the one of Neo's apartment from The Matrix that I used for previous demo this is a really nice set and thankfully it's available for free so to get it you can exit unreal and after a moment the launcher will come back up then go to Marketplace and search for apartment Tech and this is
the set that you're looking for now I've already gotten this set so it's not quite the same on my screen you'll have a button here that says add to cart or add to project and it'll be free so add to the cart and then do your cart checkout and you'll have this set to use whenever you want and it won't cost you anything now we go back to our library if I scroll down here you'll see that I've got quite a lot of stuff in here but if I get down to the right spot we'll find apartment Tech props and a button that says add to project so we click that and the project I'm working on is bliss tutorial 2. so I'll add it to that and you'll see a progress bar here that'll tell you when everything's done and now we can go back
up and select Bliss tutorial 2 and start the project back up again when your project is started you'll notice over here in the content browser there's something called Deco Matrix demo so I'm going to go to that in here there's apartment and shared so go into apartment and go into Maps and go into demonstration this is the same map I used to put myself in Neo's apartment in a previous demo to start this process you just need to copy over the professional production layout you created earlier in this tutorial into this level so start by duplicating the level just so you'll have a spare copy of it
I'll leave it named demonstration one and this is where it gets interesting when I was testing this for the tutorial I discovered a bugging Unreal Engine now by the time you get around to trying this that bug might have been fixed or it might not have the way this step should work is like this first we go back to our main level that we just finished making in the tutorial now we go up here and we select all of our compositing related stuff that's all the comp elements here in this folder on the camera and that's really all you need and we copy that
then we go down to the maps of where the apartment is go into the one we just copied and paste it now if everything was working right we'd just about be ready to start messing with this but for some reason when I click on the comp it crashes I'd suggest you save all your work and then give this a try because it might have been fixed by the time you try it but if you do get a crash this is how you work around it instead of copying everything just copy everything except for the zero one zero comp and just like before go into the apartment and paste it
now you'll need to come up into the composure window and you'll need to make a new comp slot connect the two existing ones to it by dropping them on it put it into the composure elements folder so we can keep track of it more easily and then we just remake it the same way we did at the beginning of the tutorial to connect up to the other two layers so all we have to do here is create an output to go to player viewport
create a transform pass and connect that to our compositing material and we're almost there the only other thing we have to do is tell it that we want it to render this entire apartment scene now we could select all of these objects and add it to our foreground layer but since we want to render the whole thing I'm just going to tell it to ignore the layer and render everything so now you can see here that it's rendering everything we have to do one more thing to the CG element because this level has a lot of cameras
already in it we need to make sure that the CG element is rendering from the right one so scroll down here and come in the CG element look for camera source set override and I've renamed my main camera to Matrix camera actor it's the one under camera parent yours might have a different name now you can see it's sort of rendering nothing but that'll be fixed in just a moment we just have to set the world pose to align the camera and the set we do this using the Roku tags I showed you before and when we imported our composure setup the iruko tags ended up way over here
off the set somewhere so I'm going to click the tag and this sets floor is at about 101. vertical so I'm going to move that arruko up to 101 so it'll be sitting on the floor and I'm going to just drag it over here into the actual set area you can hit G if you want all this other stuff to disappear and I'm going to position that tag roughly where I want the center of my reel set to end up then we do a world pose just like we did before find your lens file and open it and you'll be in the camera calibrator go back into unreal and find camera calibrators media plate right
here and go down to the media texture and set it to the media texture where your camera is coming in this is the same process we used before so now you can see a bit of the CG and a bit of the real world go to Noodle offset like we did before select aruco markers click on the screen you should get four numbers like this say apply to camera parent and now our camera is aligned up so that the aruko in the real world matches the aruko in our set and if we go up here to the composite we can see that if you want to see this in more detail you can just do a new
editor window in pi and there you are you can see this real uruko tag this tube here the chair and if I move the camera everything stays together you can see my chair sitting in Neo's apartment I spent a fair amount of time tuning the lighting of the Reel set the lighting of the virtual set and lining everything up nicely so that it will all fit together very cleanly obviously this can take a fair amount of time and since what we're doing here is just a basic tutorial I'm not going to try and show you that right now I'll cover that in a future tutorial so like I said at the beginning of the video I want to tell you now about how you get the 25 discount for the retracker Bliss that's the tracker I've been using
during this tutorial and it's also the tracker I used during the me and Neo's apartment demo that I posted before so here's the deal the first 25 viewers of my content who contact retractacker.co and ask for a discount using the code greg25 that's Greg 25. during the month of October we'll get 25 percent off you can purchase by going to this website but I've been told that if you're from the UK you may also have to make a special arrangement to pay the vat that's value-added tax that applies for the UK so if you're from the UK use the contact us link and still mention greg25 to get the discount the
rest of you can just go to the retract or Bliss page and hit the purchase button fill in the form and add the promotional code here greg25 and you'll be ready to go I know some of you won't think this price is exactly cheap even with the 25 discount but I want to remind you that it's cheaper than almost any other option the only thing I've seen that's less expensive is to use some kind of a VR headset tracker system like the original Vive trackers and those still end up costing you at least a thousand dollars to set up not to mention that they have a lot of issues such as jittering and difficulty in setting the system up and calibrating it that Bliss doesn't have this makes the Bliss a good solution for a school where they're setting up a learning Lab
at the same time the Bliss is accurate enough to use for TV news filming corporate videos to individuals who are trying to set up their own virtual Production Studios or pretty much any kind of virtual production you want to do if you don't have a huge budget for one of the more expensive devices this one will get you there all this information the links to the retract our website and so on are also in the description to this video so in this video we've gone from a completely blank project up to a ready to shoot virtual production set that's fairly complicated and has a tracked camera I know it took a while about an hour but a lot of people have been asking for a single video that shows end to end how to set up from nothing and I have to say that the combination of using Unreal Engine 5 plus the new Bliss tracker made everything go a lot quicker
in the past using Vives and other types of homemade trackers it would have taken another hour or two to get all this set up if you found this tutorial useful or interesting please do me a favor and hit the like and the Subscribe button very few of the people who watch these videos actually do that I want to remind you that I don't ask for any money or payment of any kind to do all this stuff the only thing I'd like you to do is press the like and the Subscribe button because that helps me get more viewers on YouTube it also lets me know that there are people out there who really appreciated having the content I'll be doing some more tutorials pretty soon one will show in more detail how the Bliss works and how to set it up another will cover things like lighting your set and
cleaning up your chroma key so that it looks nice like it did in my first demo until then I hope you guys have fun doing virtual production and I'll see you soon
Transcribed 86 blocks · Source: YouTube