Video Transcript
Unreal 5.7 NEW Camera Calibration Methods for Virtual Production
Greg Corson · Published 8 May 2026 · Duration 1:11:34 · youtu.be/z2f8iO9-sAc
In this tutorial I cover calibrating your camera/lens and finding the nodal point/tracker offset. Calibration helps make your real and virtual objects match up in the scene and lets you do FAST alignment of virtual and live sets. The nodal point/tracker offset is essential for shots where a tracker camera moves.
Auto-generated YouTube captions, grouped into ~45-second blocks. Product and technical terms may be misspelled (e.g. "Aruko" = ArUco, "Retracker Bliss", "fizz" = FIZ: focus/iris/zoom). Timestamps link to the video.
Unreal 5.7 has improved the camera calibration system. This tutorial will show you how to use it. But why do you want to calibrate your camera? When the live camera moves, the CG box stays in perfect alignment with the paper on the floor. A calibrated camera and lens make that possible. If you don't have camera tracking, you can still use your calibrated camera to instantly align your virtual set with the camera in your live studio for static shots. First, I wanted to apologize. I'm sorry it took so long to get this tutorial out the door. It turned out to be much more complicated than I originally thought because there's three different ways now to calibrate the camera and because there are a lot of new switches and settings that have been added in 5.7 of Unreal. Unfortunately, all this new
stuff hasn't been documented very well. And because there are so many new settings, it turns out that if you get them a little bit wrong, you can have problems that will hold you up for a while. Please remember to check the video description. I put a link where you can get my sample project and that will save you some time because everything in it is already set up correctly. There are also a number of reference links in the description that could help you get started or maybe cover some areas that this video doesn't. If you run into any problems along the way, please feel free to post in the comments any questions you have. I'll do my best to answer them quickly. So, what does calibrating your camera actually do for you? For one, it makes
sure your live camera and your Unreal camera exactly match in terms of focus, zoom, iris, and any distortion in the lens. So, your composits look much better. And as you saw in the opening, you can instantly align your studio camera and Unreal camera using a paper marker. And when you're using a camera tracker, the camera and virtual studio will stay aligned when the camera moves, so you won't see sliding feet or other similar problems. And if you have a lens encoder, it'll let you pull focus, zoom, and iris while filming, and the Unreal camera will match all the way through. In tutorial 3, I set up this shot entirely by eye, but that takes a lot of
time. Once your camera is calibrated, you can use other tools in Unreal to instantly line up shots like this, saving you a lot of time. If you're new to virtual production in Unreal 5.7, you probably want to watch my tutorial number one through three, as they cover some things that I'm not going to repeat here. Also remember that all my tutorials have transcripts and chapters, so you can skip over things that you already know. You can get a hold of the transcripts and the chapters by opening up the description of the video and scrolling down all the way to the bottom of it and clicking show transcript. Also, you can always click the gear icon in the YouTube video window and set the video to play at a slower or faster speed if
you like. So, let's get on with it. The first thing you need to do is collect information about your camera and lens. This includes the size of your camera's sensor and the crop factor. The size is usually just a rectangle measured in millimeters. Crop factor can be a bit more involved. Some cameras when they capture video don't use the entire sensor. For example, my camera has a 4x5 aspect ratio sensor. So when it records HD video, it records using the full width of the sensor but throws away some of the vertical pixels to make the final video 16x9 aspect ratio. Other cameras may only use the central part of the image. It all depends on what kind of camera you've got. So you need to check
the manual and find out. You also need a little information about your lens. The focal length, lens focus, zoom, and iris ranges are the main things you're interested in. These are usually printed right on the lens. The output video resolution is also important. Normally, it'll be 1080p or 4K, but you should make sure that when your camera changes resolutions, it doesn't change how much of the sensor is actually used. Some cameras will sample the sensor differently if you're on HD or 4K. Last thing you need to know is if there's any kind of image stabilization in your camera or your lens. This is often referred to as steadyshot. For virtual production, this feature needs to be turned off. Now, to get started, the first thing you'll need is a calibration board similar to one of these. The board can be as small as a single sheet of
paper. In my small studio, I'm using a 4K camera that is never more than about 3 m away from the talent. I've always been able to use a board printed on letter size or what's called tabloid size or 11 by7 inch paper. If you're trying to calibrate your lens at a longer distance or if you have a lower resolution camera, you might want to print out a bigger board. The best way to decide what size you need though is to experiment. Start with small boards that you can get printed cheaply and work your way up if you have problems. Here is an example of printing on 11 by7in paper at Federal Express. The price on the left is for black and white printing and the price on the right is for color. Here's another example from the FedEx blueprint printing service
which can print on much larger pieces of paper. Here you can see the price for a 24in x 36in print. As you can see, the price for color printing is substantially higher than the price for black and white. So, since you don't need color for these boards, be sure to specify black and white printing on normal paper, not photo paper, and not anything shiny or glossy. Many places can also print on metal or acrylic. Here's an example price from FedEx Office for color printing on metal. They might have a different price for black and white, but I'm not sure as it's not listed online, so you'll have to ask where you're going to have things printed. Most places have a lot of different options on how to print your board, and many can also mount the board for you on a material as well. Just make
sure that they know the board needs to be completely totally flat. If it has any curvature to it at all, it'll mess up your calibration results. And you want a material that won't bend or flex as you move around the studio with it in your hands. If you have a camera tracking system that can also track other objects like your board, make sure the board has enough space on it to mount whatever tracking devices or markers your system needs to track the board. On the left, you can see a board with a Vive Mars tracker mounted on the top. On the right, you can see a board with the small round optical tracking balls in the corners commonly used by systems like Vicon or Optitra. Some systems can also track a board using the large paper marker like you see here to the left of the checkerboard. If you
don't have a camera tracking system, all you're going to need is the checkerboard. You're also going to need a tripod for your camera, a stand or a helper to hold the calibration boards, a well-lit studio, a metric ruler or tape measure, and my sample project to follow along. You'll find a download link for the sample project in the description of this video. So, here is what my sample project looks like. All of these things you see in a row here are some sample actors I've made that uh can help you in your basic camera setup and other things. These two green rectangles are easy to use standins for your actual green screen. If you click on one of them and look over in the details, you'll see it has some built-in settings
here. And you can drag these or just type numbers in to change the size of the rectangle to match your green screen. These aren't anything special. They're just rectangles with some extra code added so that you can have these sliders over here to easily set the height and width. This next item is an aruko tag. This is a type of marker Unreal can detect and use to align your set in Unreal with the center of your actual studio. This next one is called an April tag. This one is used mainly with the Retracker Bliss tracker. Uh there may be some other types of trackers that would use it as well, but it's mainly there for the Bliss. This next item, which I've got in closeup
now, is just what it looks like. It's a measuring ruler. It's got tick marks measured in centimeters that are accurate inside of Unreal. If you select it like this and come over to its details, you'll see you can here change the you can change the height of the scale in meters to whatever you want. You can tell it you want it to draw the centimeter marks or not draw them to clean it up a little bit. And you can tell it if you want the centimeter marks in labeled with text. This can be very handy for calibrating a virtual production setup. You can put this scale in your virtual stage. And then when you composite in an actor,
you can compare their height to this scale to see if the camera is set up correctly to show them at the correct size. This object is pretty much what it looks like. It's an axis. And if you go over to its details, you can alter its scale to make it whatever size is convenient for you. This can be handy to attach to trackers or other objects that you want to see their orientation in the Unreal coordinate system. These next two objects which look like boxes are exactly that. The first one is a box that you can set all three sides to independent lengths by dragging these things in the details. So you can set X and Y and Z. And this tick box here you can use to set the pivot center. So, it
can be in the center of the base of the cube or it can be over here in the corner depending on your needs. And you can also change the thickness of the lines if you want them thicker or thinner to for easier viewing. The next box over here is basically the same thing, but it's for when you want all four sides of the cube to be the same size. So, this one just has one setting and it changes all the dimensions at the same time and it has the same ability to change the pivot point. This last object is called a thin axis and it's obviously called that because it's very thin. This is handy if you want an axis reference, an XYZ, and
you want to be small and unobtrusive so that it doesn't get in the way of things that you're looking at. Right now, this one's set to 5 mm, but you can turn it down to almost any degree of thinness you want. Again, you can put this on a tracker or you can put it on an alignment point both in your real studio and in Unreal to see if the two match. This last item where you see the camera and the checkerboard here is a structure of actors that I use to manage my tracking system. I start out with this tracking center. This represents the center point of your tracking system, which usually will be the center of your stage. Underneath that, I have a set of objects
designed to help track the calibrator or the checkerboard. There's this one, which you use to attach the tracker to or the live link input that shows where the tracker is. This is the parent of the calibrator. And normally that will be all zeros. And the last one is the camera calibration checkerboard itself. The reason I've got these three layers is because when you're doing calibration or other operations, sometimes it wants to change the offsets or the track points
associated with the calibrator. Now, you don't actually need this calibrator parent, but it does make things easier for you. I'll talk about this a little bit more later, but basically it's there so that when you make adjustments to the calibrator, it puts them in a place where you can easily zero them out again later. And the last item is the cine camera actor. This is your camera in Unreal that your live camera in your studio is connected to. Also in this template project, I've included a few things like a live link preset to make sure live link is properly set up when you open the project. A sample lens file. This is based on my own lens, and you'll be wanting to change that, but it lets you see how to set it up. A few materials
that I use to color various objects, and a virtual fizz, which I'll get into in a bit. One thing that's a bit different in this tutorial from my older ones is that instead of creating item by item, I'm starting out with most of the items and actors and other things already created and filled in. This should make the whole video go a lot quicker. But don't worry, I'll be showing you what needs to be filled in where. I just won't create everything from scratch while you're watching. If you need a deep dive into this, let me know and I'll try and do that later. To keep things simple, I'll be showing you how to calibrate a lens for a specific focus, iris, and zoom setting. For many people, one setting will be
enough. But if you plan to be frequently changing the focal length, the zoom, or the iris settings of your lens, you may need to do this more than once to create a fully calibrated lens. I'll explain a little more about how to do that later. It can be a bit timeconuming, but basically it's just repeating the simple calibration process that I'm going to show you in a minute. I'll talk about that towards the end of the video. I'll be showing you four different ways to calibrate a lens. The first one, all you need is a checkerboard and it doesn't require any camera tracking equipment. The other three methods require a camera that is tracked and a calibration board that is tracked. These other methods vary depending on what kind of tracking system you have. So I'll cover each one. All of these methods have the first step
in common, which is shooting images of your calibration board. So everyone should watch this first demonstration where I show how to do that as I won't be repeating it to keep the tutorial from getting too long. All the calibration methods start the same way by shooting images of your calibration board. Only the settings that you use to before you start shooting them change, and I'll be going over that. Now, let's work on getting everything ready to calibrate your camera and lens. Start by going up to the media profile icon here and clicking it to open it. This media profile is set up for my Blackmagic Deck Link card connected to my camera. So, I can just click on Blackmagic media source here and it starts coming through. Now, all this stuff that you see on the screen
is preset up for my camera. If you need help changing the setup for your own, you can go back to tutorial 2 where I talk more about media profiles. With this view open, it's a good time to make sure your camera is set up the way you want it. As you can see, I've got the checkerboard calibrator on a stand and it's in focus and well lit. I've set up my checkerboard to the distance from the camera where the talent usually stands. And I've got an f-stop of 18 to give you better depth of field. And I've got my zoom set to 24. For you, just set up things so that the focus, iris, and zoom settings of your camera are the ones you most commonly use. And so that the checkerboard is in a location where your talent usually stands. Once you've got
everything set up and in focus, now is a good time to turn autofocus off on your camera. so it won't change during calibration and to make sure that any steady shot or image stabilization functions in your camera are turned off. Now we can just close the media profile window. Now let's take a closer look at the camera and the calibration board inside of Unreal. I'm going to start by selecting the camera so that we can see it in this little window here. And I'm going to pin it so that it'll stay open. Now let's get the calibrator checkerboard set up. You can see that it's floating out here 1.5 meters away from the camera. I've set that up by going to calibrator tracker. And since for this part of the example there is no tracker, I just manually filled in a
value of 1.5 m to get it to appear out here. The calibrator parent should be all zeros. And then we go to set up the camera calibration checkerboard itself. I've spun my main view around now so that we're looking over the camera's shoulder at the checkerboard. Now click on the checkerboard to select it and come over here into details. The location, rotation, and scale should all be set to 0, 0, and one like it is here. If it's not, please reset it. Then scroll down in the details a bit till you come to number of corner rows, number of corner columns, and square side length. If you take a look at the board on the screen, you can see mine is 10 rows by 14 columns, but you can see mine is set up for 9x3.
This is because Unreal's calibrator only uses the points where two squares actually touch. So basically, count the number of rows and columns, subtract one from each, and then enter them here. The last thing to set up is the square side length. Just measure the size of your squares and enter that here in centimeters. Do the measurement as exactly as you can because if it's not correct, it will throw your calibration off. Once you've filled in all this information, the checkerboard you see on your Unreal screen should look just like the one in your studio. Next, you need to set up things so that Unreal knows the focus, iris, and zoom settings of your camera lens. Some people have rather expensive
hardware that reads this directly from the camera, but most people don't. So, I am going to show you how to do it manually. We start out by making what they call a virtual fizz. I've got one here already started. If you ever need to create one on your own, you can rightclick in the content window, go to live link, and choose make a blueprint virtual subject. So, here's what the fizz looks like once you've built it. You can see that there's a focus, iris, and zoom variable over here. And these go into the blueprint and get copied into the virtual subject frame data here. The update virtual subject static data blueprint node just tells the Unreal which of the static
data is actually valid. So, I've got it set for focal length, aperture, and focus distance. Now is a good time to set your initial starting focus iris and zoom values. You can do that by going into focus, setting the default focus distance here, the same for iris, and the same for zoom and then saving that away. Those should all be set to whatever you're planning on using on your camera right now. Then you can close the fizz up. The fizz also has to be opened up in live link. So you can go up here to Windows virtual production live link and you can see I've already hooked it up here. You would just go add source add virtual subject and then you put it in. If you
make changes to your live link setup, go up here to presets, select save as preset, and pick a file to save it as. If you save your preset under a different name, you probably want to make it the new default. So, just go up to this gear up here and you can select whatever preset file you just saved. That will make sure the live link preset is loaded when you load your project. Now, I'll quickly show you how the Cinei camera should be set up. In my sample project, it's already set up. So, let's just go over it. I'll click on it here. You can see it has several different components. The ones you care about is the lens component, which you can add to a cine camera by
pressing the add button and saying lens. And the other one you'll need is your fizz, a live link controller. The settings in these are reasonably straightforward. The lens component should be pointed at the lens file. In this case, I've already made a sample lens for you. And the evaluation should be set to use live link. Distortion source should be the lens file. And it's very important that apply distortion should be checked. If you're making a lens component from scratch, this may not be checked. So be sure not to miss it. As far as the fizz component, that is set to virtual camera roll. And you select the fizz object from your live link here. Now, let's move on to the lens file.
I've got a sample one here in component calibration. You can see it here. Before you start messing with this, you might want to rightclick, go to duplicate, and make a copy of it so you have something to fall back on if you make a mistake. If you ever need to make a lens file from scratch, you can just rightclick down here in the uh content browser, go to miscellaneous and choose lens file to make another one. For now, we'll just doubleclick and open the one that I've got here. This is the main screen of the lens calibrator. You shouldn't see any yellow text on here. If you do, that means something isn't set up quite right. And usually the text is enough explanation to find out what
it is. Normally, you'll see the raw fizz input, which you had set up when you did your fizz a minute ago. For me, it's a 150 cm focus, 22 iris, and 24 millimeter zoom. Then you'll see evaluated camera settings over here, which should be pretty much the same. Everything else is going to be more or less blank. If you don't see this mixed view of your camera and your Unreal camera, check up here to make sure cine camera actor is set up to the actual cine camera actor in your Unreal project. The media source should be media profile. It should be pointed to the input of the card you're using. Take a good look at this and make sure that both of the calibrators, the live one and the one in Unreal, have the same number of rows and columns. Also,
in this lens file, I've set up a set of curves, which should work for pretty much all of you that are setting up your lens manually. I've got a focus curve that covers taking the number in the fizz and mapping it to the same number in centimeters. And that one's will work out to 10 meters. The iris is uh the same. You put in the actual iris in the fizz and it will come through here as an f-stop. And the zoom is the same thing. If you happen to have a lens that's outside the parameters I've set up, you may have to edit these. But I think for most people, a focus distance of up to 10 meters and a focal length curve
of 1 to 1,000 should be plenty. Now, let's fill in the rest of the data about my camera and lens. For me, it's already filled in. You'll want to change this to data for your lens, of course, and camera. Uh, for me, I have a Sony 24 to 70 zoom lens on my camera. The serial number I've just put in 1 2 3 4 5. It's not particularly important unless you have a number of lenses and you want to keep track of them by serial number. And I've set spherical lens model, which should work for just about any lens you've got. Uh, if you're shooting anamorphic, you can choose an anamorphic lens model here. My sensor on my camera is actually 35.7 by 23.8 millimeters.
But when it's shooting 4K video, it doesn't use the entire sensor. It actually cuts off some of the vertical pixels so that the actual size being used is a 16x9 aspect ratio. So to set it up for my camera, I've adjusted the Y value so that together with the 35.7, I'll have a 16x9 or 1.77 aspect ratio. My camera is shooting at 4K resolution, so I've set the image resolution to 3840x 2160, and everything else is basically the defaults. This is a good time to estimate your tracking delay. If you don't have a tracker, you don't have to do this. If you do have a tracker, you might as well do this now as it
could make the later tracking steps easier. Tracking systems and video input cards take slightly different amounts of time to get their information into Unreal. So, you want to find that delay and set it up so that both the tracking system and the video will be in sync. So, with the lens calibrator open, make sure your transparency is set to 0.5 and point your camera at some place in your studio. You should get something that looks sort of like this with some of the Unreal scene displayed over top of the view of your studio. Now, just quickly pan the camera.
You can see that when I move the camera, the Unreal parts of the scene move first and the video moves second. So, we need to delay the tracking system so that they both move at the same time. Go up to the Windows section here, virtual production, and open live link. And you should have your tracking system set up at this point. Right now I'm using my Bliss tracker. So I click on that tracker. And you'll see all this extra information over here on the side. And the thing we care about is right here, the offset.
Right now, you can see that there's no delay on the tracking information, which is why it's moving before the video is because the video takes a bit longer to get into Unreal. So, let's try changing the offset to 0.04. Okay. And then try moving the camera again. You can see that there's still a bit of lag where the video moves last. So, let's make the delay a bit longer. We'll make it point seven and try moving the camera again.
Now you can see the delay is very small. So I'm going to take this to 08 and maybe to 0.9 and see how that looks. It looks at like at 0.9 we're still we're not quite in sync. So, I'm going to go back to 08.
And now everything seems to be moving in sync. So, we'll stick with this offset of 08. And I'll close the live link window. Once you've got a complete Unreal set and some talent in front of the camera, you may want to go back and readjust this. But for now, I'm just going to go to the live link window and I'm going to save this as a preset as my default.
And we're done with that. Now I'll show you the first calibration method. This one doesn't require you to have any tracking system and only calibrates the lens distortion. This is the method you want to use if you don't have a tracking system or if you have a type of tracking system that can only track the camera and nothing else. The nodal point and offset information can be measured manually and added later if you need it. Now we're ready to start calibrating. Click on lens distortion here and you'll see these things on the side here change. Make sure your calibration pattern is set to checkerboard and the calibrator is pointing at the Unreal object that makes this calibrator. Calibrator track
should be off. Camera track should be off. And come down to focal length guess and insert the guess for your current focal length of your lens. For me, it's set to 24. So, I'll put in 24. All this other stuff can pretty much stay as is just to reduce the confusion. I'm also going to set the blend up here so that all I see is the view for through my main camera. Now, all we need to do is start shooting images of this checkerboard in different places around the studio. The way you do that is to simply click on the window. And now you'll see that the a shot of the checkerboard has appeared over here. and that the checkerboard on the main screen has these little lines and dots
indicating that it found all of the corners in the checkerboard. You need to do a bunch of these shots with the checkerboard in a lot of different positions. You want shots that cover this entire frame from corner to corner. And you want the checkerboard to get shot at several different distances and in different orientations. You can rotate it and you can also turn it at different angles towards the camera. This can help with the calibration accuracy. Now, I'll give you a quick run through of all the images I shot. Remember, you can always view this in slow motion on YouTube if you really want to look at the individual ones. Here you can see all the shots that I did for this particular example. You can see that it's not quite perfect. There
are a few gaps and I didn't go quite far enough to the edges, but this still should be good enough because this lens doesn't have too much distortion to start with. Ideally, you want every spot on this screen to be covered, including right up into the corners. You can see all the shots you've taken by scrolling this area here, but they are kind of small. To complete the calibration, scroll down past all of these different images you shot and you should see these three buttons. Just click calibrate lens. You can see it working on the calibration here. And it tells me I have an error of 482
pixels, which is pretty good. Generally, anything below one pixel is okay. If it's above one pixel, you may have a problem. You'll see now down in the evaluated camera settings, it's calculated what it thinks is my actual focal length of 24.668. It's pretty normal that this won't actually exactly match what's printed on your lens. Uh those markings are rarely exact. It has the distortion parameters filled in and the image center which will normally be right in the middle at 0.5. Now let's see if the calibration is actually working right. Go up and click on the nodal offset tab and over here on the side make sure that everything is still set correctly. Under nodal offset algo choose nodal offset checkerboard
and make sure that checkerboard is pointing at the checkerboard object that you made. Now change the blend back to 50% so we can see both the Unreal camera and your live camera at the same time. Now we click on the image and you can see it's captured a bunch of points on the checkerboard. Then we click apply to calibrator parent and the two images have jumped right on top of each other. You can see that if I fade back and forth, they're lined up exactly. Now, if they're not lined up exactly, you have a problem. But if they do look pretty much like this, everything is good. to double check that everything is working like it should. You can move the checkerboard
off to one side like this. And then repeat this process where you click on it and then say apply to calibrator parent. And again, they should be perfectly aligned. If you want to take a closer look to make sure the alignment is exact, you can use the scroll wheel on a mouse to zoom in on the checkerboard. It's pretty normal to see a bit of blurriness around the edges of the squares. You can also run the transparency back and forth and see that the squares stay pretty much aligned whether you're looking at the real or the Unreal camera. You should try repeating this process with the checkerboard in several different positions at the top, bottom, and edges
of the screen as well as the corners. No matter where you put the checkerboard, you should get a good alignment. If you don't, then there's some kind of a problem in your calibration process. If you're getting a solve error above one, or if these alignment checks don't work out, don't panic. It may be something simple. Double check the uh focus, iris, and zoom settings of your camera and make sure they match what you put into the fizz. That's one of the most common problems. Sometimes someone will knock the camera focus or zoom out of whack and you'll try to do a calibration with the wrong values. Another common problem is some of the pictures you shot during the calibration process might not be good. They could have an unacceptable amount of motion blur or glare on the pattern that's thrown things off. The
best way to check for problems with your calibration image is to go back into your project folder, open up the saved folder, open camera calibration, lens distortion, and then find the folder for the date that you were working on your calibration and open it. For each of the calibrations on that date, there should be a separate folder. And if you open one up, you'll see all of the different images that you shot for that calibration. You can open those up in Windows Photo Viewer and then just flip through them and look for anything that has a problem.
particularly f photos with a large amount of motion blur or anything where the board looks distorted or too dark. If you find any bad images, just delete them along with their corresponding data file and then you can come back to the calibrator and try again. Here's how you do it again. In the calibrator, you hit this import data set button. Then find the folder for the calibration that you just did and modified to take out any bad images. Open that up. Load the session data up. And after a moment, you'll see all of the images over here appear again.
And once you've got that, you can do calibrate lens again. and hopefully it'll come out better with the bad images removed. One other thing that can cause significant problems is in the camera lens component. If you go into the lens component and scroll down here, apply distortion must be on. If it's off and we go back to the calibrator, you'll see that the images don't line up anymore. So, make sure that this switch apply distortion is on and try your test again. And you should
get an alignment like this. And that's the end of the basic calibration process. With this lens file, you'll notice that your composites will look a little better because the live camera and the Unreal camera will have matching amounts of distortion. So, you won't see as many differences between the live and the Unreal Engine camera when they're put together. Also, when you have a calibrated camera, you can use it to line up a shot. You can put a checkerboard in your live studio and put a corresponding one in your Unreal studio. And with a couple of clicks, you can make it so that the camera will be aligned the same in Unreal as it is in your live studio. I'll show you that at the end of this tutorial. Also, remember that now that you've got a complete lens file, you can take that lens file and
move it over to any other Unreal project that you're working on. Now, let's move on to calibration method two. This uses a tracked camera and a tracked calibration board. This method calibrates the lens, finds the nodal point, and the offset of the nodal point to the tracker all at once. For a tracked moving camera to line up with an Unreal background, you need to have the calibrated lens nodal point measurement and the distance from the nodal point to the tracker. If all of these aren't correct, when you pan the camera, your talent may not stay lined up with the set, and their feet may appear to slip and slide. To use this method, your tracking system needs to be able to track both the camera and the board at the same time. This method will work with any tracker. However, if you're
using Retracker Bliss, I would recommend skipping ahead to method three because it's customized specifically for use with the Retracker Bliss. Depending on your tracking system, the setup for the board will be different. On the right, you can see a Vive tracker attached just above a checkerboard. On the left, you can see a checkerboard with Viacom tracking balls around the edges of it. And on the left bottom, you can see a board with an April tag that it's used to track the board. These are just examples. Depending on your tracking system, take whatever marker or device the tracker looks at and attach it somewhere on the board and then you're pretty much ready to go. Since I use a Bliss tracker, I have a
board like this one here, which has a large April tag on it that the Bliss uses to track the position of the board. And that's right alongside the same checkerboard calibrator I used before. If you're using some other kind of tracker, you won't need this big April tag. You can just put whatever the tracker is, whether it's a Vive Puck, which could go anywhere on the board. Uh, or if you're using something like a Viccom or an Optitra, you might have little sensing spheres around the edges here that you saw in the previous photos. It'll all depend on what type of tracking system you're using. But since I have the Bliss, I'm going to show you how to hook this up for the Bliss. Hopefully, your system won't be too much different. After you put the Bliss software into your plugins folder in
Unreal, you can launch the uh Retracker Bliss settings app and do the basic setup for it. I found that I use Live Link Bliss with these settings. And then for the tag tracking, I use these settings and I put in the size of my tag. And that's all you really need to do there. Then I just doubleclick the retraer bliss app and it'll start up and give you this window and this window to let you know it's running. Uh you can check over here in the main window to see that it's detecting your tag and that it's sending uh to Unreal on the
local machine. You can also run the Bliss app on a different computer and send it over an Ethernet connection to the computer that's running Unreal. Now, let me move these things out of the way. And the first thing you need to do is set up Live Link. It's just a simple matter of clicking add source, choosing live link bliss, and then filling in some basic information. If you go to the main entry here in the bliss, you'll want to have evaluation mode on engine time and all these other things are pretty much default except for down here in the buffered settings. Uh because Bliss
sends a large number of samples every second, like over 500 in some cases, you want to have the buffer size beh,000. And this offset number is the tracking delay which is common for all trackers. That is the um tracking data usually comes in ahead of the video frames. So you'll need to eventually figure that out and fill in your delay here. It's going to be different for every system. For me it's approximately 009 seconds. And a little later I'll show you just how you figure this out. But once you're done setting things up, you should not be seeing any errors appearing down here in the bottom.
And you should have green check marks for the the camera one, which is track the tracking on your camera, and camera 101, which is the tracking on your calibration boards tag. So once you've got the tracking data coming in through live link and all these check marks here are green, we just need to go back to the main Unreal scene and connect the things that are being tracked to the objects in the scene here. And now if you take a look at the camera component, you should see the position and rotation from the tracker updating here. Then we want to get the checkerboard taken care of. So, for that, we'll go to
calibrator tracker. And once again, I've already got it set up so that it's already running, but you would go in and take the live link component controller, set it up for camera 101, which is the checkerboards tracker. And everything else here is pretty much the same as before. So, now the tracking data is coming in. You can see on the big screen that my board, the red with red checkerboards. That just makes it easier to see. Uh, is sitting out here in front of the camera and the view of the camera is looking right at them. Now, if I go over and move my camera a little bit,
you can see that the camera panned and the view changed. and the calibrator stayed in one p place which is as it should be since I didn't move that. Now go back to the unreal view and position yourself so that you're behind the camera and then just move forward a bit so that you're basically looking over the camera's shoulder. Now I've already aligned everything properly here but yours is probably not going to look like this unless you have exactly the same setup. So, select your April tag and check to make sure the tag size is the same as you set it in the Bliss configuration tool. Now, select the checkerboard.
Go over here into the details and click on top left. Your checkerboard should have should show the position of the top left. If it's not in this position, you need to make some adjustments. Go to the top of the checkerboard object and look at its rotation and location. You can see here in mine, I've got a Z rotation in there of 180 degrees. If I didn't have that, things would not be aligned right, and my top left point would actually be on the right. So, that's the first thing to fix. Change this to 180. Change the Z rotation to 180. Select top left. It should be in the top
left. Top right should be top right. Bottom left and bottom right. Now we have to do the offset calculation which is pretty easy for me. If you look here, you can see this is the origin of my April tag and this is the origin of the checkerboard. So basically what I need is to measure from this corner over to this first row and then measure from this corner down to this row where the origin of the checkerboard is. Then I go into the camera calibration checkerboard and I enter that information. For me it
was - 5.9 cm by -8 cm. Once you've done this, this view in your Unreal editor should look pretty much the same as the view through your camera. Take a good look at it. Make sure everything is aligned correctly. [snorts] Now, if you have some other kind of tracking system, what you'll probably want to do is replace this Aruko tag graphic with a graphic that's representative of your tracker, but you still have to do the the offset measurements pretty much the same. Measure from wherever the center of your
tracker is on the board to this point on the checker board on your calibration board. Then everything should be aligned correctly. From here on out, the process of doing a calibration is very similar to the first thing I showed you with just a few additions. all the information on your camera and lens stays pretty much the same regardless of the method you use. Then in lens distortion here, we have a few changes. We're still looking at the Cineamera actor and the media profile like before. The calibration pattern is still a checkerboard. The calibrator is set to the camera calibrator checkerboard from our scene.
But for this method, we turn on is calibrator tracked and is camera tracked. And down here, we say solve noodle offset. And we set an an approximate guess of the lens focal length, which mine is set to 24. So that's what I put in. From this point, we use the same method I showed you for the first calibration process. We keep moving the board around and clicking on the screen to collect images. One thing that you have to do a little bit differently for this method is Unreal recommends that you keep the camera stationary on its tripod, that you put the calibration board on some kind of a stand, and that you move only the calibration board around. Now, I have seen people at
Epic handhold the calibration board and it seems to work for them, but I usually get a better result when I put it on a stand. So, just like before, you going to take between say 20 and 60 images, enough to cover the entire screen with the calibration board. sometimes face on to the camera, sometimes closer, sometimes further away, and sometimes tilted to one side. That'll get a bet that'll help you to get a better result for your nodal offset. Now, this is what my screen looks like after capturing a bunch of images. You can see it's not perfectly covered. In fact, I only captured 24 images to do this, which really isn't enough. if you
normally will want to do more, but since this is just a demo, I didn't think I needed to do the entire process. So now I can hit calibrate lens. And in this case, I'm getting an error of three pixels. If id grabbed more images, I probably would have gotten a lower error. Ideally, you want it to be less than one. If you look down here in the corner of the camera calibrator, you can see that the nodal point offset information has been added in. Now, just like before, we can double check our results by going to nodal offset, selecting nodal offset checkerboard, selecting the Unreal object that's our checkerboard. Then we just click on the scene
and say apply to calibrator parent. And as you can see, everything has jumped into perfect alignment. And now if I reach over here, you can see everything stays in alignment when I move the camera. And just like before, if you go down into the content drawer, you can grab your lens file that you just finished and duplicate it. So you'll have a backup. If you come back here and make any changes again, this method two of calibration should work with most any kind of tracker and most any and most any kind of camera and it's probably the best thing to use for most systems.
However, Bliss also has its own calibrator and I think it actually works a little bit easier than this one just because it's customized to work with the the Bliss. So, let's get on to method three using the Bliss precision tracker. Calibration method three is for the Retracker Bliss and uses some software called the Retracker Bliss precision calibrator. This calibrates lens distortion and nodal point in two steps. If you're using a Bliss tracker, I'd recommend using this method because it's customized specifically for the Bliss, making it a bit easier and quicker to do the calibration. The Retracker Precision Calibrator only works with the Retracker Bliss. It's available from the Retracker users Discord server, which you should
get a link to when you purchase your Retracker hardware. Before using the Retracker precision calibrator, install NDI tools from this link and start the calibrator before you launch the Bliss app. Okay. Once you have the Bliss precision calibrator installed, be sure you go to the retracker bliss settings page and make sure that live link bliss is enabled. It's set to the right IP and port. And then do the same with tracked tag options. And remember, you need to put the size of your tag in centimeters right here. [snorts] Your tag is probably going to be a different size than mine. After you launch the Retracker
Calibrator app, you should see this. And then you want to launch the Retracker Bliss tracking app. And then these two live link feeds should appear. Now, we need to set up some of our data, which is very similar to what we set up for the Unreal Calibrator. We go to Unreal or we go to input data and set our frame rate here. Mine is 29.97. And we set our delay and mine is about three frames. I'm going to call mine 20 Sony 24 to 70 bliss. The serial number you really don't have to set unless you've got a lot of lenses you're trying to keep track of. My sensor size is the same as I used for Unreal.
35.7 by 20.08. I'm using manual zoom input. My lens starts at 24, which is how I'm going to set it. The focus value I'm going to set to to about a meter and a half and the iris value will be 22. In the live link section, we already have default values in here that are correct for the default values of the bliss tracker. Then we just have to tell it which of these tracking inputs are which.
In my case, camera one is the actual camera and camera 101 is the April tag that's being tracked. Now, we go to the media section. You can see that my camera is already showing up here. At the moment, there's a small bug in this calibrator. So, leave override resolution to on and the resolution set to 1920x 1080 for now. This will get you good results for both 1K and 4K. Depending on your source of video, you might have to select it. Uh, mine is from my Blackmagic Deck Link card. It's an SDI source and it's number one. If you don't see any video here, [snorts]
most likely there's some other application on your system that's running and and using the camera. This could be an instance of Unreal or some other program such as OBS or other video chat systems. If you have that problem, just close the other apps that are using video and then come back here and reopen your media, your proper media source. Now, we click on camera calibration and this is where we get to the meat of the actual calibration process. Here you want to turn on the coverage map and you want to set the size of your checkerboard which for me is 9 by 13
and my square size is 2.57. And now we're ready to start capturing photos of the board. By the way, you don't have to use a board like this one with the April tag and the checkerboard on the same board for the Bliss calibrator. Uh, I'm only using it because it's the board that I was using for the rest of this tutorial. I didn't want to make different ones. You can put the April tag and the checkerboard on two separate boards for this calibrator. Now, we just have to shoot a number of pictures of the board. We do this by just clicking in the video window. And you can see that it's detected the board and added it to the coverage map. So now we just keep moving around till we get a good coverage. You,
as usual, you want to do side to side as far as you can, top to bottom, till this coverage map covers the entire window. You also want to take some shots at a distance and some shots close up and some shots with the board twisted to the left or or to the right. By the way, if you don't have a helper who's helping you move the camera and control the computer, it can be very handy to just carry a wireless mouse in your hand so that you can do the clicks without going back to the computer each time. Now, you can see I've captured quite a few samples here, so I'm ready to let it
calculate. So, I'll hit the add to lens distortion button. And you'll notice right away the FPS numbers stop going. That means it's calculating. So, while it's doing that, let me point out, you can see I've got a pretty good number of samples here from the coverage map. Everything except right up in the corners, which are kind of hard to get sometimes. I've taken some samples at my normal focus distance. I've taken some further away and some closer up. And I've also done some with the board tilted to the left or tilted to the right, which helps increase the calibration accuracy. Now, a couple of minutes have gone by and the evaluation of the lens is finished. So, you can see under evaluated camera settings, it's showing a focal length of almost exactly
24, which is what the lens is. These evaluation parameters for distortion are just about what I'd expect. The image center is pretty much dead center and everything else looks pretty good. So now we go on to the part where we evaluate the nodal offset. I click here on nodal offset and I have the tag to the defaults and I set my tag size which is 21 five. I hit the generate button to generate the correct size and shape Aruko tag. Now you can see over here in the window that this dark tag is the generated one and obviously this one is the one on my board and they look about the same size. So we're in pretty good shape here. Now we just go on and do the
same thing that we did with the main board except this time we're shooting images of the Aruko tag. And like I said before the these two can be on completely separate boards. They don't need to both be here at the same time. The only thing you have to make sure of is when you're shooting the Aruko tag for the nodal offset, keep an eye over here on camera 101, which is that tag, and don't shoot it if it's not green. That means that it's lost tracking and it will just mess your data up if you shoot one that's not green. For the Aruko tag, we usually don't need to shoot as many images, but the general rule, as I've said before, you need to shoot it at several different distances. And at several different angles to the camera.
And now that I've shot all the images of the tag that I need, all I have to do is click add to nodal offset to calculate the proper offsets. You can see the April tag has jumped on top of the image of the April tag for the camera. And I can do a little wipe back and forth just to make sure they're perfectly aligned. And they are. And as a second test, I'm going to get up here and move the camera a bit. You'll notice that the tag stays aligned. There's a little bit of jiggling because the camera delay isn't set up just right here, but that really doesn't matter as long as they stay aligned when you stop moving. Now, I can just go back to settings, export,
and export the lens file. I'm going to put it in my documents folder, and we're done. Now, back in Unreal, all I have to do is go to the calibration folder and import my new lens file. That was in documents. And here it is. And I can open that lens file now and see it here in the calibrator and save it. Now, you can see there's some yellow warnings here in the calibrator. That's just because we haven't applied this lens file to our current camera. So, I can go back to the test map, find my cine camera actor,
go to the lens component, and select my new lens file. Select it in the fizz. Now, if we go back to here, we only have one set of yellow warnings left, and that's because we haven't put in any kind of a lens curve yet, which I don't believe you actually have to put in at this point, but I'm going to do it anyway to show you how you do it. So, I'll go to the lens file panel. So, I'm going to put in a focus point. Just hit plus. Say my input focus is one and a half
meters and my encoder mapping will be the same and add it. Now we've got a point. So you can see the focus distance is now being down here and it's not yellow. And then I just need to add an aperture. So I go to the iris, hit plus again, and I hit 22. Hit add. And now we're all okay here as well.
And save it. Now let's see if we did all this correctly. When you open up the map, you'll see your camera wherever you left it last. And you'll also see this little thing down here, which is called Aruko Center Stage. And it's basically an Arco tag with a box around it that's the same size of the tag. So, the first thing you'll want to do is go in in here and select it. Set the tag size to whatever your Aruko tag size is and then go to the cube and set it the same. Remember this is different than the April tag we used earlier. Now like it
says this Aruko is going to mark the center of your Unreal stage. So what I'm going to do is take it and move it over here a bit to till it's right in front of these two big cubes. Now, if you look at my camera view, you can see I've also got the same size Aruko sitting here on the floor of my studio. Now, what we're going to do is align this Aruko with this one. So, the camera will be in the same position in my Unreal stage as it is in my real studio. And the way I do that is really quick and easy. You go into
the calibration section here. You select whatever camera lens file you're using. Now, make sure your Sony camera actor and your correct media profile are selected. Go to nodal offset. Choose nodal offset arugo markers. Choose the Aruko center stage and then point at the main screen here and click and then you click a second time. Now you do apply to camera parent. And you can see everything's aligned now with the camera feed. And if I go back to the main map, you can see my unreal camera is now up here looking down at the Aruko center
stage tag. And I've got a rudimentary composure set up in here for this particular test. So we select cineic camera actor. Then we go into composure and turn it on. And now you can see the Aruko tag and this skeleton box lined up in my studio perfectly. And then we can test it by just moving the camera and see if the box stays stuck to the Ruko on my floor.
And you can see it does. So if everything has gone okay so far, you'll have a lens file that has a complete camera calibration in it, which you can move to other Unreal projects. The next tutorial in this series will show you how to set up an Unreal project for a green screen stage with an Unreal Engine generated set. If things don't seem to be working right for you, don't worry too hard. There are a lot of small mistakes that can cause this calibration process to go wrong. So, if you have problems, please let me know in the comments. And if possible, record a video of what's going wrong and post it somewhere so I can look at it. Here are a few of the most common things that cause this process to go wrong. Number
one is to make sure the camera settings are correct and what you think they are. Sometimes a setting will accidentally get changed such as zoom, focus, or other things. So, make sure before you start a calibration run that everything is set the way you thought. Another common issue is having the wrong measurement of your boards, such as the size of your rico tag or the size of the squares on your checkerboard. Double check these. Also, don't forget to shoot your board at several distances and at different angles to the camera. I usually try to do at least three different distances with the board turned 45° to the camera in some of them. Remember, you can always double check your process by doing a second calibration and seeing if the numbers
come out roughly the same. And that's about the end of this rather long tutorial on lens calibration. I hope it's been helpful for you. The next tutorial is going to be more about how to use this now that you've got it. I'll try to set up a complete green screen studio with an Unreal set and possibly provide additional troubleshooting hints as if they're needed. Please remember to like the video and subscribe to the channel. And if you have any comments or questions, please leave them in the comments section and I'll get back to you as soon as I can. Have fun.
Transcribed 89 blocks · Source: YouTube