Video Transcript
Camera Calibration - Virtual Production Tutorial 15
Greg Corson · 10,868 views 29 Jun 2021 #unrealengine #virtualproduction #cameracalibration · Duration 15:05 · youtu.be/XxcRbu-084M
To get good virtual production results you may need to calibrate your camera lens to remove distortion, the curving of lines that should be straight. The process gives you an accurate field-of-view and focal length for your lens too, this helps setting up the virtual camera. This tutorial shows how. There are many programs for lens calibration, here I used REtracker Calibrator from Rassi Engineering. They have offered to give out a limited number of copies of this calibrator for free, just "like" the Rassi Engineering facebook page and then request the calibrator from them through facebook messenger and mention "GregGiveaway" / rassiengineering Video index 0:00 Distortion, what it looks like
Auto-generated YouTube captions, grouped into ~45-second blocks. Product and technical terms may be misspelled. Timestamps link to the video.
hi everyone almost all camera lenses have some distortion in them that tends to make straight lines look a little bent i've exaggerated it in the picture you're looking at right now you can see it in the plumb line hanging in the doorway the curve of the door frame and some of the vertical poles this is something you generally don't want to have when you're doing virtual production any objects that are being rendered by the computer are going to have perfectly straight vertical and horizontal lines so if you put them up against a camera view like this they're not going to appear to fit in very well so what you want to do is calibrate the distortion on your lens so you can remove it just to give you a little bit better idea what this can look like i'll exaggerate it some more some lenses like fish eyes or older
gopros will bend out a lot like this and others will bend in the opposite direction more like that both of these can be corrected so everything looks pretty much straight depending on how much distortion your lens actually has and what kind of virtual production shots you're trying to accomplish you may not need to calibrate your lens to get a good result but if you're noticing objects don't quite line up or that vid in the video straight lines are appearing to be obviously curved you probably want to calibrate to fix that in general the better you know the specs of your lens the higher the quality of your virtual production results if you're not sure how much distortion your lens has the easiest way to find out is to take a
shot like this with some nice straight things in it like the door frame or better yet the plumb bob that's hanging from the door if you don't have one of these it's a nice tool to have around because it's a quick and easy way of getting a perfectly straight vertical line every time as a reference i'll put a link in the video description as to where you can get one but you can find them in most hardware stores if you don't want to buy a plumb bob you can just take any reasonably heavy weight like some washers or bag full of coins and tie it to the end of a piece of string and it'll work just as well after you've got the video going with some straight lines in it just take a straight edge like a ruler hold it up to your screen and see if the straight lines in the
picture are actually straight it's pretty easy to tell if they bend outward by very much if they do then you probably want to calibrate your lens like this there's a lot of different camera calibration software out there but they all work basically the same way you take a picture of a calibration board like the checkerboard you see in the scene right now from a bunch of different angles and positions in front of your camera and then the program processes them and returns some numbers you can use to correct the distortion it's even possible to guess at those numbers and adjust them by eye but in general you'll get a good result much easier if you use calibration software for this tutorial rossi engineering has offered to make their re tracker
camera calibration software available to a limited number of people for free if you're interested look in the description of this video to see how to get your own copy be sure to mention greg giveaway when you do the re tracker calibrator is still new and being improved it's pretty simple to use and will give you the values that you need rossi engineering also has a camera tracking solution that works with xsymmetry and will eventually be available on unreal if you're interested in other ways to do camera tracking be sure to ask them about it when you request your free calibrator software as i said earlier all calibration software works pretty much the same way by taking pictures of a calibration pattern like the checkerboard you see the trick isn't in the pattern or even
in the software it's taking good clear sharp pictures of the right size and in the right places calibration programs usually come with a pdf of a calibration pattern that you can print out make sure you print this actual size you'll probably be something like a checkerboard with squares from three to five centimeters across to get good results you want this checkerboard to be flat with no wrinkles so you have to mount it to something stiff you can use a piece of thin plywood some foam core artboard or a piece of heavy cardboard all will work as long as it's flat and stiff use tape contact cement or anything that won't discolor the paper to mount it a lot of copy shops framing shops or craft stores can print the paper for you and mount it
on a board so you don't have to do it yourself also don't try to use a checkerboard or a chessboard most of them don't have the good strong contrasting colors you need for this to work well with a plain border around them some of them also fold in the middle which makes it impossible to keep them flat if you don't have any way to print out the calibration pattern you may be able to get away with having it displayed on a tablet or a laptop screen or a monitor as long as it's clear and sharp to take the pictures find some place with good bright even lighting that won't cause reflections on your camera calibration board cameras can have a ton of different settings sensor sizes and shooting modes that can change the lens distortion or the focal length of the lens changing
the zoom of a lens will also change the calibration completely there are too many settings for me to list so i'll say this decide what settings you're going to use when doing your virtual production videos and set the camera lens up to the same settings this includes resolution because some cameras crop the image differently depending on the resolution you choose if your camera has any kind of stabilization or steady shot feature built in either in the camera or in the lens you need this to be turned off when doing virtual production if the camera has any built-in lens distortion correction you can leave that on be sure to keep track of the camera settings you used when doing your calibration shots you want the camera to be set up the same way when you do your virtual production shoot or the calibration will
be off anytime you change the camera settings particularly zoom you will have to do a new calibration with the new settings most cameras crop the images differently for still pictures so you want your camera set up to record video you can record in camera or on a computer but record with your highest quality setting to get the best results use higher shutter speeds if you can and higher f-stop settings to get sharper pictures the advantage of shooting video of the board is that you can pick out the highest quality frames from the video to get better calibration results now you take the pictures for this you can put the calibration board on a table or the floor and move the camera around it but it's usually easier and gets sharper pictures if you put the camera on a
tripod and move the board around instead once you've started recording get the board in a position where it fills about a quarter to a fifth of the screen and start moving it to different positions like you see me doing here at each position stop moving it briefly so that you'll get a nice clear shot you want to capture the board in as many positions as you can with different angles to the camera always keeping the entire board fully visible make sure you cover the entire field of view of the lens particularly at the edges or the corners where most distortion usually is now put the video on your computer and open it up in something that lets you save full size images off the video vlc is a good program for this and it's free skip through the video and start saving
out images of the board in different positions you want to end up with 20 to 30 of them make sure they're very sharp and clear in focus and have no bright reflections more is not better a smaller number of high quality images will give you a better result than a ton of average quality ones now put all your good images in a folder and take another look at the full size if any of the images don't look sharp clear and in focus throw them out from here on i'm going to use re tracker for the demo if you're using something else it's going to work pretty much the same way all you usually need to do is tell it where the pictures are to use retractor just unzip the file that it comes in and you should end up with a folder that looks like this create a folder called
capture and copy all of the pictures that you took in here then you need to set up some information about the board that you used and where the pictures are open up the calibration settings file count the number of squares horizontal and vertically on your checkerboard pattern subtract one and enter them here make sure that the capture folder path points at the folder you just made and that the file extension is whatever it is for your pictures usually jpg or png now we're all set and all we have to do to run the calibrator is to double click it it should go right to the folder where all of your pictures are and start working on them as it looks at each picker sure it'll come up like you see here and you'll see these different colored
lines showing that the program has found the corners of the squares in your pictures if it looks like these points aren't hitting the corners of the squares something's wrong and you probably won't get a good result after looking at each picture press the spacebar to go to the next one and just go through all the pictures this way it'll take a few moments to finish the calculation and then it should print successful patterns found and the number of pictures if this isn't the same as the number of pictures that you actually put into the program there must have been a bad one or two in there and you might want to remove it or reshoot the mean error here sometimes called reprojection error by some programs gives you some idea whether you were
successful generally anything under 0.2 or even 0.3 is usable if it's much higher than that you want to reshoot the pictures there must have been some of them that were kind of blurry then you just need to give it a file to write the results out to now you have everything you need to set up unreal xymetry or other programs to remove the distortion from your lens save your calibration file along with a little readme saying what the camera settings were when you took it if you're planning to shoot with different cameras different lenses or different zoom settings of one of your lenses remember that you'll need a calibration for each set of settings for really precise work you can get electronic encoders for your camera lens
that will read the settings of the zoom focus and f-stop rings and send them straight into unreal these usually come with their own calibration software though now let me show you how to enter that information into unreal for this i'm going to use my vp studio project that's on the screen now one thing you should know is that the way that you enter lens distortion data in unreal is changing they've revamped it for unreal 4.27 i expect the it to be pretty much the same but it may change a bit in location i'll try to do an update if that happens so in my project you'll see in vp studio core there's an azure video capture folder if you've set up this for blackmagic you know where that folder is going to
be for your stuff inside there'll be some media bundles so you open up the one for the main camera and this is where you're going to put the calibration information scroll down to the bottom you'll find a section called lens parameters you may have to click on some of these little triangles to expand everything out and this is more or less what it should look like when you start up with everything either 0 1 or 0.5 now i bring in my calibration file and you can see that it has similar headings for a lot of the numbers this first batch down to cy you're not going to use the rest of these need to be copied into places that make sense on this form that you see notice the k1 k2 p1 p2
k3 and then the normalized fx and y and the cx and y which go down into the f and c parts here now i've copied all that stuff over into the fields you see here and i need to set up the material that goes with all this that does the undistortion so you click the open material editor and it'll probably look more or less like this if you've got your camera running the first thing you may want to do is go up to this is valid and set it to 1 so you can see what you're doing and then all you really have to do from there is come down here where it says lens distortion say enable lens distortion and click those boxes and as soon as you click them you'll notice that the image changes a little bit
i'll do that again unless your camera is way off it's not going to be obvious also make sure this uv displacement box here is checked remember what this material is called and then you can save it and close it and also save and close the media bundle and come back and select your media plate over here in the world outliner if you've done a normal setup of my vp studio it should look something like this with the media source coming in here and the material blank what you want to do is find that material we just made and add it to the material section right here and the minute you do that you'll notice that my camera view changed a little bit
right here and is now undistorted and all the vertical lines look pretty nice and straight now my camera didn't have much distortion in the lens to start with so you may not notice the difference here depending on your lens you might see more generally wide angle lenses have much more distortion than narrower ones the last thing you want to do is set up your unreal camera in vp studio to match the field of view of your real camera that unreal gave you in the previous page to do this come up and click vp camera and you should already have your film back set to match your real camera and what you want to do is have this current horizontal fov down here match the fov that unreal gave
you on the previous page right here now because unreal doesn't allow you to type that in directly you're just going to have to adjust the current focal length until it matches up takes a bit of fiddling but i've got the correct value so you don't have to watch me do it now you can see the horizontal fov is a pretty is a close match to what was on the other page and that's basically the whole process hope this was useful and i'll see you again next time
Transcribed 20 blocks · Source: YouTube