Video Transcript
REtracker FIZZ/BLISS Tutorial (Part 3)
Marwan Rassi · 2,262 views 11 Aug 2023 · Duration 25:44 · youtu.be/c8Hr1AMLRZE
The road to perfection: This tutorial helps you integrate your lens data (Focus, Iris, Zoom) directly in Unreal Engine. There is no need for any extra software besides UE5. (90% of this tutorial is helpful for other lens encoding systems) At the moment this is the only fail-proof method that delivers the most pinpoint accurate results. https://www.retracker.co/ Credit: K. Rassi (carl.rassi15@gmail.com)
Auto-generated YouTube captions, grouped into ~45-second blocks. Product and technical terms may be misspelled. Timestamps link to the video.
[Music] hello everyone thank you for joining us for part three of this demo video in this session we will guide you through the process of lens calibration and virtual set alignment using the combination of retractor Bliss and Fizz with Unreal Engine additionally we will share some tricks to speed up the zoom lens calibration workflow so let's get started launch the main retract or Bliss application and turn on the retractor Fizz then proceed to run the unreal editor please load the live link preset we saved earlier verify whether the live link Bliss and Fizz sources are successfully receiving tracking data
since we are not currently utilizing April tag tracking it is okay for the status not to be displayed in green therefore we can leave it as it is for the time being open the lens file we previously created to review we have completed the two-point lens mapping for Focus as well as a lens mapping for the iris regarding the focal length as previously mentioned it has been deleted and we will utilize lens Distortion as a method for calculating the focal length Distortion and Image Center please take note that each time you rerun unreal editor the raw Fizz lens data will require the renormalization process as mentioned in part two of this demo video each raw Fizz input will be scaled from 0 for the minimum to one for the maximum utilize the nucleus M hand unit to turn the focus wheel Iris wheel and
zoom joystick to their respective minimum and maximum positions in order to guarantee that both extreme points of each raw Fizz input are reached kindly repeat this process several times as we proceed we will continue to enhance the focus lens mapping by adding more points let's start by adding a midpoint for the mapping curve in my case it is 180 centimeters referring to the marker on my lens Focus distance next locate about one quarter of the mapping curve and add another key point I got 120 centimeters then proceed to the third quarter of the mapping curve and add a key point there in my case it is 370 centimeters
continue to introduce additional key points to refine the mapping curve further this will ensure a more precise representation of your lens Focus distance once all the necessary key points are in place select them and set their tangents to automatic as you can see the mapping curve has now become smoother for the iris lens mapping let's maintain a straightforward two-point linear curve remember to save your progress for the lens file now let's prepare for the focal length and Distortion calibration process lenses possess diverse focal lengths ranging from wide angle to telephoto this prompts the need for checkerboards printed in various sizes to facilitate the focal length and Distortion
calibration process for example while an A3 size checkerboard might effectively serve its purpose when calibrating the minimum Zoom position however when dealing with the maximum Zoom position the A3 size checkerboard might prove either too large or Too Close Up in such cases opting for a smaller checkerboard when calibrating the maximum Zoom position allows better coverage of all areas including corners and edges within the lens View let's return to the unreal editor and select the checkerboard we created earlier foreign duplicate the checkerboard and change its material to Blue
we will use this blue checkerboard for the smaller size checkerboard in a later step [Music] I'm going to delete the atmospheric fog player start and the sky sphere since they are not required during the calibration process [Music] next temporarily add a cube to your scene select its material and search for Checker then choose the Checker material click on browse to the materials and content browser and this will direct you to the location where the material is saved double-click on the material to open the material graph editor then select all the nodes and copy them while the cube is still selected choose its
material once again and search for cross select M Crosshair from the available options next navigate to the location where the material is saved you will notice that M Crosshair is actually located inside the camera calibration content and this material can be enabled during lens calibration if you are feeling confused about what I am doing at the moment don't worry it will become clearer in the later steps now double click on the material and paste the nodes we copied earlier from the Checker material next right click with your mouse and search for overlay s overlay from the available options connect the output which is the first node from the right that we pasted earlier to the input labeled a of the overlay then connect the output of the overlay to the emissive color
this will break the original connection where it was previously attached select and connect the output of the multiply to the input labeled B of the overlay now right click on the alpha input of the overlay and choose promote to parameter from the available options change the parameter name to Checker Alpha and set the default value to 0.2 next select the density node and change its parameter name to Checker density and set the default value to 4. lastly apply the changes to the material and save it the material editor will likely prompt you with a warning message proceed with yes to apply the changes of the modified material to the original material select the cube we added earlier and delete it as we no longer need it now open the lens file we have
been working on and switch to the calibration steps in the viewport settings select overlay and change the setting from none to Crosshair this newly added Checker overlay can serve as a useful viewport guideline and enhance the workflow for the calibration process before we begin it is crucial to determine the focus distance and which size printed checkerboard that will work for both the minimum and maximum Zoom positions ensuring that the printed checkerboard remains in Focus throughout the entire Zoom range is essential using a higher aperture can help widen the range of focus [Music] maintaining consistent Focus input is a critical aspect of successful lens calibration as it directly impacts the accuracy of the Distortion and focal
length calibration taking the time to verify the focus input beforehand will prevent potential calibration errors and result in more reliable outcomes during the calibration process as you can see my focus input is 0.77 and the zoom input is currently at its minimum position let's proceed to create two key points one at the minimum Zoom position and another at the maximum Zoom position both with the same Focus input this will form a mapping curve with two key points and the cine camera in the scene is controlled by the lens file [Music] however let's delete this mapping curve to explore another scenario now let's observe what happens when we use different Focus input for the key
points for the first key point I am going to use the same Focus input we used earlier and set the zoom at its minimum position for the second key point I will use a different Focus input and set the zoom at its maximum position as a result a new Focus set has been added the first key Point associated with the minimum Zoom position is stored under Focus set 0.777 while the second key Point associated with the maximum Zoom position is stored under Focus set 0.787 however no mapping curve is formed for each Focus set and the cine camera remains unaffected and not being controlled either when calibrating a zoom lens from its minimum to maximum position maintaining the same Focus input is critical for
establishing a proper mapping curve now switch to the calibration steps mode and then select lens Distortion I am using an A3 size checkerboard for the minimum Zoom position with a raw Focus input at 0.777 which corresponds to a focus distance of 400 centimeters next navigate to the viewport settings and click on lens Distortion algo from the available options choose lens Distortion checkerboard select the camera calibration checkerboard that you created earlier checkerboard should represent the A3 sized printed checkerboard with a square size of 5.1 centimeters enable the show coverage overlay option and choose Crosshair for overlay utilize the Checker overlay as a guide to achieve good coverage of all areas
including corners and edges within the view of your lens let's start from the top right corner proceed to click on the viewport to capture images capture an appropriate total of images around 16 or more ensure that these images are taken with the board at different angles aligned directly to the camera both at a 45 degree angle and in the opposite direction once you have captured all the required images proceed with click on add to lens Distortion calibration this will calculate the focal length Distortion and image center of the current Zoom position subsequently a RMS reprojection error report will be prompted as a result of the calculation generally an error value below one is considered acceptable alright let's take a look at the evaluated value obtained from the
calculation for the minimum Zoom position it is 19.236 millimeters it appears to be quite close my focal length is around 18.5 millimeters next we will calibrate it at its maximum Zoom position ensuring that it maintains the same Focus input we used earlier as you can see the A3 size checkerboard appears too large or Too Close Up however using the A4 size checkerboard seems to work well for my maximum Zoom position [Music] in the viewport settings configure the checkerboard with the camera calibration checkerboard 2. navigate to the unreal editor and select the duplicated CG checkerboard adjust its Square size accordingly for my case
I measured the square size on my A4 size checkerboard which turned out to be 3.1 centimeters return to the calibration steps I will use a checker overlay of three by three if feel free to adjust the Checker overlay based on the size of your checkerboard and your personal preferences repeat the same process that we performed earlier click on the viewport to capture images and ensure that these images are taken with the board at different angles aligned directly to the camera both at a 45 degree angle and in the opposite direction thank you once you have captured all the required images proceed with click on add to lens Distortion calibration navigate to the
lens file panel and check if any key points have been added after the calibration as you can see a key Point has been added at the maximum Zoom position resulting in the formation of a mapping curve the evaluated value for the maximum Zoom position is 109.716 millimeters which is quite close my focal length is now 110 millimeters next we will proceed with another calibration at the midpoint of the mapping curve set your Zoom input to around 0.5 switch to the calibration steps in this case I will use an A3 sized checkerboard remember to set the CG checkerboard that represents the A3 size repeat the same process as we did for the minimum and maximum Zoom positions
[Music] once you have captured all the required images proceed by clicking on add to lens Distortion calibration navigate back to the lens file panel and you will see that we now have three key points for Zoom all using the same raw Focus input 0 for the minimum 0.5 for half and one for the maximum return to the calibration steps navigate to the nodal offset tab set the nodal offset algorithm to nodal offset checkerboard configure the calibrator by selecting camera calibrator checkerboard 1. click on the viewport resulting in the addition of a series of 2D and 3D points to a list proceed by choosing apply to calibrator this action will reposition
the CG checkerboard in the scene aligning it with the checkerboard visible through your lens as you can see the CG checkerboard and the media plate are quite well aligned and matched we are about to start the nodal offset calibration therefore please proceed by closing and restarting the Bliss main application [Music] return to the unreal editor and select the cine camera actor to verify its location and rotation values when initializing the Bliss sensor from a clean start the location values x y z should ideally be centered around zero with minimal deviation regarding the rotation the initial values for X and Y which correspond to roll and tilt may vary based on the mounting orientation
of the sensor however the value for Z which represents yaw should be around zero with a slight variation now select the April tag and verify that the tracking data for the April tag is being received and its location and rotation values are being updated as you can see the April tag in the scene is dynamically changing its position as I physically move the printed April tag here are some tricks to enhance the chances of obtaining successful results with the nodal offset the Bliss sensor is an optical tracking system tracking becomes more reliable when the surrounding area is filled with reference points during the nodal offset process it's not obligatory to conduct it right in front of your green screen experimenting with a location that rich in details can lead to better outcomes next open the lens file and ensure that
the nodal Point offset is empty and cleared of any values switch to the calibration steps and navigate to the nodal offset tab within the viewport settings choose the nodal offsets points method for the nodal offsets algorithm select April tag as the calibrator [Music] position your camera's framing so that the printed April tag Falls within the view of your lens I will be using the same raw Focus input that we have been using along with the minimum Zoom position sequence for clicking calibration points on the April tag may vary depending on the version of Unreal Engine you are using in this demo video we are using Unreal Engine version 5.2 which follows this order starting from the top left bottom left bottom right top right and finally the center reduce the lens file window slightly and
position it side by side with the live link and details panels while keeping the April tag selected keep an eye on the live link panel ensuring that the tracking data for the April tag consistently displays a green status at the same time monitor the details panel to confirm that its location and rotation values being updated throughout the nodal offset process click on the viewport within the lens file window to capture the April tags 2D and 3D points for the nodal offset calculation proceed by clicking points in the following sequence top left bottom left bottom right top right and finally the center repeat the same process while positioning the April tag at different angles and distances from the Bliss sensor your goal is to gather around five or more images however Please be
aware that increasing the number of images might result in a higher RMS reprojection error [Music] once you have done with all the required points proceed with click on add to nodal offset calibration usually the expected outcome for the RMS reprojection error should ideally be zero [Music] now switch to the lens file panel and verify the values of the nodal offset calculation the third value of the location as it should precisely represent the offset between your bliss sensor and the center of your lens along the vertical axis you can further validate this value by comparing it with the measurement obtained from the measuring tape regarding the first value of the location it should indicate the offset
between your bliss sensor and the nodal point of your lens along the forward or backward Direction therefore the value should logically not exceed the total length of your camera if the nodal offset result doesn't meet your expectations you have the option to delete it subsequently you can follow the same procedure once again to obtain a different calculation result we will now align the virtual set with the physical environment using an aruko marker please place the printed aruco marker on the floor switch to the calibration steps and navigate to the nodal offset tab under the viewport settings select the nodal offsets aruko markers for the nodal offsets algorithm choose a roko ID 8 as the calibrator adjust your camera's framing so that the printed aruko tag is within the view of your lens before proceeding select the aruko
marker from the outliner and reset its location and rotation values to zero then select the floor and set the Z value to around minus 0.02 this adjustment will improve visualization while fine-tuning the delay in the later step return to the lens file panel and click on the viewport within the lens file window this action will generate a series of 2D and 3D points in a list proceed by clicking apply to camera parent adjust the transparency until achieving a good blend between the CG aruko marker and the media plate pan your camera to either the left or right and then come to a stop verify the alignment and matching between the CG aruco marker and the marker on the floor this step will allow you to evaluate the accuracy of the nodal offset results you have
obtained earlier while rapidly panning your camera from left to right observe and measure the amount of delay navigate to the time data Monitor and make adjustments to the time correction for Bliss this adjustment will introduce amount of delay in the incoming tracking data from the Bliss sensor the timing between the video feed and the tracking data appears to be well synchronized now next I will enhance the focal length mapping by introducing additional key points the current key points for the focal length mapping were obtained through the Distortion calibration method however I will manually introduce more key points to the mapping curve using a checkerboard as a guide while achieving this through the Distortion calibration method as possible however utilizing the
Distortion calibration method would indeed lead to an increased time requirement for the calibration process switch to the calibration steps and navigate to the nodal offset tab in the viewport settings opt for the nodal offsets checkerboard as the nodal offsets algorithm and select camera checkerboard 1 as the calibrator next proceed by clicking within the viewport and then select apply to calibrator adjust the transparency until achieving a good blend between the CG checkerboard and the media plate as you can see when zooming into the maximum and zooming out to the minimum position the alignment and matching are noticeably off I will now share a trick to fix this issue undock the calibration steps panel and position it side by side to the lens
file panel then I will proceed to Simply remove the image Center values that were obtained from the earlier Distortion calibration process before making any modifications to the current lens file please create a backup copy of it in case you need to revert in the future to clear the image Center values select the image Center within the lens file panel and then click on the trash bin icon this action will reset and utilize the default values of 0.5 for both the X and Y coordinates of the image Center across the entire Zoom range although this approach might not be the conventional method it is proven effective in my case each time we make adjustments to the lens file it is necessary to update the position of the CG checkerboard to do this return to the halfway point
of the mapping curve click on the viewport within the calibration steps and then proceed to select apply to calibrator as you can observe the alignment and matching significantly improve with the default values of the image Center [Music] to fine tune the focal length drag its key point up and down until the CG checkerboard is precisely aligned and matches the checkerboard in your media plate during the process of enhancing and refining your focal length mapping curve employing linear interpolation simplifies the procedure please feel free to add key points along the mapping curve using the same raw Focus input in my case the focus raw input should always be 0.777 drag the key points up and down until
the CG checkerboard and your printed checkerboard are well aligned and matched continue to introduce additional key points to further refine the mapping curve this approach will result in a more accurate representation of the entire Zoom focal length [Music] it appears that introducing a slight delay in the lens data could achieving better synchronization to implement this navigate to the time data Monitor and proceed to make adjustments to the time correction for free d which serves as the live link for receiving the lens
data [Music] the timing between the video feed and the lens data now appears to be synchronized [Music] next continue by fine tuning the alignment and matching between the checkerboards covering the range from the minimum to the maximum Zoom position however in this phase make sure to frame your checkerboard near both the left and right edges as well as the center [Music]
foreign also remember to test the focus distance during this process [Music] alright that concludes part three of this demo video to wrap things up let's enjoy the quick demo we've put together using the lens file we have created from the lens calibration process thank you for watching [Music] foreign
[Music] [Applause] [Music] [Music] [Music]
Transcribed 32 blocks · Source: YouTube