Video Transcript
Unreal 5.7 Augmented Reality with Shadow Catchers!
Greg Corson · 2,357 views 25 Jan 2026 #unrealengine #virtualproduction · Duration 19:00 · youtu.be/iGAzSpmpUqw
Unreal 5.7 Composure now has shadow catchers so CG characters now can cast shadows onto the live camera scene. This makes augmented reality scenes much more realistic! You can match lighting in the CG scene to the lighting in your real studio so CG shadows will almost exactly match! If you haven't watched tutorials 1-2 you probably should as they cover basic setup and video in/out. I haven't repeated that info in this tutorial to keep it short and to the point. You can find all my new Unreal 5.7 tutorials in this playlist Unreal 5.7 Tutorial Playlist • Unreal 5.7 Virtual Production Tutorials 00:00 Intro to Shadow Catchers and Augmented Reality 00:16 Look! Shadows! 00:50 The live scene in the
Auto-generated YouTube captions, grouped into ~45-second blocks. Product and technical terms may be misspelled. Timestamps link to the video.
Unreal 5.7's composure has given us a big boost for augmented reality use. Before we would take a scene like this one coming in from my camera and we'd overlay a CG character like this one, but something was missing. They had no shadows. Well, now we can add accurate shadows and it's relatively simple to do. It's called a shadow catcher and I'm going to show you the basic setup for using it. Hi everyone, I'm Greg Corson and this is tutorial number three on basic Unreal composure setup for augmented reality. I'm going to show you the basics of how to set up a simple scene like this one. In my first two tutorials, I showed you how to set up camera input and how to do basic composure setup. This one uses much the same starting point. Just start with the
virtual production template as I explained in tutorial number one. I started out by setting up this simple scene for my camera. I've got a couple sheets of white paper, a small cardboard box, and a few other things set up on my desk for my camera to look at. We're going to add the augmented reality character into this scene. The second step is to go into Unreal and build a scene for the augmented reality part. In this case, I've got this character that I want to add into my camera feed. And then I also have a simple plane on the ground and one in the back in the same positions as the floor and the back wall of my video scene that you've seen already. I also have a light source and a camera that are in the same places as
they are in my real scene. Building the Unreal scene to actually match your real scene can be kind of tedious. So, I'm not going to show you the entire process because it's kind of long and boring. The best way to get everything lined up correctly is to take measurements of your real world set and then build the Unreal set to match those measurements. Keep in mind that the Unreal set doesn't have to be exactly the same size as the real world one, but it should have the same proportions. In this scene, I've just used a couple of simple planes that are part of Unreal and available from here. And I've also used the Unreal Mannequin, which you can get for free from Fab. You just go to the Fab section here, and you can type in mannequin. And just to thin
this out a little, I'll say only show free products. And this is the mannequin asset pack right here. You can just click that and import it into your project for free. So now we have to put all this together into a composure composite. Here's how we do it. You start out by going up to our media profile and selecting my Blackmagic input. And that's what the camera sees. And I can close that up. Greg from the future here. Sorry, the first time I recorded this, I left out a couple of important points that I wanted to insert at this point. So, here they are. Um, one of the things that you are going to want to do when you're working with augmented reality kind of things like this is normally you're probably used to working with the camera in
perspective mode up here. Um, but once you start projecting uh the texture from your live camera into the scene, you're going to want to be in your cine camera and to make sure that everything is from the correct angle. That makes everything very easy. So, you'll notice from here on in, I'm in cine camera mode almost the entire time. The other thing that's reasonably important is you don't want to accidentally move your cine camera once you get it lined up to match your real camera. That's also kind of easy to fix. You can just click on the cine camera over here and under transform, you click lock actor movement. That'll prevent the cine camera from being accidentally moved, which can screw your whole shot
up. Finally, when you're trying to get the lights to look correct, you want to make sure that whatever light you choose is set to ray traced mode. I show that a little later in the video. The other thing you want to mess with is obviously intensity, color, and then the uh attenuation radius, which may need to be kind of large to get the light to to not fall off before it reaches the background. And to get this sort of nice soft edged shadow, you need to mess with the source radius as well. The source radius usually starts at zero like this, making it a very hard shadow. And if you crank it up, you can make that shadow as hard or as soft as you want.
For this scene, I left it around 50 because it pretty much matches the lighting that I have on my real scene. The last thing I wanted to mention, when you start using projected textures, the scene can become a little confusing to look at because the camera image is being projected onto all the geometry. When this is happening, you can lose all track of where everything is pretty easily. But one way to fix that is to go up here to the lighting selection and change it to lighting only. That'll basically turn off all the texture maps and let you see just the 3D geometry. At the moment, that doesn't really do all that much, but you'll see uh later
on in the tutorial. I'll try to demonstrate this, but I wanted to let you know now in case you're following along and you need to unconfuse what you're looking at. And now back to the tutorial. And then I can add a composite actor here. The first thing I want to do is choose my resolution, which is 4K. and choose my camera which is already in the scene. Then I come down to my plate layer and click over here on the media texture and select the Blackmagic camera. The next thing I need to do is select the geometry that my camera will be projected onto. What this is actually doing is taking the video from my live camera and projecting it onto this three-dimensional geometry. And what I want it to project onto are these two
planes. So, I'm going to go in and select those two. And everything just got way too bright, as you can see. That's because I'm still missing one thing that I have to set up. You have to give the actors that will be catching the shadow a special material. So, I go in and I say apply lit mask material to both of these. And now you can see we're most of the way there. I have my mannequin character here, but she doesn't have a shadow yet. To give that character a shadow, I go up to the shadow reflection layer here. Turn that on. Then I select the geometry that I want to cast shadows. In this case, it's the queen mannequin. And now I've
got a shadow. You'll notice right now that the shadow doesn't take into account this box here because I haven't added it yet. I'll do that in a minute. Now, odds are once you've got everything set up like this, it's not going to look exactly perfect because things may not be in exactly the right places. Let me show you what that kind of looks like. If I select the back plane here, supposing it's in the wrong spot, then you'll see that the shadows are in the wrong spot. So, once again, we eyeball this So the shadows match up correctly. You might also have a problem with the shadows
not looking the same. For example, if the point light is in the wrong spot, you might have to raise or lower it like this. So the mannequin's shadow matches the other character's shadow. Also, to get a light with a proper shadow that looks correct, you need to make sure ray tracing is turned on. So you can select the point light and in search box in details you can just say ray [snorts] and in my case I've already got ray tracing enabled. If it was disabled you can see the shadow wouldn't be anywhere near as dark or consistent. So make sure it's on enabled. Another interesting thing about this setup is that you can actually have reflective
surfaces. In fact, you can see that my Quinn mannequin here, you can see reflections of her legs down here on the floor. And these lighter areas in the shadowing also come from the main light reflecting off the floor and onto the backsplash here. In this case, we don't really want that because in my real world, this is a piece of paper which is a mat and doesn't have any real reflectivity. So, we have to do a modification to the material to fix that. So, what I'll do is go to one of my planes here and scroll down to the material part. And I can double click on that to open it.
And right here, we can see M composite mesh lit mask. That's the material we want. Now you could just modify this directly but then it would be modified for all time. So what we want to do is make a copy of it and create a new material for ourselves. So I'll go down to my content drawer here and I'll hit m composite. And you'll see down here in the engine components we have two materials and the one we want is this one here. If you don't see these, go over to your settings here and make sure that you have engine content, plug-in content, all these things should be checked.
So then what you do is you make a copy of this just duplicate it and name it something. And then you want to drag it back up to your top level content folder where you're getting most of your work done. Now I'll open up that material. And you can see it's not too complicated, but don't let it scare you. We only have to change one thing. Now, I already had modified this once before. In the material, the default
material will have specularity set to one, which gives you that reflection from the floor. And what we want is actually zero. So, all you got to do is go in here, put in a zero, and then save it. Then we'll right click on that material we just modified and say create material instance. And that should give you something like this in your top level folder, a material and a material instance to play with. Now, all I have to do is apply that material instance to the two planes so they won't be reflective anymore. I've opened up a content browser window here so that you can see what I'm doing. I just take the material instance. I select plane
and I drag the material instance down here into the material. And you notice those reflections disappeared right away. I'll do the same thing with the second plane. And you notice the bit of the shadow that was a little off here in the center is now solid. The reason for that is that this floor is no longer reflecting like a mirror up onto this back wall. So everything looks a lot more consistent now. So the last thing we have to do to get this scene to look right is to add a shadow catcher so that Quinn's shadow will wrap around this box properly. So we'll start by adding a box.
You can see it pretty clearly now here. And then we have to get it to match the size and shape of this box. And to do that, I'm just going to fiddle with the scale of the box until it looks right. So, I'll stretch it out a bit. and stretch it vertically a bit. Or excuse me, stretch the size the other way a bit.
[snorts] until it looks right. That's about right. Notice the shadow now matches the shadow of the real box from my video camera. And the last thing I need to do to get that box to work correctly is come back up to the composure layer. And I go to my plate layer and then I add the cube to my shadow catchers list. And just like we did before, we have to add the correct material. And since this is a matte box, I'm going to use that
material we made earlier. So, I'll bring back my content browser and I'll select my cube and I'll copy over that material instance that we made a minute ago onto the cube. And that makes it behave as a shadow catcher also. And now we have the same scene we had at the beginning of the tutorial. I mentioned earlier that if your real world camera and Unreal Cine weren't in the same places, things could end up looking pretty bad. Uh, let me show you what that looks like by going to the perspective camera. So, you can see that if I move the perspective
camera around so that it's not in the same place as my real world camera, things can get pretty distorted and weird looking. One of the ways you can get around this if you're trying to adjust the 3D geometry and it looks weird is to go up here and choose lighting only. And now you'll only see the the raw 3D geometry without the extra projected textures. It's much easier to handle this, but for the most part you want to just stay in Cine Camera Actor so that everything looks correct. Now, through this tutorial, I have pretty much eyeballed getting the positions of the lights and camera correct so that everything looks right, but that can be pretty tedious to do,
particularly if you've never done it before. Uh, I wanted to do this tutorial with a minimum of extra stuff in it so the people who didn't want to get into the gory details of camera calibration and and other such things could uh still get an idea of how everything works. In a future tutorial, I'll cover additional ways that you can make the process of aligning the camera simpler and quicker, but this does require you to have a calibrated camera lens, and that can take a little time to get set up correctly. So, that'll probably be in the next tutorial I publish. With this setup, as long as the camera doesn't move, the mannequin could be animated to walk around the scene, and everything would continue to look
right. The only trick here is because we're not doing anything fancy, she can't walk behind any of the real world objects yet. That's a much more complicated setup that we'll get to in the future. The same goes for the real world objects. If that real world robot wanted to walk around the scene, it couldn't walk in front of the CG character right now, but it can walk behind them without any problem. Though the shadows might not be exactly right. Just to give you an idea what that looks like, I'll go over and stick my hand behind the CG character's head. As you can see, it looks pretty
convincing. So, this should give you a pretty good idea how to set up a basic augmented reality setup. Now, this setup can also be used with camera tracking, calibrated lenses, and a lot of other things to make it more realistic and to allow you to move the camera around while you're shooting. There are quite a few tools available that can make aligning the camera, setting up these kind of scenes, and even digitizing the real world area that you want to use as a shadow catcher. These are a bit more complicated to explain and take some setup, but if you need that kind of functionality, it can really save you a lot of time. I'll try to get into that in future videos. If there's a particular use case you want to see, please write it up in the
comments and I'll try to get to that in the future tutorials. If this is the first of my new tutorials that you've watched, remember that I've done two ahead of this that you should look at. That's tutorial one and two, which covers setting up a basic green screen and setting up video input and output. If you subscribe to my channel, you'll find a playlist titled Unreal 5.7 Virtual Production. It has all of the most recent tutorials starting with Unreal 5.7. There's a number of other videos on the channel as well, but the 5.7 ones are probably the ones you want to look at first because they're the most recent. If you found this tutorial helpful, please remember to subscribe to the channel and hit the like button on this
video. It really does help the channel out a lot. And also, please leave a comment if there's anything you have questions about or want to see in the near future, and I'll do my best to get back to you on it. So, that's all for now. I'll see you in the next tutorial. I'm Greg Corson, and have a good day.
Transcribed 25 blocks · Source: YouTube