In this post we’re going one more step into understanding layers by developing a project around using a layer mask. The first layers tutorial had a brief little intro to them and now it’s time to start really getting a grasp of what they can do. This project may take a little time because there is some detail work involved. If you’re in a rush you don’t have to do all of the fine detail stuff – you will still get a grasp of what the layer mask is doing – but you’ll get a better image if you spend some time on it.

Briefly, a layer mask is a grayscale image that “masks” the layer it is associated with. Think of painting your house. You mask off the trim to avoid getting color on it. The layer mask works the same way, where black is your “tape” and white is the color (or whatever is in the layer image). Black areas will show your lower layers while white keeps what is on the masked layer. Any greys will be partially transparent.

We’re going to take an image that a friend of mine provided which was taken in Yellowstone National Park at Mammoth Hot Springs. While it is a pretty cool picture, I wanted to see what the effect would be if the tree were in focus and the background sky and mountains were blurred. So load up the picture (Original image is at the bottom of the post) and let’s get blurry…

1. Duplicate the layer. For many reasons…I always do this. It’s just a good habit. The beginning layers tutorial described a number of ways to achieve this, but the quickest is to hit Ctrl/Cmd + J. Ok, now for more excitement – do that again so that you have three identical layers.
2. Double click on the name of the top layer and rename it to “blur”. Go up to Filters > Blur >Gausian Blurand select a good amount of blur (I chose 5.5). Add a layer mask by clicking on the layer mask icon (at the bottom of the layers window – it looks like a square with a hole in the middle). You should now have a nice little white rectangle “mask thumbnail” next to the image in your blur layer.
   

   Layer Mask is selected

   You should get used to checking which part of the layer is selected – the mask or the actual image on the layer. If you click on the image thumbnail, you will see a white border with black corners around it, indicating that the image is selected and anything you do will affect the image. Now click back on the white layer mask thumbnail and you’ll see the white and black border highlighting that thumbnail. Leave that selected for the next step – we want to work on the mask, not the image in this layer.

3. Here’s where we start to have some fun. Grab the paintbrush and select a brush size around 6 and hardness of 85%. With the black foreground color and the “layer mask” selected, paint over the top of the blurred-out tree. You should see it start to re-appear in focus. If you look closely though, you’ll notice the edges of the tree have a strange “halo” look to them. This is because the blur filter doesn’t recognize those edges and will spread the tree out to blur it with the background. So how do we fix that?
4. Go ahead and delete that blur layer and create another one (yeah I know, I really did just make you do something wrong 🙂 ) Don’t apply the blur filter just yet. If you have CS5, there’s an awesome little tool called “content aware fill”. This will let you select around the tree and fill it with background really easily. I’m still stuck with CS4, so I’m going to do the old-fashioned clone stamp method. As soon as I can get an upgrade I’ll give some more details on the fill tools (yeah, love having to budget this stuff…). Ok, so since we are going to just blur this image you don’t have to be super-accurate in this, but use the clone stamp tool to copy the background over the edges of the tree. You don’t have to completely erase the tree…just enough so that when you blur it you won’t see those fuzzy edges coming through that mask. If you’ve never really used the clone stamp tool before – it’s the one just underneath the paintbrush that kind of looks like an old wood handled rubber stamp. Basically, It copies one part of your image to another. Hold the alt key down and click somewhere in the mountain range near the tree. Now move over and click and “paint” along the edge of the tree. You should see that you are copying the part of the mountain you selected over the tree. This can get a little tricky where the mountain meets the sky, but by centering your cursor on the boundary between mountain and sky to alt-click and then moving over to the tree and still centering the cursor on that same boundary, you should be able to get a pretty good effect. This is what my layer looked like after this step.
5. Once you have that done, go ahead and apply the Gausian Blur filter and create a mask and start brushing in the mask as described earlier in step 3. You should now be getting nice crisp edges on your tree. Here’s the neat thing about the mask…you can easily undo anything you’ve masked by switching your color to white! Try it – select white as your color and brush over some of what you’ve been masking. You should see it go all blurry again. So, by switching between black and white and changing the size of your brush (another keyboard quickie – using the bracket keys “[“ and “]” will reduce or enlarge your brush size respectively) you should be able to mask out the tree from the blurred image. This is the detailed part of the project that I was talking about. If you don’t want to spend the time doing it all, that’s fine. Another keyboard shortcut I use a lot in this process is the X key. This swaps the foreground and background colors…so you can switch between black and white just by hitting X.
6. You might be thinking that you’re not entirely sure if you’ve got something masked or not – well, Photoshop has a handy helper for that. Hit the backslash “\” key, just above the enter key. Your masked area will show up highlighted in red. If your image had a lot of red tones in it you could change this highlight color by double clicking on the layer mask thumbnail. Hitting the backslash key again will toggle the highlight off. I actually find it easier to mask off more than I need and then go back in with the white to unmask up to the edges of the tree. Once that is done I’ll go and clean up the rest of the mask. In this image you can see that there is a lot of “extra” mask (the red) but the edges of the tree have been cleaned up.
7. Now if you think about how a large aperture in your lens gives a shallow focal plane, you will know that the tree isn’t the only part of the image that should be in focus. The ground that is the same distance from the viewer will also be in focus. To create this effect select a larger brush (somewhere around 70-100) and brush a black stroke across the image through the base of the tree.
   

   The final highlighted mask.

   Now reduce the opacity of the brush to around 25% and slowly expand that stroke into the distance and toward the bottom of the image. Reduce the opacity even more to 10% and really gently feather off the mask in the distance. Don’t forget to use the mask highlight “\” to get a good visual on how this is going. When you’re all done you should have a nicely blurred background and the tree and ground surrounding it will be in focus. This is what the highlighted mask should look like.

8. Just to show off a little more of the power of layer masks, select the first Layer 1 image and let’s do a slight color enhancement to it. Go up to Image>Adjustments>Curves and slightly enhance the shadows and highlights. It’s ok if you’re not familiar with the curves dialog – I’ll go over that in more detail later – but for now just click in the upper right part of the line and pull it slightly higher and then click in the lower left section of the line and pull it slightly down until you have a curve that looks something like this. If you have the “Preview” box selected you will actually see the contrast in the tree change. If you would like, feel free to play with the curve and see what effect it has. You should notice that the masked “blur” layer doesn’t change – just the layer with the tree that is showing through the masked area. This ability to adjust color and tone of one layer without affecting the rest of the image is one of the really powerful features of layer masks. You can do the same to the blurred layer – select it and do a curves adjustment on that.

Congratulations! You just created a blurry background for your picture by using layers and a layer mask. There are a lot of possibilities for using masks like this. Another very common photographic effect that can be accomplished with a layer mask is having one item in a picture be in color while the rest is black and white. Look for a quick-bit tutorial soon to see how that’s done if you need help!

Here’s the original image to play with:

Photo courtesy of Chip Summit