Refining Masks in Photoshop

Refining Selections

The marching ants that represent selections have been around forever. But it doesn’t mean that they are the best tool for the job; it’s just all we have had until recently. Some years back Adobe gave us the Refine Edge tool. In addition to viewing the selection in a different way, it gives you a chance to modify selections as well. Very rarely do you create a perfect selection on the first go around. This new tool gives you ample opportunities to fine-tune the selection before you turn it into a mask.

When you have any selection tool active, such as the Magic Wand or Quick Selection tool, and a selection active (the marching ants are visible on your screen), you will have access to the Refine Edge command in the Option Bar. This command (pictured below) will allow you to modify or refine the edges of your selection.

The advantage of working with your selections using this dialog box is that you are able to see their true edge. With just the marching ants, it is often difficult to tell how well you have selected an area.

The Refine Edge dialog box gives you many ways to preview a selection. By clicking on the View Box (circled in red) you get access to the different ways that you can view the area of the image that is selected.

By placing your cursor over the icon, you receive a description of the view. The first icon is the least useful. It is the Standard view showing marching ants.

One of the most useful is the On White view, which works well in general and for darker objects. You also may find the On Black view useful for lighter objects.



Standard View

On White View

On White View

On Black View

Radius Slider
By increasing this slider, you are increasing the area around the original edge that will be affected by the settings. The increased radius allows the edge to get bigger and become softer. This will be the effect if this is the only slider that you use. If you use further refinements in the bottom of the box, this radius amount is defining the region in which the other options will operate.

Contrast Slider
This slider’s main goal is to remove any fuzzy artifacts that may have become apparent when the radius was enlarged. Radius and contrast work together to tighten the selection or make it more detailed; but don’t turn up radius too much because that’s the job of the Feather slider. Another way to think of the radius is that it is used to create a soft enough edge for the contrast to have something to work with.

Smooth Slider
The Smooth slider does just what you think it may do. It smooths out the rough edges of a selection removing any hard edges.

Feather Slider
The Feather slider is similar to the Radius slider in that it “blurs” the edge of the selection. It differs in that it exerts no control over the region that is being worked on by the other sliders; it is chiefly used for blurring the edge. Use the Feather slider to blend your adjustment from inside the selection to outside the selection. Remember that what is white is selected and what is black is not selected. If it is a shade of gray, it is partially selected. This means that only some of the adjustment will come through.

Shift Edge
The Shift Edge slider will make your current selection edge grow outward (expand) or inward (contract). If your edge is hard, it will stay hard but just grow inward or outward. If it is soft, it retains its soft nature and contracts or expands. To get any noticeable amount of expansion, the Radius slider may need to be increased. Just increasing the Contract/Expand amount without increasing radius may produce very little movement of the edges.  Increasing the Radius slider increases the region or the area around the edge that will be affected by the Contract/Expand slider (or any of the other sliders as well). This slider comes in handy for removing halos. Click OK inside the Refine Edge dialog box to commit to the changes that you made. You will be returned to your image with the new selection still active. Remember that you may not see any visible change to the marching ants. Don’t worry, though—when you create an Adjustment Layer, the resulting mask will look just like the preview!



Refining Masks

Modifying the edges of the selection with the new Refine Edge tool is a pretty neat trick. It does have one drawback, however: visibility. The problem with working on the selection occurs when you are masking out an Adjustment Layer. The Adjustment Layer, of course, will produce a change in the image. This change may or may not be obvious at the edges of the selection. With just modifying the selection before the adjustment is made, you have no idea how each side of the selection edge will look.

If you create a good selection first, then create the Adjustment Layer and turn it into a mask, and then modify your mask, you will have a real-time visual of the effects of your edges. You will be altering your mask as it masks out (or reveals) the underlying layer or new Adjustment Layer. The ability to see the changes as you adjust is very important.

The Masks Mode in the Properties Panel allows the Refine Edge Tool Controls to work on a mask.  Once a Mask is made, click directly on the mask to change the Properties Panel from showing the adjustment to showing the Masks Controls.  It is always a good idea to click on it once (the mask itself, not the Adjustment Layer) to ensure that you are actually on the right layer and on the mask itself. This will get you into a good habit that will be beneficial to you when you begin to work with multiple Adjustment Layers and multiple images in one document.

If you accidentally double-click on the mask rather than single click, it will bring up the Layer Mask Display options box. Just click OK for now. No harm done.

6-Masks View

7-Masks View

8-Masks View

9A-Masks View

Once you click on your mask, you are able to modify it in any way that you would a grayscale image. This means you can lighten, darken, increase contrast, use the Clone Stamp tool, blur, sharpen, or apply any other number of filters to it. At the moment, however, you can’t really see the mask. This doesn’t mean you can’t affect it; you just can’t see what you are doing. There will be many times when you want to affect the mask without looking at it. One example would be when you have created an Adjustment Layer with a mask, and the new adjustment is adversely affecting the surrounding areas. By working on the mask but looking at your image, you can watch how your edits are affecting the mask. Of course, there are those times that you will want to look at the mask directly.

There are two ways you can view a mask:
1.  Press the Option key (Alt for PC), and click on the mask itself. This will overlay the mask in black and white on your image. The images to the left show the Normal view and the image after Option (Alt) clicking on the Mask view. To return to Normal view, just press the Option (Alt) key and click on the mask again.

2.  Press the backslash key on your keyboard. The backslash key is just to the left of the bracket [ ] keys. This will show the mask as a semitransparent red overlay on your image.  The color and the opacity of this overlay can be changed to suit your needs. Double-click on the mask to bring up the Layer Mask Display Options dialog box. Click OK in this box when you have made the desired changes. The mask overlay will display these new settings until you return to this box to change them. Pressing the backslash key again will return your image to Normal view.

It is beneficial to know both of these options, as neither will work 100% of the time. Sometimes, you may need to see through to your image, while other times it will be easier to work in the black-and-white mode. These are the manual techniques for viewing your mask. When you begin working in the Masks Panel, these overlay modes are also available.

Click on the Masks tab to reveal the Masks Panel. When adjusting the sliders in the Masks Panel, you should be looking directly at your image (usually at 100% magnification) rather than at the small icon of the mask in the Adjustment Layer. This allows you to see in real time the changes you are making to the mask.  In the image to the left I have made a mask of the sky and darkened it using Curves.  Notice the artifacts (circled in red) around the edge of the Washington Monument.  This can be easily fixed using the Masks Panel.

The first slider you will see is the Density slider. It is set to 100% by default. This means the mask is at full density. Blacks are black, whites are white. If you reduce this slider, you will be lightening the blacks and grays on the mask. Remember, the blacks of the mask are blocking the change occurring from that Adjustment Layer. The grays are somewhat blocking the change. The whites allow it through fully. If you lower the density of the mask, the blacks and grays are getting lighter, thus allowing more of that change through to your image.

The next slider down is the Feather slider. It works just like the Feather slider in the Refine Edge tool for selections. The Feather slider “blurs” the edge of the mask. This creates a transition zone (from black to gray to white), from the adjustment being fully on to fully off.  The Feather slider will affect smaller resolution images more drastically than larger resolution images. The image to the left shows that by simply adjusting the feather I am creating a Halo around the monument.  I will need to click on Mask Edge to get to the full range of adjustments.

The next section of the Masks Panel is the Refine area. Here you will see the buttons for Mask Edge, Color Range, and Invert. The Mask Edge button brings up the very same control panel that you get with the Refine Edge tool for selections (pictured at left).

Here it works on the mask rather than a selection. As mentioned earlier, we find that it is often easier to refine the mask after the fact instead of trying to refine the selection before hand. The reason is that you are refining the mask with the current adjustment applied, allowing you see your image while you work.

Using the Refine Mask Box

10-Masks View

1. Here I have clicked on the Mask Edge Button and the Refine Mask box pops up.

2. Next I chose the On Layers View (circled in red)







11-Masks View

3. Increase the radius until most of the artifact disappears (here I have set 9.5). Remember this increasing the area around the original edge that will be affected by the sliders below.

4. The problem with this mask is that it is just a bit too big. The curves adjustment is darkening down the sky, and it is edging into the monument.  Shifting the edge of the mask will eliminate the dark halo.  Here I have shifted the edge +7.





12-Masks View

13-Masks View

5. The upper image to the left shows how by increasing the radius and Shifting the edge I have removed the halo from around the monument.

6. By checking the Show Original Box (circled in red) you see the original image before the mask refinement (lower image).

7. When you are satisfied with your refinements, click OK to apply your changes.


You would follow the same steps to apply any of the other commands such as Smooth, or Contrast within this dialog box.




































Sometimes you may find that the whole edge, however, does not benefit from the same amount of Shifting adjustment. This could be fixed manually afterwards, by going in and painting on the mask.

On occasion, you can create a mask that has shades of gray as well as white and black. This is not uncommon when using Select > Color Range. In cases like these, you may want to subtly alter the tones in the mask.

You can adjust a mask with any adjustment (Curves, Levels, and so on) that work on brightness or contrast. Color adjustments will be grayed out when you are on a mask.

To alter the contrast of a mask (remember to click once on your mask first), choose Image > Adjustments > Curves-Do not create another adjustment layer. Here you are working on the mask itself so go up to the menu and choose Image > Adjustments > Curves. You could also use Levels. The adjustment will be reflected on your mask as you adjust. Remember, white allows your adjustment to be visible, and black restricts it. So as you increase the contrast of a mask, you are simultaneously letting more and less of the adjustment through in different areas of the image.

You can also combine the selections with masks. Let’s say that you wanted to blur a section of the mask rather than the entire thing.

With your mask active, draw a rough selection with the Lasso tool.

You need to blur the selection to ensure a good blur on the mask, so click the Refine Edge button in the Options Bar and feather the edge. Click OK.

To blur the mask, you would think you could just use the Feather slider in the Masks Panel. Not so. For some reason, the panel ignores the selection. So we will use a trusted old technique. With the selection active (and your desired mask active), select Filter > Blur > Gaussian Blur, and adjust the radius to suit your needs. Remember to go to Select > Deselect when you are finished!




Want more from Tim Cooper?

Click here to check out 
the courses he will be teaching in 2015!