Adobe After Effects is Adobe Creative Suite’s program dedicated to visual effects and motion graphics. Its extensive features and compatibility alongside the Adobe Creative Suite has established it as an industry-grade program, being used by industry professionals–both by in-studio artists and freelance artists.
How do you rotoscope in Adobe After Effects? There are two popular methods to do this: You can create masks through the pen tool, or use the rotobrush. Each method is different, but aims to provide the same result of separating your desired elements in the video.
In this article, you will learn how to rotoscope in Adobe After Effects in whichever method you choose, as well as some general rotoscoping tips you could apply whichever method or program you choose.
What Is Rotoscoping?
Rotoscoping is an animation technique used in the earliest days of 2D cel animation, where animators would trace and draw over live action footage in order to mimic natural, realistic movements.
This practice has soon extended to visual effects production, although with a slightly different purpose. The footage is still traced over frame by frame, but instead of painting over them to produce 2D animation, rotoscoping separates different elements within the shot, cutting them off from each other.
This separation of different elements allows for more manipulation of the shot–the rotoscoped elements could be moved from one place to another, transferred into another shot, or manipulated in many other different ways as one pleases.
Rotoscoping in visual effects is not the effect itself, but could be seen as a preparation for the different effects that could be done on the footage.
To go more in-depth about the topic of Rotoscoping, check out my extended article about it:
Adobe After Effects Rotoscoping Methods
There are two basic ways on how to rotoscope your footage in Adobe After Effects–using pen tool masks or using the rotobrush, with each having their own unique points, strengths, and weaknesses. Assessment of your footage is essential in order to decide which method will work best with the footage you have. Here is a quick explanation on how to rotoscope with each method.
Pen Tool Masks
Using the pen tool is one of the earliest rotoscoping methods for modern visual effects. It allows you to form detailed lines and curves that you could form into shapes, but could take a bit of a learning curve, especially if you are new to it. It could also be quite time-consuming.
In order to rotoscope using the pen tool you will need to first select your layer, then select the pen tool which could be accessed either by clicking it on the toolbar or using the shortcut G.
Start by selecting your keyframes, and trace over your element, breaking it down into smaller, simple shapes if needed. Click the stopwatch on the mask layer in order to mark it.
After doing your keyframes, go through your footage frame by frame which could be done by using the Page Up and Page Down keys on your keyboard to go one frame forward and one frame backward, respectively.
You have successfully created your mask! Now go on and do whatever you want with it.
Here is a great example of using the pen tool, especially because he explains how to separate different parts from each other instead of rotoing the whole object with just one path.
Pen tool masks, although allowing for extensive detail, could be quite frustrating and mundane, as it will take a lot of time, especially with longer pieces of footage. However, it allows you to rotoscope basically anything in the footage without relying on the software’s AI capabilities, which could not be perfect all the time.
When working with footage which does not have a great degree of contrast or definition between objects, the pen tool mask could be a good option, just make sure to simplify your workflow by making use of multiple shapes and controlling them as groups instead of individual points.
Rotobrush
Rotobrush is an Adobe After Effects tool which was made specifically for rotoscoping, making the process a lot quicker and easier than rotoscoping with the pen tool.
In order to rotoscope with the rotobrush, select it from the toolbar, and then select your layer.
Just like rotoscoping with the pen tool, start off with your keyframes. When using the rotobrush, you do not need to be as exact as you needed to be when using the pen tool.
It works a bit like the Magic Wand tool on Adobe Photoshop. You just need to paint over the elements that you want to include in your mask. It will show up as a green brush stroke on screen. If you want to remove something from the mask, hold the Alt key on your keyboard while painting over it, showing a red brush stroke on screen. Your mask will show up as a purple outline on your footage.
Once done with the keyframes, you can check the rest of the frames, but rotobrush usually already adjusts the mask by itself, making use of artificial intelligence (AI). There may be some small inconsistencies that you could adjust yourself.
After checking all your frames, you are done, and you could now do anything with your mask!
The rotobrush is a very convenient tool, and is often preferred by many users due to its simplicity and effectiveness. Although the rotobrush works well in well lit footage with some good contrast, it doesn’t perform as well in footage where the elements aren’t as well defined and contrasted. In these situations, you would be better off using the pen tool to create your masks.
I’ve quite liked this tutorial here. Its also nicely entertaining.
Rotoscoping General Tips
Before diving into the specific methods you could use when rotoscoping your footage in Adobe After Effects, here are some general tips to keep in mind whichever method you may choose. These tips, although not absolutely required, will help you to work more efficiently and get the most out of the program and your time.
Study The Shot
This step applies to most, if not all parts of visual effects and creative work. In order to effectively manipulate your footage, you must first watch it and assess the different elements you need to work with.
- Which elements do you need to rotoscope?
- Is it clearly defined in the shot?
- Is there a lot of movement?
- How long is the shot and how many frames do you need to work on?
Start With Keyframes
Any moving image is made up of frames–still images that when put together give the illusion of movement. Keyframes are simply frames where the subject holds the most significant or complex positions within the video. These frames are where the most work needs to be done.
It makes sense to start here since these are usually the frames that audiences pay attention to, often containing significant moments within the entire project. Having keyframes set up also gives you a point of reference when working on the rest of the footage. When rotoscoping, it is advised to start by identifying the keyframes, working on them, and then going through the other frames and adjusting as needed.
Use as Few Points As Possible
Especially when working with masks and using the pen tool, use as few points as possible. This method, although allowing you more control and finer points, takes more time to accomplish, and is also more prone to jitters and inconsistencies, as there is a lot you need to work on.
Each point can be controlled individually, and for complex shapes there could be a lot of them. This, compounded by hundreds, or even thousands of frames, could be overwhelming and very difficult to consistently control, not to mention the amount of time you will need to accomplish all of it, should you choose to do this.
Make Use Of Multiple Shapes
Sometimes the elements you need to rotoscope will be complex shapes doing complex movements, and trying to fit all of it into one shape and controlling it across several hundred frames could be difficult.
This could be made simpler by breaking down your element into smaller, simpler shapes that you could manipulate as a group. For example, if you are rotoscoping a person, you could break them down into specific body parts–separately rotoscoping their legs, thighs, feet, etc.
This allows for easier organization and troubleshooting mistakes, especially when working on a larger project. You can then look at the footage as different parts or segments, instead of an entire shape with hundreds of points which can be quite overwhelming.
For any more great tutorials you should also check out Adobe’s own help content on their site.
Conclusion
Rotoscoping, although quite mundane and repetitive, is an essential step in the visual effects process. It is both straightforward and complex, and opens up entire worlds of opportunity for your creativity to run free. It definitely is an essential skill to learn, especially for people who wish to pursue a career in visual effects. Congratulations on taking the first steps to learn how to do it in Adobe After Effects, and good luck on your future projects!