What will we cover in this tutorial?
How to convert the foreground characters of a live webcam feed to become cartoons, while keeping the background as it is.
In this tutorial we will show how this can be done using OpenCV and Python in a few lines of code. The result can be seen in the YouTube video below.
Step 1: Find the moving parts
The big challenge is to identify what is the background and what is the foreground.
This can be done in various ways, but we want to keep it quite accurate and not just identifying boxes around moving objects. We actually want to have the contour of the objects and fill them all out.
While this sounds easy, it is a bit challenging. Still, we will try to do it as simple as possible.
First step is to keep the last frame and subtract it from the current frame. This will give all the moving parts. This should be done on a gray scale image.
import cv2
import numpy as np
# Setup camera
cap = cv2.VideoCapture(0)
# Set a smaller resolution
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
# Just a dummy frame, will be overwritten
last_foreground = np.zeros((480, 640), dtype='uint8')
while True:
# Capture frame-by-frame
_, frame = cap.read()
# Only needed if you webcam does not support 640x480
frame = cv2.resize(frame, (640, 480))
# Flip it to mirror you
frame = cv2.flip(frame, 1)
# Convert to gray scale
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# Keep the foreground
foreground = gray
# Take the absolute difference
abs_diff = cv2.absdiff(foreground, last_foreground)
# Update the last foreground image
last_foreground = foreground
cv2.imshow('WebCam (Mask)', abs_diff)
cv2.imshow('WebCam (frame)', frame)
if cv2.waitKey(1) == ord('q'):
break
# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()
This results in the following output with a gray scale contour of the moving part of the image. If you need help installing OpenCV read this tutorial.
Step 2: Using a threshold
To make the contour more visible you can use a threshold (cv2.threshold(…)).
import cv2
import numpy as np
# Setup camera
cap = cv2.VideoCapture(0)
# Set a smaller resolution
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
# Just a dummy frame, will be overwritten
last_foreground = np.zeros((480, 640), dtype='uint8')
while True:
# Capture frame-by-frame
_, frame = cap.read()
# Only needed if you webcam does not support 640x480
frame = cv2.resize(frame, (640, 480))
# Flip it to mirror you
frame = cv2.flip(frame, 1)
# Convert to gray scale
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# Keep the foreground
foreground = gray
# Take the absolute difference
abs_diff = cv2.absdiff(foreground, last_foreground)
# Update the last foreground image
last_foreground = foreground
_, mask = cv2.threshold(abs_diff, 20, 255, cv2.THRESH_BINARY)
cv2.imshow('WebCam (Mask)', mask)
cv2.imshow('WebCam (frame)', frame)
if cv2.waitKey(1) == ord('q'):
break
# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()
Resulting in this output
Using the threshold makes the image black and white. This helps it to become easier to detect the moving parts.
Step 3: Fill out the enclosed contours
To fill out the enclosed contours you can use morphologyEx. Also, we have used dilate to make the lines more thick and enclose the part better.
import cv2
import numpy as np
# Setup camera
cap = cv2.VideoCapture(0)
# Set a smaller resolution
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
# Just a dummy frame, will be overwritten
last_foreground = np.zeros((480, 640), dtype='uint8')
while True:
# Capture frame-by-frame
_, frame = cap.read()
# Only needed if you webcam does not support 640x480
frame = cv2.resize(frame, (640, 480))
# Flip it to mirror you
frame = cv2.flip(frame, 1)
# Convert to gray scale
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# Keep the foreground
foreground = gray
# Take the absolute difference
abs_diff = cv2.absdiff(foreground, last_foreground)
# Update the last foreground image
last_foreground = foreground
_, mask = cv2.threshold(abs_diff, 20, 255, cv2.THRESH_BINARY)
mask = cv2.dilate(mask, None, iterations=3)
se = np.ones((85, 85), dtype='uint8')
mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, se)
cv2.imshow('WebCam (Mask)', mask)
cv2.imshow('WebCam (frame)', frame)
if cv2.waitKey(1) == ord('q'):
break
# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()
Resulting in the following output.
Step 4: Creating cartoon effect and mask it into the foreground
The final step is to create a cartoon version of the frame (cv2.stylization()).
frame_effect = cv2.stylization(frame, sigma_s=150, sigma_r=0.25)
And mask it out out with the foreground mask. This will result in the following code.
import cv2
import numpy as np
# Setup camera
cap = cv2.VideoCapture(0)
# Set a smaller resolution
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
# Just a dummy frame, will be overwritten
last_foreground = np.zeros((480, 640), dtype='uint8')
while True:
# Capture frame-by-frame
_, frame = cap.read()
# Only needed if you webcam does not support 640x480
frame = cv2.resize(frame, (640, 480))
# Flip it to mirror you
frame = cv2.flip(frame, 1)
# Convert to gray scale
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# Keep the foreground
foreground = gray
# Take the absolute difference
abs_diff = cv2.absdiff(foreground, last_foreground)
# Update the last foreground image
last_foreground = foreground
_, mask = cv2.threshold(abs_diff, 20, 255, cv2.THRESH_BINARY)
mask = cv2.dilate(mask, None, iterations=3)
se = np.ones((85, 85), dtype='uint8')
mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, se)
frame_effect = cv2.stylization(frame, sigma_s=150, sigma_r=0.25)
idx = (mask > 1)
frame[idx] = frame_effect[idx]
# cv2.imshow('WebCam (Mask)', mask)
cv2.imshow('WebCam (frame)', frame)
if cv2.waitKey(1) == ord('q'):
break
# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()
Step 5: Try it in real life
I must say the cartoon effect is heavy (is slow). But other than that, it works fine.