What will we cover in this tutorial?
We will continue the work of this tutorial (Create a Moving Photo Slideshow with Weighted Transitions in OpenCV). The challenge is that construction is that we pre-load all the photos we need. The reason for that, is that loading the photos in each iteration would affect the performance of the slideshows.
The solution we present in this tutorial is to load photos in a background thread. This is not straightforward as we need to ensure the communication between the main thread and the background photo loading thread is done correctly.
The result will be similar.
Already done so far and the challenge
In the previous tutorial we made great progress, creating a nice slideshow. The challenge was the long pre-loading time of the photos.
If we did not pre-load the photos, then we would need to load the photos in each iteration. Say, in the beginning of the loop, we would need to load the next photo. This would require disk access, which is quite slow. As the frame is updated quite often, this loading time will not let the new position of the photo to be updated for a fraction of a second (or more, depending the photo size and the speed of the processor). This will make the movement of the photo lacking and not run smoothly.
Said differently, in one thread, the processor can only make one thing at the time. When you tell the processor to load a photo, it will stop all other work in this program, and do that first, before updating the frame. As this can be a big task, it will take long time. As the program needs to update the frame continuously, to make the photo move, then this will be visible no matter when you tell the processor to load the image.
So how can we deal with this?
Using another thread to load the image. Having multiple threads will make it possible to do more than one thing at the time. If you have two threads, you can do two things at the same time.
Introducing threading
A Python program is by default run in one thread. Hence, it can only do one thing at the time. If you need to do more than one thing at the time, you need to use threading.
Now this sound simple, but it introduces new problems.
When working with threading a lock is a good tool to know. Basically, a lock is similar to a lock. You can enter and lock the door after you, such that no one else can enter. When you are done, you can unlock the door and leave. Then someone else can enter.
This is the same principle with a lock with threading. You can take a lock, and ensure you only enter the code after the lock, if no-one else (another thread) is using the lock. Then when you are done, you release the lock.
We need a stack of photos that can load new photos when needed.
class ImageStack:
def __init__(self, filenames, size=3):
if size > len(filenames):
raise Exception("Not enough file names")
self.size = size
self.filenames = filenames
self.stack = []
while len(self.stack) < self.size:
filename = self.filenames[random.randrange(0, len(self.filenames))]
if any(item[0] == filename for item in self.stack):
continue
self.stack.append((filename, Image(filename)))
# Lock used for accessing the stack
self.stack_lock = threading.Lock()
self.add_image_lock = threading.Lock()
def get_image(self):
self.stack_lock.acquire()
filename, img = self.stack.pop()
print(f"Get image {filename} (stack size:{len(self.stack)})")
self.stack_lock.release()
return img
def add_image(self):
self.add_image_lock.acquire()
filename = self.filenames[random.randrange(0, len(self.filenames))]
self.stack_lock.acquire()
while any(item[0] == filename for item in self.stack):
filename = self.filenames[random.randrange(0, len(self.filenames))]
self.stack_lock.release()
img = Image(filename)
self.stack_lock.acquire()
self.stack.append((filename, img))
print(f"Add image {filename} (stack size: {len(self.stack)})")
self.stack_lock.release()
self.add_image_lock.release()
The above is an image stack which has two locks. One for accessing the stack and one for adding images.
The lock for stack is to ensure that only one thread is accessing the stack. Hence if we have the code.
stack = ImageStack(filenames)
load_next_image_process = threading.Thread(target=buffer.add_image)
stack.get_image()
The code above will create an ImageStack (notice that filenames is not defined here). Then on the second line it will start a new process to add a new image. After that it will try to get an image. But here the lock comes into the picture. If the thread with add_image has acquired the stack lock, then get_image call cannot start (it will be waiting in the first line to acquire stack lock).
There are more possible situations where the lock hits in. If the 3rd line with stack.get_image acquires the stack lock before that the call to add_image reaches the lock, then add_image needs to wait until the lock is released by the stack.get_image call.
Threading is a lot of fun but you need to understand how locks work and how to avoid deadlocks.
Full code
Below you will find the full code using a threading approach to load photos in the background.
import cv2
import glob
import os
import random
import threading
class Image:
def __init__(self, filename, time=500, size=500):
self.filename = filename
self.size = size
self.time = time
self.shifted = 0.0
img = cv2.imread(filename)
height, width, _ = img.shape
if width < height:
self.height = int(height*size/width)
self.width = size
self.img = cv2.resize(img, (self.width, self.height))
self.shift = self.height - size
self.shift_height = True
else:
self.width = int(width*size/height)
self.height = size
self.shift = self.width - size
self.img = cv2.resize(img, (self.width, self.height))
self.shift_height = False
self.delta_shift = self.shift/self.time
self.reset()
def reset(self):
if random.randint(0, 1) == 0:
self.shifted = 0.0
self.delta_shift = abs(self.delta_shift)
else:
self.shifted = self.shift
self.delta_shift = -abs(self.delta_shift)
def get_frame(self):
if self.shift_height:
roi = self.img[int(self.shifted):int(self.shifted) + self.size, :, :]
else:
roi = self.img[:, int(self.shifted):int(self.shifted) + self.size, :]
self.shifted += self.delta_shift
if self.shifted > self.shift:
self.shifted = self.shift
if self.shifted < 0:
self.shifted = 0
return roi
class ImageStack:
def __init__(self, filenames, size=3):
if size > len(filenames):
raise Exception("Not enough file names")
self.size = size
self.filenames = filenames
self.stack = []
while len(self.stack) < self.size:
filename = self.filenames[random.randrange(0, len(self.filenames))]
if any(item[0] == filename for item in self.stack):
continue
self.stack.append((filename, Image(filename)))
# Lock used for accessing the stack
self.stack_lock = threading.Lock()
self.add_image_lock = threading.Lock()
def get_image(self):
self.stack_lock.acquire()
filename, img = self.stack.pop()
print(f"Get image {filename} (stack size:{len(self.stack)})")
self.stack_lock.release()
return img
def add_image(self):
self.add_image_lock.acquire()
filename = self.filenames[random.randrange(0, len(self.filenames))]
self.stack_lock.acquire()
while any(item[0] == filename for item in self.stack):
filename = self.filenames[random.randrange(0, len(self.filenames))]
self.stack_lock.release()
img = Image(filename)
self.stack_lock.acquire()
self.stack.append((filename, img))
print(f"Add image {filename} (stack size: {len(self.stack)})")
self.stack_lock.release()
self.add_image_lock.release()
def process():
path = "pics"
filenames = glob.glob(os.path.join(path, "*"))
buffer = ImageStack(filenames)
prev_image = buffer.get_image()
buffer.add_image()
current_image = buffer.get_image()
buffer.add_image()
while True:
for i in range(100):
alpha = i/100
beta = 1.0 - alpha
dst = cv2.addWeighted(current_image.get_frame(), alpha, prev_image.get_frame(), beta, 0.0)
cv2.imshow("Slide", dst)
if cv2.waitKey(1) == ord('q'):
return
for _ in range(300):
cv2.imshow("Slide", current_image.get_frame())
if cv2.waitKey(1) == ord('q'):
return
prev_image = current_image
current_image = buffer.get_image()
load_next_image_process = threading.Thread(target=buffer.add_image)
load_next_image_process.start()
process()
