Creating a Supercut Generator for the ACMI video collection
One of the most popular interactives in ACMI’s Story of the Moving Image is the Edit Line. It enables visitors to create a supercut — which is a montage of short clips — from iconic film and tv show moments. Visitors move blocks around to compile their own scene of clips. Our inspiration was fridge magnet poetry.
For a long time we wondered how we could extend that experience to make use of the ACMI video collection, and our recent work on auto-generating video image frame captions have provided us the tools to do this. So, what would it look like to deliver search results as a cut sequence?
The prototype
Using our video search code as a base, we started exploring existing video editing tools in Python to automate the process.
New York-based artist Sam Lavigne’s, Videogrep is an excellent supercut generator, but requires video files to be available locally. Seeing all of our collection videos are securely stored on cloud services in the cloud, we started building using MoviePy.
Get Simon Loffler’s stories in your inbox
Join Medium for free to get updates from this writer.
Video editing takes a long time, so our first task was to add an event stream API so we could update visitors on how progress is going:
@application.route('/supercut_progress/<task_id>')
def supercut_progress(task_id):
"""
Stream progress updates for supercut generation via EventSource.
"""
def generate():
while True:
with lock:
task = supercut_tasks.get(task_id)
if task:
if task['status'] == 'in_progress':
yield f"data: {task['progress']}\n\n"
elif task['status'] == 'saving':
yield f"data: {task['status']}\n\n"
elif task['status'] == 'completed':
if task['filename']:
yield f"data: completed {task['filename']}\n\n"
else:
yield "data: no_clips\n\n"
break
time.sleep(1) # Poll every second
return Response(generate(), mimetype='text/event-stream')Now we’re ready to hand our video search results (video URL and JSON captions) to MoviePy to cut up the original videos and splice them back together. The steps are:
- Load the videos
- Cut the sections that have the search term(s) with 0.5 seconds padding either side
- Resize the clips to the output resolution of 720p
- Add a background colour if the clip is in a different ratio to the output
- Update the frontend that we’ve completed a video and move onto the next one
- Join the clips together to make one video
- Create a poster image for the video
Here’s what that looks like in Python:
def generate_supercut_background(query, search_results, task_id):
"""
Run supercut generation in a background thread and update task status.
"""
filename = f'supercut_{slugify(query)}.mp4'
output_path = f'app/static/videos/{filename}'
total_clips = sum(
1 for result in search_results['hits']['hits']
for segment in result['_source']['transcription']['segments']
if query.lower() in segment['text'].lower()
)
total_clips += 1 # add resize
clips = []
processed_clips = 0
for result in search_results['hits']['hits']:
video_path = result['_source']['web_resource']
for segment in result['_source']['transcription']['segments']:
if query.lower() in segment['text'].lower():
video = VideoFileClip(video_path)
# Extend clip by 0.5 s on either side, but keep within the video’s bounds
start_time = max(float(segment['start']) - 0.5, 0)
end_time = min(float(segment['end']) + 0.5, video.duration)
try:
clip = video.subclipped(start_time, end_time)
clips.append(clip)
except ValueError:
pass
processed_clips += 1
progress = (processed_clips / total_clips) * 100 if total_clips > 0 else 100
with lock:
supercut_tasks[task_id]['progress'] = progress
if clips:
target_resolution = (1280, 720)
resized_clips = []
for clip in clips:
# Resize to fit within target_resolution, preserving aspect ratio
resized_clip = clip.resized(height=target_resolution[1]) # Scale based on height
if resized_clip.w > target_resolution[0]: # If too wide, scale based on width instead
resized_clip = clip.resized(width=target_resolution[0])
# Create a colored background clip
background = ColorClip(size=target_resolution, color=(0, 0, 0), duration=clip.duration)
# Center the resized clip on the background
padded_clip = CompositeVideoClip([background, resized_clip.with_position('center')])
resized_clips.append(padded_clip)
processed_clips += 1
progress = (processed_clips / total_clips) * 100 if total_clips > 0 else 100
with lock:
supercut_tasks[task_id]['status'] = 'saving'
supercut = concatenate_videoclips(resized_clips, method='compose')
Path('app/static/videos').mkdir(exist_ok=True)
supercut.write_videofile(
output_path,
codec='libx264',
audio_codec='aac',
preset='veryfast',
bitrate='2000k',
threads=multiprocessing.cpu_count(),
)
frame = supercut.get_frame(1.0)
frame_image = Image.fromarray(np.uint8(frame))
if frame_image.mode == 'RGBA':
frame_image = frame_image.convert('RGB')
frame_image.save(output_path.replace('.mp4', '.jpg'), 'JPEG')
for clip in clips:
clip.close()
supercut.close()
with lock:
supercut_tasks[task_id]['status'] = 'completed'
supercut_tasks[task_id]['filename'] = filename
else:
with lock:
supercut_tasks[task_id]['status'] = 'completed'
supercut_tasks[task_id]['filename'] = NoneDemo to teams
We presented the demo to teams that we thought might find this tool useful for their work — video teams, website editorial and collections. Their feedback was positive and that this is definitely something they would use for research and to create supercuts for different public outputs.
Some of the feedback and features they thought would make it more usable were:
- like the Edit Line interactive experience in the museum, the ability to choose multiple words to create one supercut.
- have a selection tool before the supercut is generator that allows the user to choose the snippets and the videos — maybe using a simple checkbox.
- ability to download the metadata of the videos in the supercut.
Next steps
The supercuts are cached after they’ve been generated so subsequent requests for the same search terms are returned instantly, but the waiting time while the original request is generated on CPUs is very slow. So the next obvious step is to run the generation thread on cloud or locally hosted GPUs.
We’ll also start work on the other items teams raised at the team demo.
Something else we always have in the back of our minds is making this a public experience, online, in the museum or both. As we improve the tool and iron out any issues we’ll get closer to understanding what this could be and how it could further open up the collection for our visitors.
Watch the code evolve on GitHub: github.com/ACMILabs/video-search
