This is a documentary of my family, it's past and my grandmother's storytelling.
What will play is one of 1,286,384,503 different permutations through the 5 hours of footage gathered to create this video.
My grandmother will explain it best, but I will say it in my own words here: my family's history isn't a straight line, it's a complicated web with different folds feeling closer to the tip of our tongues each time we go to feel them.
This is an attempt to archive my family's history through my grandmother's weaving.
Verna Gray is a documentary of my maternal family history and of my grandmother's storytelling. Other family members have attempted to our history but all of those versions were unmanageable given the concurrency of events.
My grandmother is a wrangler of narrative threads. Her stories weave in between decades, countries, and central figures. The way she traverses these events makes it evident that no codex could tie together the tapestry she has committed to memory. This project builds on the structure-as-tangent principles I used in my prior project: Release.
The structure, which will I'll talk about in greater depth in the technical section, takes inspiration from Italo Calvino's Invisible Cities. Instead of escaping linearity through city names, the videos transition based off of their semantic material (akin to Sam Barlow's Immortality). People, places, and decades are the stepping stones I use to skip around the video.
In the fall of 2021, I spent fours days with my grandmother recording stories about the family. The end result was eight hours of video that darted between people, places, decades, and tense. I felt the best way archive the wide breadth of stories she brought to life in our conversations was to organize it as a tree of tangents.
I did a first editing pass to cut out b-roll, silent periods, and sections that were specifically requested to be removed. This resulted in about 6 hours of usable video. During that initial cut editing process, I was developing a way to show all of this footage in a tree structure. Having done Release, I know the impact that technical constraints of a projects have on it's editing process.
I initially targeted this set of constraints:
5 hours of video, even compressed for the web, is prohibitively large. For reference, the final 720p exports of the footage still takes ~27 gigabytes of storage.
Initially I thought about having one long video that would jump to the appropriate timestamp, but it caused extremely long buffering times to load in the new section of video. Media fragments exist, but can't load multiple fragments in a single video (browsers implemented this standard to just load a single fragment at a time), so there was no tangible benefit for this use-case.
Another option is having two videos which would swap with one being the actively playing video and the other being a buffer that would load the next section and then swap in once the transition took place. This also runs into the issue of controlling loading and bandwidth. Scripts can't alter media fragments because that causes the video element to reload. Between the metadata exchange and the reindexing into the file, the management of fragment boundaries seems like too much overhead for the payoff. Without some boundary for the buffering-non-active video, it can over-buffer past the footage actually needed. This would lead to large duplication of data which yet again causes bandwidth issues.
The only reasonable method seemed to be:
First was editing the video to breaking down the raw footage into clips and loosely categorizing them. I used Adobe Premier to edit the footage. Premier allows the extension of clip metadata to include tags, which I used as a way to do more fine-grain categorizing.
I used the tagging to construct a census of topics that came up through my grandmother's stories. My initial pass was to break the footage down into logic chunks (which would ultimately be the chunks used for the video switching) and to help focus my editing on a second pass when creating transitions between the clips.
While doing the coarse edit, I made a demo of the harness that would orchestrate the video switching. Ultimately I wanted the videos to be plain-jane MPEG4s so I could leverage the html5 video api instead of having to deal with some kind of library. This meant the elements would be handled by some harness driven by a configuration to specify the clips and their transitions. Release used a transpiler to generate a bootstrap function to generate the graph walk, but I wanted to fully separate code deployment from application state for Verna Gray. So I worked out a serviceable configuration language in JSON:
"clips": {
"$some_name_for_the_clip": {
"file": "$the_clips_actual_file_path.mp4",
"transitions": [
{
"name": "$a_name_for_the_transition_to_help_debugging",
"jump_from_timestamp": 290, // <- when to leave the current clip
"jump_to_clip": "$the_name_of_the_clip_to_go_to",
"jump_to_timestamp": 0 // <- What timestamp to jump into on the new clip
}
]
}
...
The initial harness used SetInterval to queue up the transitions for the videos. The idea was
when a user kicks off the initial video, the harness would randomly choose a walk through the configuration. It would
then go step by step, keeping track of the time and set intervals for a function to run with predefined arguments and
would interact with the correct video clips. The time had to be kept during this setup because each successive transition
needed to be offset by the length of the walk that lead to it. For example, if a clip was 7 seconds long but transitioned
after a 30-second clip, its interval had to be set to 37 seconds instead of 7. The plan ultimately was a nightmare to debug.
I initially went this route because I used SetInterval in Release to manage the
dyslexia effects. It was perfect in that project because none of the effects depended on the timing of any other; they could just
happen in isolation. The chaining-dependence of clips in Verna Gray found no such convenience. The interval
centric method also made it so you couldn't pause the video. The timers didn't pause when the video did, so
they would just keep firing away without warning.
I ended up switching to using .ontimeupdate instead. This allowed the videos to read
from a manifest of transition and set up just the next jump and only transition once the
current videos timestamp matched the timestamp in the config. No timing offsets, no issues pausing video, and
a much easier time debugging. It also had the benefit of being much easier in optimizing when to get rid of videos
after they played to save memory.
Once the initial clips were cut in Premier, I used "marks" to notate the timestamps of when subjects had a good point the segue from. For example a natural pause in conversation just after finishing a point about Jack Silsby has a mark to say "we can jump from here to another clip about Jack."
I didn't mark the points of entry for the clips ("we can start talking about Jack here") because the entrance to a topic might want to be different based on the end of the clip that jumps to it.
I wish I could say I made some grand program for organizing all the transitions, but I didn't. This part, to me, really is the editing process for this project and I, having never done it at this size before, decided to not spend the time working on an interface for a process I had never done. Instead, I broke out a notebook and sketched the thing out:
I've been calling this a "Linear Grid Chart". Each line is a clip, each work is a marker with the topic name (but no real timestamp data). I wanted to avoid loops and backtracking in the edit of this footage, which mean the grid had to have a set of rules for its traversal:
The direction of the traversal over this Linear Grid Chart can best be described as "downwards and mostly left."
Again, the reason for these rules was to avoid loops. There are other ways, but this meant I have practically 0 chance of creating a loop. The reason I so doggedly wanted to avoid loops was because of what video of my grandmother repeating the same story over and over would imply in the meaning of the traversal. My grandmother is cogent, and sharp. A loop over stories would insinuate she has issues with memory or time, which isn't true. So the rules are to avoid a mistake of creating a loop all together.
Following those rules, I drew out the connections:
Other than looking like little like a John Cage composition, the traces were fairly successful as giving me a map to append to the base configuration for the harness. Using the Linear Grid Chart as a map, I did a third pass through the footage to figure out the time stamp each transition point should jump into when moving to the new clip.
My hope in hosting a deploying this project was to make a deprecation resistant object. Release has required more upkeep than I would like given the libraries it originally used (which have been abandoned), the APIs I leveraged (which were experimental when I used them and have since changed), and the audio I used which has since been handicapped by Apple's mobile browser restrictions. - Thus, I prioritized using standardized and well-accepted APIs in Verna Gray. It's deployment needed to achievable with standard POSIX tools. And the server needed to be on a development cycle an order of magnitude slower than a server built with NPM in mind.
My hope was to figure out how to host the site as a static webpage and let HTML5 tags to the lifting for the video streaming. After a quick look, Digital Ocean ended up being the hosting platform I chose. Vultr and Linode has slightly smaller Storage options (remember I need to server 27 gbs of video) and I literally cannot figure out how to estimate costs in AWS.
For the actual webserver, I am using Apache "httpd" Server
It is fast, easy to install, doesn't require yarn or npm, and is fine for static webpages
(which the project technically falls under).
That's it really. There is no js library. No reverse proxy. Nothing fancy.
Ultimately there is an easier way to make videos like this: use a video player that can buffer in multiple places.
I don't really know how to do something like that over the web, but it certainly can be done as a local app. Trying to leverage unity for the web might be able to pull it off, given the control you have over their video player. But I really don't know yet. Getting the edit outside the Linear Grid Chart and think of transitions as skips on a chronological line would free up a lot of editing burden.
I think once I settle on an editing process, building tools to auto compile a lot of this would be both easy and a great accelerator of getting this kind of work together.
There are a lot of other kinds of stories to be told in some structure/harness like this. What about stories that loop? What about stories that let you chose which path to take? What if the transition lines are generated or randomized? What if we went full cage-ian and used the transition lines from one piece on another?
This single version of the video project has roughly 1,286,384,503 different permutations. And as time goes on, and I add more to it: that will grow. I don't know when I will tackle another video project like this, but there are plenty more experiments and avenues to explore once I do.