Introduction To Visual Storytelling

In our last issue we used an AI video model to convert this first-frame:

Into this life-like video shot:

Now that you know how easy it is to bring any picture to life, it’s time for us to start exploring together a vast and fascinating topic that’s at the very core of making movies: Visual Storytelling.

First, let’s differentiate between Storytelling and Visual Storytelling.

Storytelling, as you might have guessed, is the art –and the science– of telling compelling stories.

In this newsletter we’ll be talking a lot about storytelling, since every good movie starts with a great story.

In future issues I’ll introduce you to screenwriting, and the different frameworks screenwriters and fiction writers use to “construct” great, unforgettable stories, beat-by-beat.

Visual Storytelling, on the other hand, is the art of telling stories with visuals, this is, with pictures and video shots.

Visual Storytelling is a big subject that we’ll be chewing at one small, delicious piece at a time.

Today, we’ll start our conversation –and your personal journey as a Director-in-training– by introducing you to what I like to call The Language of Visual Storytelling.

You see, every medium and art expression has their own language, and a big part of becoming an efficient creator in that art is becoming familiar and proficient with that new language.

The language of music, for instance, is made up of notes, keys, silences and harmonies.

Once you know the “letters” and “words” of this language, you can start composing your own music.

The language of sculpture is composed of materials, textures, shapes and proportions.

The language of writing is made up of words, and meaning.

The language of Visual Storytelling is made up of visual cues, such as frame composition, framing, lighting, camera angles and camera movements.

In order learn and become proficient in this new language as fast as humanly possible, we’ll be focusing on the last 2 elements –camera angles and camera movements– in a very hands-on way over the next few issues of this newsletter.

For the time being, we’ll be letting AI take care of lighting and frame composition for us, simply by including some simple nods in our prompts like “cinematic composition” or “natural lighting”.

But before getting into it, I’ll propose you get yourself in the Director’s frame of mind first.

In order to do so, you need to keep this magic question in mind at all times:

“What’s the logical next thing the spectator needs to see to follow the story?”

The answer to this simple but powerful question will guide your camera!

So far, we have crafted a tiny seed of a story and, based on that, we’ve already created a first-frame, and made it into a first-shot.

Now, we need to think about how can we take this first sneak-peak into the world inhabited by our characters, and develop it into a full-fledged scene.

In cinematography, MOVIES are made up of SEQUENCES, sequences are made up of SCENES, and scenes are made up of SHOTS.

So let’s bring our seed story back, and think a bit about what SHOTS we need to tell it:

Akiko, a 20-year old piano player, in visiting her grandmother for her birthday in rural Japan, 1943.

She’s tuning a dusty old piano in her grandparents’ home.

Grandma is standing close by, observing her silently.

She just brought Akiko a cup of green tea.

The steaming, beautiful ceramic cup, is resting on a little bamboo table beside the piano.

Akiko’s semblance appears a little worried (Will she be able to repair the piano on time?)

Ok, we already have a first shot and that’s great start because this shot answers a few important questions:

How does the house look?

How does Akiko look?

How does grandma look?

What other objects are part of the scene, how do they look and where are they located?

All of this in invaluable visual information for us!

Now we need to decide how to tell this little portion of a story we call“a scene”.

In filmmaking, a “scene” is simply an event that takes place in a single location, and, typically, during a continuous period of time.

For example, a conversation in a kitchen, a fight in a park, a character walking down a hallway, are all scenes.

A scene, as stated before, is made up of several SHOTS.

Each shot will do their part to help tell the viewer the story of what’s happening.

(By the way, when I say “shots”, it’s cinematography jargon for“video clips”)

Remember the magic question:

“What does the spectator need to see NOW to understand what’s happening?”

Deciding what shots you need is an iterative process: you read the script and you ask yourself how you can show THAT to the viewer.

Let’s do the exercise together:

Akiko, a 20-year old piano player, in visiting her grandmothers for her birthday in rural Japan, 1940’s.

This is mostly information about the main character, and the location where this scene is taking place.

Ask yourself: is there something here that I can show visually?

Right!

You can certainly show the location in Japan’s country side, maybe even an old village with structures that hint at the time period –1940s– and, of course, the exterior of the house the scene will be taking place in.

These kinds of shots are called “establishing shots” and they are commonly used at the beginning of each scene to ground the viewer in the location where things are happening.

So let’s take mental note that we need an establishing shot to start the scene.

Better yet, write it down in your “shot list”.

Let’s continue:

She’s tuning a dusty old piano in her grandparents’ home.

Remember the magic question: “What the viewer needs to see?”

We should show her doing some work on the piano, maybe trying a few notes, and adjusting the tuning pins.

This could easily be 3 or 4 shots.

First, we need a wide shot showing her doing these activities, kind of an establishing shot for the action, if you will.

But we also will need a couple close-ups:

One close-up showing her hands while playing the notes, and another one showing her hands in the act of adjusting the tuning pins inside the piano.

These two extra shots will greatly help to make what's going on clearer to the viewer

So let’s add those to the shot list too…

Akiko’s semblance appears a little worried (Will she be able to repair the piano on time?)

This calls for a close-up shot, showing the frustration on her face.

Added to the shot list…

Grandma is standing close by, observing her silently.

We need a shot for that, focusing on grandma.

Or maybe we can use that wide shot we already have showing the entire scene…

We'll decide later… (add a note about it to your shot list)

She just brought Akiko a cup of green tea.

We could certainly show this action.

Or we can start with the shot above, where grandma is standing close to Akiko and the teacup is already set in place.

Decisions, decisions!

(Welcome to the world of a Director!)

For now, let’s simply make a note of it and continue on…

The steaming, beautiful ceramic cup, is resting on a little bamboo table beside the piano.

That sounds like something that is screaming for a extreme close-up or insert shot!

(Don’t worry, we’ll cover those really soon!)

But we need to keep in mind at all times that we don’t just randomly take shots of things that look interesting…

As directors, we only take shots of the things that help the story move forward!

Remember:

Only show the spectator what they need to know at this moment in time.

Showing things that are not crucial to move the story forward will only confuse the audience and take away from the scene’s emotional impact!

Showing anything that they don’t need to know yet will likely break the flow of story.

Consequently, we’ll need to decide if showing the teacup adds to the story in any significant way.

The rule is simple: whatever doesn’t add to the story must be cut out.

If we decide it does add to the story –for instance, because showing the love of grandma is crucial to the story being told… which in this case, I think it is– then it’d make sense to also include the shot of grandma bringing the cup and delicately placing it on the bamboo table close to Akiko.

We may even decide that this element is so important for establishing the loving connection between the characters in the scene that an extra scene showing grandma in the process of carefully preparing and pouring this tea is called for.

Storytelling is, and you can see, an iterative process.

This is even more the case in AI Filmmaking, where we have the outstanding ability to try different settings and ideas along the way, something that might be too cost-prohibitive to try with a traditional film crew.

Also, in case you’re thinking: “Well, if there was a scene showing grandma preparing the tea, it would already be in the script”…

This is not necessarily true.

Even in traditional movie productions, it’s not uncommon for the Director, once on location, to decide to add extra “coverage” or shots if they feel it will add to the visual storytelling of a scene.

As a matter of fact, in movie production “scripts” are fluid and are being rewritten all the time!

Not only the Director, but actors will sometimes improvise and come up with new lines, or non-verbal cues that weren’t in the original script but which they feel convey the story or the character better.

Some of these changes will inevitably end up in the final film!

As the old adage goes in Hollywood, a movie is written at least 3 times:

The first time, the movie is written on paper by the Screenwriter.

The second time, the movie is re-written with light by the Director (aided by the Director of Photography, the actors, and the entire film crew)

And the third and final time, the movie is re-written in the edit bay by the Editor to produce the version you and I will see: the Final Cut.

Of course, when it comes to AI Filmmakers and, especially, to those 1-person movie studios just made possible by AI, this iterative creative process is even more crucial!

So, if you allow me, this is your homework:

Go over your own script just like I did above, make the decisions you need to make along the way –don’t worry: you can and will change some of them later– and prepare your own shot list.

We’ll meet again next Friday to go over our shot lists, and start creating some of these shots while we learn how to convey emotion with camera angles and camera movements!

See you then!

Leonardo

P. S. As a side note, I had the chance to participate in Runway’s Gen:48 AI Filmmaking competition over the weekend.

It was quite tough to go from idea to short film in just 48 hours while using exclusively Runway’s video tools, but it was a great experience.

I almost didn't submit my film because there was a huge blackout in Europe affecting several countries, including Spain where I live.

Fortunately, my UPS’s battery afforded me a few extra minutes to upload the film and submit it to Runway before completely dying on me.

I couldn’t even review the video to see if it had been uploaded correctly, but I did so this morning when the power finally came back and, thank God, it was fine!

(Incidentally, as electricity just came back a few hours ago, this is the reason why I’m sending you this newsletter a little late today. Sorry for that!)

You can watch it here:

Oh, and please don’t forget to share Flawlessly Human with your friends and colleagues!

That would mean the world to me!