What my creative director does

What I find surprising, and really quite incredible, is how much my AI creative director does that I never see. Today Dray made a short film of our grumpy fox doing De Niro's "You talkin' to me?" scene from Taxi Driver, and there are so many decisions and tools in that one piece of work. It all happens very quickly and behind the scenes.

I've trained Dray to get good at creative work, but it can now do much more than I can. When I get Dray to talk me through what it did technically, I'm just so impressed by how much is in it. Dray has written up the technique in a Craft piece, and I wanted to understand how it does it, so we went back through the whole chat.

All I asked was whether the fox could copy the scene, as a homage. I said I wouldn't act it myself, that lots of people have probably done impressions of it, and that the facial expressions are important. That was at 4:31 in the afternoon. The first 10 second test was on my phone at 4:46, and the cut I approved came at 5:19.

In between, Dray searched YouTube for people doing the scene and downloaded six of them. It can't watch a video, so it laid each one out as a grid of still frames and read them like a contact sheet. Two looked good enough. Then it pulled the real scene and cropped in on the faces, one frame a second, with the best impression next to De Niro. The man doing the impression grinned the whole way through and missed the offended look, so Dray dropped him and used De Niro himself.

Kling, the video model, needs one shot with no cuts in it, or the fox jumps. So Dray used ffmpeg, a free tool, to score how much each frame changes from the one before, which finds every cut, and one shot runs for 33 seconds without one. It checked frame by frame where the gun comes out, because our fox is grumpy and never menacing. When I asked for 30 seconds without bad language, it ran the sound through whisper, which writes down every word with the second it's said, so it knew where each swear word fell before it cut anything.

Then it made the picture of the fox. It took a frame from De Niro's scene and set the fox up to match it, in the same place in the frame, at the same size, in the same pale green kitchen with cereal boxes on a shelf. It wrote the prompt for the video with the same care. The prompt names the moves that matter, the raised eyebrows, the head tilts and the turn away and back, and it says "Fox paws, never human hands", because a copy of a person's hand pulls towards fingers.

A lot of this comes from having done it before. Kling works best 10 seconds at a time, and if every piece starts from the same picture, the fox jumps at each join. Dray knew that from the Bad Times fox video, so it started each piece from the last frame of the one before, and the joins don't show. It keeps notes from every job and reads them before the next one, and it reused working code from earlier projects rather than starting again.

It also checks its own work before I see it. It lays the film out as a grid, zooms in on the paws at full size, and looks at every frame around each join. When Kling refused the last piece, it tried once more, cut shorter and well clear of the gun, and then stopped and asked me. When I asked it to take out a moment where the fox's eyes went too wide, it looked at that second ten frames at a time and put the cut where his eyes were half closed on both sides, so his head doesn't jump.

Dray made more than 20 decisions, with six tools. yt-dlp found and downloaded the clips, ffmpeg did the cutting and the checking, whisper wrote down the words, Seedream made the picture, Kling made the film, and Dray wrote small scripts of its own to run them. Most of them are free. Only Seedream and Kling cost money.

I've no idea what goes on inside the image and video generators themselves, and that's another whole bag of tricks.

What impresses me is what AI can do once it's been given knowledge and training, because it figures things out for itself and makes its own judgment calls along the way.