Water World

John R. Leonetti, ASC goes down The River with a raft of small-format cameras and innovative VFX workflows

The makers of the ABC Studios/DreamWorks’ The River set out to make something not just a little creepy but really scary. Not surprising given that one of the show’s creators is Oren Peli, writer/director of the blockbuster feature Paranormal Activity. Shot by John R. Leonetti, ASC (Insidious, The Mask) the new series has that same disconcerting feeling as the Paranormal franchise – terrifying events shown with the matter-of-fact feeling of ‘found footage.’

Shooting on location in Hawaii (and Puerto Rico for the pilot) with more than 100 VFX shots per episode and a huge assortment of cameras, sensors and formats, The River breaks new ground for an episodic drama. Rather than risk sending the massive amount of image data from set to Hollywood-based facilities for VFX and post work, the producers brought much of that work to the location. Key players from Encore and Level 3 went to Hawaii to set up a mobile “lab” for managing all the camera files, with Encore VFX supervisor Stephan Fleet, VES on the set to work closely with Leonetti and his team.

As seen in the pilot episode, beloved nature-show host Dr. Emmet Cole, who has entertained millions with the intrepid adventures of his TV series Undiscovered Countries, has vanished on a boat voyage deep in the Amazon. Months later, signals are received via an emergency homing beacon, so Cole’s wife, son and a team of documentary filmmakers set out in their own boat to find him, recording their journey as they go. The look of The River is mainly defined by found footage the rescue squad has shot, as well as material captured by surveillance cameras that documents mysterious, sometimes supernatural phenomena along the way.

Leonetti says he loves “beautiful images” as much as the next cinematographer. But this series required a different mindset, one he first learned while shooting docudramas like Against the Wall (about the Attica prison riots) and The Burning Season (about strife in Brazil) for director John Frankenheimer. “Once you get into a situation where women don’t have to look beautiful,” Leonetti muses about The River’s down-and-dirty approach, “with light slamming in their face and a big backlight on them, you’re in a whole different kind of storytelling.”

Sony EX3s and F3s, ARRI ALEXAs, Silicon Imaging SI-2K, Canon 5D Mark II and 7D DSLRs and GoPros were among the many different cameras Leonetti made use of throughout the series. Explaining why he didn’t just settle on one high-end system and simulate the different looks in post, he states, “You can spend time in post and ‘dumb down’ an Alexa to make it look more like an EX3 or even a GoPro, and we did do some of that. But each of these cameras offered something different. We used GoPros in very low light, and it looks gritty and gnarly and introduces a lot of yellow and orange noise that gives you a very uneasy feeling. The DSLR material certainly looked good enough to work as surveillance footage at some very high ISOs.”

The River’s shooting style, in which five (and often many more) cameras covered a scene at once, was cost-prohibitive using high-end professional-level gear. And there were also practical reasons why it made sense to use some of the consumer cameras.

“It’s their physical size,” Leonetti explains. “We could Velcro a couple of GoPros to the ceiling and use them as surveillance cameras. You can’t do that with a bigger camera. You can’t even do it with a 7D without seeing it. We did use 7Ds, but we had to be more careful hiding those.”

He describes one scene that was covered with two GoPros and two flashlights. “A character who is a documentary cameraman has to go find a special bulb that grows under a tree,” Leonetti explains. “But he’s afraid of tunnels and has to climb under these trees to get the bulb. We fixed two GoPros back-to-back on a plate with a handle and two flashlights, one pointing in each direction.” The DP credits his operators (Mark Meyers on A-camera and Raphy Molinary on B-camera for the pilot; Tod Campbell on A-camera and Richard Cantu, SOC on B-camera for the series) for being able to go against their natural, i.e., highly trained, instincts. “They couldn’t be graceful,” Leonetti continues. “The look had to be completely spontaneous with a lot of snap zooming with the ENG lens. Pop it in; pop it back. And that was also tough on our assistants [Scott Ronnow on A-camera and Tony Nagy on B-camera]. A lot of these cameras were on auto iris. You do a stop change, and you see the stop change in the shot, which is certainly not what they are used to.”

Working with DIT Adrian Jebef (Jeff Tomcho took over DIT duties when the show moved to Hawaii) and Level 3 Post in Burbank, Leonetti wanted to see how each different rig would stand up through the color grading process. “I worked with Larry Field, who did all the final color grading for the series,” he adds, “and we set looks for each camera.”

Jebef had his work cut out for him. “Everything shot was file-based,” he says. “I brought some tower computers to the location and used them to ingest all the material from all the different kinds of media onto hard drives in their native codec and organized everything into file folders. The Alexas recorded to ProRes Log-c. The Canon DSLR material was in H.264 on the cards that go in the camera. For the EX3, we wanted to work with a more robust codec than the XDCam format, so we recorded 4:2:2 signal directly to a [Convergent Design] nanoFlash Recorder. Level 3 had set up a mobile ‘lab’ at the location, and I would hand off all the files in their native formats to them.”

One striking motif of The River is how breathlessly it cuts, within a scene, between “documentary” footage from different shooters and any number of surveillance cameras, all of which were covering the action simultaneously. The mixture of so much data, in so many formats, coupled with some 100 VFX shots per episode created a challenging set-to-post pipeline.

“From the pilot, we discovered that it made the most sense to put everything into a single visually lossless format,” says Jay Bodnar, vice president of engineering for entertainment television at Deluxe, which includes both Encore and Level 3. “The format of choice was DNX175x. It’s a robust codec that allowed us to retain all the information from every one of the native camera formats. We immediately transcoded everything, did a color grading pass [using a scaled-down Baselite system] for dailies, and we also made DNX36 versions that were more manageable for the Avid systems in the cutting room.”

DNX175x files were forwarded to Los Angeles where they were “published” to the SAN that Encore and Level 3 share. This allowed Field to do some early grading in his Lustre suite while Encore’s VFX crew could also get to files to start building CGI elements and tracking them into shots. Fleet then helped to ensure the effects duplicated the look of the live-action shots, camera quirks and all.

“If we had to put some kind of creature into a shot or add to the environment,” Fleet explains, “the final shot had to look real. If the shot was noisy or had a strange color, or the focus popped in and out, or the iris opened and closed midway through the take, we’d have to match that look when we created the effect.”

Confident that Encore could match any look Leonetti’s team could create, Fleet made no effort to constrain production for the sake of an effect. “They did everything that the VES handbook says you’re not supposed to do,” he laughs. “Auto iris, auto zoom…cameras constantly whip-panning around. And because The River is not a fantastical show, everything you see is supposed to be grounded in reality. In many ways, that can be more challenging [for VFX]. But we have great artists and compositers and a brilliant tracker.”

Being on set, Fleet could bring his VFX/post perspective to discussions about how to approach particular sequences. “For certain shots that would require a lot of compositing work, I recommended John go with the Alexa,” Fleet continues. “The sensor size and format would obviously give us more information to work with later. But he made it clear early on that this just didn’t go with the approach of the show. First, we could never be sure that Editorial would decide to use the [angle] shot with the Alexa. They needed the freedom to use whatever best told the story. And the look of something shot handheld with an Alexa and handheld with the cameras that the characters on the show are supposed to be using is different. I knew that the artists and compositers back in Hollywood could match to any of the cameras, so we didn’t have to worry so much about which camera was covering what.”

Fleet was also able to use his fluency in Autodesk Maya and Maxon’s Cinema 4D to present options to the filmmakers so they could see in a somewhat concrete form what certain ideas might look like. “There’s a quick scene in the second episode,” Fleet recalls. “A girl crosses a mirror, and there’s the image of someone else in the mirror. It’s a very quick shot and there were some discussions about exactly how much we should see in the mirror, how ethereal the image should be and that sort of thing. I was able to go to the dailies people, get the footage, put it on my laptop and give them a couple of rough options right there.”

Another sequence required that the travelers’ boat encounter a second craft. But the stretch of river the unit was using to stand in for the mighty Amazon was not wide enough for two boats to travel side by side. “Whenever we needed to see two boats in the same shot,” Fleet continues, “it would have to be done through visual effects at extreme cost. I wanted to help them not go down the path of just relying on effects to fix everything. I know that’s counter-intuitive for an effects guy, but I really wanted us to be able to focus on a smaller number of really great shots. So I took all the data I’d accumulated about cameras, chip sizes, focal length, et cetera, and recreated the geography of the scene in 3D. I was able to build something on my laptop that allowed John and the director and producers to get a rough sense of what the scene would look like from all different angles. Of course, it wasn’t exact, and things change when you go to shoot, but this way they could get a very good idea of how they could use camera placement to tell the story with the minimal number of visual effects shots.”

Drives with the DNX175 versions of everything shot, as well as copies of the original camera files, were shipped back to Los Angeles. As the episode’s offline cut developed, Field would refer back to the various “looks” for the different cameras and situations he had preset with Leonetti and do some initial color passes in his Autodesk Lustre (2012) theater. Artists and compositers, also using Maya, Cinema 4D, The Foundry’s Nuke and other VFX tools, could start the work of building CGI elements – a boat, millipedes, various creatures and environments – that would have the photo-real look the creators envisioned and also mimic the specific artifacts of the camera and format for each shot, eventually making a seamless composite.

Level 3 senior online editor Josh Baca used Avid Nitris 5.5 to keep an updated timeline of all the files as they evolved and to conform the larger DNX175x files to Editorial’s EDLs as they came in. Baca also made use of some unorthodox techniques to “age” a lot of material for the show that was shot on the ALEXA and EX3 cameras but were supposed to look like vintage analog material. “I would use a real Betacam SP deck and copy the material, and then go back and upconvert back into a digital format,” Baca says, adding that “there are ways to synthesize that look, but I don’t think any of them work as well as doing it for real.”

Summing up this groundbreaking prime-time journey, Leonetti says it’s “amazing how much we’ve advanced in visual effects and how flexible they can be. Images don’t have to be pristine – I didn’t light much at all. The visual effects could keep up with whatever we gave them. I wouldn’t want every show to look like this, but it works perfectly for The River.”

By Jon Silberg / photos by Mario Perez, Francisco Roman & Bob D’Amico