Week 17

Hi team,

This week was about rendering and about establishing a pipeline for distributing tasks of any kind onto a network of “workers”, specifically for Rendering via Deadline.

Table of contents

• Rendering
  • Alternatives
  • Thoughts
• Miniature Render Farm
• Scene Protection
• Next Week

Rendering

Rendering is the act of putting all 3d assets together and flattening it into a 2d image. As opposed to other production stages, such as modeling and rigging, it’s a long-running and computationally expensive process which makes it unsuitable for similar treatment of local validation and processing.

Instead, rendering is split up into two processes, one happening locally and the other on remote computers known as “nodes” or “workers”.

• Submission
• Processing

Submission is the act of configuring and distributing a task to one or more computers. Validation includes things such as making sure references to absolute paths are accessible to the worker

The process is as follows.

1. A Maya scene is prepared by the lighter, i.e. Emre
2. The scene is submitted for processing to a distributed queue management system, i.e. Thinkbox Software Deadline for rendering into the artists working folder.
3. Once the artist is happy with the result, the resulting images are published to make it available to the compositor via the Loader.

This two-tier approach differs from how data is published in modeling, rigging, animation and look development due to the separation between submission and processing. In other stages, these two occur simultaneously which is impractical in the case of rendering due to how much longer the processing actually takes - i.e. minutes, to hours, to days.

Once rendered, publishing happens via the command-line by browsing to the rendered images.

$ cd path\to\images
$ avalon --publish

This will trigger the same publishing GUI and visualise each individual sequence of images for publishing. Once published, sequences become assets like any other. Next week I’ll integrate the pipeline tools into Nuke for creating, loading and publishing of assets via the same tools as artists use in Maya.

Alternatives

Deadline is one of many options available. Some are specifically designed for visual effects type work, whereas others are more generic and somewhat lighter weight.

Here’s a brief summary of what is available and my current (and shallow) understanding of each.

Thoughts

A few things became apparent with this workflow.

1. How do you know which rendered images have already been published?
2. Should we copy images from working folder to published folder, considering images can occupy 1+ GB of data?
3. How do you use images - e.g. for basic compositing - prior to publishing the second time? If you do e.g. use the images straight from the working folder, then you would need to manually load the images and later transition to using the Loader once the images have been published.

To overcome these issues, I’m considering renders immediately being published once submitted and made available to the compositor, but later approved by the lighter if the submission is what he expects.

The upside of this approach is that (1) you’ll know what is published by the approved attribute, (2) there is no copying or data-duplication because images go straight to where they eventually end up and (3) transitioning from using work-in-progress images to approved images is seamless and use the same tools; no manual loading.

The downside is that the number of versions will significantly increase. Each render, even if only for testing purposes, will produce a new version potentially reaching 3-digit numbers or more when taking into account basic tests.

A way to combat this is to filter these versions out from the Loader, by default, and enable the artist to manually toggle their visibility where needed.

Miniature Render Farm

As a side note to the above, in order for development to progress smoothly I needed direct access to a reliable distributed job scheduler, ideally Deadline.

The trouble with developing for anything distributed is that you never know what’s available to you. Are machines running? Are they busy? Is the scheduler running? What version is the scheduler, and each of the available slaves? This leads to some difficult-to-debug errors, the kind that resolves itself when you run things twice or restart one or more machines.

So I put together a miniature render farm where versions and availability was pre-defined up-front.

• https://github.com/mottosso/docker-deadline

Scene Protection

A mechanism has been put in place that prevents overwriting an original published scene file in Maya.

Once a scene is published a lock is placed on it, preventing accidental overwrites.

1. A locked scene cannot be saved
2. A locked scene cannot be published
3. A lock can be overridden with cmds.remove("protect")

Before this lock, it was possible to publish an asset and make changes to the file from which it was published, making it difficult or impossible to return back to this particular version sometime in the future.

Under the hood, this mechanism is enforced by cmds.file(renameToSave=True) and by the presence of a node called protect.

I’d imagine this locking to become frustrating at first, so let’s play by ear and how to make it bearable. The goal is guaranteeing that the source file remains true to the published file. For example, if you would like to revisit the original source file for version 12 of a render, the scene referenced by this published version should ideally be identical to the one where version 12 was published from.

Next Week

I’ll wrap up the publishing of renders on Monday and hope to complete the compositing pipeline by Friday, but that depends on whether there is a compositor available and returned from holiday.

Implementing compositing will include (1) loading of published renders via the Loader, (2) creation of new compositions via Creator and (3) publishing of composites, also to Deadline via the publisher.

Otherwise, I’ll re-implement Make Available Offline and it’s uploading counterpart so as to enable full and optimal synchronisation to take place between offices.

• GitHub Issues