Week 9

Hi team,

This week has been diverse; from putting together research for (1) shot building and (2) internal review, to implementing a first draft of (3) selective sync and giving (4) cloud desktops a try.

Solving sync was originally set for Week 25 but due to pressing issues was moved up to now.

Table of contents

• Lots of New Features
• Selective Sync
• Internal Review
• Shot Building
• Cloud Desktop
  • VMWare Horizon
  • Amazon Workspaces
  • Paperspace
  • Sixa
• Publishing and Comments
• Sign-in with Google
• Two weeks with MongoDB
  • Query language, great
  • PostgreSQL, query language scary
  • RethinkDB, MongoDB without the kinks, events
  • RoboMongo, great UI

Lots of New Features

With some of the boring parts of workflow automation behind us, we can finally get to more interesting things, things that have a real effect to your day to day workflow.

New features at a glance

• Performance 16x or more performance increase in Loader when selecting multiple subsets.
• Performance Clicking around in the Loader is now asynchronous, it should feel more snappy
• Cosmetics Versions now in descending order in Loader
• Cosmetics Latest version automatically selected in Loader
• Cosmetics Database performance statistics now visible in Loader
• Fixed Manager now highlighting currently loaded version
• Fixed Mindbender -> Rigging -> Clone (World) now actually clones in worldspace
• Fixed Current time automatically set to Edit start frame in Maya on launch
• Feature Mindbender -> Reset Frame Range sets the frame range to whatever is set for the currently active project.
• Safety Names of assets limited to either characters, numbers and or underscore(s). Before, they could have been anything, including %!&*$ which would have caused issues in all sorts of places.
• Security Implements #133, lays the groundwork for restricting certain projects to certain users

Details on GitHub

Selective Sync

The main event for this week has been the implementation of Selective Sync, a mechanism for remote artists to get a hold of assets in a secure and efficient manner.

Before we look at the advantages to this approach, let’s look at what we’ve tried up till this point.

1. SyncThing
2. Google Drive

SyncThing

An open source and secure computer-to-computer mechanism with a less favourable user interface was lacking the ability for the remote artist (henceforth referred to as the client) to specify what was wanted and had to rely on what was served. The local arist (i.e. server) could define what not to include, via .stignore files but the file-based configuration file proved difficult to manage for the local artist.

Google Drive

Like SyncThing, files are automatically synchronised between two or more computers, but rather than specifying what not to include, the client could instead selectively pick what to make available offline. However, files travel computer-to-cloud-to-computer with a restrictive license.

When you upload, submit, store, send or receive content to or through our Services, you give Google (and those we work with) a worldwide licence to use, host, store, reproduce, modify, create derivative works, communicate, publish, publicly perform, publicly display and distribute such content.. Google’s Terms and Conditions

The copyright holder may upload his content at his own expense - such as King uploading their own content - but someone producing content for someone or receiving content from someone doesn’t own the copyright to said material and hence does not have the right to aree to these terms and conditions and run the risk of legal trouble.

Selective Sync

In short, SyncThing is safe but difficult whereas Google Drive is easy but unsafe.

Selective Sync solves both of these in one fell swoop by securely transferring files computer-to-computer without configuration. Files are made available as they are published and are picked via the same interface as they are loaded.

Limitations

Currently this is one-way. That is, files are made available from Mindbender to remote artists, but remote artists cannot currently make their files available to Mindbender.

This should be solved by the end of next week.

Backend

Selective sync is made possible via the Python requests library making requests to an instance of Nginx running in Docker on a local computer.

Docker provides failure redundancy in that when/if for whatever reason the instance crashes, a new instance would immediately go back online. For scale it also enables distributed instances and management of said instances via hip new tools like Kubernetes. Nginx is quick and works out-of-the-box as a file server, whilst facilitating basic authentication for both download and upload.

The database is currently running on a low-end, 2gb memory local machine and is the one I have access to remotely. Before deploying this backend, we will need a dedicated machine of at least 4gb and 20gb of hard disk space. As we scale to 40-60 we might consider having Nginx do server-side, in-memory caching of commonly accessed files and for that we’ll need as much memory as we can throw at it along with a few hundred gigabytes. Once we reach the 100-200 artists, we distribute the Docker instance to a few computers of equal capacity and should be golden.

More details on implementation and future can be found in the GitHub issue.

• Selective Sync on GitHub

Internal Review

Shotgun et.al. provide facilities for uploading, commenting and annotating on video and stills. This enables great efficiency in taking and managing notes for a given artist submission, like an animation.

I’ve been making great progress at finding a solution for how we can solve this internally.

It involves..

1. Submitting with comment and image
2. Commenting on submission
3. Notifying relevant people
4. Integrating on accepted review
5. Potentially mark as “approved” upon accepting

Technically we need..

1. A web server
2. A real-time web application, with for example React.
3. An image and video annotation mechanism

Read more about it on GitHub.

• Internal Review on GitHub

Shot Building

Research and development for how shot building is to work is under way, you can follow the thought process via the GitHub thread.

Cloud Desktop

On working towards virtual desktops in the cloud, there are a few options available.

Amazon Workspaces

I got a chance to test Amazon Workspaces, a virtual computer in the cloud.

In practice, it’s exactly like TeamViewer or VNC into another machine, the difference being primarily one thing - performance.

Where TeamViewer and friends rely on your computer (1) rendering graphics to your screen, (2) capturing your desktop 60 frames per second and (3) transmitting the results across the interwebs, a cloud-based option like Amazon Workspaces cut out the middle man and render straight to the interwebs via a technology called Teradici.

Amazon offers a few hardware options.

The test above was made on their Performance option. I haven’t yet been able to test drive their Graphics option, this is what support had to say.

For a limit increase of this type, I will need to collaborate with our Service Team to get approval. Requests for WorkSpaces limit increases fit into two categories, the first being Value, Standard, and Performance WorkSpaces, and the second just for Graphics WorkSpaces.

Sixa

Sixa isn’t an option to us, as they only permit 3 running computers per account, which in practice means no interoperability amongst more than 3 computers at a time, like log-in information or filesystem sharing.

Paperspace

The most recent and hip alternative, their focus is on high-performance workstations in the cloud for the home user, recently providing enterprise, or “team” options similar to Amazon Workspaces, such as shared disks and user management.

Their offering are somewhat more capable than Amazon’s.

At a glance, the hardware is sufficient though we may struggle with diskspace. For the hourly price of GPU+, 10 hours a day, 5 days a week that comes to roughly $85/month, whereas a fixed monthly cost of the Pro option is $58/month.

The upside to Paperspace is clearly the performance, however their servers are limited to the US, which means a greater delay between your cursor and things happening on screen. Workspaces have servers in Ireland and Frankfurt amongst other locations.

VMWare Horizon

Perhaps the most secure and configurable option would be to host our own “cloud desktop” solution, which is what VMWare Horizon is. With it, we’d have a server room of computers that artists would log onto remotely, just like they would through Paperspace and Workspaces. Odds are these services are simply reselling their own Horizon set-up at a marked up price.

With Horizon, we’d manage all resources and data locally and have full control, at the cost of more maintenance. Whether the final price tag is more or less is hard to tell, depending on who and how many is required to maintain it.

Publishing and Comments

Next week I’ll have a look at introducing comments for our publishes.

Up till now, there hasn’t been any way of actually viewing comments, had we had them. But the Loader is about to get an update in which you can visualise a written comment on the a given version to a subset.

Sign-in with Google

As we’re gathering places in need of a secure login, I’d like to try and avoid introducing additional usernames and password, especially given there would need to be one pair per service (such as (1) the asset database and (2) make available offline) per user.

So I figured we’d reuse the one everyone working at Mindbender already uses, their Email log-in.

Two weeks with MongoDB

It’s now been two weeks since I first introduced the Asset Database using MongoDB for a backend. So far it’s been very good, no apparent issues, inserting and querying for data is logical and immediate. The GUIs are currently written in such a way that they block user input whilst awaiting data from the server and so far the delay is unnoticeable, between 5-10ms on a local network, 40-50ms from London to Gothenburg and 350-1000ms when connecting from London to Los Angeles to Gothenburg.

Mongo has an excellent GUI as well, called RoboMongo which makes browsing it akin to browsing files on a file system. Very familiar.

I am however still weary due to reports of data loss with Mongo and am looking towards swapping MongoDB for RethinkDB; a real-time database with all of what Mongo offers and more. It will also enable us to have GUIs update themselves when assets are published and to support notifications of events, like when a version you care about have been published.

Another alternative is PostgreSQL, a mature and commonly used database in all sorts of scenarios. I am however a little weary of the SQL language, coming from Mongo which is passing dictionaries back and forth, and also the fact that it appears heavily over-engineered, faciliting every feature one can think of; including all of what Mongo offers.

Next Week

I’ll finish up the Available Offline feature, awaiting Peter’s return such that I can gain access to a more capable computer, something that can handle the Asset Database and now a file server.

I’ll put together the Include Source.. option for offline files along with remote publishes automatically uploading themselves onto the central Mindbender file server. That should take two to three days, after which I’ll return back to implementing a first draft of Shot Building.