Week 18

July 30, 2017

Hi team,

This week has been about reintroducing and completing file synchronisation, along with looking back at where we’ve come from and where we are right now to gain an understanding of where we are heading.

Table of contents

File Synchronisation
Cyber Security
- Haze
Looking back
Next Week

File Synchronisation

At Mindbender, files are hosted on a single, shared and central disk. Artists working from outside the walls of the main office in Gothenburg, Sweden require access to it.

“Make available offline” draws inspiration from Spotify and other distributed services where files are visible in each of the pipeline applications like the application launcher and asset loader even though the underlying files are not locally present, and that files can be made locally present by making them available offline. Under the hood, what this means is that a remote connection is made to the central database in Gothenburg to gain access to available asset, and that files are transferred from that same office across the interwebs into a matching directory path on the local machine running in e.g. Brazil.

The counterpart of this happens during publishing, once an asset has been published into the main database an upload takes place to make the underlying files available at the central disk.

As a result, assets can be used transparently regardless of origin and ultimately resides in a central location where they may be both protected and backed up.

Cyber Security

5% of my time this week was spent on pipeline functionality and user interface. The other 95% was spent on this. The invisible and possibly unnecessary, but like wearing your seatbelt as you drive, it’d be irresponsible of me not to have Mindbender buckle up against even the most plain hacks. Let’s go.

With any connection made across the interwebs comes a bundle of responsibility with regards to security. There’s been a surge of security related innovation and news in the past few years, from breaches by Chinese hackers into Sony’s files from upcoming movies and threats being made thereof, to mandatory and globally accepted standards for preventing common attacks from happening.

At the end of this journey my goal is to limit the risk of Mindbender ending up in a scenario which could swiftly put an end to it.

But if Sony can be hacked, what chance do we - as an independent company with little to no understanding of the intricacies of cyber security - do to protect ourselves? Should we even bother trying, or is it all a big waste of time?

In large part, my impression of these attacks is that they are less about hyper intelligent hackers, and more about incompetence on part of the victim. About not knowing, understanding nor caring about plain measures against hacks, such as having a just a password - any password - or involving the simplest and most common form of encryption to communication happening across the public internet. Measures that used to be difficult, expensive or both, but in recent years have gained traction and become free, such as TLS.

TLS, also known as SSL, is an encryption made when transmitting information from one computer to another. It used to be the case that in order to have such an encryption, you paid a yearly subscription for a certificate from a known and trusted source, limiting its applicability to those who understood the benefit well enough to pay for it. Out of the transfers made on the internet, my guess would be that less than 1% actually involved a certificate of any kind.

Then along came Let’s Encrypt.

A free service for automatically renewable certificates, meaning the world no longer had an excuse not to use encryption for any website. The only trouble was, even when free, setting up encryption is hard.

It used to be the case that you could expose a port (80) on your local machine, point a domain name at it - such as mindbender.com - and off you go. Anyone on the internet could type this address in and communicate with the machine. All was well with the world.

But then people (i.e. hackers) thought of clever ways to take advantage of the simplicity and implied trust, things like MITM or “Man in the middle” attacks. With it, someone could intercept data sent from one computer to another and replace what was supposed to come from mindbender.com with something malicious. It could intercept things you sent to mindbender.com, such as your password, and make note of it for later use. Done well, you would never know. The server didn’t do anything wrong, nor did the client. But because there was no encryption, communication was open to abuse.

So, what you now have to do is have a third-party there to validate each and every request made to a computer. A third-party who can provide you with a “key” that you embed into your request. The server is then given this same key from the third-party, compares it with the key you sent it, and if there is a match then you know the request came from where it says it comes from. (Not entirely accurate, take with a grain of salt)

In practice, this complicates things a great deal for me. It means I need to host on two ports, 80 and 443, have 80 “redirect” to 443 which in itself is a task. Once the computer is hosting on 443, I need to make a request to Let’s Encrypt for my certificate - a process that must be redone at regular intervals - once a month, possibly more often. With a certificate, requests to and from this computer must now also be made with encryption, which means additional software must be available on any computer interacting with the server.

Luckily, I’m not the only one having this problem.

There are a lot of smart people out there struggling with this too and some have come up with solutions - such as nginx-proxy and the companion container. These solve two problems at once; (1) I need each service exposed to the internet to come from a single computer due to the single IP address available from the office to the public internet. Services such as file synchronisation, Sentry and our MongoDB asset database. nginx-proxy enables one computer to serve multiple domains, such as files.mindbender.com and logs.mindbender.com for file synchronisation and Sentry respectively by multiplexing requests to one computer into separate server processes. The companion container listens in on requests to each domain, such as files.mindbender.com, and automatically requests a certificate from Let’s Encrypt, downloads it and exposes it to Nginx for real-time use in replying to each request made.

This was all great - complex and scary, sure - but great.

My next problem was upload. Nginx is a webserver, webservers are all about serving files. Downloading a file was no different then serving the standard index.html file for each and every website, and it was optimised for it. For our purposes, this fit well and Nginx defaulted to transmitting the file available at the absolute path provided in the URL.

wget https://files.mindbender.com/hulk/assets/Bruce/publish/modelDefault/v001/modelDefault.ma

Simple. I could keep the server secure per default as the server process itself required no write access to files and listing of directories was off, meaning no file was accessed without being handed a full path up front. Security through obscurity. With regards to performance, I knew it would never become a problem as Nginx powers a large portion of the internet, including files of relevant sizes to our industry, in the megabyte to gigabyte range.

Then along came uploading. Suddenly the server required write permissions to all or portions of our asset library. Obviously we wanted granular permissions, exposing only what was essential for it to do it’s job. But with granularity comes complexity and management. Granting it permissions to everything was simpler, but introduced the need for caution. We didn’t want Nginx to crap out on us and start deleting files (and yes, permission to write include permisson to delete) and it meant the need for additional security as we wouldn’t want hackers to delete anything either. Before write permissions, the worst a hacker could do was read files and break our NDAs to the media with their findings.

Worse yet, I struggled with having Nginx play nice with actually providing the ability to upload anything. After a few hours and without greater commitment to Nginx in particular, I turned elsewhere and found Caddy.

Caddy is competing, younger server application, promising an out-of-the-box experience with TLS certificates built-in, upload functionality and a file browser resembling the one available for Google Drive and Dropbox, but serving our files directly.

I got Caddy up and running rather quickly after some minor difficulties involving some differences in running it locally and on the target server at the offices in Gothenburg. Downloads worked, uploads as well, with a nice looking sub-path separating the two.

files.mindbender.com/upload
files.mindbender.com/download

Haze

Then TLS reared its ugly head. With Nginx multiplexing requests and fetching certificates on its own, we now had another piece on the board, one also capable of handling certificates. My understanding of how these certificates actually work is still hazy, but if I understand correctly either Nginx or Caddy must handle certificates or both do. While Nginx merely forwards requests to Caddy, I’m unsure of where certificates come into play. If Nginx manages requests, does that mean Caddy is off the hook and that communication is safe without it handling it for itself? Or will Nginx have to adhere to the certificate retrieved by Caddy by somehow listening in on this process and read from certificates retrieved by it before forwarding requests to it?

It’s unclear and will likely take another day or two of research and understanding of security. All the while a little voice in the back of my head is saying “is this necessary? Will Mindbender ever be at risk for attack, and is this a valid way to protect against it? Am I wasting my time?”

Looking Back

Back in April I laid out a 3 month plan for this project that looked like this.

See here for details.

Out of the 10 points listed, only 6 were accomplished and I’m 1/3 overdue. With Builder, Approval, Manager, and Connector missing and several additional targets introduced along the way. Let’s go through each of them.

Goal	Reached	Notes
Launcher	X	We transitioned away from the command-line interface (CLI) and into using a graphical user interface (GUI). Functionally similar, the GUI enabled finer control over assets and metadata such as start- and end-frame, the benefit of which hasn’t quite made its way to artists but has shaped the underlying software in ways that can facilitate stability and extensibility down the line - technically these benefits relate to the fact that metadata is no longer part of Windows .bat files but rather part of the asset database. In contrast to the CLI, the GUI can take on tasks including notifications, browsing and management of tasks relative individual artists, centralised process for I/O such as the downloading of assets without influence of whether your Maya session is alive.
Loader	X	We transitioned from listing files on disk to listing assets in a database and enabled use of the Loader from another office, without actually having any files present on disk. This further enabled the ability to make an asset available offline, the topic of this week. In fact all tools and user interfaces build on-top of this one database which is what makes this pipeline able to operate outside of the office, on any computer, without being limited by the fact that it isn’t within the office. We’ve also been introduced to the updated GUI by Roy over at Colorbleed. Under the hood, there is now a plug-in architecture to enable loaders to be written for pipelines that differ from ours, more on this under Avalon below.
Builder		The implementation for scene building depends on everything we’ve built so far and the difficulty has been knowing when to stop improving upon the foundation and start locking it down in order to get on with building. Creation and Loading has been drastically improved since I got started implementing the Builder, but it has turned it into a mirage. Something moving further and further away the closer it seems. - See #130 for details.
Approval		Without building, approval serves little purpose at the moment. Technically, we have the foundation necessary for this plain metadata to be added to any version of any asset, and what we could do as a “tick the box” measure for this feature it have our Loader display something like an icon next to versions marked “approved”. Low hanging fruit.
Connector		Shaders have an implicit relationship to models. When loading a shader, the shader is automatically applied to any shaders present in the scene. This is an independent mechanism that we have little control over at the moment. For example, what if you import a model after a shader? How do you affect how connections are made, and can it be made to connect other things? Do other things? For example, connecting a piece of clothing to a character. Without a known method of connecting assets, the idea of separating assets haven’t entered our workflow which can make it difficult to fully appreciate the power and flexibility of this mechanism. It is also a prerequisite to shot building in which assets are meant for non-interactive load and connection. The exact look and implementation of this feature has seen little action and is yet to be documented.
Publisher		We’ve successfully upgraded the publishing GUI to a more modern and refined look and added support for comments and warnings.
Sync		From FTP to Syncthing to Google Drive, file synchronisation is partially solved yet remains heavy handed and without support for optimisations (too little or too much being uploaded) and file permissions (anyone can delete an entire project, both locally and remote, at any given time).
VM, Cloud		We took a number of weeks to explore whether transitioning into the cloud was feasible, the result of which was “no” and we’re unlikely to return until at least 6-12 months once the technology and data center availability has improved.
Manager		The manager has seen a facelift since the merge with Colorbleed, but has yet to live up to the original plan which included making notifications about new versions of loaded assets being made available.
Creator		The merge highlighted parts of the pipeline that were tailored to Mindbender, the creator was one of them. At Mindbender, we want our own types of assets, such as HistoryLook or RenderGlobals, without necessarily affecting Colorbleed. They in turn have types of assets that are unique to them, that we wouldn’t necessarily want to adopt. So I put in place an API for writing creators as plug-ins. Each plug-in determines available types of assets along with their name and post-processes, such as boiler-plate creation of nodes.
Rendering		In the original plan, I didn’t account for rendering which in retrospect was clearly an oversight. A first draft was implemented last week where naming convention and published paths are made. There are still some kinks due to the implementation happening at the same time as Mindbender was transitioning from Maya 2017-2018, at which point Autodesk also transitioned their render layer system for the first time since 8.5? The crux (inconsistency in render layer output, sometimes prefixed with `rs_`) was later fixed in Maya 2018, which might be a better candidate for the transition, expecting it to be mainly a stability release as it contains few features or changes at all.
Compositing		Due to the month-long surprise-vacation at Mindbender, a few things fell through the cracks and compositing was one of them. The result of which is that there won’t be a compositing pipeline for the next few months whilst we work to make up for the month of lost work over the span of the coming months of part-time pipeline development. Colorbleed is of no use here as they aren’t using Nuke, but Fusion.
ftrack		We’ve also had a look at various production tracking solutions, including NIM and ftrack and ultimately decided on using ftrack. However due to no project actually having been used with it, no integration with the pipeline has been made and is unlikely to be made anytime soon. Instead, what may happen is for use of ftrack as an additional scheduling and review management platform for which it is quite well suited.

Next Week

Over the course of the next two weeks, next week consisting of only Monday and Tuesday (birthday!) I’ll wrap up what we’ve built so far. At that point I’ll be 6 weeks overdue and have 6 weeks to catch up on with regards to other projects which means busy times ahead. For you, it’s important these next two weeks are filled with testing and reporting issues you feel may prevent major work from being done. It’s not a time for new features nor cosmetics or “nice to have’s” but a time of ensuring you can get along without me.

Moving forwards, here’s what will happen. I’ve dedicated 8 hours a day on this project for the past 18 weeks. Two weeks from now, that number will be closer to 1 hour a day. Colorbleed currently makes up for another 2-3 hours a day, accounting for the 5-6 hours they spend on development unrelated to Mindbender. I expect MetaPipe to finish at Siggraph in two weeks time and spend then next few months tailoring their platform for their early adopters until eventually having time and need to dedicate resources in developing a pipeline for themselves. At that point, I expect their 3-4 developers to provide for an additional 5-10 hours a day of development effort that in some way benefit Mindbender.

In 6 months time, I expect myself, Colorbleed and MetaPipe to account for 16-24 hours a day of development effort that in some way benefit Mindbender. Development effort provided on a win-win basis without financial cost to anyone and everything to gain.

In 12-24 months I expect Mindbender to have taken on at least one project with TV series or feature film proportion at which point the work put in all around the board will be put to the test. I expect our work to have outdone most bespoke pipelines out there and for an open community to be flourishing around it, with users such as Goodbye Kansas, ILP and The Mill.