Week 8

May 14, 2017

Hi team,

This week has been about solidifying the recent addition of a database for asset metadata and fully transitioning into using the new Launcher in place of the old cmd.exe-based interface, mb.bat.

The main hurdle has been trying to maintain backwards compatibility between how you work today, and how you will work in the future. The less for you to change, the easier the transition and thus progress.

Table of contents

Good bye mb.bat
Project Configuration
Next Week
Database Layout
No Children
Object Model

Good Bye `mb.bat`

It has served it’s purpose well, but there are greater things on the horizon, and it’s time we go get’em!

Project Configuration

Much like mb.bat was first a console based application launcher, later turned into a graphical user interface, so is the .inventory.toml and .config.toml files to configuring projects later to become a web-based interface.

As plain files, they are easily editing by hand and though not ideal should suffice for the time being. The idea is that you should be able to drop these two files into any new project and once saved begin work as per usual.

These files got a lot of focus this week, as I move to making the ease of use and potential of these files greater and ensuring they remain relatively consistent once we make the switch to a web interface for them. Over the course of the past two weeks, they have gone from YAML to TOML, a choice based on the fact that the Python parser for YAML files was challenging to ship with the project whereas TOML was not.

Before, YAML

schema: "mindbender-core:inventory-1.0"

label: "The Hulk"

assets:
  - name: "Batman"
  - name = "Bruce"
    label = "Bruce Wayne"
    icon = "cutlery"

film:
  - name: "1000"
    edit_in: 1000
    edit_out: 1202

After, TOML

schema = "mindbender-core:inventory-1.0"

label = "The Hulk"

[[assets]]
name = "Batman"

[[assets]]
name = "Bruce"
label = "Bruce Wayne"
icon = "cutlery"

[[film]]
name = "1000"
edit_in = 1000
edit_out = 1202

The process remains the same, to edit a project you --load, make some changes and --save.

$ python -m mindbender.inventory --load
$ # Make some changes
$ python -m mindbender.inventory --save

To upload an existing project to the new pipeline, you --extract and --upload, like before.

$ python -m mindbender.inventory --extract
$ python -m mindbender.inventory --upload

This also applies to when you already have transitioned, but published something via mb.bat. This will also need extraction and an upload, so you just re-run the above and it will update the database accordingly.

Next Week

I’ll begin putting together a first draft of Shot Building which will become the foundation for render and layout publishing.

WARNING: Technical things ahead!

Database layout

“Document” is the official term for an “entry in the database” and represents a block of data in the JSON format.

{
	"name": "hulk",
	"type": "project"
}

Each document contains the following members.

{
	"name": "Name of document, e.g. `hulk`",
	"type": "Type of document, e.g. `asset`, `version`",
	"data": "Document metadata, e.g. `startFrame`, `label`",
	"parent": "Document parent, e.g. an asset or a project"
}

Here’s what changed about that this week.

One of the goals of having metadata, like shot lengths and relationships between a version and it’s parent subset, was to have the contents of each document laid out flat.

{
	"name": "1000",
	"type": "shot",
	"silo": "film",
	"editIn": 1012,
	"editOut": 1301
}

But one of the immediate issues with this was that in order to make use of the data related to visualising available assets, like in the Launcher and Loader, I had to explicitly exclude the editIn and editOut members. And likewise when visualising the metadata for a given asset, I had to exclude type and silo along with other members related to metadata.

This was no good, so document-related data now occupy the first level of members, whereas metadata resides in a child called data.

{
	"name": "1000",
	"type": "shot",
	"silo": "film",
	"data": {
		"editIn": 1012,
		"editOut": 1301
	}
}

No children

Another important aspect of the document structure is that a document has a parent, but parents have no children.

{
	"name": 1,
	"type": "version",
	"parent": "modelDefault"
}

{
	"name": "modelDefault",
	"type": "subset",
	"parent": "Bruce"
}

The alternative is for each document to contain both a reference to its parent and children.

{
	"name": "modelDefault",
	"type": "subset",
	"parent": "Bruce",
	"children": [1, 2, 3, 4]
}

But the problem with that approach is that when a new version is added, I must not only insert it into the database, but also make an update to the parent subset, leaving room for inconsistencies should this second write fail or otherwise become corrupt.

The question I needed answering in this first iteration was, how can I query the children of a given parent? Such as getting all versions of modelDefault?

Turns out this was rather straightforward.

subset = io.find_one({"type": "subset", "name": "modelDefault"})
all_versions = io.find({"type": "version", "parent": subset["_id"]})

This then leaves no room for error when creating new documents of any kind. However, it does have a potential performance impact, as to find every version the database now has to traverse the entire database in order to filter out those that do not match the query.

So far, with 1000+ documents, this (supposed) performance hit hasn’t made itself known and I expect that when it does I can resort to indexing.

Object Model

On disk, files share a common object model, as first described in Week 1.