Open Source Tools Distribution Project


#1

Hey everybody,

I’ve been tossing this idea around for a while and I think it’s time for me to get this going. I really want to develop a SIMPLE package distribution system that can work in most DCC apps as well as standalone. This is basically a way to getting tools rolled out to your teams and theoretically across studios. I would love to open this up to the community and have people pitch in with ideas or help out with the code.

I’ve brought this up in the slack channel and I’m usually there if you want to brainstorm. I left the description a bit more vague than necessary so we can flesh some of the subsystems together. But the idea is that this is a simple core system that pulls files down and manages dependencies, and then you can enhance it with your own packages for your own needs. Small modules that do things well. And then we can start having some shared community modules, like maya menus, perforce hooks into DCCs, logging libraries, crash reporters and all of the other stuff that noone really wants to write.

I’m planning on writing this in python because it’s the industry standard (yes, not C# or powershell), but that’s not to say that you couldn’t have a C# module that does something and gets delivered through Python. I’ve seen people use these kinds of systems even to distribute photoshop brushes or reference images, basically anything that you want to share with multiple coworkers that might not have access to a common p4, and the real trick is to get packages to be used cross studios, so we can stop the madness.

I’ve written about 3 different iterations of systems like this, and there are some out there that come close, but are a tad to complicated or unsupported (wpkg, rez) but I’m all for taking the best of those packages and bringing them into this. I’m sure the larger studios already have something like this in place and don’t really need this, but I want the packages to be usable by those devs too (python logging library should agnostic enough)

Requirements:

Small bootstrap script for installation (one or 2 files ideally)
Simple way of generating packages
Different ways of hosting packages (network drive, AWS, local drive)

General Loop:

Bootstrap script will download zipped packages and set them in local user machine, as well as their dependencies.
Environment setup package will run functions on each of the packages to add them to the user’s environment (this can be anything from adding it to sys.path, to any code that should be run during initialization of the package)

What is in scope:

Copying files locally
Running local commands on the user’s machine (move files around, run installers, etc)
Dependency management
Modules will have hooks for different parts of the deployment process (pre_deploy, post_install, run)
Multiple repository sources (general public, company wide, project specific, etc)

What is not in scope:
CMake or any more complex building systems (this could be done with a package, but not out of the box)

Let me know if you’d like to help design/contribute this and we can start a slack channel or a github depot for it.


#2

I’ve been thinking about something like that for some time now and I’d be glad to help.


#3

these are some things to consider

  • sudo or UAC permissions
  • photoshop and 3dsmax are very strict as to where they need to live and where scripts and plugins will be found and executed from

These always annoy the piss out of me

A thought on the bootstrap, if you went with something like portable python, you could embed the bootstrap package there which would make things pretty slick.


#4

For now I’m just building my bootstrap with py2exe so I don’t have to worry about users having python installed.


#5

I’d strongly recommend either splitting modes – an ‘artist’ mode and a ‘dev mode’ – or concentrating on a completely self-curating setup. apt-get, pip, and all the other neat programmer-y tools are either going to just have to be scripted anyway, or will just be misused by the artists. The number on thing to focus on is verifiable and self-contained installs: dependency mismatches cause the hardest-to-fix bugs and cost the most time to sleuth out.


#6

This might also be interesting to consider:

Go is statically compiled: a given go program has no external dependencies. That’s… attractive. OTOH using C as a bridge is gnarly and depressing.


#7

Sorry I missed these! I only got the notification for the first message!

Yes, Dev/Artist split is definitely the plan. In practice there will be a developer package that will be needed for creating/deploy the tools, but the artists will only have the bootstrap script.

I do plan on making the bootstrap script a self contained exe, it will only have an additional config file to determine your repo paths (which we might put in an Env Variable)

I’ll type something more formal up this weekend and put it up on github, but here’s the general idea

BootStrapper

All this does is download packages from repositories and copy them to your local machine. It will also do the semantic dependency matching, so if you say foo.exe install my_uv_script-1.* it will find the latest version that matches your request and pull it down and maybe run some hooks on the packages themselves like post_install and pre_install.

Python Environment Manager

This would be another package that would be part of a core set of packages that most people will likely use. This would be a python script that you can load up in your tool and what this does is run through packages and do a setup_enviroment callback. Which will do things like add it to the system path and setup any environment variables you might want setup, this would be entirely up to your package.

Dev Tools

This package will give you command line tools to deploy packages, roll back, run unit tests, and build documentation on the packages. I’d like to try to do this as a plugin system, so people can extend it with things like building code or whatever other process that would take source code and turn it into publishable artifacts. These would be tools that build machines would run.

Dev Tools UI

This package will be the pretty UI version of the dev tools, with things like wizards to make new packages, and debugging options to see what dependencies a package might have etc

Then I would expect that there would be a few other smaller modular libraries like logging and telemetry that would be used by other tools, but you could fork and make your own version and just do a higher version package or a variant package for your studio. Something like foo_logging_sidefx that would have a foo_logging python package that would just replace the core package when the environment manager is adding all of the dependencies.

Let me know if this makes sense, I’ve gotten some great questions about this already and I’m hoping to start cracking on it this weekend.


#8

Heyhey,

did anything happen with this project idea?
Interested! :slight_smile:
seb