[Python] Site package overriding

I’m beginning the process of building a pipeline from scratch. It’s a personal project, but ideally there would be a main tools site-package for things that every project will use, like modeling tools. In addition to that there would be another site-package that would be project specific. Each project will have one and they have the same structure as the global tools repository:

Global Tools
-ahtools
–maya
—animation
----animationtools.py
----modeling
------modelingtools.py
-----rigging
------riggingtools.py
–motionbuilder
—animation
----retargeting.py

ProjectA Tools
-ahtools
–maya
—animation
—modeling
—rigging
----riggingtools.py
----autorig.py
–motionbuilder
—animation
----retargeting.py

I left out all of the init.py files to keep things simple here.

Someone once told me that they were able to achieve this by adding both tools databases to sys.path on startup. They said that they just had to insert ProjectA Tools into sys.path before the Global Tools. The super awesome benefit to doing things this way is that python would look in ProjectA Tools for a module first, essentially overriding the one in the Global Tools database if it exists. If the module doesn’t exist in ProjectA Tools, like modelingtools.py for example, then the system would use the one in Global Tools, because its path is last in sys.path.

It sounds like a really smart way to structure a pipeline’s tools. This behavior reminds me of other non-destructive things like overriding methods in classes, or localizations.

The only problem is that I can’t get it to work with site-packages. I get an import error if the module being imported doesn’t exist in ProjectA Tools (animationtools.py for example). Has anyone used this design before? Were you able to solve this import error? Thanks!

So you’ve got a sys.path that looks something like (if your on windows anyway):


['', 
'c:\\ProjectA Tools',
'c:\\Global Tools', 
'c:\\Program Files\\Autodesk\\Maya2016\\bin\\python27.zip',
'c:\\Program Files\\Autodesk\\Maya2016\\Python\\DLLs',
'c:\\Program Files\\Autodesk\\Maya2016\\Python\\lib',
'c:\\Program Files\\Autodesk\\Maya2016\\Python\\lib\\plat-win',
'c:\\Program Files\\Autodesk\\Maya2016\\Python\\lib\\lib-tk',
'c:\\Program Files\\Autodesk\\Maya2016\\bin',
'c:\\Program Files\\Autodesk\\Maya2016\\Python',
'c:\\Program Files\\Autodesk\\Maya2016\\Python\\lib\\site-packages']

And when you do import maya.rigging.autorig that you get an import error?

So what is happening is that when python runs that import command, it checks through sys.path in order. When it hits ‘ProjectA Tools’, it notices that a package named maya exists, so it checks for the rigging namespace, sees that exists, looks for autorig.py, can’t find it, and fails.
Python is now done, it isn’t going to continue the search because it already found the maya namespace.
So by placing a maya package at the front of the search path, you’ve blocked any other maya packages from ever being searched.

There are some ways around this, specifically namespace packages, these let you define packages that span across multiple locations on disk, but appear to be a single node in the search tree.

Also, don’t use maya as package name, Maya already defines that namespace and you don’t want to conflict with builtin tools, very bad practice.

You might want to look at using modules for your project-specific code, it well be more portable and easier to manage:

It will also be good discipline for keeping project specific code out of the core, which is always a challenge.

Otherwise you can just use nested namespaces in a single location to do the same thing (all of these are assumed to be packages with init.pys so they nest)


+pipeline
    + core
       + animation
          + rigging
          + baking
        + modeling
        + lighting
     + project_1
        + tool_1
        + tool_2 
     + project_2
        + tool_1

As long as nothing in project_1 imports project_2 code, and nothing in core imports from either project, you are good to go and can easily spot bad imports or circular stuff without path manipulations: you just need sys.path to find pipeline. Your imports should be nice and clean, like:


from pipeline.core.animation import find_joints
from pipeline.core.modeling.subpackage import make_polygons
from project_1 import project_1_location
from project_1.animation import project_1_rig
...

+1 to modules

Great advice from Theodox, as usual. The only thing to keep in mind - avoid changing where the MAYA_MODULE_PATH environment variable points to. Some of our clients use it to make it point to their custom tools, which then clashes with out own tools installation. i.e. it’s not oursourcer friendly. If your studio has a certain size, outsourcing may happen at some point or the other - make sure your tools are ready for it :slight_smile:

edit: avoid copying stuff in C:\program files - these are the application’s directories and are, unless you are an admin, protected. Good practice would be to put any user content into C:\Users<users> and then in the appropriate Maya folders. Another option is to have an installer. Installers are extra work, but they also allow for proper uninstallation, which is nice, if your user ever wants to start over with a “clean slate” rather than doing a full Maya uninstall/re-install because he has no clue which files in C:\program files\autodesk\maya are original and which are added later. The same is, of course, possible when sticking to the C:\Users approach - your Maya install itself will remain clean and nice and you have good separation between user content and Autodesk’s content.

That’s pretty much the same logic why you don’t install software into C:\Windows directly, but into separate directories instead.