Maya external file references across branches

Morning,

How are you guys handling re-pathing of external file references in Maya?

We have our Maya project set to the root of the current P4 branch and this has afforded us “relative paths” so far. Now we are running into issues with “cross branch references”. Something like this:

Animation file: d:/p4/npc_sprint_branch/animation.mb
File Reference should be: d:/p4/npc_sprint_branch/rig.mb
File reference is: d:/p4/main_branch/rig.mb

In the above scenario the animation scene and file reference were setup in “main_branch” and then integrated to “npc_sprint_branch”.

It seems that Maya first tries to resolve the absolute reference path and only if that fails uses the project relative one. In this case I have both branches on my machine.

I’m thinking we could do something like:
• Make paths local with callback on save, load or on demand.
• Save as ASCII and run a P4 trigger that will make all paths relative on submit
• Possibly rely on an environment variable to make our paths relative, instead of the Maya project.

Do you have a working solution or thoughts on this?

Cheers,
Sune

Just did some metrics on Maya ASCII vs. Binary and load times do seem “pretty” equal.

If this test is valid, making all paths project relative (or with an environment variable: $PROJECT_ROOT\myFolder\myfile.ma) with a python script and a P4 trigger sounds like an easy way to go.

I have solved this two ways at two different companies.
The first time I encoutered this problem, I just ran a script job at File Open that checked the current project, gathered external references, and updated the paths. It was pretty simple and worked well.
The other way was to embed Environment Variables in the file paths - Maya can expand Environment Variables just fine.

Was that only for references or also things like audio and textures? And were you able to update the paths to all these things, before Maya spent time loading them?

Yeah, I tried defining one of those today, worked a treat.

I’m just concerned with having to write code that handles re-building absolute paths into environment (or project) relative ones, across a range of Maya systems (references, texture, audio, etc). Or is there an easy way to do this across the board?

The intended beauty of running a P4 pre-submit trigger, is that you would simply massage every file path in your .ma file (potentially with a few qualifications) and be done…That’s the theory at least :slight_smile:

Thanks man!

We use the subst command to create a virtual drive that points to a location on a physical disk or network share. Different teams use different drive letters so that you can have multiple projects on the same machine. All references point to the virtual drive including textures, and we have a couple simple batch files that remap drives to different locations when people work out of sandboxes or different branches.

I’ve always wished that Maya supported relative paths without the cumbersome project file management drama, or that it supported Max style search paths, but alas…

EDIT: Oh, and in regards to version control, we have Maya scripts that understand about virtual drive mappings, and help people check files out, add them, etc. Many of our C#/C++ tools do the same.

It worked for all external file references, including other maya files, texture files, etc. Personally I like this approach over the environment variables because it doesn’t need to do any conversion. Just take the path from the maya file, strip off the branch, append the current branch, test if the file exists, and move on to the next.

I’ve had reasonable luck using the Workspace command to set the active project so that when Maya attempts to resolve relative references, they go to the right location. The ‘Scenes’ and ‘Source Images’ roots get pointed at the root content folder for the branch,. I try to police saves so that all references are relativized before save (I don’t let users save a file with reference outside the project: no “E:/temp/blah.ma” for you!)) We use a global environment variable for each project and branch, and userSetup.py checks that on startup and chooses the correct project appropriately.

It helps that in our case all the material assignments etc are really done in the game, and read back in from the game material files - textures there are relative paths, and I refresh them on file load so users always get the right textures when they open the file .

It’s still possible for users to sneak around - I didn’t put a ton of work into bulletproofing it - but it takes some extra work on the user’s part. The path of least resistance usually just works.

As a side benefit, I deliver all tools and tool binaries (things like external command line tools that get called from python) to the project folders, so they can be smoothly branched.

In the past I’ve done it with SUBST and dedicated drives, and that works fine - as long as you never need to support multiple project/branch combinations at once. I found that out the hard way building a mult-project build server :frowning:

At my previous job I had to deal with multiple Projects and Branches. I just made a simple function to update file paths when files were opened. It went kinda like this:

  1. Get the Current Project and Branch by querying Environment Variables.
  2. Assemble the path up to the Project Root.
  3. Loop through each referenced asset and strip off the Project and Branch to get the relative path.
  4. combine the newly assembled Root and Relative Path.
  5. test the location exists.
  6. update the path.

All super easy stuff to manage in python script since managing the environment variables was done outside Maya.

(Note: Users could only be working in one project/branch combination at a time. If they wanted to switch branches or project, they needed to shut down the tools and relaunch them from the appropriate location).

– Steve - You get to read game material files in Maya? As in ‘Maya Export --> Game --> Back to Maya’?

– What’s a P4 Trigger? The user hits submit, and you can run scripts on the file(s) they’re checking in? Are both versions submitted?

– Our working assets are rarely branched within a project, but the games assets are pretty often. We use a config file and overrides to point the working assets to the new game asset branches (stuff doesn’t ever come back from the game into Maya). When starting a new project with old working assets, we depend on Workspace taking care of remapping, then as a 2nd option, Find/Replacing with a batch text editor. edit: Maya ASCII!!!

Subst is not an option for us, used it at EA though and it was very straight forward. We do have a regestry key that we look up on Maya startup, based on which we initialize the correct scripts environment and set the Maya project, for what ever the current projects/branch is.

So the relative pathing that Maya gives you already works… Except when it doesn’t :slight_smile: Guess I need to figure out how to modify paths on load/save then, or just go the P4 pre sub trigger route.

“What’s a P4 trigger”

It’s a script that runs on the P4 Server. There may be different kinds, but the one I’m talking about get’s triggered when you submit a change list, that has a specific file type in it. It will run a user definable script (that happens to modify the Maya files you are about to submit) and then go ahead with the submit.

At least that’s the theory, I haven’t actually done any yet.

The downside would be that it’s not project specific, meaning that it would likely be run on any submit containing Maya files on that server, so that might be a deal breaker. At our studio every branch/project is a completely self contained code base/environment/asset set, so that might be worth protecting. Also in our case we would have to ask IT to actually put it on the server… You know what that means :slight_smile:

Btw. If any of you do have code examples of modifying paths on load or save handy, that would be awesome to see.

Thanks!

this is a super simple example…

# assume this structure:
# c:/depot/MyProject/MyBranch/Content/Environments/Level01/sourceimages
import os
def constructNewFilePath(oldPath):
    project = os.getenv('PROJECT')
    branch = os.getenv('BRANCH')
    projectRoot = os.path.join(project, branch)
    relativePath = oldPath.split('Content', 1)[-1]
    newPath = os.path.join(projectRoot, 'Content', relativePath)
    return newPath

@pat: The game uses crytek .mtl files, which are XML used by the game’s shader compiler. Maya only assigns named shaders to polys, all rendering info is assigned in the Crytek editor. On file load I parse out the xml file for the game and pull out the textures that are assigned in game and create the appropriate nodes in Maya – that way users always get the game textures from the project with no worries about file locations or relativization. Plus this also makes it impossible for users to assign textures that aren’t part of the project hierarchy, since Crytek enforces that in the editor :wink:

All actual shader changes (‘make this shinier’, etc) are supposed to be done in the Crytek editor: it’s where the WYSIWG editing happens. I don’t bother trying to map maya stuff on to the crytek materials, it’s not a very 1:1 relationship, but it’s very useful for modelers to see the right textures in their scenes. I do make fake crytek materials for new shaders in Maya so that users can iterate on changing materials (although Crytek is a whiny baby about changes to the number or order of submaterials - it uses the Max multi-subobject paradigm and I have to do a bunch of nonsense behind the scenes to preserve the order of sub materials, handle deletions on either end, etc.

This has some stuff specific to the layout of our project but most of it is steal-able for any structure. 99 times out of a hundred I just make a project.ULProject.default() object and use it for all path management in an area . In rare cases I make two or more explicit ones using the root, project, branch constructor so I could, eg, map between two branches which had different directory hierarchies

The DepotProject class does the exact same stuff but for perforce so you can use it to, say, bulk-relativize a bunch of depot file paths into local disk paths

'''
classes and methods for handling the UL project structure
'''
from os import  environ, getcwd, chdir, pathsep, path, mkdir      #@UnusedImport
import posixpath                                                                    #@UnresolvedImport
from collections import deque

class ProjectError ( ValueError ):
    pass

class OutOfProjectError( ProjectError ):
    pass



class ULProject( object ):
    '''
    Represents a combination of a root directory , a project directory, and a branch which together define a working branch of the UL project tree
    '''

    VALID_PROJECTS = ['class3', 'class4']
    CONTENT_ROOT = ['Game/', 'game/', 'GAME/'] # precased for speed
    MAYA_TOOLS_LOCATION = ["tools", "dcc", "maya"]

    def __init__( self, root, project, branch ):
        self._Root = root
        self._Project = project
        self._Branch = branch

    @property
    def Root( self ): return self._Root

    @property
    def Project( self ): return self._Project

    @property
    def Branch( self ): return self._Branch

    @property
    def Content( self ): return path.join( self.Path, self.CONTENT_ROOT[0] ).replace( "\\", "/" )

    @property
    def Path( self ):
        rootPath = path.join( self.Root, self.Project, self.Branch )
        return rootPath.replace( "\\", "/" )

    def __str__( self ):
        return self.Path

    def __repr__( self ):
        return "< project : %s>" % path.join( self.Project, self.Branch )

    def contains( self, abspath ):
        '''
        Returns true if the supplied path is contained in this project
        
        Any non-absolute path is assumed to be contained, so all relative paths return true.
        
        @note: this DOES NOT check the disk -- it just indicated if the path is formally part of the project.
        '''
        # relative paths are presumed to be 'contained'
        abspath = abspath.replace( "\\", "/" ).lstrip( "/" )
        if not path.isabs( abspath ): return True #@UndefinedVariable
        p1 = path.normcase( path.normpath( self.Path ) )
        p2 = path.normcase( path.normpath( abspath ) )
        return path.commonprefix( [p1, p2] ) == p1

    def relative( self, abspath ):
        '''
        Returns an absolute path as a path relative to the project root. 
        
        if abspath is not absolute (ie, no disk path) it's returned unchanged
        '''
        # relative paths are returned unchanged
        abspath = abspath.replace( "\\", "/" ).lstrip( "/" )
        if not path.isabs( abspath ): return abspath #@UndefinedVariable
        if ( self.contains( abspath ) ):
            relpath = path.relpath( abspath, self.Path )
            return path.normpath( relpath ).replace( path.sep, posixpath.sep )
        else:
            raise OutOfProjectError( "%s is not inside project %s " % ( abspath, self.Path ) )

    def absolute( self, *relpath ):
        '''
        Return the supplied relative path as an absolute path
        
        If multiple items are supplied they are concatenated to make a path:
        
        >>> proj.absolute('rel/path')
        >>> 'C:/ul/class3/main/rel/path'
        >>> proj.absoluter('rel', 'path', 'segs')
        >>> 'C:/ul/class3/main/rel/path/segs'
        '''

        if '//depot' in relpath[0].lower():
            raise ValueError, 'Do not use ULProject instance with depot paths -- try DepotProject instance instead'
        # lstrip -- otherwise left slashes are 'absolute' and don't concatenate
        noSlash = lambda q: q.strip( "\\/" ).strip( "\\/" )
        argList = map ( noSlash, list( relpath ) )
        absCount = 0;
        for item in argList:
            if path.isabs( item ): absCount += 1
        if ( absCount > 1 ) : raise ProjectError( "supplied items contain more than one absolute path : %s" % ( ", ".join( argList ) ) )
        if ( absCount == 1 ):
            return posixpath.normpath( posixpath.join( *argList ) ).replace( '\\', '/' )
        else:
            argList.insert( 0, self.Path )
        return posixpath.normpath( posixpath.join( *argList ) ).replace( '\\', '/' )

    def asset_path( self, path ):
        '''
        Returns relative path to a file in the content directory (typically /Game/)
        '''
        relpath = self.relative( path )
        for item in self.CONTENT_ROOT:
            if relpath.startswith( item ): return relpath[len( item ):]
        return relpath

    def absolute_asset_path( self, relpath ):
        '''
        Returns an absolute path for a path relative to the content root (ie, /game/ ) directory
        '''
        if path.isabs( relpath ):
            raise ProjectError( "absolute asset path expects a path relative to the content root directory, got '%s'" % relpath )

        segments = list( path.split( relpath ) )
        c_root = self.CONTENT_ROOT[0].rstrip( "/" )
        if segments[0].lower() != c_root.lower():
            segments.insert( 0, c_root )

        return self.absolute( *segments )


    def maya_tools( self ):
        toolsPath = path.join( self.Path, *self.MAYA_TOOLS_LOCATION )
        return self.absolute( toolsPath )

    def same( self, path1, path2 ):
        '''
        Compares two paths, returns true if they absolutize to the same value. Comparison is case insensitive
        '''
        ab = self.absolute( path1 )
        ab2 = self.absolute( path2 )
        return self._pathCompare( ab, ab2 )

    def _pathCompare( self, path1, path2 ):
        p1 = path.normcase( path.normpath( path1 ) )
        p2 = path.normcase( path.normpath( path2 ) )
        return p1 == p2

    def __eq__( self, other ):
        '''
        Equality operator.  If two project objects share the same branch and same project, they are 'equal'
        '''
        try:
            return self.Project.lower() == other.Project.lower() and self.Branch.lower() == other.Branch.lower()
        except:
            return False

    @classmethod
    def default( cls ):
        return cls( environ["UL"], environ["UL_PROJECT"], environ["UL_BRANCH"] )

    @staticmethod
    def from_path( filepath ):
        filepath = filepath.replace( '/', path.sep )
        pathSegs = filepath.split( path.sep )
        for n in range( len( pathSegs ) - 1, 0 , -1 ):
            if ULProject.VALID_PROJECTS.count( pathSegs[n].lower() ):
                root = path.sep.join( pathSegs[:n] )
                project = pathSegs[n]
                branch = pathSegs[n + 1]
                return ULProject( root, project, branch )

        raise OutOfProjectError ( "%s is not in a recognized project path" % filepath )




class DepotProject( ULProject ):
        '''
        This subclass of ULProject exposes the same functions, however the paths are
        relativized and absolutized to p4 depot paths.  Thus, calling absolute
        returns something like '//depot/class3/main/path/segments', and calling
        relative will turn //depot/class3/main/relative/path' into 'relative/path'
        '''

        def __init__( self, root, project, branch ):
            self._Root = root
            self._Project = project
            self._Branch = branch
            self._Depot = '//depot'
            self._InvalidDepot = '\\depot'


        @property
        def Depot( self ):
            return self._Depot

        @property
        def DepotPath( self ):
            return posixpath.join( self.Depot, self.Project, self.Branch )

        def contains( self, abspath ):
            '''
            returns true if abspath is contained in this project. Abspath can be either a disk path or an depot path
            '''
            # relative paths are presumed to be 'contained'
            abspath = abspath.lstrip( '\\' )
            abspath = abspath.replace( "\\", "/" )
            if not path.isabs( abspath ): return True #@UndefinedVariable
            p1 = None
            if ":" in abspath:
                p1 = path.normcase( posixpath.normpath( self.Path ) )
            else:
                p1 = path.normcase( posixpath.normpath( self.DepotPath ) )

            p2 = path.normcase( posixpath.normpath( abspath ) )

            return path.commonprefix( [p1, p2] ) == p1

        def absolute( self, *relpath ):
            '''
            Return the supplied relative path as an absolute path
            
            If multiple items are supplied they are concatenated to make a path:
            
            >>> proj.absolute('rel/path')
            >>> '//depot/class3/main/rel/path'
            >>> proj.absoluter('rel', 'path', 'segs')
            >>> //depot/class3/main/rel/path/segs'
            '''

            if self.Depot.lower() in relpath[0].lower():  # this means we are a depot path
                val = posixpath.normpath( posixpath.join( *relpath ) )
                if not val.startswith( '//' ): val = "/" + val  # special-case handling in case somebody passed in '///depot'
                return val.replace( '\\', '/' )

            if self._InvalidDepot.lower() in relpath[0].lower():
                raise ProjectError, "Depot paths must start with //depot (no left slashes)"
            noSlash = lambda q: q.lstrip( "\\/" )
            argList = map ( noSlash, list( relpath ) )
            absCount = 0;
            for item in argList:
                if path.isabs( item ): absCount += 1
            if ( absCount > 1 ) : raise ProjectError( "supplied items contain more than one absolute path : %s" % ( ", ".join( argList ) ) )
            argList.insert( 0, self.DepotPath )
            return posixpath.normpath( posixpath.join( *argList ) ).replace( '\\', '/' )

        def relative( self, abspath ):
            '''
            Returns an absolute depot path as a path relative to the project root.   Abspath can be a depot path or a disk path
            
            if abspath is not absolute (ie, no disk path) it's returned unchanged
        
            '''
            # relative paths are returned unchanged
            abspath = abspath.replace( "\\", "/" )
            if ":" in abspath:
                return ULProject.relative( self, abspath )
            if not posixpath.isabs( abspath ): return abspath #@UndefinedVariable
            if ( self.contains( abspath ) ):
                relpath = path.relpath( abspath, self.DepotPath )
                return relpath.replace( path.sep, posixpath.sep )
            else:
                raise OutOfProjectError( "%s is not inside project %s " % ( abspath, self.Path ) )

        def local( self, depotpath ):
            '''
            returns the supplied path as a file system path:
            
            >>> proj.local('game')
            >>> 'C:/UL/Class3/Main/game'
            >>>
            >>> proj.local('//depo/class3/main/game')
            >>> 'C:/UL/Class3/main/game'
            
            @note:  the mapping DOES NOT use the perforce client spec to do this
            translation! It just assumes that the depot is mapped   as
            UL/UL_PROJECT/UL_BRANCH -- which is typically safe but might not
            work properly if the client mapping is not standard.  USE WITH CARE.
            '''
            relpath = self.relative( depotpath )
            return ULProject.absolute( self, relpath )

        def to_depot( self, diskpath ):
            '''
            Given a disk path or a  relative path, returns a depot path
            
            >>> proj.to_depot('C:/UL/Class3/main/game')
            >>> '//depot/Class3/main/game'
            >>> proj.to_depot('game')
            >>> '//depot/Class3/main/game'
            
            @note:  DOES NOT use p4 client to establish mapping.  USE WITH CARE
            '''
            dp = ULProject.relative( self, diskpath )
            return self.absolute( dp )

        def __repr__( self ):
            return "< DepotProject : %s>" % path.join( self.Project, self.Branch )


def create_path( newpath ):
    '''
    Creates all of the directories needed to complete a path
    
    If the final entry of the path contains a "." character it is treated as a file and ignored
    
    If the path cannot be created, raise a WindowsError
    '''
    newpath = newpath.replace( "\\", '/' )
    newpath = posixpath.normpath( newpath )
    newpath = newpath.lstrip( '/' )
    if not path.isabs( newpath ):
        raise ValueError, 'path %s is not absolute' % newpath

    # remove the file, if there is one
    finalpath, tail = posixpath.split( newpath )
    if not ( "." ) in tail:
        finalpath = newpath

    segs = deque( finalpath.split( posixpath.sep ) )
    done = []
    trunk = ""

    while segs:
        done.append( segs.popleft() )
        trunk = "/".join( done )
        if not path.exists( trunk ):
            mkdir( trunk )

Thanks for the info!

That’s some really useful pathing reference! Which I can see is what I asked for… Though I did mean to ask for more Maya specific code on what commands you run at what point (DOH!). Anyhow, I’m sure I’ll figure that side of things out :slight_smile:

Thanks both!

Uhhh… This look interesting:

cmds.filePathEditor()

a 2013 Extension addition.

This should let me know about and re-path external file references (refs, textures, audio and image planes) in one central place, likely on save. Whoot!

The chosen solution contained in this snippet of communication with Maya Support:

I need all external file references to be project/workspace relative, like this:

“d:/p4/project/branch//construction/characters/mayafile.ma “
->
“construction/characters/mayafile.ma”

Dirmap will not allow me to map from something to “” (nothing)

If I could have used environment variables, I could have used dirmap:

“d:/p4/project/branch//construction/characters/mayafile.ma “
->
“%PROJECT_PATH%/construction/characters/mayafile.ma”

Alas I need to FBX files to/from Max and MotionBuilder and both do not support environment variables in file paths

Had I wanted to do repathing with maya.cmds while the scene is loaded, cmds.filepathEditor() will handle re-pathing of the most common external file references, in one central place (new in 2013.5), but the side effect would be that the user would have to sit around and wait for resources to reload, after they are re-pathed.

The chosen solution we went with is:
• Save as .ma
• On post save/export callbacks run Python function that
o loads the Maya file and does a search and replace

Super-fast and easy (like 0.04 seconds for a 12 meg file)

Forgive me for perhaps hijacking, but I’m trying to tackle a very similar problem. We use .mb files, so post-save fixups of the bytes in the file themselves aren’t really an ideal option. I’ve been trying to do some kind of pre-save remapping of any file reference from the absolute form to a relative form (or one including an environment variable reference).

Using the Python API, I’ve been able to install a pre-save hook (via MSceneMessage.addCallback() with kBeforeSave) and within that callback get all the files via MFileIO.getFiles(), but I’m having trouble actually changing any of the file references to the values I want. What’s the best way to actually change the references via the Python (or Mel) API? I’m using Maya 2011, if that is relevant.

(I’m a programmer, not an artist, so I have very little familiarity with Maya itself)