Patching a mel command at runtime

A lot of Maya studios want to extend their Maya UIs, adding custom elements to menus or marking menus or the like.

Unfortunately, by ancient tradition, most of the Maya UI is implemented directly in MEL – so if you want to edit items you’re often reduced to finding out how to add your little bit to a menu created by some funky callback that dynamically creates the menu every time. This is hard enough that a certain studio I know you’ve heard of used to maintain a giant folder with most of the most important Maya UI files for every version they supported – installing there tools was “copy this over the contents of your maya install.”

This is not ideal on any level – but it happens for a reason.

So if you are stuck wanting to (a) edit your vanilla Maya UI and (b) retain your dignity, you can generate your own runtime replacement for the native MEL which adds insertion points where you can stick your custom stuff without having to the touch the vanilla install. The solution comes in two parts:

First, you need to extract the original mel function. Luckily Maya will tell you where the original function lives, and it’s fairly easy to extract the text:

import maya.mel
import re

def extract_mel_definition(procName : str) -> str:
    """
    Extract the original text of mel proc, so it can be redefined under a
    new name.
    """

    s = maya.mel.eval(f"whatIs {procName}")
    if not "found in" in s:
        raise RuntimeError (f"{procName} is not an original Mel procedure")

    local_copy = s.split("found in: ")[-1]

    depth = 0
    with open(local_copy, "rt") as melFile:
        text = melFile.read()

    # could modify this to also get local procs.  Worth it?
    header = re.search("global\W+proc\W+" + procName + ".*\n", text)
    counter = header.end()
    for char in text[header.end():]:
        depth += char == "{"
        depth -= char == "}"
        counter += 1
        if depth == 0:
            break

    return text[header.start(): counter]

That scrapes the text of the original Mel function into a string.

Now you can synthesize a new Mel function with the same name:


def patch_mel(procName : str, before_proc : str = None, after_proc : str = None, replace_proc: str = None):
    """
    Patches an existing MEL command, retaining the name and signature but adding
    optional callbacks which will fire before and/or after the original code.

    The patched command must have been loaded from disk (presumably from the normal Maya
    install, though this will work on anything which has an on-disk source)

    Note that if the original command relies on local commands, it may be necessary to
    replicate them.

    """

    # reset the command to its original state
    maya.mel.eval(f"source {procName}")

    proc = extract_mel_definition(procName)
    replacement_name = procName + "_orig"
    pre_callback_name = procName + "_before"
    post_callback_name = procName + "_after"

    if before_proc and pre_callback_name not in before_proc:
        raise ValueError(f"callback name should be: {pre_callback_name}")

    if after_proc and post_callback_name not in after_proc:
        raise ValueError(f"callback name should be: {post_callback_name}")

    if replace_proc and f"global proc {procName}" not in replace_proc:
        raise ValueError(f"callback name should be: {procName}")


    # the original proc logic becomes the "_orig" function,
    maya.mel.eval(proc.replace("proc " + procName, "proc " + replacement_name))
    success = "entered interactively" in maya.mel.eval("whatIs " + replacement_name)
    if not success:
        raise RuntimeError (f"failed to redefine {procName}")

    # synthesize a new procedure with same signature;
    signature = re.search("(.*)\(.*\)", proc).group(0)          # -> "global proc (string $a, string $b)"
    arg_forward = re.search("\(.*\)", signature).group(0)
    arg_forward = re.sub("(string \$)|(int \$)|(float \$)", "$", arg_forward)
    arg_forward = re.sub("\[\]", "", arg_forward)               # ->  ($a, $b)


    if not replace_proc:
        # callback genertor
        replace_proc = signature
        replace_proc += "{\n"
        replace_proc += '\t' + pre_callback_name + arg_forward + ";\n"
        replace_proc += '\t' + replacement_name + arg_forward + ";\n"
        replace_proc += '\t' + post_callback_name + arg_forward + ";\n"
        replace_proc += "}\n"

    local_signature = signature.replace("global proc", "proc")

    pre_callback = before_proc or local_signature.replace(procName, pre_callback_name) + "{}\n"
    post_callback = after_proc or local_signature.replace(procName, post_callback_name) + "{}\n"

    maya.mel.eval(pre_callback)
    maya.mel.eval(post_callback)
    maya.mel.eval(replace_proc)

This function gives you three options: before_proc is a MEL proc which fires before the script you are replacing ; after_proc fires after the original; and replace_proc replaces the original entirely. Depending on your level of MEL tolerance these could be self-contained MEL functions or they could just be two-liners that call python() on your real code. When this script executes the original procedure is renamed and the replaced by the new one which whatever mixture of callbacks and replacement you specified. The rest of Maya won’t know anything has changed and for the duration of the session your updated func is the real code.

If you need to revert during a session, you can retrieve the code from the previous step by simply sourcing the original name.

def unpatch_mel(procName: str) -> str:
    maya.mel.eval(f"source {procName}")

MEL is like getting a colonoscopy, it’s sometimes necessary but not something you want to do more than is strictly necessary. Patching your MEL UI at startup time is way better than fiddling with the on-disk contents of your Maya directory.

4 Likes