This article looks at how we can keep reusable code within a Houdini scene, and why we would want to do that. There's a download at the end but stick around for the initial discussion as it covers a few important points about:
Why storing code in the scene can be a good idea (and better than storing it on disk).
What sort of code you would and wouldn't want to store in your scene.
So first, let's look at why we might want to keep code in the Houdini scene file itself.
Version Managing Your Code And Scene
I need to mention that this article will be coming from the perspective of a TD working on a show. It's important to state this early on because the needs of a TD are very different from someone who is writing a pipeline. The key differences are:
TD Code
TD code forms part of the artistic content of the scene and is instrumental in delivering content to the client.
The code and its functionality evolve as the scene progresses with little concern given to previous behaviours.
Code may completely change for specific shots or assets.
The behaviour of a Houdini scene is often directly tied to the code it uses (e.g. a custom particle solver).
Used by a small number of people on a single project.
Pipeline Code
Pipeline code is all about infrastructure, and it takes no part of the creative deliverable.
Code behaviour is consistent and often written in a way to be backward compatible.
Functionality is homogenous across all assets and shots.
Tends to take more of a behind-the-scenes role.
The same code is deployed across multiple projects.
The takeaway here is that for TDs, the code can be just as much part of the scene as the Houdini nodes. So it seems like a good idea to keep it within the scene itself.
Let's quickly look at what the issues are with storing code externally to the scene.
Why Storing TD Code Externally Can Be Bad
At first glance, storing code externally in a file located on your $PYTHONPATH can also seem like a reasonable way to go. It means we can put the code in a Git repository and properly version manage it. It also allows us to easily share it with everyone on the job. Great, so what's the downside?
Imagine you have a shot where your Python code is used in a custom particle DOP Solver. You've saved the code in a directory on disk and you're using the Python "import" statement to import it into your solver node.
You render your latest version and get some feedback from the director that means that you need to completely change the behaviour of the solver. So you rewrite your code and commit the new version into your Git repo and send your scene off to the farm to simulate and render.
All good right? Not so fast!...
Later, you hear back that the director also wants to see a slightly tweaked version of the simulation that he saw earlier. You need your old code back. So you go back to Git and revert your changes to make a small tweak to the code. Oh... but hang on, you still have your new render running.
Too late, you've rolled back. Now that new render is broken as it has just automatically picked up the change you made when you reverted the code. Oh, and guess what... you've also just noticed an email from a co-worker to say that their mammoth 5 day render broke when you made that commit earlier on. Oops!
So I imagine you may well have seen that coming, right? And so instead, you might have done something like branching your code into a new repository located on the shot, or renamed the Python file to something like "my_solver_shot_002.py". Okay, so that might work. But let's imagine you're now a month into this show and the director says he wants to see a slight change to a render you did right at the beginning. Can you remember which version of the code it was that generated the particle behaviour in that particular scene from a month ago?
It's not a nice situation. Every scene file you make will have a dependency on a specific version of code. If the same code is shared between multiple scenes, you quickly run into problems where you might need the behaviour of the code to change for just one of the shots.
When you take it to the limit, the only thing you can do is to have every scene load its own specific Python file. In which case your Python files will be named something like "my_solver_shot_002_myscene_v001.py" which is clearly ridiculous. Not to mention, how the heck do you handle making a bug fix or managing code-based look-dev across all the different versions you're managing?
Version management of code is not easy and it's because of dependencies. If you've been coding a while, I'm sure you'll have come across the phrase "DLL Hell", if not then I'd encourage you to follow that link and get a feel for some of the problems you can run into.
Other reasons to avoid storing code externally:
In some cases, it may complicate deployment to an external cloud-based render farm, as you'll have to make sure your code is available remotely.
Sharing the scene file with other companies (e.g. out-sourcing vendors) now means you need to supply the external code too, and it might be tricky for them to get everything working as expected.
The functionality of any external code should be kept to things like pipeline, shelf-tools, basic automation, and simple library functions. These are all things that should be written once and aside from enhancements and bug fixes, the fundamental behaviour shouldn't need to change during the project.
Now we've got that out of the way, let's look at the alternative.
Storing Code In Your Scene
Keeping TD code in your Houdini scene ensures that the behaviour of the scene is defined by the scene itself and will never change because of something changing external to your scene file. This is a good thing as it removes the dependency issues we already discussed.
So that just leaves us with a few questions:
Where do we put our code?
How do I reuse code within my scene file?
How do I share my code with others or across shots?
Before we examine one way we can do this, let's just take a quick look at something which I'm always surprised people don't use more frequently.
Python Source Editor
The Python Source Editor (found under the Windows menu) lets you write Python code that will be saved into your scene file. Every time the scene file is opened, the source code will be evaluated automatically.
This means that you should nearly always place any code inside a function when using this editor, as most of the time, you'll want to call the code at a time of your choosing.
Let's look at an example. Open the Python Source Editor and enter this
def hello_world():
print "Hello world!"Click Accept or Apply. If you now open the Python Shell (also from Houdini's Windows menu), you can type this:
>>> hou.session.hello_world()
Hello world!So you could, in theory, call this code from anywhere in your scene. That's a good step in the right direction.
But we still lack the ability to share this code between other shots and artists.
A Different Approach
So what's a better way? Let's come at the problem from a workflow point of view. If I'm a TD, I want to be able to do the following:
Do something equivalent to "import my_td_module" from anywhere in the scene.
Keep the module code saved within the scene.
Be able to easily share this code with other people on the same job.
In Houdini, what things are kept in the scene, but can be easily shared? You've guessed it: HDAs.
We can create a Python Module HDA that has string parameters for the name of the Python module as well as the code for it. If we put these parameters under a multi-parm, it lets us have more than one Python module on the same node.
It might look something like this:
We'll get into how we might share this later, but first, let's look at how we can implement this HDA.
Python Module HDA
To use the Python "import" keyword to load a module from this HDA, we need it to create the module for us and put it in "sys.modules".
We'll also need to add callbacks to the HDA so the module is updated whenever we change the code, as well as when the node is loaded in a new scene.
Updating Modules on Parameter Change
First, let's address updating "sys.modules" when the module name or code is changed. To do that, we just add a callback like this to the relevant parameters:
hou.phm().load_module(kwargs)If you've not come across "kwargs" before, it's just a dictionary that Houdini provides containing useful information about the triggered event. If you're interested in seeing more, try putting:
print kwargsin here instead and see what happens.
Then, we create a Python Module script in the HDA called "load_module" that might look something like this:
import sys
import imp
def load_module(kwargs):
node = kwargs["node"]
idx_str = kwargs["script_multiparm_index"]
module_name = node.parm("module_name_" + idx_str).eval()
code = node.parm("script_" + idx_str).rawValue()
if not module_name or not code:
return
new_module = imp.new_module(module_name)
exec code in new_module.__dict__
sys.modules[module_name] = new_module
On our HDA, the parameter names are "module_name_#" and "script_#" where the hash is the number of the multi-parm instance. It means we have to replace the hash with a "1"-based index, which we can get from the "kwargs" dictionary that Houdini passed to us using the "script_multiparm_index" key.
The last three lines deal with creating the module, evaluating the code in the new module's namespace, and then storing the new module in the "sys.modules" dictionary to make it available for import using the "import" keyword.
Updating Modules On Scene Load
To deal with loading the modules when a Houdini scene is loaded, we need to add another function to the HDA's Python Module to load all the modules:
def load_all_modules(kwargs):
node = kwargs["node"]
module_count = node.parm("module_count").eval()
for idx in reversed(xrange(module_count)):
idx_str = str(idx+1)
kwargs["script_multiparm_index"] = idx_str
load_module(kwargs)
Here, we're iterating over the multi-parm in reverse order and calling the existing "load_module()" function for each module that exists on the node.
Why create the modules in reverse multi-parm order? Well, this helps in the situation where a TD wants to import a module that's also stored on the same node under a different tab in the multi-parm.
From an intuitive point of view, I think most people would expect to have the module containing the highest-level functionality in position 1 of the multi-parm tabs as it's the first one you see.
By iterating in reverse, it means that modules located in tabs towards the left of the UI can import modules towards the right. That's just how my brain works, but of course, do what makes sense for you.
We can now call that code from an "OnLoaded" callback for the HDA, like this:
import hou
node = kwargs["node"]
hda_mod = node.type().hdaModule()
hda_mod.load_all_modules(kwargs)This technique of getting the node type's "hdaModule()" is nice because it means we can keep all our code in one place and call it from the other callbacks.
If you create this HDA yourself (or download it from the link below), add some code and specify the name of the module, you should be able to import this code from anywhere inside Houdini by using the Python "import" keyword in the Python Shell.
We're done, right? Not quite! There are still a few things we need to be aware of and protect against.
Caveats and Cautions
Trying to import code from one Python Module HDA node to another has a 50:50 chance of failing. There's no guarantee on the order that the nodes are executed in, so the module might not have been created when you try to import it. If you have a dependency between modules, they should be created on the same node.
Conflicts. What happens if another Python Module HDA defines the same python module but with different code? Which wins? Again, it would be fairly arbitrary which evaluates last. I'll come back to solving this problem further down.
The module name could do with a proper regex validation to avoid spaces and other invalid characters appearing midway through the name. For simplicity, that code has been left out in the listings above. Something like this would work:
module_name = re.sub(r"[^a-zA-Z0-9_]", "", module_name)Additional validation should be done to avoid using an existing module like "os" for the module name. Imagine the confusion if you replaced it with your own code! I'd recommend adding a special attribute to the modules created by this HDA which can be tested for before overwriting its entry in "sys.modules".
If I were deploying this within a pipeline, I'd also want to have a way to clear out old modules from old scenes that are still loaded in "sys.modules". Placing something in a callback for when a scene is loaded or cleared would be a good place to do this, but that has to happen at the pipeline level and isn't something an HDA is able to take care of.
The additional validation code has been added to the downloadable HDA if you want to see how this might be done. Also, a full code listing is presented at the bottom of this article.
Sharing Code
So how would we share this code with another artist? Will it work if we put our Python Module HDA into another digital asset and share that?
Unfortunately, sharing the Python Module HDA isn't quite as easy as just bundling it into another HDA. That's because once locked inside an HDA, callback events such as "OnLoaded" and "OnCreated" no longer get triggered on our Python Module HDA.
The responsibility for getting the Python Module HDA to create its modules lies with the enclosing HDA. It will need to call the "load_all_modules()" function itself at the appropriate time.
You can call the Python Module HDA's "load_all_modules()" routine for a particular Python Module node like this:
# Get the node type
obj_cat = hou.objNodeTypeCategory()
node_type = obj_cat.nodeType("apn_python_module::1.0")
# Get the HDA's python module stored inside
hda_mod = node_type.hdaModule()
# Call our function using Houdini's kwargs convention
kwargs = {"node": hou.node('/obj/apn_python_module1')}
hda_mod.load_all_modules(kwargs)Here, I'm finding the node type using the "objNodeTypeCategory" class, getting the HDA's internal python module, and then making my own "kwargs" dictionary to point to a specific Python Module node in the scene.
If that sounds like a bit too much like hard work, then you can just make a "Reload All" button and set the "load_all_modules" as its callback, like we did the other parameters.
Now, if you want to share a Python Module HDA inside another HDA, just make sure you add an "OnCreate" and an "OnLoad" callback to it. Inside both callbacks, you should either press the "Reload All" button on the Python Module HDA or call the "load_all_modules()" function.
A call to press the button inside the event callback is super simple and would look like this:
hda_node = kwargs["node"]
hda_node.parm("apn_python_module1/reload_all").pressButton()Now, not only are you able to access your code anywhere in your scene, but you can also share your code with other people and make use of the HDA versioning system in Houdini to switch between different versions as you see fit.
Almost done. One more thing to guard against:
Python Module HDA Conflicts
Let's revisit what I mentioned above about the potential conflict that arises if we have multiple Python Module HDAs in the scene, all trying to create their own Python module with the same name but different code.
So how do we get around this issue? There are a number of ways, but the simplest would be to create a hidden attribute in the module and store the scene path to the Python Module HDA that created it.
When another Python Module HDA creates a new module, it can check to see if it already exists in "sys.modules" and if it does, it can find out which node created the module. If the node exists in the scene and the code is different, then we can show an error to the user to warn them. If the code is the same, then there's no issue. If the node no longer exists in the scene, then this node can create the new module and take ownership.
This will allow multiple instances of the same code in the scene and will complain as soon as any incompatibility occurs. Of course, there are other strategies you could use and you might find another method suits you better.
We can also use the presence of this hidden attribute to identify which Python modules are ones created by the Python Module HDA, and use it (as described earlier) to make sure we don't overwrite modules like "os".
I've added all this validation checking to the downloadable HDA and you can also see the full code below.
Download
This is the final interface of the HDA:
I've added a couple of buttons for reloading, as well as an Auto-Update checkbox in case the automatic Python compilation ever gets problematic (e.g. when reporting errors).
If you want to try this node out for yourself, you can download the complete HDA below. As the disclaimer below says, this hasn't been production tested yet, so there may be some issues I'm not aware of. Do take appropriate caution!
Python Module HDA Code
This is the fully commented contents of the final Python Module script inside the HDA.