PyTorch in Nuke: Building a Next-Node Predictor
March 10, 2025 · 6 min read

Inspired by Paul Ambrosiussen's Tensorflow experiment with next-node prediction in Houdini, I built NodeScout: a plugin that trains a graph neural network on your own Nuke scripts.
I wanted to know if it might actually spit out relevant suggestions, given my extremely limited dataset of nuke scripts (pulled off the internet, forums, and from my own sad nuke scripts).
This is how the project was structured:
- Nuke Plugin: A panel for training and predictions.
- Worker: A small C++ program, linked against the libtorch that Nuke ships. It handles training and prediction in its own process.
- Model: A Graph Attention Network (GATv2).
Most of the interesting problems ended up being on the Nuke side.
So that's where I'll start off...
How to run PyTorch on Nuke's libtorch
Nuke's Python doesn't ship the torch module, and installing it into Nuke's own site-packages isn't great either!
Training would also block Nuke's UI if it ran in-process.
So the model runs in its own process, and the plugin talks to it:
- The plugin collects the graph upstream of the selected node.
- It turns the graph into features, and sends them to the worker as JSON.
- The worker replies with the most likely node types.
- The panel lists them, and double-clicking one creates it in the graph.
My first version gave that process its own Python.
- create a venv, then
pip install torch. - That venv ended up at 5 GB, a lot of bloat for a silly plugin.
Nuke's bundled libtorch
Luckily, Nuke already forced the user to download PyTorch!
Nuke uses PyTorch under the hood for its machine-learning tools, like CopyCat and the Inference node. Nuke 17 ships libtorch 2.7.1 (PyTorch's C++ libraries) right in its install directory, along with the CUDA libraries it needs:
$NUKE_DIR/libtorch.so
$NUKE_DIR/libtorch_cpu.so
$NUKE_DIR/libtorch_cuda.so
$NUKE_DIR/libc10.so
$NUKE_DIR/libc10_cuda.so
...
What it doesn't ship is the Python side: the torch package, and libtorch_python.so, the library that binds
PyTorch's C++ core to Python.
Skipping python torch entirely
My next attempt grafted that Python half onto Nuke: install the torch wheel without its NVIDIA dependencies,
then symlink its libtorch to Nuke's. It still downloaded the whole 1 GB wheel, though, and since Foundry's libtorch is
its own build, the official bindings only worked because torch binds symbols lazily, so one missing function would
kill the process the moment anything called it.
But libtorch is a complete C++ library on its own: tensors, autograd, optimizers, and torch::nn modules.
- So the final version can skip Python torch entirely, instead running the model in a small C++ program linked directly against Nuke's libraries.
As a result, nothing needs to be downloaded at runtime.
- The program links only Nuke's own libraries, and finds them through Nuke's install directory on
LD_LIBRARY_PATH. - The build just fetches the official 2.7.1 headers, since Nuke doesn't ship them.
target_link_libraries(nodescout PRIVATE
"-Wl,--push-state,--no-as-needed"
"${NUKE_ROOT}/libtorch.so" "${NUKE_ROOT}/libtorch_cpu.so" "${NUKE_ROOT}/libc10.so"
"-Wl,--pop-state")
The one gotcha is --no-as-needed. Nothing calls libtorch.so directly, so the linker drops it by default, but
it's the library that loads the CUDA backend.
- Without it, the worker silently runs on the CPU.
Turning Nuke scripts into graphs
Training needs a lot of node graphs.
Launching Nuke for every script and walking it with the Python API could work, but it's slow.
.nk files are plain text (TCL), so we can just parse them directly!
I've written up how Nuke's .nk file structure works separately. The short version is
that connections aren't stored as a list of edges.
- Instead, each node pops its inputs off a stack, so the order nodes appear in the file determines how they connect.
Getting this right took a lot of trial and error. Some examples:
- Clones: Clones are written as references to their original node, so they need special handling.
- Group inputs: A group's
Inputnodes map to its inputs by theirnumberknob, rather than the order they appear in. - Nested groups: Groups (and gizmos) get flattened, so the model ends up seeing one graph, wired straight through.
- Passthrough nodes:
DotandNoOpnodes should be collapsed as they don't change the image.
Reusing the serializer
Training graphs come from .nk files on disk, but predictions come from the graph that's live in Nuke.
If those two paths serialize the script differently, the model could see graphs at prediction time that look wildly different from the training set.
The plugin walks upstream from the selected node with Nuke's Python API, and
rebuilds the same intermediate structures the .nk parser produces.
- Both get passed through the same serializer.
- Flattening, passthrough removal, input order, etc. etc. all match!
Building training examples
Each training example is three lists, which Python writes to a JSON file for the trainer:
x: A[num_nodes, num_features]list, with one row of features per node.edge_index: A[2, num_edges]list of connections.- Row 0 holds source node indices, and row 1 holds target node indices.
- An edge from node 3 to node 5 is the column
[3, 5].
y: The answer: the index of the node type that actually came next.
It's the same layout PyTorch Geometric (PyG) uses, which is where the model started out.
Each script produces many examples:
- Pick a target: Step through the script's nodes (every third one), treating each as the node to predict (
y). - Collect context: Walk upstream from the target, breadth-first, collecting up to 50 nodes.
- The closest nodes get collected first.
- Targets with fewer than 5 upstream nodes are skipped, since there isn't enough context to learn from.
- Build features: Each node gets a short feature vector:
- Its type (as an index into a vocabulary of node types)
- Its number of inputs, and how many of them are connected
- Its position in the node graph (
xpos,ypos), normalized per graph
- Build edges: Each connection between two collected nodes becomes a column in
edge_index.
At prediction time, the selected node is included in the context, since it's the node you're adding after.
The model
Nuke scripts are already graphs, which is why the model is a graph neural network. Each node gathers information from the nodes wired into it, a few rounds over, so it ends up carrying some context about its neighborhood. The model then averages those into one vector for the whole graph, and scores every node type by how likely it is to come next.
I prototyped it in Python with four off-the-shelf GATv2Conv layers from PyG, which learn how much weight to give each upstream node, with a small MLP on either side.
Resources
- PyTorch Geometric Documentation - Comprehensive library for graph-based deep learning.
- The PyTorch C++ API - libtorch's tensors, autograd and
torch::nn, from C++. - The Foundry's ML-Server - Framework for integrating ML with Nuke.
If you want to have a look, you can find the tool here.
Related Posts
Automatic Lens Distortion in Nuke from a Single Frame
A Nuke node that estimates lens distortion from one frame, with no grid or camera move, using a fine-tuned AnyCalib network fitted to NukeX's lens model.
Understanding Nuke's '.nk' file structure
Understanding and parsing Nuke's '.nk' file structure