Rust Blender Extension API with Hot Reloading

Blender and Rust are two of my favorite things. Over the years, I’ve tried a few ways of combining them, and here I’d like us to retrace the steps that led to one way that I’m particularly fond of:

A Blender extension that calls reloadable Rust code. Find the project with build instructions on GitHub.

As an example, this extension takes a mesh object and creates a new object consisting of point samples that are inside the mesh. To detect the inside, the extension leverages generalized winding numbers.

Note that you could achieve this entirely with Geometry Nodes in default Blender by chaining a Mesh to Volume Node with a Distribute Points in Volume.

Motivation

Let’s assume the following. We have our idea for Blender and want to build the core logic with Rust. We prefer Rust over Python, maybe because of performance, or because we want to reuse some other Rust code, or maybe just because we think that Rust strikes the best compromises of any programming language out there!

We don’t want to create a custom Blender build from source (in this case, C/C++ would probably be the better choice), and we expect to tinker in the core (Rust) implementation for the most part of the development. Also, and this is key, the application programming interface (API) is not going to change much.

In essence, we want to use Rust and have fast development cycles.

Making Choices

There are many ways to get this done. I’d like to mention one important alternative right now: (re-)start a separate local server process. If your API is going to change a lot, you might want to look into that instead.

Blender’s Extensions

Let’s make it our first goal to avoid rebuilding Blender from source. A good way to achieve this is to build an extension. Extensions are easy to distribute and install. Extensions are written in Python, and we get an extensive API with Blender’s Python Module (bpy). We can write up our extension and bundle it into a ZIP file with Blender (from the command line). This ZIP file can then be installed from within Blender.

Within the Python part, we need to do three things:

1. Register and build the user interface
2. Read and write Blender’s Data
3. Call our core logic

Python is good, but we want Rust for the core logic!

Wheels Make it Go!

So, there’s our next challenge: how can we bridge between our beloved Rust and Python, which seems to be a mandatory stopover. Since both languages have been very popular for years, it’s unsurprising that a mature solution exists: PyO3.

It seems natural to wrap our core logic into a Python package, a wheel. This allows us to import it in other Python code for automatic testing. More importantly, this is the default way for Blender extensions to load binary code. PyO3’s maturin will turn our Rust code into a wheel file.

This file needs to go into our extension’s source tree, and we need to list it in the extension’s manifest; it’ll be bundled with the ZIP file, and we’ll be able to import our module from the extension code!

Let’s visualize where we are at.

Great, right? Nope, it’s horrible!

we expect to tinker in the core implementation for the most part of the development

The issue is that this approach doesn’t allow us to iterate on changes in the core logic with any speed to speak of. And it might be worse than you think.

Dynamic Libraries are too Static

First, let’s go back to how we “magically” get to call our Rust code from Python. A crucial detail for the wrapping library crate is that it needs to have the crate type cdylib, meaning it’s going to create a shared library with the binary code compiled from our Rust code. Then, the Python side can request that this shared library be loaded by the operating system (OS), find functions by name (there’s a lookup table), and call them. That is, assuming that the application binary inferface (arguments, etc.) matches what the Python side expects.

Dynamic and shared libraries refer to the same thing, but the OS prefers to think about them as shared. The OS would rather have different calls, threads, and processes reuse and share the library as much as possible. This is nice, but it hinders us from swapping it out for a new version! While it is possible to unload a shared library in theory, it isn’t easy to do reliably in practice. For example, the dlclose function Linux provides won’t work if any of the library’s symbols are still “in use”.

We finally arrive at the issue: we cannot change anything in our core logic and use it in our running Blender process. Instead, we have to shut down Blender, rebuild everything, and start again. Only to find that some cache still has our old version! Explicitly deactivate our extension, rinse, repeat.

This is annoying, especially if you have to reproduce some state in Blender to test a particular thing.

Finally, Reloadable Code

Actually, for the Python side, hot-reloading is readily available. We can use VSCode with this super helpful addon for extension development. Not only does this alleviate the need for explicitly building the ZIP with Blender on the command line (and installing it from disk), but it also gives us the option to reload our Python code!

So, that is the Python side done.

For the Rust side, we can go with David Wheeler: “All problems in computer science can be solved by another level of indirection.” There is a hot-lib-reloader solution in pure Rust that we can use between our core logic and our wrapping crate!

I’m not entirely sure how it works, but I believe the rather cheeky trick is to constantly make new names for the shared library that we want to be reloadable. This way, the operating system is forced to load the latest version from disk. Of course, the upper layers of our Matryoshka extensions are none the wiser, which is good!

So, that is the Rust side done, as well!

Live With the Choices

Preventing Race Conditions

Actually, we’re not quite done with the Rust side. With our trickery, we’re angering the race-condition-gods, and we need to place at least one synchronization primitive to appease them.

Unfortunately, there are several subtle ways to mess this up because there is a thread from the hot-lib-reloader whose task is to replace our library, and it acts asynchronously. Here are some ideas of what might go wrong:

• Some of the binary code we’re replacing is currently running
• We have handed references to some structs in Rust to Python, and now their layout has changed

And there’s more: even though Python would traditionally allow only one call to our module at a time, due to its global interpreter lock (GIL), that isn’t necessarily the case anymore. And in our case, we’re going to get parallel calls from Blender and its render engine.

For once, we’re happy about the persistence of the shared libraries. It’s not possible to pull the shared library from under our feet while we’re still using it!

We’re not going to deal with all issues in a foolproof way, but the following tricks get us there most of the way.

1. Use a “context” trait to expose core logic
2. Allow a single context instance to exist at one time
3. Protect this instance with a mutex
4. Explicitly synchronize the library-reloading-thread

We think about this context as our reloadable library. Now we’re jumping ahead to show a few relevant places in the code: we’re left with a single function that needs to be hot-reloaded:

#[unsafe(no_mangle)]
pub fn create_context() -> Box<dyn Context> {
    println!("creating new rust context");
    Box::new(Impl)
}

And the spicy part of the code that deals with reloading looks like this:

let lib_observer = rust_hot_reload::subscribe();
loop {
    // wait for reload and block it
    let update_blocker = lib_observer.wait_for_about_to_reload();

    // wait for any library calls to finish and block further calls
    let mut context_guard = LOCK.lock().unwrap();

    // cleanup
    let context = context_guard.take();

    // this should block until any threads are joined and any resources are dropped
    drop(context);

    // let the library update commence and wait until it's finished
    drop(update_blocker);
    lib_observer.wait_for_reload();

    // fresh context with new lib version
    *context_guard = Some(rust_hot_reload::create_context());

    // context_guard is dropped and calls can continue
}

The middle part here is where we can still mess up. If we have any threads or resources in our core logic that might cause trouble when it’s replaced, they need to be joined and cleaned up when the context is dopped in a blocking fashion.

The Catch

I’ve hinted at this a few times; now it’s time to stop beating around the bush. Even with our fancy-schmancy reloading schemes, we should not hot-reload if we change our API (aka. context trait). If we do this anyway, all bets are off, and undefined behavior (UB) ensues. Let me point to the hot-lib-reloader blog post again for further caveats. Whether this partial reloadability is worth giving up certain safety guarantees is for you to decide.

Of course, we can regain the safety we’re losing as soon as we compile without hot reloading.

Overview of the Code

Here, we examine the top-level structure, summarize the build and reload process, and conclude with testing the hot reloading.

We have four different Rust directories (crates) serving various purposes and a single python directory. Let’s visualize this!

1. rust_api

   This crate defines context trait; it’s not reloadable.

2. rust_core

   This crate implements the context trait; we implement the core logic here, and it’s reloadable.

3. rust_hot

   This crate exposes one reloadable function that creates an instance of the context (and re-exports the context trait). We don’t need to edit this; it only exists to facilitate splitting the API and core logic. This crate is also the one we’ll rebuild for reloading.

4. rust_wrap

   This crate is on the boundary to Python. Here, we rely on PyO3 to create a Python module and also utilize the hot-lib-reloader to orchestrate the reloading of the context creation function.

For the first build and, whenever we change something in the API, we let maturin create a new wheel from rust_wrap and build rust_hot using the standard cargo command. If Blender is running a prior version of our extension, disable the extension and shut down the Blender process as well.

However, for any subsequent changes to our rust_core, we only need to rebuild rust_hot using the standard cargo command.

See it in Action

Now we can tinker in our core logic to our heart’s content and immediately see the result in Blender. Check this out:

Conclusion

First, let’s address the drawbacks: this approach is clunky to get started, and it doesn’t alleviate the growing pains of the API. If we reload with a changed API, we’re toast (we get UB). And if we’re not careful in cleaning up our context, we might get bitten by zombie threads.

But once it’s up and running, we can write Rust code as if it were a natively supported language! We have direct function calls without any serialization overhead and benefit from the Rust ecosystem to accomplish nearly everything we want, from high-performance numeric parallel GPU tasks to complex networking.

Furthermore, we could build on existing solutions that work and primarily focus on plumbing. Shoutout to PyO3, the Blender Development addon, hot-lib-reloader, and, of course, Blender, Python, and Rust! ❤❤❤

Thank you for reading!