Using C++ as a Scripting Language, part 12

8 min readFeb 3, 2024

RPC, but with less drudgery

Hello again. Welcome to another crazy-mans blog post about using C++ as a scripting language sandboxed in a low-latency RISC-V emulator.

Previously

Modern RPC is often schema-based message serialization over some kind of convenience E2E library. Eg. gRPC. Depending on your program/architecture, it might be the best and easiest option. It also might mean you have to implement everything 4 times over, if you have a game script with host/guest-side implementations and then also the server/client-side protocol message implementations.

Implementing everything 4 times over is a lot of work. Let’s be real.

Protobuf messaging

Protobuf and other schema-based libraries are extremely convenient if you only have to implement them to serialize the data once, and deserialize it once.

message Message {
  oneof cmds {
    // Initial responses from server
    IntroResponse intro_resp = 1;
    IntroError    intro_err  = 2;
    // Intros & Chat
    IntroducePlayer intro_player = 3;
    PlayerLeft      player_left = 4;
    GlobalChat      chat = 5;
    ...
}

The start of a protobuf message that can contain one of each of those sub-messages. I’m using it for my client-server protocol. This is very convenient as there’s lots of stuff going on behind the scenes that needs a message. Extending and modifying the protocol has become very easy.

However, in order for scripts to talk to each others, I began thinking in terms of engine-events, so that I ended up designing things in my game script to try to take advantage of event-generating side-effect-like behavior in order to more easily trigger remote events that would lead to eg. sounds playing on other nearby clients. Sounds awful, right?

So, I thought about ‘what if I can just send a byte array message’ to nearby clients? That would be an easy protocol message to make. But, not only did I write the game engine, I wrote the emulator the script is running on, and also the scripting system on top of it. Not recommended. One of the decisions I made early is to link statically, so that everything has a fixed address. I also send each script program from the server to each client on login. They all share the same exact programs, and memory layout.

RPC in sandboxed C++

For my use-case, I wanted something tiny that carries just enough information to be useful in 99% of cases. But it needed to be made such that the end-result would improve my day-to-day incremental work on my game. I didn’t want a large switch case with a protocol message for every little thing. Well, I say that, and that is exactly what I have now: hundreds of different messages. It would be incredible if I could avoid all the network protocol drudgery, while also making it possible to for example create 100 particles with just one message. Instead of let’s say 100 messages, or one big message. Gives me the mega-shader jeebies.

An example: Whenever the player destroys something, particles should be created, and some sound to commemorate the occasion. One could imagine that each block is different, and creating engine-side frameworks, or even just custom callbacks to handle this is literally busywork. However, if it’s made easy enough, it could be a 5 minute job.

Right now, whenever a block is destroyed, the game engine calls a script function to let it determine what it means:

PUBLIC void stdRemovedBlock(int x, int y, int z, Item item, Block removed)
{
 // ... create particles and sound locally

 Game::remote_call(x, y, z, [item, g] (int x, int y, int z)
 {
  if (Game::is_server()) return;

  GridWalker gw(x, y, z);
  gw.remove();

  const vec3 center = {x+0.5f, y+0.5f, z+0.5f};
  for (int i = 0; i < 100; i++)
  {
   Particles::spawn(item, center, vec3::randNormal3(0.1f), g);
  }
 });
}

And after executing the local portion, we can just tell everyone else to do the same, except the server. This particular example is assuming clients are not antagonistic. Note: I’ll never have PvP and this is a sandbox game. I don’t trust clients, but I’m allowing them to make any change they want inside their own instance (piece of the world). And the script is fully sandboxed.

So, what’s happening here? The remote_call() function takes a C++ lambda, which can have capture storage. And that capture storage can be forwarded across the network. Other clients are running the same exact program, with necessarily the exact same lambda function, and so we can execute the same lambda, with the same capture storage. All you need is a fixed-capture-size function wrapper.

I looked around a little bit, and I see that at least one implementation before me has implemented RPC with lambda capture, namely UPC++. The library seems like a high-quality abstraction for serious work in distributed computing and HPC. Their RPC seems to also be able to forward complex arguments to remote hosts, unlike my simple variant which primarily carries the location of the caller (x, y, z) and the by-value lambda capture. One distinguishing feature here is that I am doing this from sandbox to sandbox, maintaining its integrity.

struct Data {
    char str[64] = "Hello, world!";
} data;
Game::remote_call<Data>(gw, data,
[] (GridWalker gw, Data& data) {
    print("Remote call at ", gw.getX(), ", ", gw.getY(), ", ", gw.getZ(),
          " with data: ", data.str, "\n");
});

The above prints some ‘Hello, world!’ on each client near the RPC location:

[std] says: Remote call at 542, 127, 549 with data: Hello, world!
[std] says: Remote call at 538, 127, 549 with data: Hello, world!
[std] says: Remote call at 539, 127, 549 with data: Hello, world!
[std] says: Remote call at 540, 127, 549 with data: Hello, world!

The template variant (shown above) allows passing larger amounts of data, but would require you to serialize your complex data. I still haven’t really used it, as the fixed-size 24-byte lambda capture has been large enough for all my remote calls so far. I mostly transport floats and integers, and this can capture 6 of them, while also passing the location (x, y, z) outside of the capture. But hey, maybe the server can broadcast assets with the template variant?

Redundant messages

As a consequence of this implementation, I am making a list of redundant protocol messages that I will remove in the future, eg. playing a sound at a (x,y,z). Playing sounds is usually a message that happens after another sequence of events, anyway. Like above, the removal or building a block. In some cases, one could deduce (guess) when a sound should be played, in order to reduce traffic. But we want to have all the nice things. The script allows us to group these things together into a single function that just does the right things, no matter how few or many there are. Even player footsteps would be OK to handle this way, as turning it into a script function allows us to expand on it later (eg. adding simple particles).

Ignoring footsteps, the end result of this type of RPC is a general reduction in network messages. And, of course, it makes sending remote work between server and clients, and vice versa, completely trivial. It’s insane how easy it has become, really.

The network message to make the above RPC is fairly small. I measured the block removal function call to be a 79 bytes protocol message. I know I can make it smaller with some protobuf trickery, but it’s OK. It plays the right sound, creates ~100 particles, as you saw earlier.

Errors have natural context

If something goes to shit, we can see the offending function with backtrace in the log, unlike a random protobuf message. So, what do I mean by that? Well a random protobuf message might tell you it failed to find/produce a particle due to unknown ID or something, but it will not tell you the purpose of producing it. Unless you send metadata with each message. When you call a function, it‘s going to be in debuginfo, meaning it has a demangle-able name. Also, you can always try to deduce line information too. Just pick a simple debuginfo format. Bottom line: Pick your complexity level, but it’s going to work.

The proof is in the pudding:

    Game::remote_call(gw,
    [blk, pcnt] (GridWalker gw) {
      asm("unimp");
    });

Let’s crash when walking creates sound and particles. From another client:

Exception when calling:
  api::Game::remote_call(api::GridWalker, Function<void (api::GridWalker)>)::{lambda(int, int, int, void*, unsigned long, void*)#1}::_FUN(int, int, int, void*, unsigned long, void*) (0x401344)
Backtrace:
-> [0] 0x0000000000401770 + 0x014: void Function<void (api::GridWalker)>::trampoline<stdPlayerMoved::{lambda(api::GridWalker)#1}>(Function<void (api::GridWalker)>::Storage, api::GridWalker)
-> [1] 0x0000000000401344 + 0x040: api::Game::remote_call(api::GridWalker, Function<void (api::GridWalker)>)::{lambda(int, int, int, void*, unsigned long, void*)#1}::_FUN(int, int, int, void*, unsigned long, void*)
-> [-] 401344 + 0: api::Game::remote_call(api::GridWalker, Function<void (api::GridWalker)>)::{lambda(int, int, int, void*, unsigned long, void*)#1}::_FUN(int, int, int, void*, unsigned long, void*)

Exception: Unimplemented instruction executed  (data: c00)
>>> [0x401784] c0001073 UNIMP
>>> Machine registers:
[PC   401784] [RA 00401384] [SP 21641FB0] [GP 004413D0] [TP 00441238] 
[LR 00000004] [TMP1 00000035] [TMP2 00000000] [SR0 FFFFFFFFFFFFFEB7] [SR1 0001002E] 
[A0 21641FC0] [A1 6A00000023] [A2 00000004] [A3 01541FE0] [A4 00000009] 
[A5 00401770] [A6 21641FC0] [A7 0000FFFF] [SR2 FFFFFFFFFFFFFF69] [SR3 00000052] 
[SR4 0000FFFF] [SR5 00000000] [SR6 00000000] [SR7 00000000] [SR8 00000000] 
[SR9 00000000] [SR10 00000000] [SR11 00000000] [TMP3 00000006] [TMP4 00000038] 
[TMP5 00407580] [TMP6 00000038] 
Program page: [0x0000000000401784] Readable: [x]  Writable: [ ]  Executable: [x]
Stack page: [0x0000000021641FB0] Readable: [x]  Writable: [x]  Executable: [ ]

It’s crashing in:

void Function<void (api::GridWalker)>::trampoline<stdPlayerMoved::{lambda(api::GridWalker)#1}>(Function<void (api::GridWalker)>::Storage, api::GridWalker)

Which we can shorten to:

stdPlayerMoved::{lambda(api::GridWalker)#1

Mystery solved. It’s also possible to automatically start GDB in order to remotely debug the sandbox when it crashes, but I have yet to need it. I could also spam some line information using any of the dwarf libraries. Again, you can pick your debuginfo format using -g<format>.

Conclusion

So, what can we say about this kind of RPC?

It transmits type-safe data by-value, and optionally larger serialized data
It lets us program remote clients and the server instance, directly from inside the script
— Specifically, you can only reach your own instance of the world and other players within that instance
It allows tracing and debugging remote events, where they used to be random pieces of protocol messages.
— Easily see the source of an error without having to start an investigation into which part of the server sent it.
It is only safe to implement if it is backed by a reliable sandbox
It’s probably a good idea to restrict it to the server when clients are considered antagonistic

Considering how often I’ve been using it since I made it, I know that I struck gold.

-gonzo