An Introduction to Low-Latency Scripting for Game Engines, Part 3

Scripting with Nelua, a systems language

Scripting with Nelua

Originally I thought I would spend part 3 just writing down manual function call and argument handling, but I realized that I’m not using manual argument handling anymore. It’s all replaced with convenience functionality. So, I thought why not write about something else.

Nelua (Native Extensible Lua)

Nelua transpiles to C, and is then compiled and linked. Exactly what you want if you want to maintain a fast scripting environment, while at the same time script in a simple Lua-like language.

If we keep the same host program as before, and just replace the RISC-V script executable can we use Nelua? It turned out to be fairly easy.

You can find the whole project here. Remember to modify build.sh to use your particular RISC-V compiler!

1. A build script

First we need a build script that takes a Nelua program and builds a static RISC-V executable.

#!/usr/bin/env bash
set -e
CC=riscv64-unknown-elf-gcc
source="${1:-program}"

cfile=.build/output.c
binfile=.build/output.elf
mkdir -p .build

nelua --ldflags="-static" --cflags="api.c -O2 -g3 -Wall -Wextra" -o $binfile $source.nelua

The only out-of-the-ordinary argument here being that I’m passing the C API in cflags. Couldn’t figure out how else to add it. The program also needs to be statically built.

2. A C API

I wrote a C API that instantiated 32 functions that calls out to the host. I did this in order to not have to do anything in the C API other than it being there. So, think of C API as containing numbered functions that call into the game engine, each passing its number. dyncall1 -> first callable in the game engine. And so on. This is the API file if interested.

These C functions can be externed by other files (and even other languages), and at that time we can decide what kind of prototype the symbol will have. What I mean by that is, even if I “say” that dyncall1 is of type int(const char*) in api.c, that doesn’t mean anything in another file. And on top of that, in this case, the ultimate meaning comes from how the game engine expects the function to work. As an example, dyncall1 can be a void(void) in api.c, it could be int(int) in myprogram.nelua, and finally, it could be uint32_t dyncall1(uint64_t, int, int, int) in the game engine. One of these we can safely ignore, and that is the definition in api.c. It’s enough that it exists. It makes sure the function invokes the game engines callable with the same number. Now we just have to make sure that our Nelua program uses the same prototype as the engine expects.

3. A Nelua API

This is where we will import the C functions instantiated by the C API, and pretend that they are no longer void() but instead match our expectations in the game engine. For example, the first callable is expected to be int(int).

So, no matter how we end up implementing the first callable, it should call dyncall1 with an integer argument, and take an integer as return value.

require 'string'

global MyData = cstring

global api = @record{}
global api.exit: function(status: integer)
global api.dyncall1: function(n: integer): integer
global api.dyncall2: function(s1: string, s2: string)
global api.dyncall_empty: function()
global api.dyncall_data: function(data: MyData)

local function fast_exit(status: cint): void <cimport>
end
api.exit = function(status: integer)
  fast_exit(status);
end

local function dyncall1(n: cint): cint <cimport>
end
local function dyncall2(str1: cstring, size1: csize, str2: cstring): void <cimport>
end
local function dyncall3(): void <cimport>
end
local function dyncall4(data: MyData, size: csize, data: MyData): void <cimport>
end

api.dyncall1 = function(n: integer): integer
  return dyncall1(n)
end
api.dyncall2 = function(s1: string, s2: string)
  dyncall2(s1, #s1, s2)
end
api.dyncall_empty = dyncall3
api.dyncall_data = function(data: MyData)
  dyncall4(data, 1, data)
end

That’s the whole example API, written in beginners Nelua. Yes, I’m a total beginner. In it we create a namespace for the API called simply api, and it’s a record (a table?) that contains functions. Afterwards we import a few C API functions and we then implement the Nelua API by calling those C functions according to what the host engine expects.

For example, to create the first int(int) function, we import the dyncall1 C function, and then implement api.dyncall1 so that it calls that function:

-- Import C function
local function dyncall1(n: cint): cint <cimport>
end
-- Nelua API function
api.dyncall1 = function(n: integer): integer
  return dyncall1(n)
end

We are forwarding the return value from dyncall1 back to the caller of api.dyncall1. We have an extra step because the types cint and integer are not necessarily the same.

With this we can now require("api") in our Nelua main file and use the API:

require("api")

print("Hello from Nelua!")

local n = api.dyncall1(0x12345678)
api.dyncall2("Hello, Vieworld!", "A zero-terminated string!");

print("dyncall1 returned: ", n)
api.exit(0) -- prevent global destructors from running

local function test1(a: cint, b: cint, c: cint, d: cint) <cexport, codename 'test1'>
  print("test1 called with: ", a, b, c, d)
  return a + b + c + d
end

local function test2() <cexport, codename 'test2'>
  local a: [4]integer <volatile> = {1,2,3,4}
end

local function test3(str: cstring) <cexport, codename 'test3'>
  print("test3 called with: ", str)
end

local Data = @record{
  a: int32,
  b: int32,
  c: int32,
  d: int32,
  e: float32,
  f: float32,
  g: float32,
  h: float32,
  i: float64,
  j: float64,
  k: float64,
  l: float64,
  buffer: [32]cchar
}
local function test4(d: Data) <cexport, codename 'test4'>
  local str: cstring = tostring(&d.buffer[0])
  print("test4 called with: ", d.a, d.b, d.c, d.d, d.e, d.f, d.g, d.h, d.i, d.j, d.k, d.l, str)
end

local function test5() <cexport, codename 'test5'>
  local d : MyData = "Hello, World!"
  api.dyncall_data(d)
end

local function bench_dyncall_overhead() <cexport, codename 'bench_dyncall_overhead'>
  api.dyncall_empty()
end

I think this is fairly good!

Some noteworthy things:

1. If we want a function to be callable from the engine, it needs to be marked as cexport.
2. We shouldn’t return normally from main() as Nelua doesn’t want to print anymore. Instead we use api.exit(), which is noreturn.
3. Printing a fixed-size buffer was not easy to figure out, but it turned out that you can convert it to a cstring:
   local str: cstring = tostring(&d.buffer[0])
4. The regular Nelua integer is 64-bit. So, use int32 when appropriate.

And that’s about it. With this, we can now run the Nelua program just like our C++ program from the previous blog posts.

Hello from Nelua!
dyncall1 called with argument: 0x12345678
dyncall2 called with arguments: 'Hello, Vieworld!' and 'A zero-terminated string!'
dyncall1 returned:  42
>>> myscript initialized.
test1 called with:  1 2 3 4
test1 returned: 10
Call overhead: 11ns
Benchmark: std::make_unique[1024] alloc+free  Elapsed time: 16ns
test3 called with:  Oh, no! An exception!
test4 called with:  1 2 3 4 5.0 6.0 7.0 8.0 9.0 10.011.0 12.0 Hello, World!
Benchmark: Overhead of dynamic calls  Elapsed time: 12ns
dyncall_data called with args: 'Hello, World!' and 'Hello, World!'

The performance seems to be quite good, running this on my laptop!

This example can be found under the libriscv gamedev example.

I’ve implemented “scripting” for pretty much every systems language, and even a few others that you can imagine. Some languages are designed for this purpose, like Lua, but it is not a systems language. Now, perhaps we can add Nelua to that list? It is likely the simplest systems language to work in. Next time I will try to write an example Nim program.

-gonzo