I haven't posted in a while so figured I would start off with one of my weird side projects. There is an indie video game creator named Eric Hartman who goes by BadSpot online. He created Blockland and Age of Time. These games are special to me because Blockland was what got me into software development (which is now my career) and Age of Time is the video game I met my wife on.

Although Blockland went on to become a successful retail game selling thousands of copies Age of Time however faded into obscurity. Because of this I was curious if anyone even player Age of Time anymore. The official server is still online and you can still connect to it but it has been empty every time I connect.

Well I decided to create a bot that would connect to the Age of Time official server and bridge chat over into Discord so I could monitor player activity. It also will track player statistics to a MariaDB database (kills, deaths, arrests, race times, etc). I wanted to see if it was possible to use Node-RED as the brains for the whole project while having the game client feed data into it using scripts written in the game engine's native language.

You can view this results of this effort over on the Unofficial Age of Time Discord group. You can also download Age of Time and go interact with the bot yourself.

TorqueScript

This is the scripting language used inside Torque Game Engine that both Age of Time and Blockland are built on top of. It's actually quite a powerful scripting system that allows people to make plugins and mods for games built using Torque Game Engine. It should also be noted that LLMs such as ChatGPT and Claude are actually pretty good at generating working TS code.

Now TorqueScript does have various limitations and can be quite cumbersome to do some simple tasks. So I wanted to find some way to have an external system communicate with the game client instead. Originally I was going to use HTTP connection objects to communicate with an external system but then I stumbled across the TCPObject class. This is actually a far better way of communicating because I can connect to a remote system and send/receive updates live. I also remembered that Node-RED has built in support for TCP connections. Since Node-RED is a low-code environment for easily prototyping and connecting different things it seemed to be a perfect fit for this. I'm already using Node-RED to automate all of my smart home and always looking for new an interesting ways to use it.

Building a Node-RED client in Torque Script

This was an interesting task. I needed something that would connect to a remote TCP server running on Node-RED and provide methods for easily retrieving and sending messages. Because of this I created the NodeRED Integration for Torque Script.

This allows you to register callbacks for incoming messages as well as send outgoing messages. It also handles automatically reconnecting with a connection backoff for repeated failures. Usage example:

    getNodeRED().connect("localhost:1881");
    if(!getNodeRED().hasReceiveCallback("socket_data_in")) {
        getNodeRED().addReceiveCallback("socket_data_in");
    }
    getNodeRED().send("player_message|Skylord|Hey Buddy!");

    function socket_data_in(%msg) {
        error("Received: " @ %msg);
    }

Now we can send and receive data from Node-RED we need a sane way of serializing the data we send over to Node-RED.

Serializing the data

TorqueScript didn't really have anything out of the box to do this so I had to find something that would work. JSON would be great since Node-RED is NodeJS based is has great built-in support for it already. Problem is TorqueScript has no native support for JSON. Luckily qoh over on GitHub had already ran down this rabbit hole and created jettison.cs for handling this.

Now Jettison works very well for serializing JSON but deserializing it was incredibly slow. Luckily data coming back from Node-RED is usually very simple so I opted for doing a basic format:

<action>|<data>

Essentially converting the data into a string and concatenating it with pipes |. This sped up the communication significantly and also is a bonus for being easy to diagnose similar to JSON.

So for example we run this function whenever we get an incoming player message:

function TrackPlayerMessage(%local, %name, %msg)
{
	if(!$BOT::ENV::ENABLED || !$BOT::ENV::NODE_RED::SEND_MESSAGES) {
		return false;
	}

	if($BOT::ENV::NODE_RED::SEND_MESSAGES) {
		%data = JettisonObject();
		%data.set("action", "string", "player_message");
		%data.set("isLocal", "boolean", %local);
		%data.set("name", "string", %name);
		%data.set("message", "string", %msg);
		NodeRED::send(jettisonStringify("object", %data));
		%data.delete();
	}
}

Important note: Make sure you delete the Jettison objects after sending them as TS will not garbage collect them after references are dropped. I figured this out when my game client would slowly creep up in memory usage over time until it crashed.

And here is an example of us parsing incoming messages from Node-RED:

function socket_data_in(%msg) {
	if(%msg $= "server_data") { // return server statistics
		sendServerData();
	} else if(%msg $= "game_data") {
		sendGameData();
	} else if(%msg $= "reload_scripts") {
		%reload=exec("base/PlayerTracker.cs");
		NodeRED::send("reload|" @ %reload);
	} else {
		error("Unknown NodeRED command received: " @ %msg);
		NodeRED::send("command_unknown|" @ %msg);
	}
}

// bind callback if it hasn't been yet
if(!NodeRED::has_receive_callback("socket_data_in")) {
	NodeRED::add_receive_callback("socket_data_in");
}

So now we have communication between our game client and Node-RED we can start making our Discord bridge and handling statistics.

Telnet Console

Another cool thing I found while digging through old Torque Game Engine source code was the telnetSetParameters method that lets you enable a telnet server you can connect to to view and send commands to the in-game client console.

I added the ability to run commands on the bot from Discord and wanted to be able to output the console lines that appear while the command is running. This way you can get sort of a full console experience from within the Discord chat.

So I have the following lines in my Torque Script to enable this functionality:

// start telnet console
if($BOT::ENV::TELNET::ENABLED) {
	telnetSetParameters($BOT::ENV::TELNET::PORT, $BOT::ENV::TELNET::CONSOLE_PASS, $BOT::ENV::TELNET::LISTEN_PASS);
}

Now Node-RED can connect to this telnet port with a provided password and have complete access to the console using the following flow:

Node-RED telnet client for Torque Game Engine console

Node-RED telnet client for Torque Game Engine console - flow.json

Gist262588213843476

Now we can easily inject console commands and get the full response. Way better solution over using eval() from Torque Script (which seems to only return the last defined variable in the called function if no return is defined).

Node-RED

There was various iterations before I came to this layout. At first I was using link out and in nodes that come with Node-RED and did not like how it looked. I actually ended up creating node-red-contrib-event-listener so I could convert incoming messages to events. This allowed me to easily break down the logic for handling player messages, server messages, etc.

Part of these flows is tracking statistics into a MySQL database. This way you can see the last time someone connected, their total kills/deaths, how fast they completed Log Challenge, etc. I even added in-game chat commands that people can use to query for this information. For example a player can say !lastseen PlayerName to see when someone was last online.

I also went a step further and made it so whenever someone completes an Age of Time race the bot will send a message about how well you did versus other players:

Myndi finished the Level 1 Race in 16:45.37
DiscordBot: Finished in position 9/12 overall, 1/1 among your attempts. Best time: 146.11s, your best: 1005.37s. Congrats!

The reason I added this was because the built in tracking system for this is bugged. If the server is up for more than a specific number of days it starts counting time incorrectly. People that complete challenges during this time are able to get a really buggy number. BadSpot doesn't really reset this very often so I wanted a better way of tracking this for players (and for my own curiosity to see what is actually possible in this game).

Running the game client in Docker

So now we have a bot that sends and receives data to Node-RED which then stores it in a database and forwards player/server messages to Discord. Now we need a way to run the client that takes very little resources and can run headless in a Docker container.

For this purpose I created aot-wine-x11-novnc-docker.

Limiting CPU usage

Docker containers do have built in support for locking containers to specific cores and that is nice but I want to go even further and prevent the container from using more than 10% of a single core.

So to accomplish this I turned to cpulimit which is a linux tool for limiting a process's CPU usage. I tested various different CPU levels until I found the lowest value that wouldn't crash the game client.

I went further and made it so that there is a cpulimit file you can set a number into to change the limit dynamically. I then hooked this up into Node-RED so that when a player connects it ups the limit so the bot has access to more resources. When everyone disconnects except the bot the CPU limit it restored back to the minimum.

If you are interested in diving into the code for this it can be found here.

Another interesting find I made was with setting the resolution to an invalid amount. This causes the game window to just disapear and stop rendering saving us even more CPU cycles. The VNC server that the container runs is at a very low resolution so if you try to go fullscreen it actually fails to do so but the game client continues running just fine albeit at a much lower CPU usage since rendering has essentially stopped.

Handling game crashes

I am using supervisord in the container to auto restart Age of Time client when it crashes. I started running into the issue that the game client doesn't fully crash and instead will pop up the winedbg window. Because the process didn't exit supservisord never restarts it.

To fix this I updated the script mentioned before for handling the cpulimit to also look for a window with title Program Error and if it exists force kill it.

    AOT_PID=$(pidof AgeOfTime.exe)

    # if "Program Error" dialog found kill aot and winedbg
    # as supervisor thinks it's still running
    if wmctrl -l | awk '{$3=""; $2=""; $1=""; print $0}' | grep '^\s*Program Error$'; then
        kill $AOT_PID
        kill $(pidof winedbg)
        sleep 2
        continue
    fi

Now when winedbg detects the crash and pops up the winedbg window we just force close it causing supervisord to detect the crash and restart the game client.

Conclusion

I'm actually pleasantly surprised at how easy it was to throw something together using the Node-RED's low-code environment. I've been running it for several years now without any major issues.

I built a docker container for running the game instance, Torque Script code for running a bot and connecting to Node-RED, Node-RED flows for sending messages to discord and tracking data, and finally a module for Node-RED to make handling events much easier just to keep my flows sane. It was an interesting project and touched on a bunch of things that I really enjoy messing with.

It actually sparked some interest into Age of Time for old players since we added it to a Discord channel with a bunch of veterans. It was fun to see everyone hop on and say hi to each other again. That alone made the project worth it to me.