Ever have a game server lock up but not fully crash so it doesn't trigger a restart? I've been having this issue recently with my Sven Co-op server and I found a great fix using Node-RED that I want to share with others. It can be applied to almost any game server easily.
This post is designed to restart a game server that is running in a container (whether it be on the same host or on another host) but you can easily modify this to run any sort of restart command or process you want if your game server isn't running in a container (such as running SSH terminal commands on the specified server to get the job done).
Requirements
- Node-RED installed
- node-red-contrib-gamedig installed in Node-RED (I made this package, contributions are welcome!)
- node-red-contrib-dockerode installed in Node-RED
- Some sort of game server running in a container you want to restart.
Configuring node-red-contrib-dockerode in Node-RED
So first off you need to configure node-red-contrib-dockerode in Node-RED to connect to the docker server running the game server you want to be able to restart. This Node-RED plugin gives us access to a bunch of container data and calls that we will be using to get the job done.
If the game server and Node-RED are installed on the same server and the docker server is configured to use a socket file (which is usually the default setup) you can simply pass the /var/run/docker.sock
(this is the path on Unraid, it may differe for your system so make sure to look it up) to the Node-RED container and configure node-red-contrib-dockerode to use that file.
Otherwise you will need to enable the remote API for the docker server the game server is running on. You can see this page for reference on how to do this (you will need to search to figure out how to do it on your specific system). You will then need configure node-red-contrib-dockerode to point to the host and port of the docker host and you are good to go (test it to make sure).
To configure dockerode place one of the node's from the package in the working area and double click it. From there it should get you access to configuring the plugin.
The Flow
The Node-RED flow is super easy and builds on top of my other post for tracking game server data and storing it in InfluxDB (you can actually combine both of them to build a nice server tracker & auto restart bundle).
Here is the flow. Import it using the hamburger menu in Node-RED:
[{"id":"c0975f20.5aa74","type":"function","z":"5ef5d69c.ad1238","name":"Offline Check","func":"if(msg.payload != 'online') {\n msg.sven_server_try_count = flow.get('sven_server_try_count') || 0;\n msg.sven_server_try_count += 1;\n} else {\n msg.sven_server_try_count = 0;\n}\n\nnode.status({fill:(msg.sven_server_try_count > 0 ? \"red\" : \"green\"),shape:\"dot\",text:\"failed tries: \" + msg.sven_server_try_count});\n\n\nif(msg.sven_server_try_count >= 12) {\n return msg;\n}\n\nflow.set('sven_server_try_count', msg.sven_server_try_count);","outputs":1,"noerr":0,"x":1360,"y":720,"wires":[["9265b9a.83ca448"]]},{"id":"5771e1a.627672","type":"docker-container-actions","z":"5ef5d69c.ad1238","name":"Container Restart","config":"d06d964c.564c68","container":"SvenCOOP","action":"restart","cmd":"","x":1990,"y":700,"wires":[["5c8bd477.5115dc"]]},{"id":"9265b9a.83ca448","type":"docker-inspect","z":"5ef5d69c.ad1238","name":"","config":"d06d964c.564c68","container":"SvenCOOP","x":1570,"y":720,"wires":[["6efb6cc5.e56364"]]},{"id":"6efb6cc5.e56364","type":"switch","z":"5ef5d69c.ad1238","name":"Is container running","property":"payload.State.Status","propertyType":"msg","rules":[{"t":"eq","v":"running","vt":"str"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":1790,"y":720,"wires":[["5771e1a.627672"],["d46d1cf5.c94bb"]]},{"id":"5c8bd477.5115dc","type":"function","z":"5ef5d69c.ad1238","name":"format email","func":"var newMsg = {\n topic: \"Sven server auto restart\",\n payload: \"Sven server auto restarted after failure to query \"+flow.get('sven_server_try_count')+\" times.\"\n};\nreturn newMsg;","outputs":1,"noerr":0,"x":2170,"y":700,"wires":[["f2284380.bf9bf","d46d1cf5.c94bb"]]},{"id":"f2284380.bf9bf","type":"e-mail","z":"5ef5d69c.ad1238","server":"smtp.mailgun.org","port":"465","secure":true,"tls":true,"name":"[email protected]","dname":"Send email","x":2350,"y":700,"wires":[]},{"id":"d46d1cf5.c94bb","type":"change","z":"5ef5d69c.ad1238","name":"","rules":[{"t":"set","p":"sven_server_try_count","pt":"flow","to":"0","tot":"num"}],"action":"","property":"","from":"","to":"","reg":false,"x":2070,"y":760,"wires":[[]]},{"id":"b31d892a.fe6d08","type":"comment","z":"5ef5d69c.ad1238","name":"Restart Sven Coop server if offline for too long and send email","info":"","x":1500,"y":680,"wires":[]},{"id":"346c9ca7.040934","type":"query-game-server","z":"5ef5d69c.ad1238","name":"","server_type":"hldm","host":"server.skylar.tech","port":"27018","halt_if":"","max_attempts":"","socket_timeout":"","attempt_timeout":"","x":1130,"y":720,"wires":[["c0975f20.5aa74"]]},{"id":"51b352b8.0cbb3c","type":"cron","z":"5ef5d69c.ad1238","name":"Every 20 seconds","crontab":"*/20 * * * * *","x":890,"y":720,"wires":[["346c9ca7.040934"]]},{"id":"d06d964c.564c68","type":"docker-config","z":"","host":"/var/run/docker.sock","port":"2375"}]
This is an exact copy of what I am using for auto restarting my Sven Co-op server. In order for you to use it we will need to change some nodes to match your setup:
- Change interval to however often you want the server to be checked. Also change how many attempts are tried on the server before a restart by modifying the
Offline Check
function block. - If you change the flow variable used to track failed server query counts you need to update it in the various nodes it is used in. These are the
Offline Check
function node,Format Email
function node, and the change node at near the end of the flow that resets the value back to zero. - Both the
Container Inspect/Health
andContainer Restart
nodes need to be changed to point to your docker server and the target server container (It shows a nice autocomplete field with what containers are on the host which makes it easy and hard to mess up). - The email message in the
Format Email
function block needs to be updated to contain the text appropriate for your server.
Once you get those changed you now have a functioning server restart tool built inside of Node-RED!
Conclusion
This tool has been super handy for the various servers I host (especially ones that run custom maps that can cause some strange issues). I hope others find it useful for keeping their servers online as well.
One thing I want to add in the future is so it emails me which map was last playing on the server when it crashed. I store all my server data in InfluxDB already so it is as simple as writing a query to get that data. This would be handy for my Sven Co-op server that has over 2,400+ maps that I just haven't had time to test all of them. This makes it so now it will log problem maps to my email for me to investigate further later.
If you have any feedback, suggestions, or need help feel free to post below. I love to hear back from my readers.