Creating a Multiclass Classification Endless Runner Car Game

Let's go through the process of making an amazing car driving game with random road segments and multiclass classification in the browser.

Motivation

I had a project in mind to predict car crashes with machine learning and data analysis. The initial idea involved the installation of sensors in vehicles and the collection of extensive data to understand driver behavior. However, due to the high associated costs, this approach seemed impractical. I got excited by the potential of the emerging technology known as the Internet of Things (IoT).

At this juncture, a need arose to explore more cost-effective alternatives. Thus, the concept of creating a car crash simulator took shape. Following a tutorial, I successfully simulated car movement on a road. However, delving into more complex aspects such as multi-car interactions and collision deformation akin to systems like Rigs of Rods or BeamNG.drive proved to be intricate and demanding endeavors, requiring significant investment of time and effort. Consequently, a decision was made to maintain simplicity, resulting in the development of a basic car racing game featuring low-poly cars.

An intriguing notion was to replicate real-world road distributions dynamically, inspired by navigation apps like Waze. OpenStreetMap in conjunction with the Overpass API emerged as a potential solution for this task, although it was left as a prospect for future work. To maintain simplicity, the road generation process was kept random. Nonetheless, technical challenges surfaced; instances arose where the car's speed surpassed the rendering capability of the device, leading to pavement rendering issues. Employing an object pooling technique was a possible remedy, but this approach made the road pattern predictable and monotonous. Ultimately, the frequency of road generation was fine-tuned through trial and error, seeking an optimal balance that could potentially be hardware-dependent.

Tools

• Blender
• Godot Engine
• GDScript
• TensorFlow.js
• JavaScript

Making-of

I was guided by this video tutorial for doing the groundwork:

https://www.youtube.com/watch?v=B5vE-nNszxA

3D Models

I made my own car in Blender and changed the car of the tutorial. I had no idea of how to do this, I only played with the tool putting basic shapes. I took the following course on Udemy for learning a bit: Low-Poly Vehicle Design in Blender for Unity Game Developers.

I've been an enthusiast of blender since I was a teenager, it prompted me to learn Python with the hope of doing games like Yo Frankie!.

And this is the car I made:

As you can see the car is composed of basic shapes with cuts and some extrusions. I had to set up the origin of the wheels and the handle to make these objects rotate around their center. After the job is done I export the model in glTF format. Here are my export options:

In format I selected glTF embedded, and in Geometry Appliy Modifiers to export the modifiers that I applied in blender like the mirror modifier to do half of the work in some cases and get better symmetry.

Video game programming

Now let's go with the waited Godot, in the main scene there is a track, this is made of curves filled by polygons, the polygons have a gray image with yellow dashed lines as the material. I imported the car and put it in the center of the scene, when the game runs we start adding the segments the car will walk on, the car runs over these segments and subsequently others will be generated.

The Godot game engine facilitate a lot of tasks, it done all of the physics of the car for me, I only had to program the rotation of the handle. When the car left the pavement the program activate a counter and when this counter reach a threshold the game is over.

Natural user interface

At this point there is a typical game but what if the game gets a bit more immersive, I was amazed playing the Pac-Man demonstration with TensorFlow.js. I modified the source code and removed one class from the classifier to predict only three: right, left and up. The game exported with Godot is embedded in an iframe and the web page has to pass the classified pose to the iframe, for this I used the method postMessage(). The data set used for training is little, it only contains a bunch of photos of me, with more photos of different people it will produce more accurate results and prevent overfitting.

In Godot we receive the message from the classifier in the following way thanks to the JavaScript Interface:

var _callback = JavaScript.create_callback(self, "_on_gesture")

func _ready():
	JavaScript.get_interface("window").addEventListener('message', _callback)

func _on_gesture(args):
	var message = String(args[0].data)
	if message == "left":
		steer_val = 0.3
	elif message == "right":
		steer_val = -0.3
	else:
		steer_val = 0

To send a message to the parent container we use the following code:

var window = JavaScript.get_interface("window")
window.parent.postMessage(score)

This way the game runtime is capable of sending messages to the page, although it runs in an isolated iframe with WebAssembly and WebGL.

The tensorflow classifier has a memory leak, when it runs the memory usage grows and grows until my computer freezes, the next image illustrates the problem:

Sorry, my system monitor was in Spanish but this is easy to understand: Memoria is Memory and Intercambio is Swap.

After searching the internet I found a solution that worked for me. I enclosed the prediction loop with the functions startScope() and endScope() of the tensorflow engine.

/**
 * @license
 * Copyright 2018 Google LLC. All Rights Reserved.
 * Copyright 2022 RealisMagical. All Rights Reserved.
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

async function predict() {
  ui.isPredicting();
  while (isPredicting) {
    tf.engine().startScope();
    const img = await getImage();
    const embeddings = truncatedMobileNet.predict(img);
    const predictions = model.predict(embeddings);
    const predictedClass = predictions.as1D().argMax();
    const classId = (await predictedClass.data())[0];
    img.dispose();
    ui.predictClass(classId);
    await tf.nextFrame();
    tf.engine().endScope();
  }
  ui.donePredicting();
}

With this the memory consumption was stabilized, now we have a steady amount of memory after the game initializes, as depicted in the following screenshot:

Narrative of the game

In a distant realm that bridged the boundaries between the earthly and the ethereal, a wondrous place known as "Heavenly Highways" existed. Here, the spirits of once-beloved cars found eternal joy as they soared on celestial roads, embodying the very essence of freedom and exhilaration. These spirits retained their vehicular forms, shimmering with vibrant hues and radiant energy, eager to traverse the breathtaking landscapes that stretched as far as the eye could see.

In this enchanting haven, a unique and vibrant spirit named Auroride resided. Auroride had been a cherished vintage car, cherished by its owner during its time on Earth. Now, in the Heavenly Highways, it embodied the epitome of happiness. Its radiant aura exuded a sense of euphoria, and its tireless engine hummed with an enchanting melody, infusing the very air with an atmosphere of joy.

Auroride delved into a series of journeys through realms that mirrored various landscapes of its earthly existence. Each realm was a virtual representation of the car's cherished memories. From bustling city streets to tranquil countryside roads, Auroride revisited the moments that had defined its earthly life. Through these experiences, players would navigate challenges and overcome obstacles.

As players progressed, they would unlock new abilities and customizations for Auroride, adding to the car's radiant splendor and enhancing its gameplay capabilities. These upgrades symbolized the joy and growth that the spirit had experienced throughout its existence.

The game blended elements of adventure, racing, and puzzle-solving, creating an immersive experience that resonated with themes of joy, nostalgia, and the enduring connections between humans and their cherished vehicles.

"SpiritRide: Road to Heavenly Bliss" offered players not only a visually stunning and gameplay-rich experience but also a poignant narrative that celebrated the power of happy memories, the thrill of the open road, and the eternal bond between a spirit and its earthly vessel. As players guided Auroride on its journey, they would come to understand that even in the afterlife, the spirit of a car could find happiness by embracing the memories that had once fueled its earthly adventures.

Source code

The source code of the game is available at: https://github.com/realismagical/spirit-ride

The game project is completely free and open source under the MIT license. I promise to improve this over time. Any suggestion or contribution is welcome.

2023-07-14