Tools to develop C++ CodinGame Bots Efficiently

When I started to develop bots on CodinGame, I quickly missed some features to be efficient. For example, I wanted to pass specific options to the compiler, split my code into multiple files, and run the bot in a debugger. I thus decided to write, debug and run code locally. Since I work on my computer, I could also build and use tools to improve my productivity, as I am accustomed to doing when I code. I know that setting up a C++ project on a computer is not the easiest and the funniest thing to do. This setup can repel some people that would thus avoid working locally.

To help you quickstart your CodinGame activities on your computer, I wrote a C++ starter project and a VS Code extension. This article will tell you how to install and use them. You will also see some of the tools I built, such that you could code your own.

Note 1: Some scripts, functionalities, and code are missing from this article. One of the reasons is it is advised on CodinGame forum not to distribute tools that will fetch replays. Since the website API is public, you can easily download your last matches to analyze the decisions made by your bot. However, if many people would do it without considering the workload it imposes on CodinGame servers, the API would not be public any longer. Other tools are missing because I did not write them correctly yet. If you want to share some ideas, feel free to contact me 😄. Finally, some tool gives too much competitive edge to be shared. It would be better for you to craft your tools to be more efficient.

Note 2: Until very recently, I developed exclusively on Linux. As such, I may have overlooked specificities on Windows. If you encounter any issues or have suggestions to improve Windows support, let me know.

1. Installation

This section will guide you to install the required tools and the VS Code extension I wrote. That VS Code extension expects you to install the following on your computer:

• VS Code (of course 😆)
• C/C++ VS Code extension,
• cmake to generate a build configuration for your projects,
• ninja to build your projects; this generator is fast and provides compile_commands.json which is used by IntelliSense,
• compdb to extend the compile_commands.json file with headers, to improve the indexing done by IntelliSense,
• clang to perform the indexing done by IntelliSense.

Additionally, to use the starter project I provide, you need:

• lld, an efficient C++ linker (you can get rid of this requirement by modifying one line in the CMakeLists.txt file),
• python, to execute scripts.

On Linux, your package manager will install everything for you, except the C/C++ VS Code extension you can get from VS Code itself.

On Windows, start by downloading and running installers for VS Code, cmake, ninja, and python. Then install compdb, and make sure it is in the search path (on my computer, the path to add was C:\Users\atom\AppData\Roaming\Python\Python39\Scripts). Next, download and run an installer found on LLVM. This installer will take care of clang and lld, among other nice things.

Now we can install the two VS Code extensions. In VS Code, type Ctrl+Shift+X to open the extensions panel. Then type C++ in the search box, find an install the following extension:

C/C++ extension for VS Code, a nice extension you must use in VS Code!

For the CodinGame extension I wrote, you have to download in on the github repository. Then type Ctrl+Shift+X again in VS Code to open the extension panel, click on the three dots at the top right, and select Install from VSIX... in the menu. Select the downloaded extension, and let VS Code install it for you.

The last thing you need is a starter project. This project will serve as a basis for new bots you will create. Have a look at the requirements for such a project in the README of the extension repository on GitHub to create your own, or fetch a copy of the one I provide.

2. Configuration

The VS Code extension has some settings documented here. Most of the settings are related to the organization of your projects. Mine are organized like so:

• codingame/ which is my root path, and its absolute path is stored inside the setting rootPath
  • bot1/ the folder for the bot bot1
  • bot2/ the folder for the bot bot2
  • ...
  • tools/ where I store everything that is not specific to a bot
    • include/ contains all my generic headers. If you have a different organization, put the path of that folder into the includePath setting
    • lib/ contains all libraries I use in my tools. If you have a different organization, put the path of that folder into the libPath setting
    • scripts/ for scripts that fetch replay, analyze them, test the bot, and so on
    • starter/ the starter project. If you have a different organization, put the path of that folder into the starterPath setting

Then, you have three settings in the extension to customize the build of the bot. cmakeExtra is to pass some extra arguments to CMake when configuring the build. cCompilerPath and cppCompilerPath are to make sure you use the compiler you want. If not defined, CMake will use the default compiler found on your system. I recommend using Clang since it works nicely on all platforms, provides nice error and warning messages, and comes with a lot of useful tools.

Finally, you have four settings relative to CodinGame. gamePassword and gamerEmail are used to store your credentials to connect to CodinGame. While gameId and isMulti are per-project settings, to store respectively the identifier of the bot on CodinGame (it's the last part of the URL when you are in CodinGame's IDE) and if the bot is for a multiplayer game. You can see how to define those per-project settings on lines 2 and 3 of that file.

3. Using the extension

The extension commands are currently limited. It can create, open, configure, archive, and send bots. The starter project I provide enables the indexing of your source code by VS Code. Thus, some cool features of the IDE are activated, such as finding references, refactoring, and error checking. There is also a task to build the bot from the IDE.

3.1 Creating and opening

In the IDE, bring the command palette with the Ctrl+Shift+P shortcut. In the palette textbox, start to type Create and select the CodinGame: Create New Project command. Then, type the name of the new bot and press enter. A new project with the given name will be created, using the starter project as a basis.

As for all commands, if an error happened during the execution of the command, you will see a notification in a popup located in the lower right corner of VS Code.

Creating a new c++ bot for CodinGame has never been easier.

Similarly, you can open any bot project in your rootPath folder with the command CodinGame: Open Bot Project. The command checks for you if the name you give is an actual bot name.

3.2 Configuring and building

When you create a new bot project or add a new file to the project, you need to configure it. The configuration makes sure the build system is up to date and produces the .vscode/compile_commands.json file. This file is crucial to let VS Code parse and analyze your project. It is crafted by CMake when your generator is either Ninja or Make, then improved by compdb with header files. To configure your bot, select CodinGame: Configure Build in the command palette. You have then to choose the build type used to configure the build. Debug is to be reserved when you hunt down bugs, as it is slower to execute. Dev is your go-to build type, as it is fast enough and style offers some debug capacities. Release is for performance-focused builds, ideal when you are comparing different versions of your bot in an arena.

Configuring a bot and then building it. Once configured, you can browse your code using Ctrl+Click on symbols or headers, among other cool things. The build merges a multiple file bot into a single file, ready to be sent to CodinGame.

Sometimes, the indexing of your files will fail. The indexing is when VS Code uses the .vscode/compile_commands.json file to parse your files and build the IntelliSense database to enable advanced features (right click on a symbol to see some of those features). You can notice a failure when the IDE reports many errors in the source code, even if you did not change anything. In such a situation, I find it useful to do the following:

1. delete the .vscode/compile_commands.json file,
2. execute the C/C++: Reset IntelliSense Database command in the command palette,
3. configure your bot,
4. execute the C/C++: Reset IntelliSense Database command again.

The start project I provide has a .vscode/tasks.json file. This file can launch a build using Ninja in a terminal of VS Code and allows the IDE to parse errors to navigate between them. To build your bot, use the Ctrl+Shift+B shortcut. At the end of the build, a python script is executed to merge the bot files into a single file. The resulting file is placed into the package/ folder, ready to be sent to CodinGame. The produced binaries are in the bin/ folder.

3.3 Archive bot version and send current version to CodinGame

I keep previous versions of my bot in the package/ folder. That way, I can build them and let them fight to see if a new version is significatively better than previous ones. To create a new version, you have to select the CodinGame: Save Current Version command in the command palette. The new version will be automatically added to the build, thanks to the PreviousBotVersions.cmake file.

I prefer not to do menial tasks manually. For such tasks, I rely on automation (or maybe I am just lazy 😁). This is why there is a command to send the current bot version to CodinGame. Use the CodinGame: Sent Your Bot Code to CodinGame command from the command palette to store the bot code on CodinGame servers. This command takes some time to complete as the only way I found to save the code on CodinGame is to request a play in CodinGame's IDE. The advantage is that you do not need to have an active session open on CodinGame.

4. Python scripts to exploit replays

When different bots fight in CodinGame arena, the results are summarized in a JSON file named replay. The viewer in the IDE uses this file to show you how the fight went. The replay contains whatever you sent to stdout and stderr, the input sent to you by the referee, and the summary of the turn assembled by the referee. A section of the replay contains the name of the players for this match and their final scores.

As mentioned in the introduction, I will not give you a script to automatically collect replays. However, I am sure you could build such a script yourself if you are motivated. I cannot stress enough how valuable those replays could be! Indeed, a common practice on CodinGame is to collect the last replays of a bot to analyze, debug, validate, and tune that bot.

To illustrate the power of replays, I share with you a small script I made to validate my bot for the card game Legend of Code & Magic. My bot kept doing wrong actions, and I did not want to proofread my code multiple times. I noticed the referee was kind enough to list the wrong actions my bot makes every turn. So I thought it would be simpler to fetch replays, find turns where I made wrong actions, and list them all. Then, in the replays, I would find the initial state I serialized in stderr for the turns I made wrong actions. With this state serialization, I could run the bot in the debugger, find the error, fix it and validate the fix. After a few iterations, I removed all the bugs from the bot and got promoted to the gold league. In the following script, 3726978 is my identifier on CodinGame. To find your identifier, execute the CodinGame: Get CodinGame Id command in the command palette.
Show/Hide Code

#!/usr/bin/env python
import argparse
import json
import sys
import os

sys.path.append(os.path.join(sys.path[0], '..', '..', 'tools', 'scripts'))
from utils.progress import progress_bar


def __analyze_replay(folder_path):
    paths_to_clean = []
    for directory_path, _, file_names in os.walk(folder_path):
        for file_name in file_names:
            if file_name.endswith('.json'):
                paths_to_clean.append(os.path.join(directory_path, file_name))

    paths_to_clean.sort()

    illegal_actions = []
    defeats = []
    timeouts = []
    victory_by_timeout_count = 0
    victory_count = 0
    defeat_count = 0
    match_count = 0

    print('Analyzing replay from folder {}'.format(folder_path))
    for replay in progress_bar(paths_to_clean, '  Progress', length=60):
        with open(replay, 'r') as infile:
            content = json.load(infile)

        relative_replay_path = os.path.relpath(replay, folder_path)
        try:
            my_index = 0 if content['agents'][0]['codingamer']['userId'] == 3726978 else 1
        except:
            my_index = 1
        my_var_index = '${}'.format(my_index)

        this_replay_illegal_actions = []
        for i, frame in enumerate(content['frames']):
            summary = frame.get('summary', None)
            if summary and '[Warning]' in summary and my_var_index in summary:
                summary = summary.replace('[Warning]', '')
                summary = summary.replace('\n', ';')
                this_replay_illegal_actions.append((i+6, summary))

        if this_replay_illegal_actions:
            illegal_actions.append(
                (relative_replay_path, this_replay_illegal_actions))

        if content['ranks'][0] == my_index:
            victory_count += 1
            if content['scores'][1 - my_index] < 0:
                victory_by_timeout_count += 1
        else:
            defeat_count += 1
            defeats.append(relative_replay_path)
            if content['scores'][my_index] < 0:
                timeouts.append(relative_replay_path)
        match_count += 1

    print('  Summary: victories {} (by timeouts {}), defeats {} (by timeouts {}), win rate {:03.2f}'.format(
        victory_count, victory_by_timeout_count, defeat_count, len(timeouts),
        float(victory_count) / float(match_count) * 100.0))
    if illegal_actions:
        print('  Illegal actions')
        for replay, actions in illegal_actions:
            if len(actions) == 1:
                print('   - {}: frame {}, {}'.format(replay, actions[0][0], actions[0][1]))
            else:
                print('   - {}'.format(replay))
                for frame, action in actions:
                    print('    - frame {}, {}'.format(frame, action))


if __name__ == '__main__':
    parser=argparse.ArgumentParser('replay_analyzer')
    parser.add_argument(
        'input', metavar='INPUT', type=str, nargs='+',
        help='list of replay folder paths to clean')
    settings=parser.parse_args()

    for folder_path in settings.input:
        if not os.path.exists(folder_path):
            print('skipping folder {}: path does not exist'.format(folder_path))
            continue
        __analyze_replay(folder_path)

Since the replay structure could be different between games, I tend to write a script for each game. To maximize the usefulness of replays, I add the following information in stderr for each turn:

• a serialization of the initial state, such that I can load the turn to test in on my computer
• a tag if I am expecting to win the game and another one if I am expecting to lose,
• a tag if the bot crashed and some info to find the crash reason,
• a serialization of the expected final state, if this makes sense for the game, to compare it with the initial state of the next turn.

The crash tag is here to distinguish between a timeout and a crash. The referee does not make such a distinction: if the bot did not answer in due time, the referee considers that the bot time outed. By catching POSIX signals inside the bot, I can print a particular string in stderr after a crash to ease the distinction. This info allows me to efficiently find and fix crashes by parsing replays.

Tags for expected victory and defeat are handy for example to debug a MCTS solver. Let us suppose that the solver states that we are going to win the game. If the game ends with a defeat, this means that there is a bug in the MCTS or the solver.

Conclusion

In this article, I introduced a VS Code extension and a starter project to quickly start to write C++ bots for CodinGame on your computer. With those, you can write bots in multiple files, enable indexing, configure and build bots locally, and more. I also mentioned how to exploit replays to improve your bot and to make replays even more powerful.

With such tools, I have seen my efficiency, as well as my ranking, improved. I hope it would be the same for you! Again, if you have any issues or any suggestions, please let me know. You can reach me on GitHub for the extension or the starter project.