The test runner builds user’s code (based on the Starter Code) with the corresponding Dockerfile to build a Test Runner Image.

The test runner script is run within a container based on this image.

Test Runner Script

The test runner script roughly follows this flow:

  • Builds the test runner image.
    • This step is only run it a test runner image doesn’t exist already or is being re-built.
    • Logs from this step are prefixed with [build].
  • Copies in user’s latest code, if it has changed since the image was built.
  • If any files in CODECRAFTERS_DEPENDENCY_FILE_PATHS have changed, triggers a re-build of the test runner image (i.e. back to step 1)
  • Copies any files in /app-cached to $CODECRAFTERS_SUBMISSION_DIR if present.
  • Sets CODECRAFTERS_SUBMISSION_DIR to /app
    • This value is subject to change in the future, hence the use of an environment variable.
  • Runs /app/.codecrafters/compile.sh if it exists.
    • Logs from this step are prefixed with [compile].
    • This script only runs once even if we’re testing multiple stages.
  • Sets CODECRAFTERS_TEST_CASES_JSON based on the current test run.
  • Runs the tester program (example)
    • Logs from this step are streamed directly with no prefix, the tester program is expected to add prefixes like “[stage-1]”, “[stage-2]” etc. to its logs.
    • The user’s program is assumed to be present at /app/.codecrafters/run.sh.
    • The exit code of this step (0 or 1) is used to determine if the test run passed or failed.

Performance

We use three different techniques to ensure user’s code is built & run as fast as possible. Depending on the language, you may not need to use all of these techniques.

1. Dockerfile Caching

The test runner will look for a CODECRAFTERS_DEPENDENCY_FILE_PATHS environment variable in the image. It will only re-build the image if the files listed in the variable change.

A Dockerfile can contain steps to install dependencies, like cargo install for Rust, and npm install for JavaScript.

Since files like Cargo.toml & package.json don’t change often, this helps avoid re-installing dependencies on every run.

Any Dockerfile steps that emit output will be printed to the user’s test run logs with the [build] prefix.

[build] logs

Notes:

  • If you want a Dockerfile step’s output to not be included in [build] logs, you can add > /dev/null to the end of the command.
  • If a CODECRAFTERS_DEPENDENCY_FILE_PATHS environment variable is not present, the test runner will never re-build the image.

2. Compilation

If the user’s repo contains /.codecrafters/compile.sh, it will be run once before invoking the tester.

This is useful with compiled languages like Rust, where the first compilation can take a long time, but running the compiled binary is fast.

Logs from this script will be printed to the user’s test run logs with the [compile] prefix.

[compile] logs

More on this in in Test Runner Interface.

3. Cached Files

If any files are placed in /app-cached, they will be copied to /app folder (overwriting any of the user’s code) before running the tester.

This can be used to store files required for incremental compilation in languages that support it.