Remote Debugging CI/CD Pipelines

One of the key features of NUKE is that you can run and debug builds locally as opposed to repeatedly triggering CI/CD pipeline with a lot of logging all over the place. Based on broad feedback, this solves more than 90% of the cases where YAML and other schemas are difficult to troubleshoot. While the vast majority of developers has yet to experience this advantage, we will try to go a step further.

The real missing part for CI/CD automation is a local development experience.

Sounds counter-intuitive but being able to run the E2E CI/CD pipeline locally and validate failures saves engineers from the 20-minute push, coffee, failure, fix loop.

— r3kc4h0rt3r‏ (@retrohack3r) July 21, 2022

In practice, differences between your local and CI/CD environment are unavoidable. Even with dockerized environments, you can't rule out misconfiguration of ports, volumes, or services. Beyond build automation, many of us have also faced flaky tests and race conditions that never occurred locally. Overall, problems in CI/CD pipelines are very tedious and time-consuming to investigate.

In this post, we will take a look at a proof-of-concept that allows you to attach the debugger to any remote process running in your CI/CD environment. Our tools of choice are JetBrains Rider and GitHub Actions, but the principles work much the same with any other tools.

Preparing for Remote Debugging

Enabling remote debugging is as simple as adding an attribute to your Build class:

Debugging the Build

[EnableRemoteDebugging(WaitForDebugger = true)]
class Build : NukeBuild
{
    Target Compile => _ => _
        .Executes(() => { /* ... */ });
}

Debugging other Processes

[EnableRemoteDebugging]
class Build : NukeBuild
{
    Target Test => _ => _
        .Executes(() => { /* ... */ });
}

tip

If you want to debug another processes, you have to manually wait for the debugger to be attached in the respective code path:

SpinWait.SpinUntil(() => Debugger.IsAttached, TimeoutInMilliseconds);

Once the build is running in your CI/CD environment, it will print detailed connection instructions to the log output:

[INF] SSH tunnel opened: ssh -t matkoch.nuke.build
[INF] Fingerprint: MD5:74:8b:e4:12:3b:b0:db:1e:88:4c:22:fd:ee:d9:b3:40
[INF] SSH Configuration:

Host matkoch.nuke.build
  ...

Before you can connect, you need to go through a one-time SSH setup. This includes updating your .ssh/config as above and creating a new SSH session configuration in JetBrains Rider:

Attaching to a Remote Process

After completing the initial setup, you can connect to the remote host:

The SSH service on the remote machine will verify your identity using your public key, which can be obtained from GitHub or from a database. That makes sure that only authorized users can connect.

From the list of .NET processes, you can select any process to attach the debugger. In this example, we will attach to the build process, but it could also be a dotnet test process:

Debugging works exactly the same on the remote host as on your local machine. You can step through your code, inspect variables, and even execute commands in the immediate window:

The logging call from above will immediately execute in the GitHub Actions runner:

Creating Terminal Sessions

Beyond debugging .NET processes, you can also spin up simple SSH terminal session:

ssh -t matkoch.nuke.build

This comes in handy when you need to verify the file structure, check on the content of specific files, or just explore your CI/CD environment:

Conveniently, you'll also switch directly to the root directory once you join the terminal session.

Related Work

The community has built projects like upterm and tmate, which have been very inspiring. However, both of these are limited to just terminal sessions and don't allow to attach to any process you'd wish for. In addition, transmission is done in plaintext through a relay server, which means that the public relay server can be a security concern. Alternatively, you can also self-host an instance.

Conclusion

Everyone remembers at least one of those WTF moments, when absolutely nothing that happened in the CI/CD pipeline made any sense. How much time, money, and nerve did it take? Once more, NUKE pushes the limits of what's possible in build automation. With a simple attribute, you can enable remote debugging and attach to any process or just spin up a terminal session for your investigations.

What was your latest WTF moment? Would this feature have helped? Make sure to let us know in the comments!