When we run the rnode Docker image created by sbt, the process has /opt/docker as current working directory by default. Rnode files are in this directory (e.g. /opt/docker/bin/rnode).
When a node crashes, for instance on event like this
interesting files are put in the current working directory and these files are lost when container stops.
We should modify the created image to run rnode from a designated directory so that it'll be easy to bind-mount it into host filesystem. We cannot just bind-mount /opt/docker because the image has rnode files there.
Let's use either /var/lib/rnode or /root/.rnode because people are already used to bind mount one of these two to persist node logs. Let's also make /var/lib/rnode symlink to /root/.rnode in Docker image so that there's no confusion where does the rnode actually store logs.
We can't force people to do it but we definitely need to persist logs and everything else that gets dumped into current working directory in all the nodes we run, including bootstrap ones.
The problem could be that /opt/docker/bin/rnode shell script could be changing directory. This script and the corresponding Docker image are generated by sbt-assembly-plugin. It may not be that straightforward to bend the plugin but it should be investigated. In the worst case, the rnode Docker image could be created by extending Docker image generated by sbt-assembly-plugin.