I recently ran into an interesting situation that required me to run a Python script repeatedly with different inputs on a remote server. Of course, with any SSH session, there is always the possibility of a timeout which would kill any running jobs. Normally, I would simply deploy a program and use an & at the end of the command, allowing the job to run in the background even after I logged out of my SSH session. Seeing that I had multiple scripts to run, and could simply adjust my inputs with a for loop, I created a bash script that repeatedly called my Python code. This was pretty straightforward and I deployed the script with an & before logging out of my SSH session to let the job complete.
After an hour or so I logged back into my remote machine to find a log of the script filled with errors, missing library warnings and just generally bad lines that I didn’t expect to see. After some investigation, I realized that the Bash script continued to run. However, the Python scripts were being called without any of my user level libraries. Furthermore, the system Python version was being used, not my newer version that is also in my user space. Once I saw this, I realized that when I logged out, I needed a way to allow for my user level libraries and Python installation to stay active.
Some quick google-fu and a few Stack Overflow pages brought up the screen command in Unix as a possible solution. I’ll leave the details of the command to it’s man page, and instead detail how I used it to solve my problem.
Using screen to keep Python going:
The screen command is quite powerful for SSH sessions and provides a little something for everyone. Essentially, I was concerned with keeping my user space active when I logged out. To do so, I used the following process in my SSH session:
- screen
- source ~/.login (Can also be source ~/.profile depending upon system.)
- Run whatever script or jobs you need.
- Ctrl + A d
The first command launches the screen environment, which will appear as a new window to you in your terminal. It is effectively running a new window within your SSH session. However, you will need to run the source command and load either your login or profile scripts to get your user level environment loaded. Once this is complete, simply run whatever job or script you need to. Finally, the trick to keeping these jobs running is to use the command Ctrl + A, then press d. This will detach your screen session from your active window. Checking your running processes, you should still see your scripts running along with a screen process. This means that it is running effectively. You are now free to log off the system while your jobs continue to run.
Picking Up Where You Left Off:
If you want to go back and check a detached screen, you can use the command screen -x, even if you have logged off and back on again. To exit the screen window, you can simply type exit, or just kill it through the kill command if you don’t have a need to go back into the session directly.