Recently I found myself having to execute some long running tasks (running find with -exec on a directory with millions of files) on a linux server over SSH. This can cause a bit of an issue if you want to go away and do something else or if you have an unreliable internet connection. This is because as soon as you SSH session disconnects the current process ends. This happens wether you close the connection yourself or the frustrating packet_write_wait: Connection to some.ip port 22: Broken pipe error. There are a couple of solutions to this:

Screen

Screen is a program that allows you to have multiple “windows” open without having to create separate SSH sessions for each process. I had always meant to learn how to use this program but there were a lot of options and I always put it off, however like many programs I found there are only a few options you need to learn to get work done.

To get started first SSH into the server we need to run the task on and run screen.

In my case I got this: -bash: screen: command not found

Not a problem, just install with sudo yum install screen and select yes when asked to confirm.

installing screen I’m on CentOS so used yum, for debian based distros use apt-get

Now it’s installed lets try again. You may be presented with a welcome screen but you can just tap space or enter to continue. This will take to to a new empty shell. From here you can run tasks a normal. Lets say you are trying to find all .txt files in your home directory you might use something like find ~/ -name *.txt and depending on how many files it has to search through this may take a while. Not to worry. Press Ctrl+a then d and your screen will be “detached” and you will be returned to the main shell:

[detached from7938.pts-2.newbury]
[craig@newbury ~]$

From here you can close your SSH connection and come back in your own time. To “re-attach” simply screen -r yourSessionId in my case this would be screen -r from7938.pts-2.newbury If you can’t remember the session ID you can also run screen -ls which will show you something like:

[craig@newbury ~]$ screen -ls
There is a screen on:
7938.pts-2.newbury (Detached)
1 Socket in /var/run/screen/S-craig.

[craig@newbury ~]$

Next run screen -r 7938.pts-2.newbury and you will be re-attached back to the session.

Theres also a handy feature to name sessions. From within the session type Ctrl+a

then :sessionname mysessionname

Nohup This is another program that allows you to run programs without the risk of them being killed on disconnect. In-fact they way I have been using this I haven’t needed to come back, leaving it run on it’s won was fine so nohup some-command & did the job fine. The & tells the OS to put the job into the background. If you need to see if its still running then jobs -l will do the trick for you. You will get output similar to this:

jobs -l
[1]+ 13100 Running nohup find / &

Where “13100” is the PID and “nohup find / &” is the command you ran. You can still kill the process, say for example services start having latency/responsiveness issues by using kill 13100 (SIGTERM/Graceful process end) or kill -9 13100 (SIGKILL/Forcing the process to end).