Ubuntu: I think I fork-bombed my server with a cron job - what's happening here and how do I recover?



Question:

I scheduled a simple python script to run every minute under the user foo

SHELL=/bin/bash  BASH_ENV=/home/foo/.bash_profile    * * * * * python /home/foo/slack-bot/main.py 2>&1 /home/foo/slack-bot.log  

After a few mins I got these errors and was soon after kicked off my server

-bash: fork: Cannot allocate memory  -bash: wait_for: No record of process 32012  -bash: fork: Cannot allocate memory  -bash: wait_for: No record of process 32012  ....  

Uh oh. So I tried logging in as root and sudo-ing as foo

sudo -iu foo  sudo: unable to execute /bin/bash: Cannot allocate memory  

Great.

I then removed the offending line from the foo user's crontab file, and also killed all processes with slack in the name

kill $(ps aux | grep 'slack' | awk '{print $2}')  

I now get a different error, but I still can't sudo that user:

sudo -iu foo  -bash: xmalloc: .././copy_cmd.c:86: cannot allocate 32 bytes (450560 bytes allocated)  

What's going on here? I'm sure there's an error in my cron schedule which I'll investigate, but what kind of behavior causes the out of memory fork errors? What's the best way to recover besides a hard-reset?

Thanks!


Solution:1

Its a common launch pattern - depends on what you need ... you may

  • increase time between process launch to increase likelihood prior has finished
  • identify if prior launch is still running if so terminate
  • create a long running daemon instead of a short running process
  • launch - do work then terminate - have a watcher process spin up new launch only when it discovers its not running ( a la supervisord )

without knowing the use case its hard to proceed


Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Previous
Next Post »