Tutorial :Total memory used by Python process?



Question:

Is there a way for a Python program to determine how much memory it's currently using? I've seen discussions about memory usage for a single object, but what I need is total memory usage for the process, so that I can determine when it's necessary to start discarding cached data.


Solution:1

Here is a useful solution that works for various operating systems, including Linux, Windows 7, etc.:

import os  import psutil  process = psutil.Process(os.getpid())  print(process.memory_info().rss)  

On my current Python 2.7 install, the last line should be

print(process.get_memory_info()[0])  

instead (there was a change in the API).


Solution:2

For Unixes (Linux, Mac OS X, Solaris) you could also use the getrusage() function from the standard library module resource. The resulting object has the attribute ru_maxrss, which gives peak memory usage for the calling process:

>>> resource.getrusage(resource.RUSAGE_SELF).ru_maxrss  2656 # peak memory usage (bytes on OS X, kilobytes on Linux)  

The Python docs aren't clear on what the units are exactly, but the Mac OS X man page for getrusage(2) describes the units as bytes. The Linux man page isn't clear, but it seems to be equivalent to the information from /proc/self/status, which is in kilobytes.

The getrusage() function can also be given resource.RUSAGE_CHILDREN to get the usage for child processes, and (on some systems) resource.RUSAGE_BOTH for total (self and child) process usage.

resource is a standard library module.

If you only care about Linux, you can just check the /proc/self/status file as described in a similar question.


Solution:3

On Windows, you can use WMI (home page, cheeseshop):

  def memory():      import os      from wmi import WMI      w = WMI('.')      result = w.query("SELECT WorkingSet FROM Win32_PerfRawData_PerfProc_Process WHERE IDProcess=%d" % os.getpid())      return int(result[0].WorkingSet)  

On Linux (from python cookbook http://code.activestate.com/recipes/286222/:

import os  _proc_status = '/proc/%d/status' % os.getpid()    _scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,            'KB': 1024.0, 'MB': 1024.0*1024.0}    def _VmB(VmKey):      '''Private.      '''      global _proc_status, _scale       # get pseudo file  /proc/<pid>/status      try:          t = open(_proc_status)          v = t.read()          t.close()      except:          return 0.0  # non-Linux?       # get VmKey line e.g. 'VmRSS:  9999  kB\n ...'      i = v.index(VmKey)      v = v[i:].split(None, 3)  # whitespace      if len(v) < 3:          return 0.0  # invalid format?       # convert Vm value to bytes      return float(v[1]) * _scale[v[2]]      def memory(since=0.0):      '''Return memory usage in bytes.      '''      return _VmB('VmSize:') - since      def resident(since=0.0):      '''Return resident memory usage in bytes.      '''      return _VmB('VmRSS:') - since      def stacksize(since=0.0):      '''Return stack size in bytes.      '''      return _VmB('VmStk:') - since  


Solution:4

On unix, you can use the ps tool to monitor it:

$ ps u -p 1347 | awk '{sum=sum+$6}; END {print sum/1024}'  

where 1347 is some process id. Also, the result is in MB.


Solution:5

Heapy (and friends) may be what you're looking for.

Also, caches typically have a fixed upper limit on their size to solve the sort of problem you're talking about. For instance, check out this LRU cache decorator.


Solution:6

I like it, thank you for @bayer. I get a specific process count tool, now.

# Megabyte.  $ ps aux | grep python | awk '{sum=sum+$6}; END {print sum/1024 " MB"}'  87.9492 MB    # Byte.  $ ps aux | grep python | awk '{sum=sum+$6}; END {print sum " KB"}'  90064 KB  

Attach my process list.

$ ps aux  | grep python  root       943  0.0  0.1  53252  9524 ?        Ss   Aug19  52:01 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid  root       950  0.6  0.4 299680 34220 ?        Sl   Aug19 568:52 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid  root      3803  0.2  0.4 315692 36576 ?        S    12:43   0:54 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid  jonny    23325  0.0  0.1  47460  9076 pts/0    S+   17:40   0:00 python  jonny    24651  0.0  0.0  13076   924 pts/4    S+   18:06   0:00 grep python  

Reference


Solution:7

Using sh and os to get into python bayer's answer.

float(sh.awk(sh.ps('u','-p',os.getpid()),'{sum=sum+$6}; END {print sum/1024}'))  

Answer is in megabytes.


Solution:8

han = win32api.OpenProcess(win32con.PROCESS_QUERY_INFORMATION|win32con.PROCESS_VM_READ, 0, os.getpid())  process_memory = int(win32process.GetProcessMemoryInfo(han)['WorkingSetSize'])  


Solution:9

Current memory usage of the current process on Linux, for python 2 and 3 and pypy:

import os  def getCurrentMemoryUsage():      ''' Memory usage in kB '''        f = open('/proc/{}/status'.format(os.getpid()))      memusage = int(f.read().split('VmRSS:')[1].split('\n')[0][:-3].strip())      f.close()        return memusage  

I've tested it on Linux 4.4 and 4.9, but any Linux version should work.

Looking in man proc and searching for the info on the /proc/$PID/status file, it mentions minimum versions for some fields (like Linux 2.6.10 for "VmPTE"), but the "VmRSS" field (which I use here) has no such mention. Therefore I assume it has been in there since an early version.


Solution:10

Below is my function decorator which allows to track how much memory this process consumed before the function call, how much memory it uses after the function call, and how long the function is executed.

import time  import os  import psutil      def elapsed_since(start):      return time.strftime("%H:%M:%S", time.gmtime(time.time() - start))      def get_process_memory():      process = psutil.Process(os.getpid())      return process.get_memory_info().rss      def track(func):      def wrapper(*args, **kwargs):          mem_before = get_process_memory()          start = time.time()          result = func(*args, **kwargs)          elapsed_time = elapsed_since(start)          mem_after = get_process_memory()          print("{}: memory before: {:,}, after: {:,}, consumed: {:,}; exec time: {}".format(              func.__name__,              mem_before, mem_after, mem_after - mem_before,              elapsed_time))          return result      return wrapper  

So, when you have some function decorated with it

from utils import track    @track  def list_create(n):      print("inside list create")      x = [1] * n      return x  

You will be able to see this output:

inside list create  list_create: memory before: 45,928,448, after: 46,211,072, consumed: 282,624; exec time: 00:00:00  

Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Previous
Next Post »