Ccna final exam - java, php, javascript, ios, cshap all in one. This is a collaboratively edited question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
Sunday, May 27, 2012
Linux: How to measure actual memory usage of an application or process?
How do you measure the memory usage of an application or process in Linux? I've read here that "ps" is not an accurate tool to use for this intent.
With ps or similiar tools you will only get the amount of memory pages allocated by that process. This number is correct, but:
a) does not reflect the actual amount of memory used by the application, only the amount of memory reserved for it
b) can be misleading if pages are shared, for example by several threads or by using dynamically linked libraries
If you really want to know what amount of memory your application actually uses, you need to run it within a profiler. For example, valgrind can give you insights about the amount of memory used, and, more importantly, about possible memory leaks in your program.
In recent versions of linux, use the smaps subsystem. For example, for a process with a PID of 1234:
cat /proc/1234/smaps
It will tell you exactly how much memory it is using at that time. More importantly, it will divide the memory into private and shared, so you can tell how much memory your instance of the program is using, without including memory shared between multiple instances of the program.
This is an excellent summary of the tools and problems: archive.org link
I'll quote it, so that more devs will actually read it.
If you want to analyse memory usage of the whole system or to thoroughly analyse memory usage of one application (not just its heap usage), use exmap. For whole system analysis, find processes with the highest effective usage, they take the most memory in practice, find processes with the highest writable usage, they create the most data (and therefore possibly leak or are very ineffective in their data usage). Select such application and analyse its mappings in the second listview. See exmap section for more details. Also use xrestop to check high usage of X resources, especially if the process of the X server takes a lot of memory. See xrestop section for details.
If you want to detect leaks, use valgrind or possibly kmtrace.
If you want to analyse heap (malloc etc.) usage of an application, either run it in memprof or with kmtrace, profile the application and search the function call tree for biggest allocations. See their sections for more details.
There isn't a single answer for this because you can't pin point precisely the amount of memory a process uses. Most processes under linux use shared libraries. For instance, let's say you want to calculate memory usage for the 'ls' process. Do you count only the memory used by the executable 'ls' ( if you could isolate it) ? How about libc? Or all these other libs that are required to run 'ls'?
You could argue that they are shared by other processes, but 'ls' can't be run on the system without them being loaded.
Also, if you need to know how much memory a process needs in order to do capacity planning, you have to calculate how much each additional copy of the process uses. I think /proc/PID/status might give you enough info of the memory usage AT a single time. On the other hand, valgrind will give you a better profile of the memory usage throughout the lifetime of the program
If your code is in C or C++ you might be able to use getrusage() which returns you various statistics about memory and time usage of your process.
Not all platforms support this though and will return 0 values for the memory-use options.
Instead you can look at the virtual file created in /proc/[pid]/statm (where [pid] is replaced by your process id. You can obtain this from getprocessid()).
This file will look like a text file with 7 integers. You are probably most interested in the first (all memory use) and sixth (data memory use) numbers in this file.
Valgrind is amazing if you have the time to run it. valgrind --tool=massif is The Right Solution.
However, I'm starting to run larger examples, and using valgrind is no longer practical. Is there a way to tell the maximum memory usage (modulo page size and shared pages) of a program?
On a real unix system, /usr/bin/time -v works. On linux, however, this does /not/ work.
If you want something quicker than profiling with Valgrind and your kernel is older and you can't use smaps, a ps with the options to show the resident set of the process (with "ps -o rss,command") can give you a quick and reasonable aproximation of the real amount of non-swapped memory being used.
A good test of the more "real world" usage is to open the application, then run "vmstat -s" and check the "active memory" statistic. Close the application, wait a few seconds and run "vmstat -s" again. However much active memory was freed was in evidently in use by the app.
With ps or similiar tools you will only get the amount of memory pages allocated by that process. This number is correct, but:
ReplyDeletea) does not reflect the actual amount of memory used by the application, only the amount of memory reserved for it
b) can be misleading if pages are shared, for example by several threads or by using dynamically linked libraries
If you really want to know what amount of memory your application actually uses, you need to run it within a profiler. For example, valgrind can give you insights about the amount of memory used, and, more importantly, about possible memory leaks in your program.
Hard to tell for sure, but here are two "close" things that can help.
ReplyDelete$ ps aux
will give you Virtual Size (VSZ)
You can also get detailed stats from /proc filesystem by going to /proc/$pid/status
The most important is the VmSize, which should be close to what "ps aux" gives.
/proc/19420$ cat status
Name: firefox
State: S (sleeping)
Tgid: 19420
Pid: 19420
PPid: 1
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
FDSize: 256
Groups: 4 6 20 24 25 29 30 44 46 107 109 115 124 1000
VmPeak: 222956 kB
VmSize: 212520 kB
VmLck: 0 kB
VmHWM: 127912 kB
VmRSS: 118768 kB
VmData: 170180 kB
VmStk: 228 kB
VmExe: 28 kB
VmLib: 35424 kB
VmPTE: 184 kB
Threads: 8
SigQ: 0/16382
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000020001000
SigCgt: 000000018000442f
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
Cpus_allowed: 03
Mems_allowed: 1
voluntary_ctxt_switches: 63422
nonvoluntary_ctxt_switches: 7171
In recent versions of linux, use the smaps subsystem. For example, for a process with a PID of 1234:
ReplyDeletecat /proc/1234/smaps
It will tell you exactly how much memory it is using at that time. More importantly, it will divide the memory into private and shared, so you can tell how much memory your instance of the program is using, without including memory shared between multiple instances of the program.
Try the pmap command:
ReplyDeletepmap -x <process pid>
This is an excellent summary of the tools and problems: archive.org link
ReplyDeleteI'll quote it, so that more devs will actually read it.
If you want to analyse memory usage of the whole system or to thoroughly analyse memory usage of one application (not just its heap usage), use exmap. For whole system analysis, find processes with the highest effective usage, they take the most memory in practice, find processes with the highest writable usage, they create the most data (and therefore possibly leak or are very ineffective in their data usage). Select such application and analyse its mappings in the second listview. See exmap section for more details. Also use xrestop to check high usage of X resources, especially if the process of the X server takes a lot of memory. See xrestop section for details.
If you want to detect leaks, use valgrind or possibly kmtrace.
If you want to analyse heap (malloc etc.) usage of an application, either run it in memprof or with kmtrace, profile the application and search the function call tree for biggest allocations. See their sections for more details.
There is no easy way to calculate this.
ReplyDeleteBut some people have tried to get some good answers:
ps_mem.py
ps_mem.py at github
There isn't a single answer for this because you can't pin point precisely the amount of memory a process uses. Most processes under linux use shared libraries. For instance, let's say you want to calculate memory usage for the 'ls' process. Do you count only the memory used by the executable 'ls' ( if you could isolate it) ? How about libc? Or all these other libs that are required to run 'ls'?
ReplyDeletelinux-gate.so.1 => (0x00ccb000)
librt.so.1 => /lib/librt.so.1 (0x06bc7000)
libacl.so.1 => /lib/libacl.so.1 (0x00230000)
libselinux.so.1 => /lib/libselinux.so.1 (0x00162000)
libc.so.6 => /lib/libc.so.6 (0x00b40000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00cb4000)
/lib/ld-linux.so.2 (0x00b1d000)
libattr.so.1 => /lib/libattr.so.1 (0x00229000)
libdl.so.2 => /lib/libdl.so.2 (0x00cae000)
libsepol.so.1 => /lib/libsepol.so.1 (0x0011a000)
You could argue that they are shared by other processes, but 'ls' can't be run on the system without them being loaded.
Also, if you need to know how much memory a process needs in order to do capacity planning, you have to calculate how much each additional copy of the process uses. I think /proc/PID/status might give you enough info of the memory usage AT a single time. On the other hand, valgrind will give you a better profile of the memory usage throughout the lifetime of the program
If your code is in C or C++ you might be able to use getrusage() which returns you various statistics about memory and time usage of your process.
ReplyDeleteNot all platforms support this though and will return 0 values for the memory-use options.
Instead you can look at the virtual file created in /proc/[pid]/statm (where [pid] is replaced by your process id. You can obtain this from getprocessid()).
This file will look like a text file with 7 integers. You are probably most interested in the first (all memory use) and sixth (data memory use) numbers in this file.
#!/bin/ksh
ReplyDelete#
# Returns total memory used by process $1 in kb.
#
# See /proc/NNNN/smaps if you want to do something
# more interesting.
#
IFS=$'\n'
for line in $(</proc/$1/smaps)
do
[[ $line =~ ^Size:\s+(\S+) ]] && ((kb += ${.sh.match[1]}))
done
print $kb
Valgrind is amazing if you have the time to run it. valgrind --tool=massif is The Right Solution.
ReplyDeleteHowever, I'm starting to run larger examples, and using valgrind is no longer practical. Is there a way to tell the maximum memory usage (modulo page size and shared pages) of a program?
On a real unix system, /usr/bin/time -v works. On linux, however, this does /not/ work.
If you want something quicker than profiling with Valgrind and your kernel is older and you can't use smaps, a ps with the options to show the resident set of the process (with "ps -o rss,command") can give you a quick and reasonable aproximation of the real amount of non-swapped memory being used.
ReplyDeleteA good test of the more "real world" usage is to open the application, then run "vmstat -s" and check the "active memory" statistic. Close the application, wait a few seconds and run "vmstat -s" again. However much active memory was freed was in evidently in use by the app.
ReplyDelete