Cpustat – Monitors CPU Utilization by Running Processes in Linux

Cpustat is a powerful system performance measure program for Linux, written using Go programming language. It attempts to reveal CPU utilization and saturation in an effective way, using The Utilization Saturation and Errors (USE) Method (a methodology for analyzing the performance of any system).

It extracts higher frequency samples of every process being executed on the system and then summarizes these samples at a lower frequency. For instance, it can measure every process every 200ms and summarize these samples every 5 seconds, including min/average/max values for certain metrics.

Cpustat outputs data in two possible ways: a pure text list of the summary interval and a colorful scrolling dashboard of each sample.

How to Install Cpustat in Linux

You must have Go (GoLang) installed on your Linux system in order to use cpustat, click on the link below to follow the GoLang installation steps that is if you do not have it installed:

1. Install GoLang (Go Programming Language) in Linux

Once you have installed Go, type the go get command below to install it, this command will install the cpustat binary in your GOBIN variable:

# go get github.com/uber-common/cpustat

How to Use Cpustat in Linux

When the installation process completes, run cpustat as follows with root privileges using the sudo command that is if your controlling the system as a non-root user, otherwise you’ll get the error as shown:

$ $GOBIN/cpustat
This program uses the netlink taskstats interface, so it must be run as root.

Note: To run cpustat as well as all other Go programs you have installed on your system like any other commands, include GOBIN variable in your PATH environment variable. Open the link below to learn how to set the PATH variable in Linux.

This is how cpustat works; the /proc directory is queried to get the current list of process IDs for every interval, and:

• for each PID, read /proc/pid/stat, then compute difference from previous sample.
• in case it’s a new PID, read /proc/pid/cmdline.
• for each PID, send a netlink message to fetch the taskstats, compute difference from previous sample.
• fetch /proc/stat to get the overall system stats.

Again, each sleep interval is adjusted to account for the amount of time consumed fetching all of these stats. Furthermore, each sample also records the time it took to scale each measurement by the actual elapsed time between samples. This attempts to account for delays in cpustat itself.

When run without any arguments, cpustat will display the following by default: sampling interval: 200ms, summary interval: 2s (10 samples), showing top 10 procs, user filter: all, pid filter: all as shown in the screenshot below:

$ sudo $GOBIN/cpustat 

From the output above, the following are the meanings of the system-wide summary metrics displayed before the fields:

• usr – min/avg/max user mode run time as a percentage of a CPU.
• sys – min/avg/max system mode run time as a percentage of a CPU.
• nice – min/avg/max user mode low priority run time as a percentage of a CPU.
• idle – min/avg/max user mode run time as a percentage of a CPU.
• iowait – min/avg/max delay time waiting for disk IO.
• prun – min/avg/max count of processes in a runnable state (same as load average).
• pblock – min/avg/max count of processes blocked on disk IO.
• pstart – number of processes/threads started in this summary interval.

Still from the output above, for a given process, the different columns mean:

• name – common process name from /proc/pid/stat or /proc/pid/cmdline.
• pid – process id, also referred to as “tgid”.
• min – lowest sample of user+system time for the pid, measured from /proc/pid/stat. Scale is a percentage of a CPU.
• max – highest sample of user+system time for this pid, also measured from /proc/pid/stat.
• usr – average user time for the pid over the summary period, measured from /proc/pid/stat.
• sys – average system time for the pid over the summary period, measured from /proc/pid/stat.
• nice – indicates current “nice” value for the process, measured from /proc/pid/stat. Higher means “nicer”.
• runq – time the process and all of its threads spent runnable but waiting to run, measured from taskstats via netlink. Scale is a percentage of a CPU.
• iow – time the process and all of its threads spent blocked by disk IO, measured from taskstats via netlink. Scale is a percentage of a CPU, averaged over the summary interval.
• swap – time the process and all of its threads spent waiting to be swapped in, measured from taskstats via netlink. Scale is a percentage of a CPU, averaged over the summary interval.
• vcx and icx – total number of voluntary context switches by the process and all of its threads over the summary interval, measured from taskstats via netlink.
• rss – current RSS value fetched from /proc/pid/stat. It is the amount of memory this process is using.
• ctime – sum of user+sys CPU time consumed by waited for children that exited during this summary interval, measured from /proc/pid/stat.

Note that long running child processes can often confuse this measurement, because the time is reported only when the child process exits. However, this is useful for measuring the impact of frequent cron jobs and health checks where the CPU time is often consumed by many child processes.

• thrd – number of threads at the end of the summary interval, measured from /proc/pid/stat.
• sam – number of samples for this process included in the summary interval. Processes that have recently started or exited may have been visible for fewer samples than the summary interval.

The following command displays the top 10 root user processes running on the system:

$ sudo $GOBIN/cpustat -u root

To display output in a fancy terminal mode, use the -t flag as follows:

$ sudo $GOBIN/cpustat -u roo -t

To view the top x number of processes (the default is 10), you can use the -n flag, the following command shows the top 20 Linux processes running on the system:

$ sudo $GOBIN/cpustat -n 20 

You can also write CPU profile to a file using the -cpuprofile option as follows and then use the cat command to view the file:

$ sudo $GOBIN/cpustat -cpuprofile cpuprof.txt
$ cat cpuprof.txt

To display help info, use the -h flag as follows:

$ sudo $GOBIN/cpustat -h

Find additional info from the cpustat Github Repository: https://github.com/uber-common/cpustat

Advertisement

Linux security alert: Bug in sudo’s get_process_ttyname() [ CVE-2017-1000367 ]

Skip to contCa

There is a serious vulnerability in sudo command that grants root access to anyone with a shell account. It works on SELinux enabled systems such as CentOS/RHEL and others too. A local user with privileges to execute commands via sudo could use this flaw to escalate their privileges to root. Patch your system as soon as possible.

It was discovered that Sudo did not properly parse the contents of /proc/[pid]/stat when attempting to determine its controlling tty. A local attacker in some configurations could possibly use this to overwrite any file on the filesystem, bypassing intended permissions or gain root shell.
From the description

We discovered a vulnerability in Sudo’s get_process_ttyname() for Linux:
this function opens “/proc/[pid]/stat” (man proc) and reads the device number of the tty from field 7 (tty_nr). Unfortunately, these fields are space-separated and field 2 (comm, the filename of the command) can
contain spaces (CVE-2017-1000367).

For example, if we execute Sudo through the symlink “./ 1 “, get_process_ttyname() calls sudo_ttyname_dev() to search for the non-existent tty device number “1” in the built-in search_devs[].

Next, sudo_ttyname_dev() calls the function sudo_ttyname_scan() to search for this non-existent tty device number “1” in a breadth-first traversal of “/dev”.

Last, we exploit this function during its traversal of the world-writable “/dev/shm”: through this vulnerability, a local user can pretend that his tty is any character device on the filesystem, and
after two race conditions, he can pretend that his tty is any file on the filesystem.

On an SELinux-enabled system, if a user is Sudoer for a command that does not grant him full root privileges, he can overwrite any file on the filesystem (including root-owned files) with his command’s output,
because relabel_tty() (in src/selinux.c) calls open(O_RDWR|O_NONBLOCK) on his tty and dup2()s it to the command’s stdin, stdout, and stderr. This allows any Sudoer user to obtain full root privileges.

A list of affected Linux distro

1. Red Hat Enterprise Linux 6 (sudo)
2. Red Hat Enterprise Linux 7 (sudo)
3. Red Hat Enterprise Linux Server (v. 5 ELS) (sudo)
4. Oracle Enterprise Linux 6
5. Oracle Enterprise Linux 7
6. Oracle Enterprise Linux Server 5
7. CentOS Linux 6 (sudo)
8. CentOS Linux 7 (sudo)
9. Debian wheezy
10. Debian jessie
11. Debian stretch
12. Debian sid
13. Ubuntu 17.04
14. Ubuntu 16.10
15. Ubuntu 16.04 LTS
16. Ubuntu 14.04 LTS
17. SUSE Linux Enterprise Software Development Kit 12-SP2
18. SUSE Linux Enterprise Server for Raspberry Pi 12-SP2
19. SUSE Linux Enterprise Server 12-SP2
20. SUSE Linux Enterprise Desktop 12-SP2
21. OpenSuse, Slackware, and Gentoo Linux

How do I patch sudo on Debian/Ubuntu Linux server?

To patch Ubuntu/Debian Linux apt-get command or apt command:
$ sudo apt update
$ sudo apt upgrade

How do I patch sudo on CentOS/RHEL/Scientific/Oracle Linux server?

Run yum command:
$ sudo yum update

How do I patch sudo on Fedora Linux server?

Run dnf command:
$ sudo dnf update

How do I patch sudo on Suse/OpenSUSE Linux server?

Run zypper command:
$ sudo zypper update

How do I patch sudo on Arch Linux server?

Run pacman command:
$ sudo pacman -Syu

How do I patch sudo on Alpine Linux server?

Run apk command:
# apk update && apk upgrade

How do I patch sudo on Slackware Linux server?

Run upgradepkg command:
# upgradepkg sudo-1.8.20p1-i586-1_slack14.2.txz

How do I patch sudo on Gentoo Linux server?

Run emerge command:
# emerge --sync
# emerge --ask --oneshot --verbose ">=app-admin/sudo-1.8.20_p1"

Prev

Impermanence in Linux – Exclusive (By Hari Iyer)

Impermanence, also called Anicca or Anitya, is one of the essential doctrines and a part of three marks of existence in Buddhism The doctrine asserts that all of conditioned existence, without exception, is “transient, evanescent, inconstant”

On Linux, the root of all randomness is something called the kernel entropy pool. This is a large (4,096 bit) number kept privately in the kernel’s memory. There are 2⁴⁰⁹⁶ possibilities for this number so it can contain up to 4,096 bits of entropy. There is one caveat – the kernel needs to be able to fill that memory from a source with 4,096 bits of entropy. And that’s the hard part: finding that much randomness.

The entropy pool is used in two ways: random numbers are generated from it and it is replenished with entropy by the kernel. When random numbers are generated from the pool the entropy of the pool is diminished (because the person receiving the random number has some information about the pool itself). So as the pool’s entropy diminishes as random numbers are handed out, the pool must be replenished.

Replenishing the pool is called stirring: new sources of entropy are stirred into the mix of bits in the pool.

This is the key to how random number generation works on Linux. If randomness is needed, it’s derived from the entropy pool. When available, other sources of randomness are used to stir the entropy pool and make it less predictable. The details are a little mathematical, but it’s interesting to understand how the Linux random number generator works as the principles and techniques apply to random number generation in other software and systems.

The kernel keeps a rough estimate of the number of bits of entropy in the pool. You can check the value of this estimate through the following command:

cat /proc/sys/kernel/random/entropy_avail

A healthy Linux system with a lot of entropy available will have return close to the full 4,096 bits of entropy. If the value returned is less than 200, the system is running low on entropy.

The kernel is watching you

I mentioned that the system takes other sources of randomness and uses this to stir the entropy pool. This is achieved using something called a timestamp.

Most systems have precise internal clocks. Every time that a user interacts with a system, the value of the clock at that time is recorded as a timestamp. Even though the year, month, day and hour are generally guessable, the millisecond and microsecond are not and therefore the timestamp contains some entropy. Timestamps obtained from the user’s mouse and keyboard along with timing information from the network and disk each have different amount of entropy.

How does the entropy found in a timestamp get transferred to the entropy pool? Simple, use math to mix it in. Well, simple if you like math.

Just mix it in

A fundamental property of entropy is that it mixes well. If you take two unrelated random streams and combine them, the new stream cannot have less entropy. Taking a number of low entropy sources and combining them results in a high entropy source.

All that’s needed is the right combination function: a function that can be used to combine two sources of entropy. One of the simplest such functions is the logical exclusive or (XOR). This truth table shows how bits x and y coming from different random streams are combined by the XOR function.

Even if one source of bits does not have much entropy, there is no harm in XORing it into another source. Entropy always increases. In the Linux kernel, a combination of XORs is used to mix timestamps into the main entropy pool.

Generating random numbers

Cryptographic applications require very high entropy. If a 128 bit key is generated with only 64 bits of entropy then it can be guessed in 2⁶⁴ attempts instead of 2¹²⁸ attempts. That is the difference between needing a thousand computers running for a few years to brute force the key versus needing all the computers ever created running for longer than the history of the universe to do so.

Cryptographic applications require close to one bit of entropy per bit. If the system’s pool has fewer than 4,096 bits of entropy, how does the system return a fully random number? One way to do this is to use a cryptographic hash function.

A cryptographic hash function takes an input of any size and outputs a fixed size number. Changing one bit of the input will change the output completely. Hash functions are good at mixing things together. This mixing property spreads the entropy from the input evenly through the output. If the input has more bits of entropy than the size of the output, the output will be highly random. This is how highly entropic random numbers are derived from the entropy pool.

The hash function used by the Linux kernel is the standard SHA-1 cryptographic hash. By hashing the entire pool and and some additional arithmetic, 160 random bits are created for use by the system. When this happens, the system lowers its estimate of the entropy in the pool accordingly.

Above I said that applying a hash like SHA-1 could be dangerous if there wasn’t enough entropy in the pool. That’s why it’s critical to keep an eye on the available system entropy: if it drops too low the output of the random number generator could have less entropy that it appears to have.

Running out of entropy

One of the dangers of a system is running out of entropy. When the system’s entropy estimate drops to around the 160 bit level, the length of a SHA-1 hash, things get tricky, and how they effect programs and performance depends on which of two Linux random number generators are used.

Linux exposes two interfaces for random data that behave differently when the entropy level is low. They are /dev/random and /dev/urandom. When the entropy pool becomes predictable, both interfaces for requesting random numbers become problematic.

When the entropy level is too low, /dev/random blocks and does not return until the level of entropy in the system is high enough. This guarantees high entropy random numbers. If /dev/random is used in a time-critical service and the system runs low on entropy, the delays could be detrimental to the quality of service.

On the other hand, /dev/urandom does not block. It continues to return the hashed value of its entropy pool even though there is little to no entropy in it. This low-entropy data is not suited for cryptographic use.

The solution to the problem is to simply add more entropy into the system.

Hardware random number generation to the rescue?

Intel’s Ivy Bridge family of processors have an interesting feature called “secure key.” These processors contain a special piece of hardware inside that generates random numbers. The single assembly instruction RDRAND returns allegedly high entropy random data derived on the chip.

It has been suggested that Intel’s hardware number generator may not be fully random. Since it is baked into the silicon, that assertion is hard to audit and verify. As it turns out, even if the numbers generated have some bias, it can still help as long as this is not the only source of randomness in the system. Even if the random number generator itself had a back door, the mixing property of randomness means that it cannot lower the amount of entropy in the pool.

On Linux, if a hardware random number generator is present, the Linux kernel will use the XOR function to mix the output of RDRAND into the hash of the entropy pool. This happens here in the Linux source code (the XOR operator is ^ in C).

Third party entropy generators

Hardware number generation is not available everywhere, and the sources of randomness polled by the Linux kernel itself are somewhat limited. For this situation, a number of third party random number generation tools exist. Examples of these are haveged, which relies on processor cache timing, audio-entropyd and video-entropyd which work by sampling the noise from an external audio or video input device. By mixing these additional sources of locally collected entropy into the Linux entropy pool, the entropy can only go up.

/etc/security/limits.conf file – In A Nutshell

The /etc/security/limits.conf file contains a list line where each line describes a limit for a user in the form of:

<Domain> <type> <item> <shell limit value>

Where:

• <domain> can be:
  • an user name
  • a group name, with @group syntax
  • the wildcard *, for default entry
  • the wildcard %, can be also used with %group syntax, for maxlogin limit
• <type> can have the two values:
  • “soft” for enforcing the soft limits (soft is like warning)
  • “hard” for enforcing hard limits (hard is a real max limit)
• <item> can be one of the following:
  • core – limits the core file size (KB)
• <shell limit value> can be one of the following:
  • core – limits the core file size (KB)
  • data – max data size (KB)
  • fsize – maximum file size (KB)
  • memlock – max locked-in-memory address space (KB)
  • nofile – Maximum number of open file descriptors
  • rss – max resident set size (KB)
  • stack – max stack size (KB) – Maximum size of the stack segment of the process
  • cpu – max CPU time (MIN)
  • nproc – Maximum number of processes available to a single user
  • as – address space limit
  • maxlogins – max number of logins for this user
  • maxsyslogins – max number of logins on the system
  • priority – the priority to run user process with
  • locks – max number of file locks the user can hold
  • sigpending – max number of pending signals
  • msgqueue – max memory used by POSIX message queues (bytes)
  • nice – max nice priority allowed to raise to
  • rtprio – max realtime priority
  • chroot – change root to directory (Debian-specific)

 

• Sigpending – examine pending signals.

sigpending () returns the set of signals that are pending for delivery to the calling thread (i.e., the signals which have been raised while blocked). The mask of pending signals is returned in set.

sigpending() returns 0 on success and -1 on error

 

credits :- Sagar Salunkhe

Linux KVM: Disable virbr0 NAT Interface

The virtual network (virbr0) used for Network address translation (NAT) which allows guests to access to network services. However, NAT slows down things and only recommended for desktop installations. To disable Network address translation (NAT) forwarding type the following commands:

Display Current Setup

Type the following command:
# ifconfig
Sample outputs:

virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:7921 (7.7 KiB)

Or use the following command:
# virsh net-list
Sample outputs:

Name                 State      Autostart
-----------------------------------------
default              active     yes       

To disable virbr0, enter:
# virsh net-destroy default
# virsh net-undefine default
# service libvirtd restart
# ifconfig