← Back to migrated posts

🔧 Little Background About SAR

• Originally published on old blog

Note: This post was originally published in 2014. While the core concepts remain valid, modern system monitoring often uses more advanced tools. However, SAR remains a fundamental system administration tool available on most Linux systems.

By default, Linux and Unix machines store the SAR (System Activity Reporter) output for 9 days. It will be stored inside the /var/log/sa/ directory.

SAR File Storage

Here's what you'll typically see in the SAR directory:

[thiru@localhost ~]$ ls -ltr /var/log/sa/sa*|grep -v sar
-rw-r--r-- 1 root root 338928 Jan 13 23:50 /var/log/sa/sa13
-rw-r--r-- 1 root root 338928 Jan 14 23:50 /var/log/sa/sa14
-rw-r--r-- 1 root root 338928 Jan 15 23:50 /var/log/sa/sa15
-rw-r--r-- 1 root root 338928 Jan 16 23:50 /var/log/sa/sa16
-rw-r--r-- 1 root root 338928 Jan 17 23:50 /var/log/sa/sa17
-rw-r--r-- 1 root root 338928 Jan 18 23:50 /var/log/sa/sa18
-rw-r--r-- 1 root root 338928 Jan 19 23:50 /var/log/sa/sa19
-rw-r--r-- 1 root root 338928 Jan 20 23:50 /var/log/sa/sa20
-rw-r--r-- 1 root root  82560 Jan 21 05:40 /var/log/sa/sa21

We can extract the memory utilization, CPU, swap, and I/O from the stored SAR report by providing the specified SAR report file.

Memory Utilization Analysis

To analyze RAM memory utilization:

[thiru@localhost ~]$ sar -r -f /var/log/sa/sa20
10:40:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
10:50:01 PM 28089004 6510764 18.82 468820 386136 32505732 108 0.00 0
11:00:01 PM 28087392 6512376 18.82 470460 386144 32505732 108 0.00 0
11:10:01 PM 28081460 6518308 18.84 472020 390504 32505732 108 0.00 0
11:20:01 PM 28079972 6519796 18.84 473672 390504 32505732 108 0.00 0
11:30:02 PM 28078236 6521532 18.85 475384 390512 32505732 108 0.00 0
11:40:01 PM 28076872 6522896 18.85 477040 390508 32505732 108 0.00 0
11:50:01 PM 28073836 6525932 18.86 478784 390560 32505732 108 0.00 0
Average: 13386344 21213424 61.31 227578 15170261 32505732 108 0.00

Understanding Memory Output

  • kbmemfree: Available physical memory in KB
  • kbmemused: Used physical memory in KB
  • %memused: Percentage of memory used
  • kbbuffers: Memory used by kernel buffers in KB
  • kbcached: Memory used by page cache in KB
  • kbswpfree: Available swap space in KB
  • kbswpused: Used swap space in KB
  • %swpused: Percentage of swap space used

CPU Utilization Analysis

To analyze CPU utilization:

[thiru@localhost ~]$ sar -u -f /var/log/sa/sa20
05:40:01 AM CPU %user %nice %system %iowait %steal %idle
05:50:02 AM all 80.62 0.00 1.67 0.12 0.00 17.59
06:00:02 AM all 81.22 0.00 1.70 0.09 0.00 17.00
06:10:01 AM all 81.39 0.00 1.69 0.10 0.00 16.82
06:20:02 AM all 81.59 0.00 1.72 0.12 0.00 16.58
06:30:01 AM all 80.31 0.00 1.71 0.09 0.00 17.90
06:40:01 AM all 80.88 0.00 1.82 0.11 0.00 17.19
06:50:01 AM all 81.05 0.00 1.87 0.05 0.00 17.04
07:00:01 AM all 80.31 0.00 2.07 0.03 0.00 17.58
07:10:01 AM all 80.80 0.00 2.30 0.11 0.00 16.79
07:20:01 AM all 79.90 0.00 2.27 0.08 0.00 17.74
07:30:01 AM all 79.97 0.00 2.33 0.06 0.00 17.64
07:40:02 AM all 80.84 0.00 2.59 0.06 0.00 16.52
07:50:01 AM all 79.75 0.00 2.42 0.05 0.00 17.79
08:00:01 AM all 81.13 0.00 2.21 0.10 0.00 16.56
08:10:02 AM all 81.71 0.00 1.74 0.08 0.00 16.46

Understanding CPU Output

  • %user: Percentage of CPU time spent in user mode
  • %nice: Percentage of CPU time spent in user mode with nice priority
  • %system: Percentage of CPU time spent in kernel mode
  • %iowait: Percentage of CPU time spent waiting for I/O
  • %steal: Percentage of CPU time stolen by hypervisor (in virtual environments)
  • %idle: Percentage of CPU time spent idle

I/O Statistics Analysis

To analyze disk I/O statistics:

[thiru@localhost ~]$ sar -b -f /var/log/sa/sa20
10:40:01 PM tps rtps wtps bread/s bwrtn/s
10:50:01 PM 0.00 0.00 0.00 0.00 0.00
11:00:01 PM 0.00 0.00 0.00 0.00 0.00
11:10:01 PM 0.00 0.00 0.00 0.00 0.00
11:20:01 PM 0.00 0.00 0.00 0.00 0.00
11:30:02 PM 0.00 0.00 0.00 0.00 0.00
11:40:01 PM 0.00 0.00 0.00 0.00 0.00
11:50:01 PM 0.00 0.00 0.00 0.00 0.00

Understanding I/O Output

  • tps: Total transfers per second
  • rtps: Read transfers per second
  • wtps: Write transfers per second
  • bread/s: Blocks read per second
  • bwrtn/s: Blocks written per second

Network Statistics Analysis

To analyze network interface statistics:

[thiru@localhost ~]$ sar -n DEV -f /var/log/sa/sa20
10:40:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
10:50:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:00:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:10:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Understanding Network Output

  • IFACE: Network interface name
  • rxpck/s: Packets received per second
  • txpck/s: Packets transmitted per second
  • rxkB/s: Kilobytes received per second
  • txkB/s: Kilobytes transmitted per second
  • rxcmp/s: Compressed packets received per second
  • txcmp/s: Compressed packets transmitted per second
  • rxmcst/s: Multicast packets received per second

Common SAR Command Options

Here are some useful SAR command options:

# CPU utilization
sar -u [interval] [count]

# Memory utilization
sar -r [interval] [count]

# I/O statistics
sar -b [interval] [count]

# Network statistics
sar -n DEV [interval] [count]

# Load average
sar -q [interval] [count]

# Process statistics
sar -w [interval] [count]

# All statistics
sar -A [interval] [count]

# Read from file
sar -f /var/log/sa/sa[day]

Real-time Monitoring

To monitor system activity in real-time:

# Monitor CPU every 5 seconds, 10 times
sar -u 5 10

# Monitor memory every 10 seconds, 6 times
sar -r 10 6

# Monitor all statistics every 30 seconds, 4 times
sar -A 30 4

SAR Configuration

The SAR data collection is typically configured in /etc/cron.d/sysstat:

# Run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib/sa/sa1 1 1

# Generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib/sa/sa2 -A

Benefits of SAR

  • Built-in Tool: Available on most Linux distributions by default
  • Historical Data: Provides historical system performance data
  • Low Overhead: Minimal impact on system performance
  • Comprehensive: Covers CPU, memory, I/O, and network statistics
  • Automated Collection: Runs automatically via cron jobs
  • Text-based: Easy to parse and analyze programmatically

Limitations of SAR

  • Limited Retention: Only keeps data for 9 days by default
  • Basic Visualization: No built-in graphical interface
  • Fixed Intervals: Data collection intervals are predetermined
  • System-level Only: Doesn't provide application-level metrics
  • No Alerting: No built-in alerting capabilities

Modern Alternatives

While SAR is still useful, modern alternatives provide more features:

  • Prometheus: Time-series database with powerful querying
  • Grafana: Advanced visualization and dashboarding
  • Datadog: Cloud-based monitoring with AI-powered insights
  • New Relic: Application performance monitoring
  • Nagios: Infrastructure monitoring with alerting

Key Takeaways

  • SAR is a fundamental system monitoring tool available on most Linux systems
  • It provides comprehensive system performance data with minimal overhead
  • Historical data helps identify performance trends and capacity planning
  • SAR data can be combined with visualization tools for better insights
  • Understanding SAR output is essential for system administration