System Configuration Analysis Skill

This skill provides detailed guidance for analyzing system configuration from sosreport archives, including OS information, installed packages, systemd services, and SELinux/AppArmor settings.

When to Use This Skill

Use this skill when:

• Analyzing the /sosreport:analyze command's system configuration phase
• Investigating service failures or misconfigurations
• Verifying package versions and updates
• Checking security policy settings (SELinux/AppArmor)
• Understanding system state and configuration

Prerequisites

• Sosreport archive must be extracted to a working directory
• Path to the sosreport root directory must be known
• Understanding of Linux system administration

Key Configuration Data Locations in Sosreport

1. System Information:

- uname - Kernel version - etc/os-release - OS distribution and version - uptime - System uptime - proc/uptime - Uptime in seconds - sos_commands/release/ - Release information

1. Package Information:

- installed-rpms - RPM packages (RHEL/Fedora/CentOS) - installed-debs - DEB packages (Debian/Ubuntu) - soscommands/yum/ - Yum/DNF information - soscommands/rpm/ - RPM database queries

1. Service Status:

- soscommands/systemd/systemctllist-units - All units - soscommands/systemd/systemctllist-units--failed - Failed units - soscommands/systemd/systemctlstatus--all - Detailed service status - soscommands/systemd/systemctllist-unit-files - Unit files

1. SELinux:

- soscommands/selinux/sestatus - SELinux status - soscommands/selinux/getenforce - Current enforcement mode - sos_commands/selinux/selinux-policy - Policy information - var/log/audit/audit.log - SELinux denials

1. AppArmor (if applicable):

- sos_commands/apparmor/ - AppArmor configuration - etc/apparmor.d/ - AppArmor profiles

1. System Configuration Files:

- etc/ - System-wide configuration - etc/sysctl.conf or etc/sysctl.d/ - Kernel parameters - etc/security/limits.conf - Resource limits

Implementation Steps

Step 1: Analyze System Information

1. Check OS version and distribution:

``bash if [ -f etc/os-release ]; then cat etc/os-release fi ``

1. Get kernel version:

``bash if [ -f uname ]; then cat uname elif [ -f proc/version ]; then cat proc/version fi ``

1. Check system uptime:

``bash if [ -f uptime ]; then cat uptime elif [ -f proc/uptime ]; then # Parse uptime from proc/uptime (seconds) awk '{printf "%.2f days\n", $1/86400}' proc/uptime fi ``

1. Extract key system details:

- OS name and version - Kernel version - System architecture (x86_64, aarch64, etc.) - Uptime (days)

1. Check for outdated kernel or OS:

- Compare kernel version with current stable - Note if system hasn't been rebooted in a very long time (>365 days) - Identify if OS version is EOL

Step 2: Analyze Installed Packages

1. List installed packages:

```bash # For RPM-based systems if [ -f installed-rpms ]; then cat installed-rpms fi

# For DEB-based systems if [ -f installed-debs ]; then cat installed-debs fi ```

1. Extract key package versions:

```bash # Important system packages grep -E "^(kernel|systemd|glibc|openssh|openssl)" installed-rpms 2>/dev/null

# Or use awk to parse package name and version awk '{print $1}' installed-rpms | head -20 ```

1. Check for known problematic versions:

- Security vulnerabilities (if known CVEs) - Buggy package versions - Compatibility issues

1. Identify package manager issues:

``bash # Check yum/dnf logs for errors if [ -d soscommands/yum ]; then grep -i "error\|fail" soscommands/yum/* 2>/dev/null fi ``

1. Count packages and categorize:

- Total packages installed - Key package versions (kernel, systemd, glibc, etc.) - Recently updated packages (if timestamps available)

Step 3: Analyze Service Status

1. List all systemd units:

``bash if [ -f soscommands/systemd/systemctllist-units ]; then cat soscommands/systemd/systemctllist-units fi ``

1. Identify failed services:

``bash if [ -f soscommands/systemd/systemctllist-units--failed ]; then cat soscommands/systemd/systemctllist-units--failed elif [ -f soscommands/systemd/systemctllist-units ]; then grep "failed" soscommands/systemd/systemctllist-units fi ``

1. Check service details:

``bash # Parse detailed status for failed services if [ -f soscommands/systemd/systemctlstatus--all ]; then # Extract service names and their status grep -E "●|Active:" soscommands/systemd/systemctlstatus--all | head -50 fi ``

1. Count services by state:

``bash # Count running, failed, inactive services if [ -f soscommands/systemd/systemctllist-units ]; then awk '{print $4}' soscommands/systemd/systemctllist-units | sort | uniq -c fi ``

1. Identify critical service failures:

- System services (systemd-*, dbus, NetworkManager) - Application services (httpd, nginx, postgresql, etc.) - Custom services

1. Extract failure reasons from logs:

``bash # For each failed service, find related log entries grep -i "failed to start\|service.*failed" soscommands/logs/journalctl--no-pager 2>/dev/null | head -20 ``

Step 4: Analyze SELinux Configuration

1. Check SELinux status:

``bash if [ -f soscommands/selinux/sestatus ]; then cat soscommands/selinux/sestatus fi ``

1. Get SELinux mode:

``bash if [ -f soscommands/selinux/getenforce ]; then cat soscommands/selinux/getenforce fi ``

1. Check for SELinux denials:

```bash # Look for AVC denials in audit log if [ -f var/log/audit/audit.log ]; then grep "avc.*denied" var/log/audit/audit.log | head -50 fi

# Or in journald logs grep -i "selinux.denied\|avc.denied" soscommands/logs/journalctl--no-pager 2>/dev/null | head -20 ```

1. Parse denial information:

- Extract denied operations (read, write, execute, etc.) - Identify source and target contexts - Note which services are affected

1. Check for SELinux booleans:

``bash if [ -f soscommands/selinux/getsebool-a ]; then cat soscommands/selinux/getsebool-a fi ``

1. Identify SELinux issues:

- SELinux in permissive mode (may hide errors) - SELinux disabled (security concern) - Frequent AVC denials (policy may need adjustment) - Context mismatches

Step 5: Check System Configuration

1. Review kernel parameters:

``bash # Check sysctl settings if [ -f soscommands/kernel/sysctl-a ]; then cat soscommands/kernel/sysctl-a elif [ -d etc/sysctl.d ]; then cat etc/sysctl.d/*.conf 2>/dev/null fi ``

1. Check resource limits:

```bash if [ -f etc/security/limits.conf ]; then grep -v "^#\|^$" etc/security/limits.conf fi

# Check limits.d directory if [ -d etc/security/limits.d ]; then cat etc/security/limits.d/*.conf 2>/dev/null fi ```

1. Review boot parameters:

``bash if [ -f soscommands/boot/grub2-editenvlist ]; then cat soscommands/boot/grub2-editenvlist elif [ -f proc/cmdline ]; then cat proc/cmdline fi ``

1. Check systemd configuration:

``bash # Look for systemd configuration overrides if [ -d etc/systemd/system ]; then find etc/systemd/system -name "*.conf" 2>/dev/null fi ``

Step 6: Generate System Configuration Summary

Create a structured summary with the following sections:

1. System Information:

- OS name and version - Kernel version - Architecture - System uptime - Last boot time

1. Package Summary:

- Total packages installed - Key package versions (kernel, systemd, glibc, openssl, openssh) - Known problematic packages (if any) - Package manager issues

1. Service Status:

- Total services - Running services count - Failed services count - List of failed services with reasons - Critical service status

1. SELinux/AppArmor:

- SELinux status (enabled/disabled) - SELinux mode (enforcing/permissive) - Denial count - Top denied operations - Policy recommendations

1. Configuration Issues:

- Kernel parameter anomalies - Resource limit issues - Boot parameter problems - Configuration file errors

Error Handling

1. Missing configuration files:

- Different distributions have different file locations - Some files may not be collected based on sosreport options - Document missing data in summary

1. Package manager variations:

- Handle both RPM and DEB systems - Account for different package naming conventions - Support multiple package managers (yum, dnf, apt)

1. SELinux vs AppArmor:

- Check which MAC system is in use - Analyze accordingly - Note if both or neither are present

1. Systemd vs init:

- Older systems may use init instead of systemd - Check for both service management systems - Adapt analysis based on what's present

Output Format

The system configuration analysis should produce:

SYSTEM CONFIGURATION SUMMARY
============================

SYSTEM INFORMATION
------------------
OS: {os_name} {os_version}
Kernel: {kernel_version}
Architecture: {arch}
Uptime: {uptime_days} days ({last_boot_time})

Status: {OK|WARNING|CRITICAL}
Notes:
  - {system_info_note}

INSTALLED PACKAGES
------------------
Total Packages: {count}

Key Package Versions:
  kernel: {version}
  systemd: {version}
  glibc: {version}
  openssl: {version}
  openssh-server: {version}

Status: {OK|WARNING|CRITICAL}
Issues:
  - {package_issue_description}

SYSTEMD SERVICES
----------------
Total Units: {total}
Active: {active_count}
Failed: {failed_count}
Inactive: {inactive_count}

Failed Services:
  ● {service_name}.service - {description}
    Reason: {failure_reason}
    Last Failed: {timestamp}

  ● {service_name}.service - {description}
    Reason: {failure_reason}
    Last Failed: {timestamp}

Status: {OK|WARNING|CRITICAL}
Recommendations:
  - {service_recommendation}

SELINUX
-------
Status: {enabled|disabled}
Mode: {enforcing|permissive|disabled}
Policy: {policy_name}

AVC Denials: {count} denials found

Top Denied Operations:
  [{count}x] {operation} on {target} by {source}
  [{count}x] {operation} on {target} by {source}

SELinux Booleans: {count} custom settings

Status: {OK|WARNING|CRITICAL}
Issues:
  - {selinux_issue_description}

Recommendations:
  - {selinux_recommendation}

KERNEL PARAMETERS
-----------------
Key sysctl Settings:
  vm.swappiness: {value}
  net.ipv4.ip_forward: {value}
  kernel.panic: {value}

Custom Parameters: {count} custom settings found

Status: {OK|WARNING|CRITICAL}
Notes:
  - {kernel_param_note}

RESOURCE LIMITS
---------------
Custom Limits Found: {count}

{user_or_group}  {type}  {item}  {value}

Status: {OK|WARNING}
Notes:
  - {limits_note}

CRITICAL CONFIGURATION ISSUES
-----------------------------
{severity}: {issue_description}
  Evidence: {file_path}
  Impact: {impact_description}
  Recommendation: {remediation_action}

RECOMMENDATIONS
---------------
1. {actionable_recommendation}
2. {actionable_recommendation}

DATA SOURCES
------------
- OS Info: {sosreport_path}/etc/os-release
- Kernel: {sosreport_path}/uname
- Packages: {sosreport_path}/installed-rpms
- Services: {sosreport_path}/sos_commands/systemd/systemctl_list-units
- SELinux: {sosreport_path}/sos_commands/selinux/sestatus
- Audit Log: {sosreport_path}/var/log/audit/audit.log

Examples

Example 1: Failed Service Analysis

# List failed services
$ cat sos_commands/systemd/systemctl_list-units_--failed
  UNIT                    LOAD   ACTIVE SUB    DESCRIPTION
● httpd.service          loaded failed failed Apache Web Server
● postgresql.service     loaded failed failed PostgreSQL database

# Find failure reason in logs
$ grep "httpd.service" sos_commands/logs/journalctl_--no-pager | grep -i "failed\|error"
Jan 15 10:23:45 server systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Jan 15 10:23:45 server systemd[1]: httpd.service: Failed with result 'exit-code'
Jan 15 10:23:45 server httpd[12345]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:80

# Interpretation: httpd failed because port 80 is already in use

Example 2: SELinux Denial Analysis

# Check for AVC denials
$ grep "avc.*denied" var/log/audit/audit.log | head -5
type=AVC msg=audit(1705320245.123:456): avc: denied { write } for pid=1234 comm="httpd" name="index.html" dev="sda1" ino=789012 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

# Interpretation:
# - httpd (web server) was denied write access
# - Target file: index.html with context user_home_t
# - Issue: Web server trying to write to user home directory
# - Solution: Fix file context or move file to proper location

Example 3: Package Version Check

# Check for specific package versions
$ grep "^openssl" installed-rpms
openssl-1.1.1k-7.el8_6.x86_64
openssl-libs-1.1.1k-7.el8_6.x86_64

$ grep "^kernel" installed-rpms
kernel-4.18.0-425.el8.x86_64
kernel-4.18.0-477.el8.x86_64
kernel-core-4.18.0-425.el8.x86_64
kernel-core-4.18.0-477.el8.x86_64

# Interpretation:
# - OpenSSL version 1.1.1k (check for known CVEs)
# - Multiple kernels installed (good for rollback)
# - Current kernel is 4.18.0-477 (from uname)

Tips for Effective Analysis

1. Check service dependencies: Failed service may be due to dependency failure
2. Correlate with logs: Service failures often have detailed errors in logs
3. Verify configurations: Check service config files for syntax errors
4. Consider timing: When did service fail? Correlate with system events
5. SELinux context matters: File contexts must match policy expectations
6. Package versions: Compare with known good/bad versions
7. Uptime significance: Very long uptime may mean missed security updates

Common Configuration Patterns and Issues

1. Service dependency failure: ServiceB fails because ServiceA is not running
2. Port conflict: Service fails to bind - port already in use
3. Permission denied: Service can't access required files/directories
4. SELinux blocking: Service denied access by SELinux policy
5. Missing dependencies: Required package not installed
6. Configuration error: Syntax error in config file
7. Resource limits: Service hits ulimit (open files, processes, etc.)
8. Outdated kernel: Running kernel doesn't match installed packages

Configuration Issue Severity Classification

See Also

• Logs Analysis Skill: For detailed service failure log analysis
• Resource Analysis Skill: For resource limit issues
• Network Analysis Skill: For network service configuration