diff --git a/Makefile b/Makefile index 470769f0..918fe0dc 100644 --- a/Makefile +++ b/Makefile @@ -200,3 +200,36 @@ clean: sweep clean-tools-cache clean-tools-cache: @echo "Removing cached tool binaries..." rm -rf tools-cache + +# Generate command documentation from help text +COMMANDS := report benchmark metrics telemetry flamegraph lock config update extract +SUBCOMMANDS := metrics:trim config:restore + +.PHONY: docs +docs: perfspect + @echo "Generating command documentation..." + @mkdir -p docs + @echo '# perfspect' > docs/perfspect.md + @echo '' >> docs/perfspect.md + @echo '```text' >> docs/perfspect.md + @./perfspect --help >> docs/perfspect.md + @echo '```' >> docs/perfspect.md + @for cmd in $(COMMANDS); do \ + echo " $$cmd"; \ + echo "# perfspect $$cmd" > docs/perfspect_$$cmd.md; \ + echo '' >> docs/perfspect_$$cmd.md; \ + echo '```text' >> docs/perfspect_$$cmd.md; \ + ./perfspect $$cmd --help >> docs/perfspect_$$cmd.md; \ + echo '```' >> docs/perfspect_$$cmd.md; \ + done + @for sub in $(SUBCOMMANDS); do \ + cmd=$${sub%%:*}; \ + subcmd=$${sub##*:}; \ + echo " $$cmd $$subcmd"; \ + echo "# perfspect $$cmd $$subcmd" > docs/perfspect_$${cmd}_$${subcmd}.md; \ + echo '' >> docs/perfspect_$${cmd}_$${subcmd}.md; \ + echo '```text' >> docs/perfspect_$${cmd}_$${subcmd}.md; \ + ./perfspect $$cmd $$subcmd --help >> docs/perfspect_$${cmd}_$${subcmd}.md; \ + echo '```' >> docs/perfspect_$${cmd}_$${subcmd}.md; \ + done + @echo "Documentation generated in docs/" diff --git a/README.md b/README.md index 7cbb8acc..661ac223 100644 --- a/README.md +++ b/README.md @@ -28,30 +28,32 @@ Usage: ### Commands -| Command | Description | -| ------- | ----------- | -| [`metrics`](#metrics-command) | CPU core and uncore metrics | -| [`report`](#report-command) | System configuration and health | -| [`benchmark`](#benchmark-command) | Performance benchmarks | -| [`telemetry`](#telemetry-command) | System telemetry | -| [`flamegraph`](#flamegraph-command) | Software call-stacks as flamegraphs | -| [`lock`](#lock-command) | Software hot spot, cache-to-cache and lock contention | -| [`config`](#config-command) | Modify system configuration | +| Command | Description | Reference | +| ------- | ----------- | --------- | +| [`metrics`](#metrics-command) | CPU core and uncore metrics | [options](docs/perfspect_metrics.md) | +| [`report`](#report-command) | System configuration and health | [options](docs/perfspect_report.md) | +| [`benchmark`](#benchmark-command) | Performance benchmarks | [options](docs/perfspect_benchmark.md) | +| [`telemetry`](#telemetry-command) | System telemetry | [options](docs/perfspect_telemetry.md) | +| [`flamegraph`](#flamegraph-command) | Software call-stacks as flamegraphs | [options](docs/perfspect_flamegraph.md) | +| [`lock`](#lock-command) | Software hot spot, cache-to-cache and lock contention | [options](docs/perfspect_lock.md) | +| [`config`](#config-command) | Modify system configuration | [options](docs/perfspect_config.md) | > [!TIP] -> Run `perfspect [command] -h` to view command-specific help text. +> Run `perfspect [command] -h` to view command-specific help text. See [`perfspect -h`](docs/perfspect.md) for global options. + +Additional commands: [`update`](docs/perfspect_update.md) checks for and applies application updates (Intel network only), and [`extract`](docs/perfspect_extract.md) extracts embedded resources (for developers). #### Metrics Command The `metrics` command generates reports containing CPU architectural performance characterization metrics in HTML and CSV formats. Run `perfspect metrics`. -![screenshot of the TMAM page from the metrics command HTML report, provides a description of TMAM on the left and a pie chart showing the 1st and 2nd level TMAM metrics on the right](docs/metrics_html_tma.png) +![screenshot of the TMAM page from the metrics command HTML report, provides a description of TMAM on the left and a pie chart showing the 1st and 2nd level TMAM metrics on the right](docs/images/metrics_html_tma.png) ##### Live Metrics The `metrics` command supports two modes -- default and "live". Default mode behaves as above -- metrics are collected and saved into report files for review. The "live" mode prints the metrics to stdout where they can be viewed in the console and/or redirected into a file or observability pipeline. Run `perfspect metrics --live`. -![screenshot of live CSV metrics in a text terminal](docs/metrics_live.png) +![screenshot of live CSV metrics in a text terminal](docs/images/metrics_live.png) ##### Metrics Without Root Permissions @@ -65,7 +67,7 @@ Once the configuration changes are applied, use the `--noroot` flag on the comma ##### Refining Metrics to a Specific Time Range -After collecting metrics, you can generate new summary reports for a specific time interval using the `metrics trim` subcommand. This is useful when you've collected metrics for an entire workload but want to analyze only a specific portion, excluding setup, teardown, or other unwanted phases. +After collecting metrics, you can generate new summary reports for a specific time interval using the [`metrics trim`](docs/perfspect_metrics_trim.md) subcommand. This is useful when you've collected metrics for an entire workload but want to analyze only a specific portion, excluding setup, teardown, or other unwanted phases. The time range can be specified using either absolute timestamps (seconds since epoch) or relative offsets from the beginning/end of the data. At least one time parameter must be specified. @@ -85,13 +87,13 @@ $ ./perfspect metrics trim --input perfspect_2025-11-28_09-21-56 --start-time 17 The `metrics` command can expose metrics via a Prometheus compatible `metrics` endpoint. This allows integration with Prometheus monitoring systems. To enable the Prometheus endpoint, use the `--prometheus-server` flag. By default, the endpoint listens on port 9090. The port can be changed using the `--prometheus-server-addr` flag. Run `perfspect metrics --prometheus-server`. See the [example daemonset](docs/perfspect-daemonset.md) for deploying in Kubernetes. -See `perfspect metrics -h` for the extensive set of options and examples. +See [`perfspect metrics -h`](docs/perfspect_metrics.md) for the extensive set of options and examples. #### Report Command -The `report` command generates system configuration reports in a variety of formats. All categories of information are collected by default. See `perfspect report -h` for all options. +The `report` command generates system configuration reports in a variety of formats. All categories of information are collected by default. See [`perfspect report -h`](docs/perfspect_report.md) for all options. -![screenshot of a small section of the HTML report from the report command](docs/report_html.png) +![screenshot of a small section of the HTML report from the report command](docs/images/report_html.png) It's possible to report a subset of information by providing command line options. Note that by specifying only the `txt` format, it is printed to stdout, as well as written to a report file. @@ -119,7 +121,7 @@ The `benchmark` command runs performance micro-benchmarks to evaluate system hea ./perfspect benchmark --no-summary # Exclude system summary from output ``` -See `perfspect benchmark -h` for all options. +See [`perfspect benchmark -h`](docs/perfspect_benchmark.md) for all options. | Benchmark | Description | | --------- | ----------- | @@ -133,9 +135,9 @@ See `perfspect benchmark -h` for all options. #### Telemetry Command -The `telemetry` command reports CPU utilization, instruction mix, disk stats, network stats, and more on the specified target(s). All telemetry types are collected by default. To choose telemetry types, see the additional command line options (`perfspect telemetry -h`). +The `telemetry` command reports CPU utilization, instruction mix, disk stats, network stats, and more on the specified target(s). All telemetry types are collected by default. To choose telemetry types, see the additional command line options ([`perfspect telemetry -h`](docs/perfspect_telemetry.md)). -![screenshot of the CPU utilization chart from the HTML output of the telemetry command](docs/telemetry_html.png) +![screenshot of the CPU utilization chart from the HTML output of the telemetry command](docs/images/telemetry_html.png) ##### Additional Telemetry via Environment Variables @@ -151,20 +153,20 @@ The following optional telemetry sources can be enabled via environment variable #### Flamegraph Command -Software flamegraphs are useful in diagnosing software performance bottlenecks. Run `perfspect flamegraph` to capture a system-wide software flamegraph. +Software flamegraphs are useful in diagnosing software performance bottlenecks. Run `perfspect flamegraph` to capture a system-wide software flamegraph. See [`perfspect flamegraph -h`](docs/perfspect_flamegraph.md) for all options. > [!TIP] > By default, flamegraphs are collected using the `cycles:P` event. To analyze different performance aspects, use the `--perf-event` flag to specify an alternative perf event (e.g., `cache-misses`, `instructions`, `branches`, `context-switches`, `mem-loads`, `mem-stores`, etc.). -![screenshot of a flamegraph from the HTML output of the flamegraph command](docs/flamegraph.png) +![screenshot of a flamegraph from the HTML output of the flamegraph command](docs/images/flamegraph.png) #### Lock Command -As systems contain more and more cores, it can be useful to analyze the Linux kernel lock overhead and potential false-sharing that impacts system scalability. Run `perfspect lock` to collect system-wide hot spot, cache-to-cache and lock contention information. Experienced performance engineers can analyze the collected information to identify bottlenecks. +As systems contain more and more cores, it can be useful to analyze the Linux kernel lock overhead and potential false-sharing that impacts system scalability. Run `perfspect lock` to collect system-wide hot spot, cache-to-cache and lock contention information. Experienced performance engineers can analyze the collected information to identify bottlenecks. See [`perfspect lock -h`](docs/perfspect_lock.md) for all options. #### Config Command -The `config` command provides a method to view and change various system configuration parameters. Run `perfspect config -h` to view the parameters that can be modified. +The `config` command provides a method to view and change various system configuration parameters. Run [`perfspect config -h`](docs/perfspect_config.md) to view the parameters that can be modified. > [!WARNING] > Misconfiguring the system may cause it to stop functionining. In some cases, a reboot may be required to restore default settings. @@ -189,7 +191,7 @@ Configuration recorded to: perfspect_2025-12-01_14-30-45/gnr_config.txt ##### Restoring Configuration -The `config restore` subcommand restores configuration from a previously recorded file. This is useful for reverting changes or applying a known-good configuration across multiple systems. +The [`config restore`](docs/perfspect_config_restore.md) subcommand restores configuration from a previously recorded file. This is useful for reverting changes or applying a known-good configuration across multiple systems. Example: diff --git a/docs/flamegraph.png b/docs/images/flamegraph.png similarity index 100% rename from docs/flamegraph.png rename to docs/images/flamegraph.png diff --git a/docs/metrics_html_tma.png b/docs/images/metrics_html_tma.png similarity index 100% rename from docs/metrics_html_tma.png rename to docs/images/metrics_html_tma.png diff --git a/docs/metrics_live.png b/docs/images/metrics_live.png similarity index 100% rename from docs/metrics_live.png rename to docs/images/metrics_live.png diff --git a/docs/report_html.png b/docs/images/report_html.png similarity index 100% rename from docs/report_html.png rename to docs/images/report_html.png diff --git a/docs/telemetry_html.png b/docs/images/telemetry_html.png similarity index 100% rename from docs/telemetry_html.png rename to docs/images/telemetry_html.png diff --git a/docs/perfspect.md b/docs/perfspect.md new file mode 100644 index 00000000..ad1fa902 --- /dev/null +++ b/docs/perfspect.md @@ -0,0 +1,42 @@ +# perfspect + +```text +PerfSpect (perfspect) is a multi-function utility for performance engineers analyzing software running on Intel Xeon platforms. + +Usage: + perfspect [command] [flags] + +Examples: + Generate a configuration report: $ perfspect report + Collect micro-architectural metrics: $ perfspect metrics + Generate a configuration report on a remote target: $ perfspect report --target 192.168.1.2 --user elaine --key ~/.ssh/id_rsa + Generate configuration reports for multiple remote targets: $ perfspect report --targets ./targets.yaml + +Use "perfspect [command] --help" for more information about a command. + +Commands: + report Collect configuration data from target(s) + benchmark Run performance benchmarks on target(s) + metrics Collect performance metrics from target(s) + telemetry Collect system telemetry from target(s) + flamegraph Collect flamegraph data from target(s) + lock Collect kernel lock data from target(s) + config Modify system configuration on target(s) + +Other Commands: + update Update the application (Intel network only) + extract Extract the embedded resources (for developers) + +Flags: + --debug enable debug logging and retain temporary directories + -h, --help help for perfspect + --log-stdout write logs to stdout + --noupdate skip application update check + --output string override the output directory + --syslog write logs to syslog instead of a file + --tempdir string override the temporary target directory, must exist and allow execution + -v, --version version for perfspect + +Additional help topics: + perfspect +``` diff --git a/docs/perfspect_benchmark.md b/docs/perfspect_benchmark.md new file mode 100644 index 00000000..3c32260d --- /dev/null +++ b/docs/perfspect_benchmark.md @@ -0,0 +1,44 @@ +# perfspect benchmark + +```text +Run performance benchmarks on target(s) + +Usage: perfspect benchmark [flags] + +Examples: + Run all benchmarks: $ perfspect benchmark + Run specific benchmarks: $ perfspect benchmark --speed --power + Benchmark remote target: $ perfspect benchmark --target 192.168.1.1 --user fred --key fred_key + Benchmark multiple targets:$ perfspect benchmark --targets targets.yaml + +Flags: + Benchmark Options: + --all run all benchmarks (default: true) + --speed CPU speed benchmark (default: false) + --power power consumption benchmark (default: false) + --temperature temperature benchmark (default: false) + --frequency turbo frequency benchmark (default: false) + --memory memory latency and bandwidth benchmark (default: false) + --numa NUMA bandwidth matrix benchmark (default: false) + --storage storage performance benchmark (default: false) + Other Options: + --no-summary do not include system summary in output (default: false) + --storage-dir existing directory where storage performance benchmark data will be temporarily stored (default: /tmp) + --format choose output format(s) from: all, html, xlsx, json, txt (default: [all]) + Remote Target Options: + --target host name or IP address of remote target + --port port for SSH to remote target + --user user name for SSH to remote target + --key private key file for SSH to remote target + --targets file with remote target(s) connection details. See targets.yaml for format. + Advanced Options: + --input ".raw" file, or directory containing ".raw" files. Will skip data collection and use raw data for reports. + +Global Flags: + --debug enable debug logging and retain temporary directories (default: false) + --log-stdout write logs to stdout (default: false) + --noupdate skip application update check (default: false) + --output override the output directory + --syslog write logs to syslog instead of a file (default: false) + --tempdir override the temporary target directory, must exist and allow execution +``` diff --git a/docs/perfspect_config.md b/docs/perfspect_config.md new file mode 100644 index 00000000..b1a7dba2 --- /dev/null +++ b/docs/perfspect_config.md @@ -0,0 +1,73 @@ +# perfspect config + +```text +Sets system configuration items on target platform(s). + +USE CAUTION! Target may become unstable. It is up to the user to ensure that the requested configuration is valid for the target. There is not an automated way to revert the configuration changes. If all else fails, reboot the target. + +Usage: perfspect config [flags] + +Examples: + Set core count on local host: $ perfspect config --cores 32 + Set multiple config items on local host: $ perfspect config --core-max 3.0 --uncore-max 2.1 --tdp 120 + Record config to file before changes: $ perfspect config --c6 disable --epb 0 --record + Restore config from file: $ perfspect config restore gnr_config.txt + Set core count on remote target: $ perfspect config --cores 32 --target 192.168.1.1 --user fred --key fred_key + View current config on remote target: $ perfspect config --target 192.168.1.1 --user fred --key fred_key + Set governor on remote targets: $ perfspect config --gov performance --targets targets.yaml + +Flags: + General Options: + --cores number of physical cores per processor + --llc LLC size in MB + --tdp maximum power per processor in Watts + --core-max SSE frequency in GHz + --core-max-buckets SSE frequencies for all core buckets in GHz (e.g., 1-40/3.5, 41-60/3.4, 61-86/3.2) + --epb energy perf bias from best performance (0) to most power savings (15) + --epp energy perf profile from best performance (0) to most power savings (255) + --gov CPU scaling governor (performance, powersave) + --elc efficiency latency control (latency, power) [SRF+] + Uncore Frequency Options: + --uncore-max maximum uncore frequency in GHz [EMR-] + --uncore-min minimum uncore frequency in GHz [EMR-] + --uncore-max-compute maximum uncore compute die frequency in GHz [SRF+] + --uncore-min-compute minimum uncore compute die frequency in GHz [SRF+] + --uncore-max-io maximum uncore IO die frequency in GHz [SRF+] + --uncore-min-io minimum uncore IO die frequency in GHz [SRF+] + Prefetcher Options: + --pref-l2hw L2 HW [all] (enable, disable) + --pref-l2adj L2 Adj [all] (enable, disable) + --pref-dcuhw DCU HW [all] (enable, disable) + --pref-dcuip DCU IP [all] (enable, disable) + --pref-dcunp DCU NP [all] (enable, disable) + --pref-amp AMP [SPR,EMR,GNR,DMR] (enable, disable) + --pref-llcpp LLCPP [GNR,DMR] (enable, disable) + --pref-aop AOP [GNR] (enable, disable) + --pref-l2p L2P [DMR] (enable, disable) + --pref-homeless Homeless [SPR,EMR,GNR] (enable, disable) + --pref-llc LLC [SPR,EMR,GNR] (enable, disable) + --pref-llcstream LLC Stream [SRF,CWF] (enable, disable) + C-State Options: + --c6 C6 (enable, disable) + --c1-demotion C1 Demotion (enable, disable) + Other Options: + --no-summary do not print configuration summary + --record record the current configuration to a file to be restored later + Remote Target Options: + --target host name or IP address of remote target + --port port for SSH to remote target + --user user name for SSH to remote target + --key private key file for SSH to remote target + --targets file with remote target(s) connection details. See targets.yaml for format. + +Subcommands: + restore: Restore system configuration from file + +Global Flags: + --debug enable debug logging and retain temporary directories (default: false) + --log-stdout write logs to stdout (default: false) + --noupdate skip application update check (default: false) + --output override the output directory + --syslog write logs to syslog instead of a file (default: false) + --tempdir override the temporary target directory, must exist and allow execution +``` diff --git a/docs/perfspect_config_restore.md b/docs/perfspect_config_restore.md new file mode 100644 index 00000000..78cb84de --- /dev/null +++ b/docs/perfspect_config_restore.md @@ -0,0 +1,36 @@ +# perfspect config restore + +```text +Restores system configuration from a file that was previously recorded using the --record flag. + +The restore command will parse the configuration file, validate the settings against the target system, +and apply the configuration changes. By default, you will be prompted to confirm before applying changes. + +Usage: perfspect config restore [flags] + +Examples: + Restore config from file on local host: $ perfspect config restore gnr_config.txt + Restore config on remote target: $ perfspect config restore gnr_config.txt --target 192.168.1.1 --user fred --key fred_key + Restore config without confirmation: $ perfspect config restore gnr_config.txt --yes + +Arguments: + file: path to the configuration file to restore + +Flags: + General Options: + --yes skip confirmation prompt + Remote Target Options: + --target host name or IP address of remote target + --port port for SSH to remote target + --user user name for SSH to remote target + --key private key file for SSH to remote target + --targets file with remote target(s) connection details. See targets.yaml for format. + +Global Flags: + --debug enable debug logging and retain temporary directories (default: false) + --log-stdout write logs to stdout (default: false) + --noupdate skip application update check (default: false) + --output override the output directory + --syslog write logs to syslog instead of a file (default: false) + --tempdir override the temporary target directory, must exist and allow execution +``` diff --git a/docs/perfspect_extract.md b/docs/perfspect_extract.md new file mode 100644 index 00000000..6492c469 --- /dev/null +++ b/docs/perfspect_extract.md @@ -0,0 +1,21 @@ +# perfspect extract + +```text +Extract the embedded resources (for developers) + +Usage: + perfspect extract [flags] + +Use "perfspect extract [command] --help" for more information about a command. + +Flags: + -h, --help help for extract + +Global Flags: + --debug enable debug logging and retain temporary directories + --log-stdout write logs to stdout + --noupdate skip application update check + --output string override the output directory + --syslog write logs to syslog instead of a file + --tempdir string override the temporary target directory, must exist and allow execution +``` diff --git a/docs/perfspect_flamegraph.md b/docs/perfspect_flamegraph.md new file mode 100644 index 00000000..5c9396fc --- /dev/null +++ b/docs/perfspect_flamegraph.md @@ -0,0 +1,40 @@ +# perfspect flamegraph + +```text +Collect flamegraph data from target(s) + +Usage: perfspect flamegraph [flags] + +Examples: + Flamegraph from local host: $ perfspect flamegraph + Flamegraph from remote target: $ perfspect flamegraph --target 192.168.1.1 --user fred --key fred_key + Flamegraph from multiple targets: $ perfspect flamegraph --targets targets.yaml + Flamegraph for cache misses: $ perfspect flamegraph --perf-event cache-misses + +Flags: + Options: + --duration number of seconds to run the collection. If 0, the collection will run indefinitely. Ctrl+c to stop. (default: 0) + --frequency number of samples taken per second (default: 11) + --pids comma separated list of PIDs. If not specified, all PIDs will be collected (default: []) + --perf-event perf event to use for native sampling (e.g., cpu-cycles, instructions, cache-misses, branches, context-switches, mem-loads, mem-stores, etc.) (default: cycles:P) + --asprof-args arguments to pass to async-profiler, e.g., $ asprof start -i . (default: -t -F probesp+vtable) + --max-depth maximum render depth of call stack in flamegraph (0 = no limit) (default: 0) + --format choose output format(s) from: all, html, txt, json (default: [html]) + --no-summary do not include system summary table in report (default: false) + Remote Target Options: + --target host name or IP address of remote target + --port port for SSH to remote target + --user user name for SSH to remote target + --key private key file for SSH to remote target + --targets file with remote target(s) connection details. See targets.yaml for format. + Advanced Options: + --input ".raw" file, or directory containing ".raw" files. Will skip data collection and use raw data for reports. + +Global Flags: + --debug enable debug logging and retain temporary directories (default: false) + --log-stdout write logs to stdout (default: false) + --noupdate skip application update check (default: false) + --output override the output directory + --syslog write logs to syslog instead of a file (default: false) + --tempdir override the temporary target directory, must exist and allow execution +``` diff --git a/docs/perfspect_lock.md b/docs/perfspect_lock.md new file mode 100644 index 00000000..9284c7fe --- /dev/null +++ b/docs/perfspect_lock.md @@ -0,0 +1,34 @@ +# perfspect lock + +```text +Collect kernel lock data from target(s) + +Usage: perfspect lock [flags] + +Examples: + Lock inspect from local host: $ perfspect lock + Lock inspect from remote target: $ perfspect lock --target 192.168.1.1 --user fred --key fred_key + Lock inspect from multiple targets: $ perfspect lock --targets targets.yaml + +Flags: + Options: + --duration number of seconds to run the collection (default: 10) + --frequency number of samples taken per second (default: 11) + --package create raw data package (default: false) + --format choose output format(s) from: all, html, txt (default: [all]) + --no-summary do not include system summary table in report (default: false) + Remote Target Options: + --target host name or IP address of remote target + --port port for SSH to remote target + --user user name for SSH to remote target + --key private key file for SSH to remote target + --targets file with remote target(s) connection details. See targets.yaml for format. + +Global Flags: + --debug enable debug logging and retain temporary directories (default: false) + --log-stdout write logs to stdout (default: false) + --noupdate skip application update check (default: false) + --output override the output directory + --syslog write logs to syslog instead of a file (default: false) + --tempdir override the temporary target directory, must exist and allow execution +``` diff --git a/docs/perfspect_metrics.md b/docs/perfspect_metrics.md new file mode 100644 index 00000000..80c1d320 --- /dev/null +++ b/docs/perfspect_metrics.md @@ -0,0 +1,66 @@ +# perfspect metrics + +```text +Collect performance metrics from target(s) + +Usage: perfspect metrics [flags] [-- application args] + +Examples: + Metrics from local host: $ perfspect metrics --duration 30 + Metrics from local host in CSV format: $ perfspect metrics --format csv + Metrics from remote host: $ perfspect metrics --target 192.168.1.1 --user fred --key fred_key + Metrics for "hot" processes: $ perfspect metrics --scope process + Metrics for specified processes: $ perfspect metrics --scope process --pids 1234,6789 + Start workload and collect metrics: $ perfspect metrics -- /path/to/workload arg1 arg2 + Metrics adjusted for transaction rate: $ perfspect metrics --txnrate 100 + "Live" metrics: $ perfspect metrics --live + +Arguments: + application (optional): path to an application to run and collect metrics for + +Flags: + Collection Options: + --duration number of seconds to run the collection. If 0, the collection will run indefinitely. (default: 0) + --scope scope of collection, options: system, process, cgroup (default: system) + --pids comma separated list of process ids. If not provided while collecting in process scope, "hot" processes will be monitored. (default: []) + --cids comma separated list of cids. If not provided while collecting at cgroup scope, "hot" cgroups will be monitored. (default: []) + --filter regular expression used to match process names or cgroup IDs when in process or cgroup scope and when --pids or --cids are not specified + --count maximum number of "hot" or "filtered" processes or cgroups to monitor (default: 5) + --refresh number of seconds to run before refreshing the "hot" or "filtered" process or cgroup list. If 0, the list will not be refreshed. (default: 30) + --cpus range of CPUs to monitor. If not provided, all cores will be monitored. + Output Options: + --granularity level of metric granularity. Only valid when collecting at system scope. Options: system, socket, cpu. (default: system) + --format output formats, options: txt, csv, json, wide (default: [csv]) + --live print metrics to stdout in one output format specified with the --format flag. No metrics files will be written. (default: false) + --txnrate number of transactions per second. Will divide relevant metrics by transactions/second. (default: 0) + --prometheus-server enable prometheus metrics server (default: false) + --prometheus-server-addr address (e.g., host:port) to start Prometheus metrics server on (implies --prometheus-server true) (default: :9090) + Advanced Options: + --list show metric names available on this platform and exit (default: false) + --metrics a comma separated list of quoted metric names to include in output (default: []) + --eventfile perf event definition file. Will override default event definitions. Must be used with --metricfile. + --metricfile metric definition file. Will override default metric definitions. Must be used with --eventfile. + --interval event collection interval in seconds (default: 5) + --muxinterval multiplexing interval in milliseconds (default: 125) + --noroot do not elevate to root (default: false) + --raw write raw perf events to file (default: false) + --input path to a file or directory with json file containing raw perf events. Will skip data collection and use raw data for reports. + --no-summary do not include system summary table in report (default: false) + Remote Target Options: + --target host name or IP address of remote target + --port port for SSH to remote target + --user user name for SSH to remote target + --key private key file for SSH to remote target + --targets file with remote target(s) connection details. See targets.yaml for format. + +Subcommands: + trim: Filter existing metrics to a time range + +Global Flags: + --debug enable debug logging and retain temporary directories (default: false) + --log-stdout write logs to stdout (default: false) + --noupdate skip application update check (default: false) + --output override the output directory + --syslog write logs to syslog instead of a file (default: false) + --tempdir override the temporary target directory, must exist and allow execution +``` diff --git a/docs/perfspect_metrics_trim.md b/docs/perfspect_metrics_trim.md new file mode 100644 index 00000000..bcf6edd0 --- /dev/null +++ b/docs/perfspect_metrics_trim.md @@ -0,0 +1,30 @@ +# perfspect metrics trim + +```text +Filter existing metrics to a time range + +Usage: + perfspect metrics trim [flags] + +Examples: + Skip first 30 seconds: $ perfspect metrics trim --input perfspect_2025-11-28_09-21-56 --start-offset 30 + Skip first 10 seconds and last 5 seconds: $ perfspect metrics trim --input perfspect_2025-11-28_09-21-56 --start-offset 10 --end-offset 5 + Use absolute timestamps and specific CSV: $ perfspect metrics trim --input perfspect_2025-11-28_09-21-56/myhost_metrics.csv --start-time 1764174327 --end-time 1764174351 + +Flags: + --end-offset int seconds to exclude from the end of the data + --end-time int absolute end timestamp (seconds since epoch) + -h, --help help for trim + --input string path to the directory or specific metrics CSV file to trim (required) + --start-offset int seconds to skip from the beginning of the data + --start-time int absolute start timestamp (seconds since epoch) + +Global Flags: + --debug enable debug logging and retain temporary directories + --log-stdout write logs to stdout + --noupdate skip application update check + --output string override the output directory + --syslog write logs to syslog instead of a file + --tempdir string override the temporary target directory, must exist and allow execution + +``` diff --git a/docs/perfspect_report.md b/docs/perfspect_report.md new file mode 100644 index 00000000..62c9bee5 --- /dev/null +++ b/docs/perfspect_report.md @@ -0,0 +1,67 @@ +# perfspect report + +```text +Collect configuration data from target(s) + +Usage: perfspect report [flags] + +Examples: + Data from local host: $ perfspect report + Specific data from local host: $ perfspect report --bios --os --cpu --format html,json + All data from remote target: $ perfspect report --target 192.168.1.1 --user fred --key fred_key + Data from multiple targets: $ perfspect report --targets targets.yaml + +Flags: + Categories: + --all report configuration for all categories (default: true) + --system-summary System Summary (default: false) + --host Host (default: false) + --bios BIOS (default: false) + --os Operating System (default: false) + --software Software Versions (default: false) + --cpu Processor Details (default: false) + --prefetcher Prefetchers (default: false) + --isa Instruction Sets (default: false) + --accelerator On-board Accelerators (default: false) + --power Power Settings (default: false) + --cstates C-states (default: false) + --frequency Maximum Frequencies (default: false) + --sst Speed Select Technology Settings (default: false) + --uncore Uncore Configuration (default: false) + --elc Efficiency Latency Control Settings (default: false) + --memory Memory Configuration (default: false) + --dimm DIMM Population (default: false) + --netconfig Network Configuration (default: false) + --nic Network Cards (default: false) + --disk Storage Devices (default: false) + --filesystem File Systems (default: false) + --gpu GPUs (default: false) + --gaudi Gaudi Devices (default: false) + --cxl CXL Devices (default: false) + --pcie PCIE Slots (default: false) + --cve Vulnerabilities (default: false) + --process Process List (default: false) + --sensor Sensor Status (default: false) + --chassisstatus Chassis Status (default: false) + --pmu Performance Monitoring Unit Status (default: false) + --sel System Event Log (default: false) + --kernellog Kernel Log (default: false) + Other Options: + --format choose output format(s) from: all, html, xlsx, json, txt (default: [all]) + Remote Target Options: + --target host name or IP address of remote target + --port port for SSH to remote target + --user user name for SSH to remote target + --key private key file for SSH to remote target + --targets file with remote target(s) connection details. See targets.yaml for format. + Advanced Options: + --input ".raw" file, or directory containing ".raw" files. Will skip data collection and use raw data for reports. + +Global Flags: + --debug enable debug logging and retain temporary directories (default: false) + --log-stdout write logs to stdout (default: false) + --noupdate skip application update check (default: false) + --output override the output directory + --syslog write logs to syslog instead of a file (default: false) + --tempdir override the temporary target directory, must exist and allow execution +``` diff --git a/docs/perfspect_telemetry.md b/docs/perfspect_telemetry.md new file mode 100644 index 00000000..2d463a3e --- /dev/null +++ b/docs/perfspect_telemetry.md @@ -0,0 +1,52 @@ +# perfspect telemetry + +```text +Collect system telemetry from target(s) + +Usage: perfspect telemetry [flags] + +Examples: + Telemetry from local host: $ perfspect telemetry + Telemetry from remote target: $ perfspect telemetry --target 192.168.1.1 --user fred --key fred_key + Memory telemetry for 60 seconds: $ perfspect telemetry --memory --duration 60 + Telemetry from multiple targets: $ perfspect telemetry --targets targets.yaml + +Flags: + Categories: + --all collect telemetry for all categories (default: true) + --cpu monitor cpu utilization (default: false) + --ipc monitor IPC (default: false) + --cstate monitor C-States residency (default: false) + --frequency monitor cpu frequency (default: false) + --power monitor power (default: false) + --temperature monitor temperature (default: false) + --memory monitor memory (default: false) + --network monitor network (default: false) + --storage monitor storage (default: false) + --irqrate monitor IRQ rate (default: false) + --kernel monitor kernel (default: false) + --instrmix monitor instruction mix (default: false) + Other Options: + --format choose output format(s) from: all, html, xlsx, json, txt (default: [html]) + --duration number of seconds to run the collection. If 0, the collection will run indefinitely. Ctrl+c to stop. (default: 0) + --interval number of seconds between each sample (default: 2) + --instrmix-pid PID to monitor for instruction mix, no PID means all processes (default: 0) + --instrmix-frequency number of instructions between samples, default is 10,000,000 when collecting system wide and 100,000 when collecting for a specific PID (default: 10000000) + --no-summary do not include system summary table in report (default: false) + Remote Target Options: + --target host name or IP address of remote target + --port port for SSH to remote target + --user user name for SSH to remote target + --key private key file for SSH to remote target + --targets file with remote target(s) connection details. See targets.yaml for format. + Advanced Options: + --input ".raw" file, or directory containing ".raw" files. Will skip data collection and use raw data for reports. + +Global Flags: + --debug enable debug logging and retain temporary directories (default: false) + --log-stdout write logs to stdout (default: false) + --noupdate skip application update check (default: false) + --output override the output directory + --syslog write logs to syslog instead of a file (default: false) + --tempdir override the temporary target directory, must exist and allow execution +``` diff --git a/docs/perfspect_update.md b/docs/perfspect_update.md new file mode 100644 index 00000000..47ef6cc5 --- /dev/null +++ b/docs/perfspect_update.md @@ -0,0 +1,21 @@ +# perfspect update + +```text +Update the application (Intel network only) + +Usage: + perfspect update [flags] + +Use "perfspect update [command] --help" for more information about a command. + +Flags: + -h, --help help for update + +Global Flags: + --debug enable debug logging and retain temporary directories + --log-stdout write logs to stdout + --noupdate skip application update check + --output string override the output directory + --syslog write logs to syslog instead of a file + --tempdir string override the temporary target directory, must exist and allow execution +```