From 892e13904fe50c7d3b1e07c37b253c67590699b6 Mon Sep 17 00:00:00 2001 From: Gabriel Corado Date: Tue, 21 May 2024 18:09:57 -0300 Subject: [PATCH] Update troubleshooting guides with debug service usage (#41396) * refactor(docs): update docs to include debug service usage * docs: code review suggestions * docs: update debug service intro definitions * Apply suggestions from code review Co-authored-by: Paul Gottschling * docs(diagnostics): move teleport-debug-config section up in the guide --------- Co-authored-by: Paul Gottschling --- .../config-reference/debug-service.yaml | 3 + .../diagnostics/teleport-debug-config.mdx | 6 ++ .../management/admin/troubleshooting.mdx | 43 ++++++++++-- .../pages/management/diagnostics/profiles.mdx | 66 +++++++++++++++++-- docs/pages/reference/cli/teleport.mdx | 3 + docs/pages/reference/config.mdx | 8 +++ 6 files changed, 119 insertions(+), 10 deletions(-) create mode 100644 docs/pages/includes/config-reference/debug-service.yaml create mode 100644 docs/pages/includes/diagnostics/teleport-debug-config.mdx diff --git a/docs/pages/includes/config-reference/debug-service.yaml b/docs/pages/includes/config-reference/debug-service.yaml new file mode 100644 index 00000000000..f47f3bbc6a7 --- /dev/null +++ b/docs/pages/includes/config-reference/debug-service.yaml @@ -0,0 +1,3 @@ +# These settings apply to the Debug service: +debug_service: + enabled: true diff --git a/docs/pages/includes/diagnostics/teleport-debug-config.mdx b/docs/pages/includes/diagnostics/teleport-debug-config.mdx new file mode 100644 index 00000000000..44e470774fb --- /dev/null +++ b/docs/pages/includes/diagnostics/teleport-debug-config.mdx @@ -0,0 +1,6 @@ + +If your Teleport configuration is not placed on the default path +(`/var/lib/teleport`), you must to specify its location to the CLI command +using the `-c/--config` flag. + + diff --git a/docs/pages/management/admin/troubleshooting.mdx b/docs/pages/management/admin/troubleshooting.mdx index 295120f99f7..8955ef6e4fa 100644 --- a/docs/pages/management/admin/troubleshooting.mdx +++ b/docs/pages/management/admin/troubleshooting.mdx @@ -18,6 +18,39 @@ services such as the Application Service and Database Service. ## Step 1/3. Enable verbose logging +To change log levels in Teleport, you can use either of the following methods: + +- Debug Service: Allows on-the-fly log level adjustments without restarting the + instance, which is ideal for troubleshooting sessions. +- Updating configuration: Involves updating the Teleport configuration file and + restarting the instance. + + + +The Teleport Debug Service allows administrators to dynamically manage log levels +without restarting the instance. The service, enabled by default, ensures +local-only access and must be consumed from inside the same instance. + +To change the instance log level use the `teleport debug set-log-level` command: + +```code +$ teleport debug set-log-level DEBUG +Changed log level from "INFO" to "DEBUG". + +$ kubectl -n teleport exec my-pod -- teleport set-log-level DEBUG +Changed log level from "INFO" to "DEBUG". +``` + +If you're unsure what is the current level you can retrieve it using +`teleport debug get-log-level`. + +After troubleshooting, remember to turn the log level back to avoid generating +unnecessary logs. + +(!docs/pages/includes/diagnostics/teleport-debug-config.mdx!) + + + To diagnose problems, you can configure the `teleport` process to run with verbose logging enabled by passing it the `-d` flag. `teleport` will write logs to stderr. @@ -33,6 +66,12 @@ teleport: Restart the `teleport` process to apply the modified log level. Logs will resemble the following (these logs were printed while joining a server to a cluster, then terminating the `teleport` process on the server): + + + +Debug logs include the file and line number of the code that emitted the log, so +you can investigate (or report) what a `teleport` process was doing before it ran into +problems. Here's an example: ``` DEBU [NODE:PROX] Agent connected to proxy: [aee1241f-0f6f-460e-8149-23c38709e46d.tele.example.com aee1241f-0f6f-460e-8149-23c38709e46d teleport-proxy-us-west-2-6db8db844c-ftmg9.tele.example.com teleport-proxy-us-west-2-6db8db844c-ftmg9 localhost 127.0.0.1 ::1 tele.example.com 100.92.90.42 remote.kube.proxy.teleport.cluster.local]. leaseID:4 target:tele.example.com:11106 reversetunnel/agent.go:414 @@ -43,10 +82,6 @@ DEBU [NODE:PROX] Pool is closing agent. leaseID:2 target:tele.example.com:11106 DEBU [NODE:PROX] Pool is closing agent. leaseID:3 target:tele.example.com:11106 reversetunnel/agentpool.go:238 ``` -Debug logs include the file and line number of the code that emitted the log, so -you can investigate (or report) what a `teleport` process was doing before it ran into -problems. - diff --git a/docs/pages/management/diagnostics/profiles.mdx b/docs/pages/management/diagnostics/profiles.mdx index 08c9c7a7011..ec104d7df72 100644 --- a/docs/pages/management/diagnostics/profiles.mdx +++ b/docs/pages/management/diagnostics/profiles.mdx @@ -7,20 +7,74 @@ Teleport leverages Go's diagnostic capabilities to collect and export profiling data. Profiles can help identify the cause of spikes in CPU, the source of memory leaks, or the reason for a deadlock. -## Enable profiling +## Using the Debug Service + +The Teleport Debug Service enables administrators to collect diagnostic profiles +without enabling pprof endpoints at startup. The service, enabled by default, +ensures local-only access and must be consumed from inside the same instance. + +`teleport debug profile` collects a list of pprof profiles. It outputs a +compressed tarball (`.tar.gz`) to STDOUT. You decompress it using `tar` or +direct the result to a file. + +By default, it collects `goroutine`, `heap` and `profile` profiles. + +Each profile collected will have a correspondent file inside the tarball. For +example, collecting `goroutine,trace,heap` will result in `goroutine.pprof`, +`trace.pprof`, and `heap.pprof` files. + +```code +# Collect default profiles and save to a file. +$ teleport debug profile > pprof.tar.gz +$ tar xvf pprof.tar.gz + +# Collect default profiles and decompress it. +$ teleport debug profile | tar xzv -C ./ + +# Collect "trace" and "mutex" profiles and save to a file. +$ teleport debug profile trace,mutex > pprof.tar.gz + +# Collect profiles setting the profiling time in seconds +$ teleport debug profile -s 20 trace > pprof.tar.gz +``` + +(!docs/pages/includes/diagnostics/teleport-debug-config.mdx!) + +If you're running Teleport on a Kubernetes cluster you can directly collect +profiles to a local directory without an interactive session: + +```code +$ kubectl -n teleport exec my-pod -- teleport debug profile > pprof.tar.gz +``` + +After extracting the contents, you can use `go tool` commands to explore and +visualize them: + +```code +# Opens the terminal interactive explorer +$ go tool pprof heap.pprof + +# Opens the web visualizer +$ go tool pprof -http : heap.pprof + +# Visualize trace profiles +$ go tool trace trace.pprof +``` + +## Using diagnostics endpoints The profiling endpoint is only enabled if the `--debug` flag is supplied. (!docs/pages/includes/diagnostics/diag-addr-prereqs-tabs.mdx flags="--debug" !) -## Collecting profiles +### Collecting profiles Go's standard profiling endpoints are served at `http://127.0.0.1:3000/debug/pprof/`. Retrieving a profile requires sending a request to the endpoint corresponding to the desired profile type. When debugging an issue it is helpful to collect a series of profiles over a period of time. -### CPU +#### CPU CPU profile shows execution statistics gathered over a user specified period: ``` code @@ -31,7 +85,7 @@ $ curl -o cpu.profile http://127.0.0.1:3000/debug/pprof/profile?seconds=30 $ go tool pprof -http : cpu.profile ``` -### Goroutine +#### Goroutine Goroutine profiles show the stack traces for all running goroutines in the system: @@ -43,7 +97,7 @@ $ curl -o goroutine.profile http://127.0.0.1:3000/debug/pprof/goroutine $ go tool pprof -http : goroutine.profile ``` -### Heap +#### Heap Heap profiles show allocated objects in the system: @@ -55,7 +109,7 @@ $ curl -o heap.profile http://127.0.0.1:3000/debug/pprof/heap $ go tool pprof -http : heap.profile ``` -### Trace +#### Trace Trace profiles capture scheduling, system calls, garbage collections, heap size, and other events that are collected by the Go runtime over a user specified period of time: diff --git a/docs/pages/reference/cli/teleport.mdx b/docs/pages/reference/cli/teleport.mdx index 7f262feaad0..5c5777242d4 100644 --- a/docs/pages/reference/cli/teleport.mdx +++ b/docs/pages/reference/cli/teleport.mdx @@ -32,6 +32,9 @@ The primary commands for the `teleport` CLI are as follows: | `teleport start` | Starts the `teleport` process in the foreground using the current shell session, including any services configured by the [configuration YAML file](../config.mdx). | | `teleport status` | Prints the status of the current active Teleport SSH session. | | `teleport version` | Prints the current release version of the Teleport binary installed on your system. | +| `teleport debug set-log-level` | Changes instance log level. | +| `teleport debug get-log-level` | Fetches instance current log level. | +| `teleport debug debug profile` | Export the application profiles (pprof format). | For more information on subcommands when working with the `teleport` cli, use the `--help` option or `teleport --help`. diff --git a/docs/pages/reference/config.mdx b/docs/pages/reference/config.mdx index 82e2ee4d7f0..8c19a425137 100644 --- a/docs/pages/reference/config.mdx +++ b/docs/pages/reference/config.mdx @@ -178,6 +178,14 @@ These settings apply to the Jamf Service: (!docs/pages/includes/config-reference/jamf-service.yaml!) ``` +### Debug Service + +These settings apply to the Debug Service + +```yaml +(!docs/pages/includes/config-reference/debug-service.yaml!) +``` + ## Configuration versions In order to avoid breaking existing configurations, Teleport's configuration is