teleport/docs/pages/machine-workload-identity/troubleshooting.mdx

---
title: Machine & Workload Identity Troubleshooting Guide
description: Troubleshooting common issues with Machine & Workload Identity
sidebar_label: Troubleshooting
tags:
 - conceptual
 - mwi
 - infrastructure-identity
page_type: troubleshooting
---

This page describes common issues that you may come across when setting up
Machine & Workload Identity (MWI) and how to work around or resolve them.

## A bot failed to renew a certificate due to a "generation mismatch"

### Symptoms

The bot will log an error like this:

```text
ERROR: renewable cert generation mismatch: stored=3, presented=2
```

Subsequent connection attempts by the bot may see errors like the following:
```text
ERROR: failed direct dial to auth server: auth API: access denied [00]
"\tauth API: access denied [00], failed dial to auth server through reverse tunnel: Get \"https://teleport.cluster.local/v2/configuration/name\": Get \"https://example.com:3025/webapi/find\": x509: cannot validate certificate for example.com because it doesn't contain any IP SANs"
"\tGet \"https://teleport.cluster.local/v2/configuration/name\": Get \"https://example.com:3025/webapi/find\": x509: cannot validate certificate for example.com because it doesn't contain any IP SANs"
```

In particular, note the message `auth API: access denied`.

In self-hosted Teleport deployments, the Teleport Auth Service will also provide
some additional context:

```text
[AUTH]      WARN lock targeting BotInstanceID:"0fa4d626-1837-4aee-a5d0-79856d4223ab" is in force: The bot instance example/0fa4d626-1837-4aee-a5d0-79856d4223ab has been locked due to a certificate generation mismatch, possibly indicating a stolen certificate. auth/apiserver.go:224
```

### Explanation

<Admonition type="note">
This applies only to bots using the `token` join method, which makes use of
one-time use shared secrets. Provider-specific join methods, such as GitHub,
AWS IAM, etc will not be locked in this fashion unless another instance of the
bot uses `token` joining.
</Admonition>

Machine & Workload Identity (with token-based joining) uses a certificate
generation counter to detect potentially stolen renewable certificates. Each
time a bot fetches a new renewable certificate, the Auth Service increments the
counter, stores it on the backend, and embeds a copy of the counter in the
certificate.

If the counter embedded in your bot certificate doesn't match the counter
stored in Teleport's Auth Service, the renewal will fail and the bot instance
will be automatically [locked](../identity-governance/locking.mdx).

Renewable certificates are exclusively stored in the bot's internal data
directory, by default `/var/lib/teleport/bot`. It's possible to trigger this by
accident if multiple bots are started using the same internal data directory, or
if this internal data is otherwise being shared between multiple `tbot`
processes.

Additionally, if a bot fails to save its freshly renewed certificates (for
example, due to a filesystem error) and crashes, it will attempt a renewal
with old certificates and trigger a lock.

### Resolution

Before unlocking the bot, try to determine if either of the two scenarios
described above apply. If the certificates were stolen, there may be
underlying security concerns that need to be addressed.

Otherwise, first ensure only one `tbot` process is using the internal data
directory. Multiple bots can be run on a single system, but separate data
directories must be configured for each.

Additionally, ensure the internal data is not being shared with or copied to any
other nodes, for example via a shared NFS volume. If you'd like to share
certificates between nodes, only copy or share content from destination
directories (usually `/opt/machine-id`) rather than the internal data directory
(by default, `/var/lib/teleport/bot`).

Once you have addressed the underlying cause, follow these steps to reset a
locked bot:
 1. Remove the lock on the bot instance
 1. Reset the bot's generation counter by creating a new bot instance

To remove the lock, first find and remove the lock targeting the bot instance.
For this example, we'll assume the bot is named `example`:

```code
$ tctl get locks
kind: lock
metadata:
  id: 1658359514703080513
  name: 5cee949f-5203-4f3b-9805-dac35d798a16
spec:
  message: The bot instance example/0fa4d626-1837-4aee-a5d0-79856d4223ab has been locked
    due to a certificate generation mismatch, possibly indicating a stolen certificate.
  target:
    bot_instance_id: 0fa4d626-1837-4aee-a5d0-79856d4223ab
version: v2

$ tctl rm lock/5cee949f-5203-4f3b-9805-dac35d798a16
```

Next, use `tctl bots instances add` to generate a new join token for the
preexisting bot `example`:
```code
$ tctl bots instances add example
```

Finally, reconfigure the local `tbot` instance with the new token and restart
it. It will detect the new token and automatically reset its internal data
directory. The bot will be issued a new bot instance UUID once connected, and
the generation counter will be reset.

## `tbot` shows a "bad certificate error" at startup

### Symptoms

Restarting a `tbot` process outputs a log like the following:

```text
INFO [TBOT]      Successfully loaded bot identity, valid: after=2022-07-21T21:49:26Z, before=2022-07-21T22:50:26Z, duration=1h1m0s | kind=tls, renewable=true, disallow-reissue=false, roles=[bot-test], principals=[-teleport-internal-join], generation=2 tbot/tbot.go:281
ERRO [TBOT]      Identity has expired. The renewal is likely to fail. (expires: 2022-07-21T22:50:26Z, current time: 2022-07-25T20:18:33Z) tbot/tbot.go:415
WARN [TBOT]      Note: onboarding config ignored as identity was loaded from persistent storage tbot/tbot.go:288
ERRO [TBOT]      Failed to resolve tunnel address Get "https://auth.example.com:3025/webapi/find": x509: cannot validate certificate for auth.example.com because it doesn't contain any IP SANs reversetunnel/transport.go:90
ERRO [TBOT]      Failed to resolve tunnel address Get "https://auth.example.com:3025/webapi/find": x509: cannot validate certificate for auth.example.com because it doesn't contain any IP SANs reversetunnel/transport.go:90
ERROR: failed direct dial to auth server: Get "https://teleport.cluster.local/v2/configuration/name": remote error: tls: bad certificate
"\tGet \"https://teleport.cluster.local/v2/configuration/name\": remote error: tls: bad certificate, failed dial to auth server through reverse tunnel: Get \"https://teleport.cluster.local/v2/configuration/name\": Get \"https://auth.example.com:3025/webapi/find\": x509: cannot validate certificate for auth.example.com because it doesn't contain any IP SANs"
"\tGet \"https://teleport.cluster.local/v2/configuration/name\": Get \"https://auth.example.com:3025/webapi/find\": x509: cannot validate certificate for auth.example.com because it doesn't contain any IP SANs"
```

In particular, note the log line: "Identity has expired. The renewal is likely to
fail."

### Explanation

Token-joined bots are unable to reauthenticate to the Teleport Auth Service once
their certificates have expired. Tokens in token-based joining (as opposed to
AWS IAM and other join methods) can only be used once, so when the bot's
internal certificates expire, it will not be able to connect.

When a bot's identity expires, certain parameters associated with the bot on the
Auth Service must be reset and a new joining token must be issued. The simplest
way to accomplish this is by removing and recreating the bot, which purges all
server-side data and issues a new joining token.

### Resolution

Use `tctl bots instances add` to create a new one-time use token for the bot:

```code
$ tctl bots instances add example
```

Copy the resulting join token into the existing bot config—either the
`--token` CLI flag or the `onboarding.token` parameter in `tbot.yaml`—and
restart the bot. It will detect the new token and rejoin the cluster as normal.

## SSH connections fail with `ssh: handshake failed: ssh: unable to authenticate`

### Symptoms

When attempting to connect to a node via SSH, connections fail with an error
like the following:

```code
$ ssh -F /opt/machine-id/ssh_config [email protected]
ERROR: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

ERROR: unable to execute tsh
executing `tsh proxy`
exit status 1

kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
```

In particular, note the `ssh: unable to authenticate` message.

### Explanation

This can occur when attempting to log into the node as a user not listed as a
principal on the SSH certificate.

You can verify this by viewing the `tbot` logs and looking for the log message
when impersonated certificates for the matching services were renewed.

In the following example, the only principal listed for the identity in
`/opt/machine-id` is `alice` (via the `access` role):
```text
INFO [TBOT]      Successfully renewed impersonated certificates for directory /opt/machine-id, valid: after=2022-07-21T21:49:26Z, before=2022-07-21T22:50:26Z, duration=1h1m0s | kind=tls, renewable=false, disallow-reissue=true, roles=[access], principals=[alice -teleport-internal-join], generation=0 tbot/renew.go:630
```

However, the SSH command attempted to log in as `bob`.

### Resolution

Ensure the bot identity is allowed to log in as the requested user by taking any
of the following actions:

 - Changing the SSH command to log in as an allowed user
 - Modifying the `access` role to allow the `bob` principal
 - Adding a role granting login via the `bob` principal

Note that if roles are added or modified, the certificates will need to be
renewed for the changes to take effect. The bot will renew certificates on its
own after the renewal interval (by default, 20 minutes), but you can trigger a
renewal immediately by either restarting the `tbot` process or sending it a
reload signal:

```code
## If using systemd, you can restart the process:
$ systemctl restart tbot
## Alternatively, you can send `tbot` a reload signal directly:
$ pkill -sigusr1 tbot
```

## Database requests fail with `database "example" not found`, but the database exists

### Symptoms

When requesting certificates for Teleport-protected
[databases](../enroll-resources/database-access/database-access.mdx), the certificate request
fails with an error like the following:

```text
ERROR: Failed to generate impersonated certs for directory /opt/machine-id: database "example" not found
database "example" not found
```

However, the database exists and can be seen by regular users via `tsh`:

```code
$ tsh db ls
Name       Description Allowed Users Labels  Connect
---------- ----------- ------------- ------- -------
example                [alice]       env=dev
```

### Explanation

Unlike regular Teleport users, Machine & Workload Identity bot users are granted
only minimal Teleport [RBAC permissions](../reference/access-controls/roles.mdx) and are not
allowed to view or list databases by default unless granted permission via one
or more roles.

### Resolution

{/* vale messaging.protocol-products = NO */}
Per the [Machine & Workload Identity Database Access Guide](./access-guides/databases.mdx), ensure at
least one role providing database permissions has been granted to the
output service listed in the error.
{/* vale messaging.protocol-products = YES */}

For example, note the `rules` section in the following example role:
```yaml
kind: role
version: v5
metadata:
  name: machine-id-db
spec:
  allow:
    db_labels:
      '*': '*'
    db_names: [example]
    db_users: [alice]
    rules:
      - resources: [db_server, db]
        verbs: [read, list]
```

Ensure the bot has a role that grants it at least these RBAC rules. If desired
you can examine bot roles with `tctl` to ensure the necessary `rules` have been
granted:

```code
$ tctl get role/machine-id-db
```

If the role is missing database permissions, it can be modified in your text
editor:

```code
$ tctl edit role/machine-id-db
```

Edit the role, then save and close the file to apply your changes.

(!docs/pages/includes/create-role-using-web.mdx!)

Once fixed, restart or reload the `tbot` clients for the updated role to take
effect.

If the bot was not granted the role initially, the simplest solution is to
delete and recreate the bot, being sure to include the role in the `--roles=...`
flag:

```code
$ tctl bots rm example
$ tctl bots add example --roles=foo,bar,machine-id-db
```

## Destination kubernetes_secret: `identity-output` must be a directory in exec plugin mode

By default, when outputting a Kubernetes identity, `tbot` outputs make use of a Kubernetes exec
plugin to always provide the latest version of the credentials.

When outputting a Kubernetes identity to a Kubernetes secret, however, it is important to disable
the use of the `exec` plugin by adding `disable_exec_plugin: true` to the output. This means that
a static `kubeconfig` file with embedded short-lived credentials is written instead:

```yaml
services:
  - type: kubernetes
    # Specify the name of the Kubernetes cluster you wish the credentials to
    # grant access to.
    kubernetes_cluster: example-k8s-cluster
    # Required when outputting a Kubernetes identity to a Kubernetes secret.
    disable_exec_plugin: true
    destination:
      type: kubernetes_secret
      # For this guide, identity-output is used as the secret name.
      # You may wish to customize this. Multiple outputs cannot share the same
      # destination.
      name: identity-output
```

Failure to add the `disable_exec_plugin` flag will result in a warning being displayed:
`Destination kubernetes_secret: identity-output must be a directory in exec plugin mode`.

## Configuring `tbot` for split DNS proxies

When you have deployed your Proxy Service in such a way that it is
accessible via two different DNS names, e.g. an internal and external address,
you may find that a `tbot` that is configured to use one of these addresses may
attempt to use the other address and that this may cause connections to fail.

This is because `tbot` queries an auto-configuration endpoint exposed by the
Proxy Service to determine the canonical address to use when connecting.

To fix this, set a variable of `TBOT_USE_PROXY_ADDR=yes` in the environment of the
`tbot` process. This configures `tbot` to prefer using the address that you have
explicitly provided. This only functions correctly in cases where TLS
routing/multiplexing is enabled for the Teleport cluster.

## Using `tbot` in an environment without an accessible Teleport Proxy Service

By default, when using the `identity` output service, the `tbot` client queries
the Proxy Service to fetch information about available Teleport clusters,
including any available [leaf clusters](../core-concepts.mdx#trusted-clusters),
which is required to generate a usable `ssh_config`.

If running in an environment where `tbot` is configured to communicate directly
with the Teleport Auth Service via the `auth_server` configuration option, but
cannot communicate with the Teleport Proxy Service, the identity output service
will fail to generate an `ssh_config` with an error like the following:

```
2026-01-21T18:11:16.320-07:00 ERRO [TBOT:PROX] Failed to ping proxy error:[
ERROR REPORT:
Original Error: *url.Error Get &#34;https://teleport.example.com:443/webapi/find&#34;: dial tcp 1.2.3.4:443: connect: connection refused
Stack Trace:
	github.com/gravitational/teleport/[email protected]/client/webclient/webclient.go:167 github.com/gravitational/teleport/api/client/webclient.doWithFallback
	github.com/gravitational/teleport/[email protected]/client/webclient/webclient.go:218 github.com/gravitational/teleport/api/client/webclient.findWithClient
	github.com/gravitational/teleport/[email protected]/client/webclient/webclient.go:192 github.com/gravitational/teleport/api/client/webclient.Find
	github.com/gravitational/teleport/lib/tbot/internal/caching_proxy_pinger.go:104 github.com/gravitational/teleport/lib/tbot/internal.(*CachingProxyPinger).Ping
	github.com/gravitational/teleport/lib/tbot/services/identity/output.go:198 github.com/gravitational/teleport/lib/tbot/services/identity.(*OutputService).generate
	github.com/gravitational/teleport/lib/tbot/internal/loop.go:161 github.com/gravitational/teleport/lib/tbot/internal.RunOnInterval
	github.com/gravitational/teleport/lib/tbot/services/identity/output.go:113 github.com/gravitational/teleport/lib/tbot/services/identity.(*OutputService).Run
	github.com/gravitational/teleport/lib/tbot/bot/bot.go:103 github.com/gravitational/teleport/lib/tbot/bot.(*Bot).Run.func2
	golang.org/x/[email protected]/errgroup/errgroup.go:93 golang.org/x/sync/errgroup.(*Group).Go.func1
	runtime/asm_amd64.s:1693 runtime.goexit
User Message: Get &#34;https://teleport.example.com:443/webapi/find&#34;: dial tcp 1.2.3.4:443: connect: connection refused] internal/caching_proxy_pinger.go:110
```

As in many cases the `ssh_config` is required for downstream applications to
function, this is expected behavior. However, if you do not require the
`ssh_config`, you can disable it by setting `ssh_config: "off"` in the settings
for your identity output service:

```yaml
version: v2
auth_server: teleport.example.com:3025
onboarding:
  token: '...snip...'
  join_method: token
storage:
  type: directory
  path: ./storage
services:
  - type: identity
    destination:
      type: directory
      path: ./destination
    ssh_config: "off"
```

If desired, you can [read more about the identity output service config
options](../reference/machine-workload-identity/configuration.mdx#identity).

Note that other `tbot` services may also attempt to connect to the Proxy
Service, and may fail similarly if blocked. For best compatibility, the `tbot`
client should generally be given network access to the Proxy Service, even if
configured to connect to the Auth Service directly.