Spring Boot Application Secrets — Rotating Credentials Without Downtime

Why rotation matters and why it's painful

Credentials are stolen through breaches, insider threats, logging accidents, and repository exposure. A credential with no expiry, once stolen, provides indefinite access. Regular rotation limits the window of exposure for any stolen credential.

The reason rotation doesn't happen: it requires a restart in most Spring Boot configurations. spring.datasource.password is read at startup, bound to HikariCP, and never re-read. Rotating the database password requires stopping the application, updating the configuration, and restarting — a deployment event that teams avoid under production load.

The alternative: credentials that can be rotated while the application is running, without restarting, without dropping connections, without a deployment.

Database credential rotation — the connection pool challenge

HikariCP opens connections using the credentials it received at startup. When the database password changes, existing connections still work (they're already authenticated), but new connections will fail with authentication errors as the pool attempts to grow or replace connections.

The dual-credential rotation pattern. Instead of replacing the old credential immediately, maintain two valid credentials simultaneously during the rotation window:

Step 1: Create new credential alongside old (both valid)
Step 2: Update application to know about both
Step 3: Drain old connections, new connections use new credential
Step 4: Remove old credential from database

For PostgreSQL, this means creating a new user or updating the password of a second user role:

-- Step 1: Create the new credential
CREATE USER app_v2 WITH PASSWORD 'new-secure-password-xyz';
GRANT ALL PRIVILEGES ON DATABASE myapp TO app_v2;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO app_v2;

-- Application switches to app_v2

-- Step 4: Remove old credential
DROP USER app_v1;

Spring Cloud Vault — dynamic database credentials. HashiCorp Vault generates short-lived database credentials automatically. Each application instance gets a unique credential that expires after a configured duration. Vault handles rotation transparently:

spring:
  cloud:
    vault:
      uri: https://vault.internal:8200
      authentication: kubernetes
      kubernetes:
        role: order-service
      database:
        enabled: true
        role: order-service-role  # Vault database role
        backend: database

@Bean
@ConfigurationProperties("spring.datasource")
public DataSourceProperties dataSourceProperties(VaultTemplate vaultTemplate) {
    // Vault provides credentials that expire after 1 hour
    // Spring Cloud Vault renews them before expiry
    return new DataSourceProperties();
}

With Vault dynamic secrets:

• Credentials are unique per application instance — a breach of one instance doesn't expose credentials used by other instances
• Credentials expire automatically — no manual rotation required
• Audit logs show which instance used which credential

Refreshing HikariCP credentials. For environments without Vault, Spring Cloud's @RefreshScope combined with a credential refresh endpoint:

@Configuration
@RefreshScope  // re-initializes when /actuator/refresh is called
public class DataSourceConfig {

    @Value("${spring.datasource.password}")
    private String password;

    @Bean
    @RefreshScope
    public DataSource dataSource(
            @Value("${spring.datasource.url}") String url,
            @Value("${spring.datasource.username}") String username,
            @Value("${spring.datasource.password}") String password) {

        HikariConfig config = new HikariConfig();
        config.setJdbcUrl(url);
        config.setUsername(username);
        config.setPassword(password);
        config.setMaximumPoolSize(20);
        return new HikariDataSource(config);
    }
}

After updating the password in Kubernetes Secrets or AWS SSM:

# Update the Kubernetes secret
kubectl create secret generic db-credentials \
  --from-literal=password="new-password" \
  --dry-run=client -o yaml | kubectl apply -f -

# Trigger Spring Cloud Config refresh on all instances
kubectl exec -it pod/order-service-xxx -- \
  curl -X POST http://localhost:8080/actuator/refresh

/actuator/refresh triggers @RefreshScope beans to re-initialize with new property values. The DataSource bean re-creates with the new password. HikariCP gracefully closes old connections and opens new ones with the updated credentials.

The coordination challenge: all instances must be refreshed before the old credential is invalidated. Use a rolling refresh with overlap:

# Refresh instance 1, verify connections work
# Refresh instance 2, verify connections work
# ...
# Only after all instances refreshed: invalidate old credential

JWT secret rotation

JWT tokens are signed with a secret (HMAC) or private key (RSA/EC). Rotating the secret invalidates all outstanding tokens — every logged-in user is logged out.

The graceful JWT rotation pattern accepts both old and new secrets during the rotation window:

@Component
public class JwtService {

    private final List<String> activeSecrets;  // multiple secrets supported

    public JwtService(@Value("${jwt.secrets}") List<String> secrets) {
        this.activeSecrets = secrets;
    }

    public String issue(UserDetails user) {
        // Sign with the first (newest) secret
        return Jwts.builder()
            .setSubject(user.getUsername())
            .setIssuedAt(new Date())
            .setExpiration(Date.from(Instant.now().plus(Duration.ofHours(24))))
            .signWith(Keys.hmacShaKeyFor(activeSecrets.get(0).getBytes()))
            .compact();
    }

    public Claims verify(String token) {
        // Try each secret — tokens signed with any active secret are valid
        for (String secret : activeSecrets) {
            try {
                return Jwts.parserBuilder()
                    .setSigningKey(Keys.hmacShaKeyFor(secret.getBytes()))
                    .build()
                    .parseClaimsJws(token)
                    .getBody();
            } catch (JwtException ignored) {
                // Try next secret
            }
        }
        throw new InvalidTokenException("Token signature invalid");
    }
}

Configuration:

# During rotation: both secrets active
jwt:
  secrets:
    - new-secret-xyz-very-long-and-random
    - old-secret-abc-very-long-and-random

# After rotation window (all old tokens expired): only new secret
jwt:
  secrets:
    - new-secret-xyz-very-long-and-random

The rotation window is the token TTL. For 24-hour tokens, maintain both secrets for 24 hours — all tokens issued before rotation expire naturally, and new tokens use the new secret. After the window, remove the old secret.

For OAuth2 with JWKs: rotate the RSA/EC key pair using the authorization server's key rotation endpoint. Spring Security's resource server handles multiple active JWKS keys automatically — it tries each key in the JWKS until one validates the token's kid header.

API key rotation — zero downtime

API keys for third-party services (Stripe, SendGrid, Twilio) often have no overlap period — the old key is invalidated immediately when the new one is issued.

@Component
@RefreshScope
public class StripeClient {

    private final Stripe stripe;

    public StripeClient(@Value("${stripe.api-key}") String apiKey) {
        Stripe.apiKey = apiKey;  // Stripe uses a static global — problematic
        this.stripe = new Stripe();
    }

    // Stripe's static global means all instances in the JVM share the key
    // @RefreshScope only helps if each request creates a new Stripe instance
}

Stripe's static API key model is a problem for dynamic refresh. The practical approach:

@Component
public class StripeClient {

    @Value("${stripe.api-key}")
    private volatile String apiKey;  // volatile for visibility across threads

    public PaymentIntent createPaymentIntent(long amount, String currency) {
        RequestOptions options = RequestOptions.builder()
            .setApiKey(apiKey)  // per-request key, not global static
            .build();
        return PaymentIntent.create(
            PaymentIntentCreateParams.builder()
                .setAmount(amount)
                .setCurrency(currency)
                .build(),
            options
        );
    }

    // Called after credential refresh
    @EventListener(EnvironmentChangeEvent.class)
    public void onCredentialRefresh(EnvironmentChangeEvent event) {
        if (event.getKeys().contains("stripe.api-key")) {
            log.info("Stripe API key refreshed");
            // apiKey is read from @Value which is re-evaluated on refresh
        }
    }
}

Using RequestOptions instead of Stripe.apiKey global avoids the static state problem — each API call uses the current key from the property, which is updated by Spring Cloud's refresh mechanism.

Vault dynamic secrets — the complete solution

HashiCorp Vault's dynamic secrets generate credentials on demand with automatic expiry and renewal. Spring Cloud Vault integrates natively:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-vault-config</artifactId>
</dependency>

spring:
  cloud:
    vault:
      uri: https://vault.internal:8200
      authentication: kubernetes
      kubernetes:
        role: order-service
        kubernetes-path: auth/kubernetes
      config:
        lifecycle:
          enabled: true    # auto-renew leases before expiry
          min-renewal: 10s # renew when 10s before expiry
          expiry-threshold: 1m  # treat as expiring when < 1m left
      generic:
        default-context: order-service  # path: secret/order-service
      database:
        enabled: true
        role: readonly
        backend: database

With this configuration:

• Vault issues unique database credentials for each application instance
• Credentials expire after the configured TTL (e.g., 1 hour)
• Spring Cloud Vault renews credentials before they expire, transparently
• If an instance is compromised, only its credentials are affected — and they expire

The database Vault role must be pre-configured:

vault write database/roles/order-service-role \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl=1h \
  max_ttl=24h

Vault creates a new PostgreSQL role for each credential request, grants it the permissions defined in creation_statements, and expires it after default_ttl. The application never knows about or manages the credential lifecycle.

The rotation runbook

For teams without Vault, a manual rotation runbook:

1. Generate new credential (strong, random)
2. Update Kubernetes Secret or AWS SSM Parameter
3. Trigger /actuator/refresh on all instances (rolling, not simultaneous)
4. Verify new connections use new credential (check application logs)
5. For JWT: wait for token TTL (e.g., 24 hours)
6. Remove old credential from secrets manager
7. Revoke old credential at the source (database, API provider)

Document this runbook. Run it on a schedule (quarterly for most credentials) rather than waiting for a breach to force it. Rotation on a schedule is routine; rotation under incident conditions is chaotic.

The ideal end state: credentials that rotate automatically with no human intervention, short TTLs that limit the damage window, and audit logs that show exactly which credential was used when. Vault dynamic secrets achieve this. For everything else, @RefreshScope with a documented runbook is the pragmatic intermediate step.