Merge pull request #830 from Dstack-TEE/fix/kms-unattested-rpc-cert

feat(kms): make RPC certificate attestation configurable

Author: kvinwang

docs/security/public-security-reports.md

@@ -49,7 +49,7 @@ These issues were filed as concrete vulnerability reports, security audit findin
 | [#606](https://github.com/Dstack-TEE/dstack/issues/606) App keys and decrypted env files world-readable | Open | Valid hardening, open | Tightening secret-bearing file writes to owner-only permissions (`0600`) is valid defense-in-depth work with no expected compatibility cost |
 | [#607](https://github.com/Dstack-TEE/dstack/issues/607) `gateway_app_id = "any"` disables gateway identity pinning | Closed | Not a production vulnerability | `gateway_app_id` is KMS contract configuration and is publicly auditable. Production deployments must not use `"any"`. No code fix was applied |
 | [#608](https://github.com/Dstack-TEE/dstack/issues/608) `auth_api.type = "dev"` allows all authorization | Closed | Not a production vulnerability | Dev auth is measured runtime configuration, not a production mode. Production must use webhook/on-chain authorization. No code fix was applied |
-| [#609](https://github.com/Dstack-TEE/dstack/issues/609) `quote_enabled = false` bypasses attestation | Closed | Not a production vulnerability | The flag is measured in runtime configuration and should fail production attestation policy. No code fix was applied |
+| [#609](https://github.com/Dstack-TEE/dstack/issues/609) `quote_enabled = false` bypasses attestation | Closed | Not a production vulnerability | The flag was measured in runtime configuration and would fail production attestation policy, so no code fix was applied at the time. The setting has since been retired; current KMS uses `attest_rpc_cert`, which defaults to `true`, to control RPC certificate attestation |
 | [#610](https://github.com/Dstack-TEE/dstack/issues/610) Unauthenticated bootstrap endpoint can overwrite root keys | Closed | Not a production vulnerability | The bootstrap endpoint does not accept caller-supplied root key material. Root keys are generated server-side, and the operator chooses which result to publish. No code fix was applied |
 | [#611](https://github.com/Dstack-TEE/dstack/issues/611) Unauthenticated `/finish` endpoint can shut down KMS onboard service | Closed | Not a production vulnerability | The onboard service is a short-lived setup flow. Premature shutdown causes operator retry, not persistent compromise or data loss. No code fix was applied |
 | [#612](https://github.com/Dstack-TEE/dstack/issues/612) Gateway `register_cvm` prefers stale `app_info` over live attestation | Closed | Not a production vulnerability | Cert-embedded `app_info` is extracted from attestation and signed by KMS. Preferring it avoids redundant extraction and is not a trust bypass. No code fix was applied |

docs/security/security-best-practices.md

@@ -84,7 +84,7 @@ Example app-compose.json:
 
 Development settings are intentionally easy to audit, but they are not production-safe. A production deployment should satisfy all of the following:
 
-- KMS quote verification remains enabled. Do not deploy production KMS with `quote_enabled = false`.
+- The KMS attests its own RPC certificate. Do not deploy production KMS with `attest_rpc_cert = false`.
 - KMS authorization uses webhook/on-chain policy. Do not use `auth_api.type = "dev"` with real key material.
 - The KMS contract pins a concrete gateway app id. Do not use `gateway_app_id = "any"` for production traffic.
 - TEE quotes are evaluated by deployment policy, including TCB status and expected OS/application measurements.

docs/security/security-model.md

@@ -307,7 +307,7 @@ The one case dstack does not leave to downstream is a genuinely invalid TCB: `dc
 
 ### Development modes are auditable, not production-safe
 
-dstack keeps several development switches as runtime or on-chain configuration rather than Cargo feature flags. Examples include KMS `quote_enabled = false`, `auth_api.type = "dev"`, and KMS contract `gateway_app_id = "any"`. These settings exist for local development and integration tests, not for production deployments.
+dstack keeps several development switches as runtime or on-chain configuration rather than Cargo feature flags. Examples include KMS `attest_rpc_cert = false`, gateway `core.debug.insecure_skip_attestation = true`, KMS `auth_api.type = "dev"`, and KMS contract `gateway_app_id = "any"`. These settings exist for local development and integration tests, not for production deployments.
 
 This is intentional. Runtime configuration that affects the trust boundary is visible in attestation measurements or public contract state. Cargo feature gates are not automatically more auditable because feature unification can enable a feature through a dependency graph, and the resulting runtime behavior is not represented as a measured deployment setting.
 

dstack/kms/kms.toml

@@ -31,6 +31,13 @@ site_name = ""
 # outside a TEE (e.g. local dev/testing) where the local guest agent socket
 # is unavailable.
 enforce_self_authorization = true
+# Whether the KMS embeds an attestation in its own RPC certificate. Set false
+# only for local dev/testing where the KMS runs outside a TEE. This narrows what
+# the KMS asserts about itself and relaxes no verification: quotes presented to
+# the KMS are still fully checked, a guest that accepts an unattested
+# certificate simply does not extend mr-kms, and another KMS refuses to onboard
+# from one.
+attest_rpc_cert = true
 # AMD SEV-SNP key/cert release remains disabled unless this local KMS gate is
 # explicitly enabled. External auth policy must still allow the verified
 # BootInfo before any sensitive material is returned.

dstack/kms/src/config.rs

@@ -55,6 +55,18 @@ pub(crate) struct KmsConfig {
     pub aws_nitro_tpm_key_release: bool,
     #[serde(default)]
     pub site_name: String,
+    /// Whether the KMS embeds an attestation in its own RPC certificate.
+    /// Defaults to `true`. Set `false` only for local dev/testing where the KMS
+    /// runs outside a TEE and cannot reach a guest agent socket.
+    ///
+    /// This narrows what the KMS asserts about itself; it relaxes no
+    /// verification anywhere. Quotes presented *to* the KMS are still fully
+    /// checked, so key release stays gated, and relying parties keep their own
+    /// policy: a guest accepts an unattested KMS certificate but then does not
+    /// extend `mr-kms`, so a remote verifier can still tell, and another KMS
+    /// refuses to onboard from one.
+    #[serde(default = "default_true")]
+    pub attest_rpc_cert: bool,
     /// Whether trusted RPCs require the KMS to first attest itself to its
     /// own auth API. Defaults to `true` (strict). Set `false` only for local
     /// dev/testing where the KMS runs outside a TEE and cannot reach a guest
@@ -183,6 +195,7 @@ pub(crate) struct OnboardConfig {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use rocket::figment::providers::{Format, Toml};
 
     #[test]
     fn default_config_parses_with_admin_disabled_and_no_hash() {
@@ -198,4 +211,32 @@ mod tests {
         );
         assert!(!config.admin.insecure_no_auth);
     }
+
+    #[test]
+    fn rpc_cert_is_attested_by_default() {
+        let figment = load_config_figment(None);
+        let config: KmsConfig = figment
+            .focus("core")
+            .extract()
+            .expect("kms.toml must parse into KmsConfig");
+        assert!(
+            config.attest_rpc_cert,
+            "the KMS must attest its own RPC certificate unless explicitly told not to"
+        );
+    }
+
+    #[test]
+    fn omitting_the_key_keeps_the_attested_default() {
+        // Configs written before this key existed must keep working, and must
+        // land on the attested side.
+        #[derive(Deserialize)]
+        struct Probe {
+            #[serde(default = "default_true")]
+            attest_rpc_cert: bool,
+        }
+        let probe: Probe = Figment::from(Toml::string(""))
+            .extract()
+            .expect("an absent attest_rpc_cert must parse");
+        assert!(probe.attest_rpc_cert);
+    }
 }

dstack/kms/src/main.rs

@@ -116,6 +116,14 @@ async fn main() -> Result<()> {
     let figment = config::load_config_figment(args.config.as_deref());
     let config: KmsConfig = figment.focus("core").extract()?;
 
+    if !config.attest_rpc_cert {
+        warn!(
+            "attest_rpc_cert = false; the KMS RPC certificate carries no attestation, so \
+             guests cannot verify which KMS they are talking to and will not extend \
+             mr-kms. Intended for local development only"
+        );
+    }
+
     if config.onboard.enabled && !config.keys_exists() {
         info!("Onboarding");
         run_onboard_service(config.clone(), figment.clone()).await?;
@@ -126,6 +134,12 @@ async fn main() -> Result<()> {
 
     info!("Updating certs");
     if let Err(err) = onboard_service::update_certs(&config).await {
+        if config.attest_rpc_cert {
+            return Err(err).context(
+                "Failed to reissue the attested KMS RPC certificate; refusing to start with a \
+                 potentially unattested certificate",
+            );
+        }
         warn!("Failed to update certs: {err}");
     };
 

dstack/kms/src/onboard_service.rs

@@ -27,6 +27,7 @@ use ra_tls::{
 };
 use safe_write::safe_write;
 use sha2::Digest;
+use tracing::info;
 
 use crate::{
     config::KmsConfig,
@@ -76,7 +77,7 @@ impl OnboardRpc for OnboardHandler {
         ensure_self_kms_allowed(&self.state.config, &self.state.attestation_verifier)
             .await
             .context("KMS is not allowed to bootstrap")?;
-        let keys = Keys::generate(&request.domain)
+        let keys = Keys::generate(&request.domain, self.state.config.attest_rpc_cert)
             .await
             .context("Failed to generate keys")?;
 
@@ -355,12 +356,20 @@ struct Keys {
 }
 
 impl Keys {
-    async fn generate(domain: &str) -> Result<Self> {
+    async fn generate(domain: &str, attest_rpc_cert: bool) -> Result<Self> {
         let tmp_ca_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?;
         let ca_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?;
         let rpc_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?;
         let k256_key = SigningKey::random(&mut rand::rngs::OsRng);
-        Self::from_keys(tmp_ca_key, ca_key, rpc_key, k256_key, domain).await
+        Self::from_keys(
+            tmp_ca_key,
+            ca_key,
+            rpc_key,
+            k256_key,
+            domain,
+            attest_rpc_cert,
+        )
+        .await
     }
 
     async fn from_keys(
@@ -369,6 +378,7 @@ impl Keys {
         rpc_key: KeyPair,
         k256_key: SigningKey,
         domain: &str,
+        attest_rpc_cert: bool,
     ) -> Result<Self> {
         let tmp_ca_cert = CertRequest::builder()
             .org_name("Dstack")
@@ -386,20 +396,32 @@ impl Keys {
             .key(&ca_key)
             .build()
             .self_signed()?;
-        let pubkey = rpc_key.public_key_der();
-        let report_data = QuoteContentType::RaTlsCert.to_report_data(&pubkey);
-        let response = app_attest(report_data.to_vec())
-            .await
-            .context("Failed to get quote")?;
-        let attestation = VersionedAttestation::from_bytes(&response.attestation)
-            .context("Invalid attestation")?;
+        // The only place the KMS embeds its own attestation. Skipping it lets
+        // the KMS run outside a TEE for development; it does not affect the
+        // verification of quotes presented *to* the KMS, which is a separate
+        // path (main_service::ensure_app_attestation_allowed) and stays on.
+        let attestation = if attest_rpc_cert {
+            let pubkey = rpc_key.public_key_der();
+            let report_data = QuoteContentType::RaTlsCert.to_report_data(&pubkey);
+            let response = app_attest(report_data.to_vec()).await.context(
+                "failed to get a quote for the KMS RPC certificate. The KMS attests \
+                     itself through the dstack guest agent, so it must run inside a dstack \
+                     CVM. For local development set attest_rpc_cert = false",
+            )?;
+            Some(
+                VersionedAttestation::from_bytes(&response.attestation)
+                    .context("Invalid attestation")?,
+            )
+        } else {
+            None
+        };
 
         // Sign WWW server cert with KMS cert
         let rpc_cert = CertRequest::builder()
             .subject(domain)
             .alt_names(&[domain.to_string()])
             .special_usage("kms:rpc")
-            .maybe_attestation(Some(&attestation))
+            .maybe_attestation(attestation.as_ref())
             .key(&rpc_key)
             .build()
             .signed_by(&ca_cert, &ca_key)?;
@@ -483,7 +505,15 @@ impl Keys {
             KeyPair::from_pem(&tmp_ca_key_pem).context("Failed to parse tmp CA key")?;
         let ecdsa_key =
             SigningKey::from_slice(&root_k256_key).context("Failed to parse ECDSA key")?;
-        Self::from_keys(tmp_ca_key, ca_key, rpc_key, ecdsa_key, domain).await
+        Self::from_keys(
+            tmp_ca_key,
+            ca_key,
+            rpc_key,
+            ecdsa_key,
+            domain,
+            cfg.attest_rpc_cert,
+        )
+        .await
     }
 
     fn store(&self, cfg: &KmsConfig) -> Result<()> {
@@ -527,12 +557,22 @@ pub(crate) async fn update_certs(cfg: &KmsConfig) -> Result<()> {
     let domain = domain.trim();
 
     // Regenerate certificates using existing keys
-    let keys = Keys::from_keys(tmp_ca_key, ca_key, rpc_key, k256_key, domain)
-        .await
-        .context("Failed to regenerate certificates")?;
-
-    // Write the new certificates to files
+    let keys = Keys::from_keys(
+        tmp_ca_key,
+        ca_key,
+        rpc_key,
+        k256_key,
+        domain,
+        cfg.attest_rpc_cert,
+    )
+    .await
+    .context("Failed to regenerate certificates")?;
+
+    // Write the new certificates to files. This runs on every start, so a
+    // hand-placed certificate is replaced -- say so, because the old silence
+    // made that look like the file had survived.
     keys.store_certs(cfg)?;
+    info!("Reissued the KMS RPC certificate for {domain}");
 
     Ok(())
 }
@@ -541,7 +581,7 @@ pub(crate) async fn bootstrap_keys(cfg: &KmsConfig, verifier: &AttestationVerifi
     ensure_self_kms_allowed(cfg, verifier)
         .await
         .context("KMS is not allowed to auto-bootstrap")?;
-    let keys = Keys::generate(&cfg.onboard.auto_bootstrap_domain)
+    let keys = Keys::generate(&cfg.onboard.auto_bootstrap_domain, cfg.attest_rpc_cert)
         .await
         .context("Failed to generate keys")?;
     keys.store(cfg)?;

test-suites/full-stack-compose/scripts/runner.sh

@@ -697,7 +697,7 @@ stop_network_probe() {
 assert_no_insecure_shortcuts() {
   log "auditing rendered manifests for production trust settings"
   local manifest
-  local forbidden='quote_enabled[[:space:]]*=[[:space:]]*false|enforce_self_authorization[[:space:]]*=[[:space:]]*false|verify[[:space:]]*=[[:space:]]*false'
+  local forbidden='quote_enabled[[:space:]]*=[[:space:]]*false|attest_rpc_cert[[:space:]]*=[[:space:]]*false|insecure_skip_attestation[[:space:]]*=[[:space:]]*true|enforce_self_authorization[[:space:]]*=[[:space:]]*false|verify[[:space:]]*=[[:space:]]*false'
   if grep -R -E "$forbidden" "$WORK_DIR"/*.app-compose.json; then
     die "rendered app manifest contains a forbidden development trust setting"
   fi

tools/dev-stack.sh

@@ -181,6 +181,8 @@ mandatory = false
 
 [core]
 cert_dir = "$CERTS_DIR"
+attest_rpc_cert = false
+enforce_self_authorization = false
 
 [core.gpu]
 enabled = $VMM_ENABLE_GPU
@@ -189,7 +191,6 @@ enabled = $VMM_ENABLE_GPU
 type = "dev"
 
 [core.onboard]
-quote_enabled = false
 address = "127.0.0.1"
 port = $KMS_RPC_LISTEN_PORT
 auto_bootstrap_domain = "$KMS_DOMAIN"
@@ -215,7 +216,9 @@ mandatory = false
 [core]
 kms_url = "https://localhost:$KMS_RPC_LISTEN_PORT"
 rpc_domain = "$GATEWAY_DOMAIN"
-run_in_dstack = false
+
+[core.debug]
+insecure_skip_attestation = true
 
 [core.sync]
 enabled = false