GCP Secret Manager Migration Guide

Current Status

We're in the process of migrating from .env files to GCP Secret Manager for centralized secret management. This is being done incrementally to avoid breaking existing functionality.

Migration Policy:

✅ New secrets: MUST use GCP Secret Manager
⏸️ Existing secrets: Will be migrated during dedicated refactor (to avoid extensive retesting)
🔄 Fallback: Local .env values are used if GCP is unavailable (for local development)

Architecture Overview

Backend (Node.js)

Utility: backend/src/utils/secretManager.js
Fallback: Checks process.env[SECRET_NAME] first, then GCP Secret Manager
Initialization: Only creates GCP client if GCP_PROJECT_ID is set

Python Services (with Berglas)

Tool: Berglas (binary in container)
Format: Secrets referenced as sm://PROJECT_ID/SECRET_NAME
Resolution: Berglas resolves secrets at container startup
Dockerfile: Uses ENTRYPOINT ["/bin/berglas", "exec", "--"] to wrap the main command

Quick Reference: Adding a New Secret

Step 1: Add Secret to GCP Secret Manager

# Using gcloud CLI
echo -n "your-secret-value" | gcloud secrets create SECRET_NAME \
    --data-file=-\
    --project=opspilot-865d9 \
    --replication-policy="automatic"

# Verify it was created
gcloud secrets versions access latest --secret=SECRET_NAME --project=opspilot-865d9

Step 2: Grant Access to Service Account

# Find your service account email
gcloud projects get-iam-policy opspilot-865d9 \
    --flatten="bindings[].members" \
    --format="table(bindings.members)" \
    | grep serviceAccount

# Grant access (replace with your service account email)
gcloud secrets add-iam-policy-binding SECRET_NAME \
    --member="serviceAccount:YOUR-SA@opspilot-865d9.iam.gserviceaccount.com" \
    --role="roles/secretmanager.secretAccessor" \
    --project=opspilot-865d9

Step 3A: Use in Node.js Backend

import { getSecret } from '../utils/secretManager.js';

// In your service or route
const mySecret = await getSecret('SECRET_NAME', process.env.GCP_PROJECT_ID);

Example:

// backend/src/services/newService.js
import { getSecret } from '../utils/secretManager.js';

export async function initializeNewService() {
  const apiKey = await getSecret('NEW_API_KEY', process.env.GCP_PROJECT_ID);
  // Use apiKey...
}

Step 3B: Use in Python Services (Berglas)

In docker-compose.yml:

services:
  your-service:
    build:
      context: ./python-services/your-service
    environment:
      GCP_PROJECT_ID: opspilot-865d9
      SECRET_NAME: sm://opspilot-865d9/SECRET_NAME  # Berglas format
      GOOGLE_APPLICATION_CREDENTIALS: "/gcp/key.json"
    volumes:
      - /path/to/service-account-key.json:/gcp/key.json:ro
    # Don't override 'command' - let Dockerfile ENTRYPOINT with Berglas run

In Python code:

import os

# Berglas will have already resolved this to the actual value
secret_value = os.environ.get("SECRET_NAME")

Important: Do NOT override the command in docker-compose if using Berglas, as it needs the ENTRYPOINT to run.

Step 4: Add Local Fallback for Development

In .env (for local development only):

# Local fallback (not committed to git)
SECRET_NAME=local-dev-value

In .env.example (committed to git as documentation):

# SECRET_NAME - Description of what this secret is for
# Can be set locally or retrieved from GCP Secret Manager
SECRET_NAME=your-secret-here

Currently Migrated Secrets

✅ Already in GCP Secret Manager:

Secret Name	Service	Status
`TYPESENSE_API_KEY`	Backend	✅ Migrated (M32.1)
`BACKEND_API_TOKEN`	Backend	✅ Migrated (M32.2)
`DB_USER`	Cognee, Backend	✅ In GCP
`DB_PASSWORD`	Cognee, Backend	✅ In GCP
`DB_HOST`	Cognee, Backend	✅ In GCP
`DB_PORT`	Cognee, Backend	✅ In GCP
`DB_NAME`	Cognee, Backend	✅ In GCP
`GEMINI_API_KEY`	Multiple services	✅ In GCP
`COGNEE_LLM_PROVIDER`	Cognee Service	✅ In GCP
`COGNEE_LLM_MODEL`	Cognee Service	✅ In GCP
`COGNEE_EMBEDDING_PROVIDER`	Cognee Service	✅ In GCP
`COGNEE_EMBEDDING_MODEL`	Cognee Service	✅ In GCP

⏸️ Still in `.env` (to be migrated later):

Secret Name	Service	Migration Priority
`FIREBASE_API_KEY`	Frontend/Backend	Medium
`FIREBASE_PROJECT_ID`	Frontend/Backend	Medium
`SUPABASE_URL`	Backend	Low
`SUPABASE_ANON_KEY`	Backend	Low
`NEWS_API_KEY`	Backend	High (M32.3)
`DOCUMENT_AI_PROCESSOR_ID`	Ingestion Service	Medium
`GCS_INPUT_BUCKET`	Ingestion Service	Low
`GCS_OUTPUT_BUCKET`	Ingestion Service	Low

Berglas Setup for Python Services

Dockerfile Template (with Berglas)

FROM python:3.12-slim

# Install Berglas (ARM64 for M1/M2 Macs, AMD64 for production)
ENV BERGLAS_VERSION=2.0.6
ADD https://github.com/GoogleCloudPlatform/berglas/releases/download/v${BERGLAS_VERSION}/berglas_${BERGLAS_VERSION}_linux_arm64.tar.gz /tmp/berglas.tar.gz
RUN tar -xzf /tmp/berglas.tar.gz -C /tmp && \
    mv /tmp/berglas /bin/berglas && \
    chmod +x /bin/berglas && \
    rm /tmp/berglas.tar.gz

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . /app

# Use Berglas as entrypoint to resolve secrets
ENTRYPOINT ["/bin/berglas", "exec", "--"]
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Important Notes:

Architecture: Use linux_arm64 for M1/M2 Macs, linux_amd64 for production servers
ENTRYPOINT: Must be Berglas for secret resolution
CMD: Your actual application command
Volume: Mount service account key at /gcp/key.json
Environment: Set GOOGLE_APPLICATION_CREDENTIALS=/gcp/key.json

Common Issues & Solutions

Issue 1: "statfs /app/service-account-key.json: no such file or directory"

Cause: Service account key not mounted correctly in container

Solution:

# In docker-compose.yml
volumes:
  - /full/path/to/service-account-key.json:/gcp/key.json:ro

environment:
  GOOGLE_APPLICATION_CREDENTIALS: "/gcp/key.json"

Issue 2: Berglas not resolving secrets

Cause: command override in docker-compose bypassing ENTRYPOINT

Solution: Remove the command override or include Berglas:

# Option 1: Remove command (use Dockerfile's ENTRYPOINT + CMD)
# Option 2: Include Berglas in command
command: ["/bin/berglas", "exec", "--", "uvicorn", "main:app", "--host", "0.0.0.0"]

Issue 3: "Permission denied" accessing secrets

Cause: Service account doesn't have roles/secretmanager.secretAccessor

Solution:

gcloud secrets add-iam-policy-binding SECRET_NAME \
    --member="serviceAccount:YOUR-SA@opspilot-865d9.iam.gserviceaccount.com" \
    --role="roles/secretmanager.secretAccessor" \
    --project=opspilot-865d9

Issue 4: Secrets work in GCP but not locally

Cause: No local fallback in .env

Solution: Add fallback to .env for development:

# .env
SECRET_NAME=local-dev-value

Testing Secret Access

Test Node.js Secret Access

// backend/src/test-secret.js
import { getSecret } from './utils/secretManager.js';

async function testSecret() {
  try {
    const secret = await getSecret('TEST_SECRET', 'opspilot-865d9');
    console.log('✅ Secret retrieved successfully');
    console.log('Value:', secret.substring(0, 5) + '...');
  } catch (error) {
    console.error('❌ Failed to retrieve secret:', error.message);
  }
}

testSecret();

Run:

cd backend
node src/test-secret.js

Test Python/Berglas Secret Access

# Start the service
podman-compose up cognee-service

# Check logs
podman-compose logs cognee-service | grep -i secret

# Should see Berglas resolving secrets

Best Practices

1. Never Commit Secrets

✅ Add to .gitignore: .env, service-account-key.json
✅ Use .env.example for documentation only
✅ Store actual values in GCP Secret Manager

2. Use Descriptive Secret Names

✅ STRIPE_API_KEY_PROD
❌ KEY1, SECRET

3. Document New Secrets

Update this guide
Add to .env.example with description
Document in service's README

4. Rotate Secrets Regularly

# Create new version
echo -n "new-secret-value" | gcloud secrets versions add SECRET_NAME \
    --data-file=-\
    --project=opspilot-865d9

# Verify new version
gcloud secrets versions list SECRET_NAME --project=opspilot-865d9

5. Use Secret Labels

gcloud secrets create SECRET_NAME \
    --labels=environment=production,service=backend \
    --project=opspilot-865d9

Migration Checklist for New Secrets

When adding a new secret:

Create secret in GCP Secret Manager
Grant access to service account
Add to docker-compose.yml with sm:// prefix (Python) or use getSecret() (Node.js)
Add local fallback to .env for development
Document in .env.example
Update this guide's "Currently Migrated Secrets" table
Test locally and in container
Remove any hardcoded values from code
Update documentation

Example: Adding a New API Key

Let's say you need to add a new API key for a payment service:

1. Create in GCP

echo -n "sk_live_abc123xyz" | gcloud secrets create STRIPE_API_KEY_PROD \
    --data-file=-\
    --labels=environment=production,service=backend \
    --project=opspilot-865d9

2. Grant Access

gcloud secrets add-iam-policy-binding STRIPE_API_KEY_PROD \
    --member="serviceAccount:opspilot@opspilot-865d9.iam.gserviceaccount.com" \
    --role="roles/secretmanager.secretAccessor" \
    --project=opspilot-865d9

3. Use in Code

// backend/src/services/paymentService.js
import { getSecret } from '../utils/secretManager.js';
import Stripe from 'stripe';

let stripeClient;

export async function initializeStripe() {
  const apiKey = await getSecret('STRIPE_API_KEY_PROD', process.env.GCP_PROJECT_ID);
  stripeClient = new Stripe(apiKey);
  return stripeClient;
}

4. Add Local Fallback

# .env
STRIPE_API_KEY_PROD=sk_test_local123

5. Document

# .env.example
# STRIPE_API_KEY_PROD - Stripe API key for payment processing
STRIPE_API_KEY_PROD=your-stripe-key-here

Full Migration Plan (Future)

When ready to fully migrate all secrets:

Phase 1: Preparation

Audit all .env files across services
Create inventory of all secrets
Create all secrets in GCP Secret Manager
Grant necessary IAM permissions

Phase 2: Backend Migration

Migrate Firebase keys
Migrate Supabase keys
Migrate API keys (News, Document AI, etc.)
Update all service initializations

Phase 3: Python Services Migration

Update all Python Dockerfiles with Berglas
Update docker-compose.yml with sm:// references
Test each service individually

Phase 4: Testing & Validation

Comprehensive integration testing
Load testing with GCP Secret Manager
Monitor secret access logs
Verify no performance degradation

Phase 5: Cleanup

Remove all .env files (except .env.example)
Update deployment documentation
Update CI/CD pipelines
Archive old secrets

Estimated Time: 1 week Dependencies: Stable production environment, comprehensive test coverage

Monitoring & Auditing

Check Secret Access Logs

# View secret access logs
gcloud logging read "resource.type=secretmanager.googleapis.com/Secret" \
    --project=opspilot-865d9 \
    --limit=50 \
    --format=json

# Check for failed access attempts
gcloud logging read "resource.type=secretmanager.googleapis.com/Secret AND severity>=ERROR" \
    --project=opspilot-865d9 \
    --limit=50

Set Up Alerts

# Alert on unauthorized access attempts
# (Configure in GCP Console > Logging > Log-based Alerts

References

Support

For questions or issues:

Check this guide first
Review existing implementations (M32.1, M32.2)
Check GCP Secret Manager console for permissions
Review container logs for Berglas errors
Consult backend/src/utils/secretManager.js implementation

Last Updated: October 2025 Status: Active - Incremental migration in progress

Current Status​

Architecture Overview​

Backend (Node.js)​

Python Services (with Berglas)​

Quick Reference: Adding a New Secret​

Step 1: Add Secret to GCP Secret Manager​

Step 2: Grant Access to Service Account​

Step 3A: Use in Node.js Backend​

Step 3B: Use in Python Services (Berglas)​

Step 4: Add Local Fallback for Development​

Currently Migrated Secrets​

✅ Already in GCP Secret Manager:​

⏸️ Still in .env (to be migrated later):​

Berglas Setup for Python Services​

Dockerfile Template (with Berglas)​

Important Notes:​

Common Issues & Solutions​

Issue 1: "statfs /app/service-account-key.json: no such file or directory"​

Issue 2: Berglas not resolving secrets​

Issue 3: "Permission denied" accessing secrets​

Issue 4: Secrets work in GCP but not locally​

Testing Secret Access​

Test Node.js Secret Access​

Test Python/Berglas Secret Access​

Best Practices​

1. Never Commit Secrets​

2. Use Descriptive Secret Names​

3. Document New Secrets​

4. Rotate Secrets Regularly​

5. Use Secret Labels​

Migration Checklist for New Secrets​

Example: Adding a New API Key​

1. Create in GCP​

2. Grant Access​

3. Use in Code​

4. Add Local Fallback​

5. Document​

Full Migration Plan (Future)​

Phase 1: Preparation​

Phase 2: Backend Migration​

Phase 3: Python Services Migration​

Phase 4: Testing & Validation​

Phase 5: Cleanup​

Monitoring & Auditing​

Check Secret Access Logs​

Set Up Alerts​

References​

Support​