Google Pub/Sub Destination Setup

Publish detection alerts to Google Cloud Pub/Sub topics for event-driven processing.

Google Pub/Sub Destination Setup

Publish detection alerts to Google Cloud Pub/Sub topics for event-driven processing.

Prerequisites

Google Cloud project with Pub/Sub API enabled
Service account with Pub/Sub Publisher permissions
Pub/Sub topic created

Setup Steps

1. Enable Pub/Sub API

Go to Google Cloud Console
Select your project
Navigate to APIs & Services → Library
Search for "Cloud Pub/Sub API"
Click Enable

2. Create Pub/Sub Topic

Using gcloud CLI:

gcloud pubsub topics create security-detections \
  --project=your-project-id

Or via Console:

Navigate to Pub/Sub → Topics
Click Create Topic
Enter topic ID (e.g., security-detections)
Click Create

3. Create Service Account

gcloud iam service-accounts create security-detections-publisher \
  --display-name="Security Detections Publisher" \
  --project=your-project-id

4. Grant Pub/Sub Publisher Permission

gcloud pubsub topics add-iam-policy-binding security-detections \
  --member="serviceAccount:security-detections-publisher@your-project-id.iam.gserviceaccount.com" \
  --role="roles/pubsub.publisher" \
  --project=your-project-id

5. Create Service Account Key

gcloud iam service-accounts keys create ~/sa-key.json \
  --iam-account=security-detections-publisher@your-project-id.iam.gserviceaccount.com \
  --project=your-project-id

Important: Save the entire JSON file contents. You'll need this for configuration.

6. Configure in Query.ai

Contact your Query.ai administrator to configure the Google Pub/Sub destination with:

Required Configuration:

Project ID
Topic ID
Service Account Key JSON (stored securely)

Optional Configuration:

Timeout in seconds (default: 30)

Message Format

Message Data

The message data is Base64-encoded JSON containing all detection fields:

{
  "detection_id": 123,
  "detection_name": "Suspicious Login Attempts",
  "description": "Multiple failed login attempts detected",
  "severity": "HIGH",
  "outcome": "MATCHED",
  "match_count": 5,
  "replay_link": "https://app.query.ai/replay/123",
  "ran_at": "2025-01-15T10:00:00Z",
  "range_start": "2025-01-15T09:00:00Z",
  "range_end": "2025-01-15T10:00:00Z",
  "run_id": "run-456",
  "run_type": "SCHEDULED",
  "errors": [],
  "match_operator": "GREATER_THAN",
  "match_threshold": 0,
  "match_eagerness": "EXHAUSTIVE",
  "match_exhaustiveness": "COMPLETED",
  "search_id": "search-abc-123",
  "trace_id": "1-abc-def"
}

Message Attributes

Messages include attributes for filtering:

detection_id - Detection configuration ID
severity - Detection severity (CRITICAL, HIGH, MEDIUM, LOW)
outcome - Detection outcome (MATCHED, NOT_MATCHED, ERROR)
run_type - Type of run (SCHEDULED, MANUAL)

Service Account Key Format

The service account key must be provided as a complete JSON object:

{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "abc123...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "security-detections-publisher@your-project-id.iam.gserviceaccount.com",
  "client_id": "123456789",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/..."
}

Testing

Test Service Account

Test publishing a message:

# Set environment variable with service account key
export GOOGLE_APPLICATION_CREDENTIALS=~/sa-key.json

# Test publishing
gcloud pubsub topics publish security-detections \
  --message='{"test": "message from Query.ai"}' \
  --project=your-project-id

Expected: Message ID returned

Test with Detection

Create a test detection with low threshold
Add Google Pub/Sub destination
Click Run Now
Verify message appears in topic

Subscribing to Messages

Create Subscription

gcloud pubsub subscriptions create security-detections-sub \
  --topic=security-detections \
  --project=your-project-id

Pull Messages

gcloud pubsub subscriptions pull security-detections-sub \
  --auto-ack \
  --limit=10 \
  --project=your-project-id

Python Subscriber Example

from google.cloud import pubsub_v1
import json
import base64

project_id = "your-project-id"
subscription_id = "security-detections-sub"

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(project_id, subscription_id)

def callback(message):
    # Decode message data
    data = json.loads(base64.b64decode(message.data).decode())
    
    # Access attributes
    detection_id = message.attributes.get("detection_id")
    severity = message.attributes.get("severity")
    
    print(f"Detection {detection_id}: {data['detection_name']}")
    print(f"Severity: {severity}, Matches: {data['match_count']}")
    print(f"Replay: {data['replay_link']}")
    
    message.ack()

streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)
print(f"Listening for messages...")

try:
    streaming_pull_future.result()
except KeyboardInterrupt:
    streaming_pull_future.cancel()

Filtering Messages

Create filtered subscriptions:

Filter by Severity

gcloud pubsub subscriptions create high-severity-sub \
  --topic=security-detections \
  --message-filter='attributes.severity="HIGH" OR attributes.severity="CRITICAL"' \
  --project=your-project-id

Filter by Outcome

gcloud pubsub subscriptions create matched-detections-sub \
  --topic=security-detections \
  --message-filter='attributes.outcome="MATCHED"' \
  --project=your-project-id

Troubleshooting

Error	Solution
`403 Forbidden`	Verify service account has `roles/pubsub.publisher` on topic
`404 Not Found`	Check project_id and topic_id are correct
`401 Unauthorized`	Verify service account key JSON is valid and complete
`Invalid JWT`	Check private_key in service account key is properly formatted
Messages not appearing	Verify topic exists, check subscription, review logs

Configuration Options

Required

project_id

Google Cloud project ID
Format: String (e.g., "my-project-123")

topic_id

Pub/Sub topic ID
Format: String (e.g., "security-detections")

service_account_key (secret)

Service account key JSON as object
Stored securely in AWS Secrets Manager

Optional

timeout

Request timeout in seconds
Default: 30
Maximum: 300 (5 minutes)

Security Best Practices

Never Commit Service Account Keys: Always store in Secrets Manager
Use Least Privilege: Grant only roles/pubsub.publisher
Rotate Keys: Rotate service account keys every 90 days
Separate Accounts: Use different service accounts per environment
Monitor Usage: Review service account activity in Cloud Audit Logs
Delete Unused Keys: Remove old service account keys

Resources

Updated 2 days ago