Skip to main content

Event Streaming with GCP Pub/Sub

This guide covers real-time event streaming setup for the Service Account authentication method using GCP Pub/Sub and Audit Logs.

Why use GCP Pub/Sub?

Event streaming via GCP Pub/Sub provides significant advantages over the OAuth 2.0 Admin SDK method:

  • Real-time delivery: Events are pushed immediately as they occur, rather than polled
  • Full audit log history: Access to complete historical data without API retention limits
  • No webhook expiration: Unlike Admin SDK's 6-hour webhook limit, Pub/Sub subscriptions are persistent
  • Higher reliability: No event loss during webhook renewal windows
Using OAuth 2.0?

If you're using the OAuth 2.0 authentication method, event streaming is configured automatically through the Admin SDK Reports API. See the OAuth 2.0 event streaming section in the main guide for details and limitations.

STEP 1: Enable data sharing​

info

You must use a super administrator account.

  1. In the Google Workspace Admin Console, go to 'Menu' > 'Account' > 'Account settings' > 'Legal and compliance'. Scroll down to Share data with Google Cloud services and click to Enable the service. For more details, read this extended guide.

STEP 2: GCP configuration​

  1. Create a new GCP project to host the Pub/Sub topics and subscriptions:
$ export ORGANIZATION_ID=<YOUR_ORGANIZATION_ID>
$ gcloud projects create audit-logs-slashid-export --organization=${ORGANIZATION_ID}
$ export PROJECT_ID=$(gcloud projects list --filter=name="audit-logs-slashid-export" --format="value(projectId)")
  1. Create a new Pub/Sub topic:
$ gcloud pubsub topics create gworkspace-audit-logs --project=${PROJECT_ID}
  1. Create an organization-level sink:
$ gcloud logging sinks create workspace-audit-sink \
    pubsub.googleapis.com/projects/${PROJECT_ID}/topics/gworkspace-audit-logs \
    --include-children --organization=${ORGANIZATION_ID} \
    --log-filter='logName:"organizations/${ORGANIZATION_ID/logs/cloudaudit.googleapis.com"'
  1. Grant permissions to the sink, so it can write to the Pub/Sub topic:
$ gcloud pubsub topics add-iam-policy-binding gworkspace-audit-logs \
  --member=$(gcloud logging sinks describe workspace-audit-sink --organization=${ORGANIZATION_ID} --format="value(writerIdentity)") \
  --role=roles/pubsub.publisher --project=$PROJECT_ID
  1. Create a push subscription your HTTP endpoint:
$ gcloud pubsub subscriptions create gworkspace-audit-subscription \
    --topic=gworkspace-audit-logs \
    --push-endpoint='https://slashid.com/nhi/events?token=<YOUR_SLASHID_SECRET_INGESTION_TOKEN>' \
    --ack-deadline=60 --project=$PROJECT_ID