Skip to main content

GCP Integration Guide

The following guide covers the steps to integrate your Google Cloud Platform (GCP) data with Vega Cloud.

Pre-Requisites​

In order to create the necessary resources to integrate with the Vega Platform you will need:

  • A GCP Project, preferably within an Organization
    • GCP IAM Role (roles/owner) granted to the user performing the Integration
  • The Project that you are deploying to must have Billing Enabled and be linked to the same Cloud Billing Account containing data to export to BigQuery.
  • To enable or modify GCP Billing Exports to the BigQuery Dataset, the user configuring the Vega Integration requires:
    • To enable billing and link a project to a Billing Account the following role is required:
      • Billing Account Administrator (roles/billing.admin)
    • The following GCP Role for the Cloud Billing Account to configure exports:
      • Billing Account Costs Manager (roles/billing.costsManager)
    • BigQuery User GCP Role (roles/bigquery.user) in the Project being used to house the Billing export
  • The ability to create GCP Roles and Bind Members at the Organization or Project level depending on your needs, choose a path below
  • Certain Google Cloud Service APIs need to be enabled (see below)

Deployment​

Step 1: BigQuery Billing Export​

First we need to determine if a Detailed usage cost BigQuery Billing export has already been enabled, as the Vega Service Account we create later will live in the same project as the export.

  1. From the Google Cloud console homepage, open the Project picker from the dropdown (hotkey Ctrl+O)
    • If you use a GCP Organization:
      • Copy the Organization ID to a text editor from the ID column for use later in these integration steps
      • Select your GCP Organization from the list of resources
    • If you do not use GCP Organizations, select a Project that is linked with your billing account
  2. From the left-hand navigation or the search bar, navigate to Billing
  3. If presented with This project has no billing account:
    • Select Link a billing account
    • Select the Billing account from the dropdown that you wish to integrate with Vega and then Set account
  4. If presented with a Billing Table, select the Billing account name of the Billing account you wish to integrate with Vega
  5. From the left-hand navigation under Cost Management select Billing export
  6. Within the BigQuery export tab, under Detailed usage cost, if the export is:
Enabled
  1. Copy the Project name and Dataset name to a text editor
  2. Click the Dataset name link to navigate to the Dataset within BigQuery
  3. In the Dataset info tab, find the Data location and copy it to your text editor
Disabled
  1. From the Project picker, select a Project to contain your Billing export and Vega Service Account
  2. From the left-hand navigation or search bar, navigate to BigQuery
  3. In the Explorer pane of BigQuery, click the vertical ellipses next to the Project Id and select Create dataset
    1. Fill in the Dataset ID, eg. VegaDataset
    2. For Location type select Multi-region and then select a region, eg. US
      Note: Multi-region is preferred as GCP will automatically backfill current and previous months billing data if it is the first time an export has been enabled.
  4. Select Create dataset
  5. Navigate to Billing
  6. Under Detailed usage cost, click Edit settings
  7. Select the Project and then the Dataset that we just created, then Save

Step 2: Enable Google Cloud Service APIs​

The below Google Cloud Service APIs need to be enabled for the Google Cloud Project that will contain the BigQuery Billing Dataset.

You will need to enable the following Google Cloud Service APIs:

  • Cloud Resource Manager API (cloudresourcemanager.googleapis.com)
    • Reads and updates metadata for resource containers
  • Compute Engine API (compute.googleapis.com)
    • Read VMs and volumes
  • Cloud Billing API (cloudbilling.googleapis.com)
    • Ingest billing data programmatically
  • BigQuery API (bigquery.googleapis.com)
    • Ingest BigQuery data
  • Recommender API (recommender.googleapis.com)
    • Returns service recommendations for cost savings, security, and optimizations
  • Cloud Storage API (storage.googleapis.com)
    • Allows interactions with Cloud Storage Buckets
  • Identity and Access Management API (iam.googleapis.com)
    • Manages identity and access control for GCP resources
  • Cloud Monitoring API (monitoring.googleapis.com)
    • Manages your Cloud Monitoring data and configurations

First, you will need to select the Project that contains/will contain your Detailed usage cost BigQuery Billing export

  • From the Google Cloud console homepage, open the Project picker from the dropdown (hotkey Ctrl+O) and select the Project

To enable these APIs choose one of the two methods below:

Google Cloud CLI (recommended)
  1. Activate the Cloud Shell terminal in the top right corner of the Console (hotkey G then S)
  2. Verify you have the correct Project set in your Cloud Shell terminal, to the right of your gcp_user@cloudshell: 
    gcp_user@cloudshell:~ (project_id)$
    If you do not see the correct Project ID in your Cloud Shell terminal use the following command substituting in the PROJECT_ID that contains your BigQuery export: 
    gcloud config set project PROJECT_ID
  3. Execute the following command to enable the necessary GCP Service APIs: 
    gcloud services enable cloudresourcemanager.googleapis.com compute.googleapis.com cloudbilling.googleapis.com bigquery.googleapis.com recommender.googleapis.com storage.googleapis.com iam.googleapis.com monitoring.googleapis.com
Google Cloud User Interface (web browser)
  1. Select the Project that contains your BigQuery Billing export
  2. On the Google Cloud Console home page, navigate to APIs & Services > Library
  3. In the API Library Search box, enter the name of the API you want to enable. Then press Enter to execute the search
  4. In the list that appears, click the API name to navigate to that API page. If the API is not already enabled, click Enable
  5. After you enable the given API, the console displays a details page for that API
  6. Repeat this step for each of the Service APIs listed above
  7. Navigate to the console Home page
  8. For each API you want to enable, navigate back to the home page and repeat these steps

Step 3: Create Custom Role​

For the custom roles there are two separate permissions sets depending on the SKU you have with Vega.

  • vInform: This is Vega's standard offering, these permissions are used for collecting metadata and metric data on Project(s) within your Organization to enrich Recommendations within the Vega Platform.
    vInform link

  • vOperate: Contains the permissions necessary for Vega's Enterprise Parking solution.
    vOperate link

note

If you are a vOperate customer, you will need to create Custom Roles for both the vInform and vOperate permission sets.

  1. From the GCP console, select the Project that contains your BigQuery export from the Project picker
  2. Activate the Cloud Shell terminal in the top right corner of the Console (hotkey G then S)
  3. Verify you have the correct Project set in your Cloud Shell terminal, to the right of your gcp_user@cloudshell:
gcp_user@cloudshell:~ (project_id)$

If you do not see the correct Project ID in your Cloud Shell terminal use the following command substituting in the PROJECT_ID that contains your BigQuery export:

gcloud config set project PROJECT_ID

  1. Create a custom permissions file in the Cloud Shell Terminal:
    Copy the contents from the vInform link:
    vInform link
    In the Cloud Shell terminal:

    1. nano vega-inform.yaml Press Enter
    2. Ctrl+V to paste the contents copied from the link above
    3. Ctrl+X to exit
    4. Y to save buffer
    5. Press Enter to accept the current filename

  2. Create the custom role:
    Below there are two methods for creating the custom role, at the Organization (recommended) or at the Project. The most common integration is to create the custom role at the Organization, as this will allow the Service Account that we create to collect metric data from all Projects within your Organization from one service account.

note

If you are a vOperate customer, you will need to create Custom Roles for both the vInform and vOperate permission sets.

Create custom role at the Organization (recommended)

Supplement in the ORGANIZATION_ID we collected earlier and the file name used in step 4 of this Create Custom Role section, then run the following command in your Cloud Shell terminal:

gcloud iam roles create vegaInformRole --organization=ORGANIZATION_ID --file=vega-inform.yaml

Operate only

gcloud iam roles create vegaOperateRole --organization=ORGANIZATION_ID --file=vega-operate.yaml

After successfully creating the role the last three lines of your output should resemble:

name: organizations/11234567890/roles/vegaInformRole
stage: GA
title: Vega Inform Role

Copy the contents of the name variable for use later, eg: organizations/11234567890/roles/vegaInformRole

Create custom role at the Project

Supplement in the PROJECT_ID we collected earlier and the file name used in step 4 of this Create Custom Role section, then run the following command in your Cloud Shell terminal:

gcloud iam roles create vegaInformRole --project=PROJECT_ID --file=vega-inform.yaml

Operate only

gcloud iam roles create vegaOperateRole --project=PROJECT_ID --file=vega-operate.yaml

After successfully creating the role the last three lines of your output should resemble:

name: projects/PROJECT_ID/roles/vegaInformRole
stage: GA
title: Vega Inform Role

Copy the contents of the name variable for use later, eg: projects/PROJECT_ID/roles/vegaInformRole

Step 4: Create Service Account​

The Service Account should be created in the same Project that contains your BigQuery Billing export.

  1. Select the Project containing your BigQuery Billing export from the Project picker
  2. Navigate to IAM & Admin from the Navigation menu and then select Service Accounts
  3. Select + Create service account
  4. Give the Service account a name, eg: vega-sa
  5. Provide a description, eg: Vega Platform Data Retrieval Account
  6. Select Create and continue
  7. In the Permissions (optional) modal from the Select a role dropdown, find the previously created custom role
note

It can occasionally take some time for the newly created role to become searchable by name via the Select a role dropdown. You can paste in the role id we copied down earlier to bypass this wait, eg: organizations/11234567890/roles/vegaInformRole

  1. Select + Add another role and add BigQuery Job User
  2. Click Continue and then Done
  3. Find your newly created Service Account in the Service accounts section and copy the Email to your text editor, this will be needed in a later section.
  4. To the right side of the Service Account row, click the vertical ellipses in the Actions column and select Manage keys
  5. From the Add key dropdown select Create new key, select Key type of JSON and then click Create
note

This will automatically download the JSON key for the Service Account to your computer. This will be needed to add your Project(s) to the Provider Accounts section of the Vega Platform.

Step 5: Apply Dataset-scoped Role for BigQuery Billing Export​

  1. Select the Project containing your BigQuery Billing export from the Project picker
  2. Navigate to BigQuery and drill down into the Explorer window by toggling the triangle next to the Project ID
  3. Select the Dataset used for the export from the Explorer window
  4. On the right side of the screen select Sharing > Permissions
  5. Select Add principal
  6. Under New principals *, find the recently created Service Account (you can use the email address we copied down earlier in the search box)
  7. Under Assign roles in the Role * dropdown, select BigQuery Data Viewer and then click Save

Step 6: Create Billing Export Bucket​

  1. Select the Project containing your BigQuery Billing export from the Project picker
  2. Navigate to Buckets by entering Buckets into the search bar
  3. Click + Create
  4. Give your bucket a globally unique name, we suggest using something that includes the Organization ID we copied down earlier, eg: vega-billing-export-123456789012
note

Copy the bucket name to your text editor as you will need to provide this to Vega for billing data ingestion.

  1. Under Location type select the same region as the BigQuery Billing export Dataset from Step 1, eg: Multi-region:US
  2. Click Create
  3. View the Bucket details by clicking the hyperlink of the Bucket name
  4. Select the Permissions tab and then + Grant access
  5. Under New principals *, select the Service Account we created earlier
  6. Under Assign roles in the Select a role * dropdown, select Storage Object User and then click Save
note

Vega recommends setting a lifecycle rule on the bucket to minimize storage costs.

(Optional) Create Lifecycle rule
  1. Still within the Bucket details from the previous step, select the Lifecycle tab
  2. Click Add a rule
  3. Under Select an action select Delete object and then click Continue
  4. Under Select object conditions > Set Conditions, check the box for Age
  5. Set your desired number of days and then click Create
note

You can set the age to as little as 1 day, Vega recommends 7 days

Step 7: Add Role Binding(s)​

Depending on your Vega SKU (vInform/vOperate), you will need to create one or more role bindings. Depending on where the role was created, you'll need to use a different command for the role binding.

note

If you are a vOperate client with multiple Projects in your GCP Organization and would like to only deploy these permissions to a subset of Projects, use the Project-scoped role binding on each Project you would like to utilize Vega's Enterprise Parking solution with. If you would like vOperate permissions on all Projects within your GCP Organization, use the Organization-scoped role binding.

Organization binding | vInform

If the Inform role you created was at the Organization:

With the Project containing the vega-sa Service Account selected, run the following command supplementing in ORGANIZATION_ID, SA_EMAIL and CUSTOM_ROLE(this is the console output from the role we created previously)

gcloud organizations add-iam-policy-binding ORGANIZATION_ID --member='serviceAccount:SA_EMAIL' --role='CUSTOM_ROLE' --condition=None

Your constructed command should resemble: gcloud organizations add-iam-policy-binding 11234567890 --member='serviceAccount:vega-sa@gcp-project-id.iam.gserviceaccount.com' --role='organizations/11234567890/roles/vegaInformRole' --condition=None

Project binding | vInform

If the Inform role you created was at the Project:

With the Project containing the vega-sa Service Account selected, run the following command supplementing in PROJECT_ID, SA_EMAIL and CUSTOM_ROLE(this is the console output from the role we created previously)

gcloud projects add-iam-policy-binding PROJECT_ID --member='serviceAccount:SA_EMAIL' --role='CUSTOM_ROLE' --condition=None

Your constructed command should resemble: gcloud projects add-iam-policy-binding gcp-project-id --member='serviceAccount:vega-sa@gcp-project-id.iam.gserviceaccount.com' --condition=None

Step 8: Provide Integration Details to Vega​

In order to configure the BigQuery Billing export for ingestion into the Vega platform there are a few items you'll need to provide to your Vega Customer Success Manager or Onboarding Engineer:

  1. The Project Id where the Service Account was created
    • If your BigQuery Dataset is located in a different Project than the Service Account, Vega will need that Project Id as well.
  2. The BigQuery Billing export Dataset Id
  3. The BigQuery Billing export Table Id
  4. The Bucket Name created in Step 6

Copy Table Id

note

You can acquire the Project Id, Dataset Id and Table Id by clicking the three-dot ellipses next to the table name in the BigQuery Explorer and selecting 'Copy ID'

warning

If you do not yet have access to the Vega Platform you will need to provide Vega with the JSON key file associated with the Service Account via encrypted email. Other options for secure file transfer are available, please reach out to your Vega Customer Success Manager.

Adding GCP Projects to the Vega Platform​

Use the process below to add your Google Cloud Projects to the Vega Platform. Provider Accounts will use the Service Account named vega-sa that was created in Step 4.

  1. Go to Settings and Add a Provider Account (Select Google Cloud Platform.)
  2. Enter your GCP information in the dialog:
    1. Paste the service account JSON into the Service Account field.
    2. (Optional) Enter a Vega Account Alias, a friendly name for the account.

GCP Account Create

tip

If you aren't ready for your accounts to be ingested by Vega yet, you can toggle the 'Enabled' switch to 'Disabled' to prevent data ingestion.

  1. Click Link Account to link the account and start the discovery process.
note

If you have multiple projects, repeat the above steps for each project or alternatively use the Bulk Upload feature to add multiple projects at once.