Skip to main content

GCP Integration Guide

The following guide covers the steps to integrate your Google Cloud Platform (GCP) data with Vega Cloud using GCP Deployment Manager to manage the resources.

Pre-Requisites​

In order to create the necessary resources to integrate with the Vega Platform you will need:

  • A GCP Project, preferably within an Organization

  • The Project that you are deploying to must have Billing Enabled and be linked to the same Cloud Billing Account containing data to export to BigQuery

  • A GCP Project with the GCP IAM Role roles/owner granted to the user performing the Integration

    • GCP Deployment Manager is unable to manage IAM Roles and Bindings out of the box and requires temporarily elevated permissions to perform these actions, additional information can be found in the GCP Deployment Manager documentation.

  • To enable GCP Billing Exports to the BigQuery Dataset, the user configuring the Vega Integration requires:

    • One of the following GCP Roles for the Cloud Billing Account to configure exports for:
      • Billing Account Administrator (roles/billing.admin)
      • Billing Account Costs Manager (roles/billing.costsManager)
    • BigQuery User GCP Role (roles/bigquery.user) in the Project being used to house the Billing Export

  • The ability to create GCP Roles and Bind Members at the Organization or Project level depending on your needs, choose a path below.

    • Organization level GCP Role (recommended)

      To create a GCP Role at the Organization level (recommended) then the following is required:

      • GCP IAM Role roles/resourcemanager.organizationAdmin

        • this is used to temporarily elevate permissions of the GCP API Service Account that exists by default in all Projects
    • Project level GCP Role

      To create a GCP Role at the Project level (not recommended) then the following is required:

      • roles/iam.roleAdmin at the Project we are using for the Vega Integration

  • Certain Google Cloud Service APIs need to be enabled (see below)

Note: granting additional permissions to the GCP API Service Account does not grant Vega additional permissions. It allows Deployment Manager to manage IAM Resources on your behalf. These Roles may be revoked once the resources have been deployed.

Google Cloud Service APIs​

The below Google Cloud Service APIs need to be enabled for the Google Cloud Project that will contain the BigQuery Billing Dataset.

You will need to enable the following Google Cloud Service APIs:

  • Cloud Resource Manager API (cloudresourcemanager.googleapis.com)
    • Reads and updates metadata for resource containers.
  • Compute Engine API (compute.googleapis.com)
    • Read VMs and volumes.
  • Cloud Billing API (cloudbilling.googleapis.com)
    • Ingest billing data programmatically.
  • BigQuery API (bigquery.googleapis.com)
    • Ingest BigQuery data.
  • Recommender API (recommender.googleapis.com)
    • Returns service recommendations for cost savings, security, and optimizations.
  • Cloud Storage API (storage.googleapis.com)
    • Allows interactions with Cloud Storage Buckets.
  • Deployment Manager V2 API (deploymentmanager.googleapis.com)
    • Deploy resources from a configuration.
  • Identity and Access Management API (iam.googleapis.com)
    • Manages identity and access control for GCP resources.

To enable these APIs choose a method below choose one of the two methods below.

Google Cloud CLI (recommended)
  1. Execute this command to enable the necessary Service APIs
gcloud services enable cloudresourcemanager.googleapis.com compute.googleapis.com cloudbilling.googleapis.com bigquery.googleapis.com recommender.googleapis.com storage.googleapis.com deploymentmanager.googleapis.com iam.googleapis.com
Google Cloud User Interface (web browser)
  1. Navigate to the Google Cloud Console to the library of APIs.
  2. On the Google Cloud Console home page, navigate to APIs & Services > Library.
  3. Search for the API you want to enable.
  4. In the API Library Search box, enter the name of the API you want to enable. Then press Enter to execute the search.
  5. Enable the given API.
  6. In the list that appears, click the API name to navigate to that API page. If the API is not already enabled, click Enable.
  7. After you enable the given API, the console displays a details page for that API.
  8. Repeat this step for each of the Service APIs listed above.
  9. Navigate to the console Home page.
  10. For each API you want to enable, navigate back to the home page and repeat these steps.

Deployment Manager

Vega currently supports GCP Roles at two levels of the organizational hierarchy - Organizations and Projects. The recommended method is to deploy GCP Roles to the Organization to avoid needing to recreate the GCP Role and Service Accounts for each Project you wish to add to the Vega Platform.

Vega Products and the Resources Deployed​

Templates​

The Vega GCP Deployment Manager Resource Templates are comprised of Jinja files, you only need the files that are relevant to your environment and deployment.

File NameDescriptionLink
v-inform.jinjaThis is the base resource template to be deployed, it enables read access to your Project(s) to collect resource metrics and generate recommendations for your environment.link
billing-export.jinjaThis is another base resource template to be deployed, it enables Vega to collect your usage data within GCP. This enables financial and usage visualizations along with amortization of costslink
v-operate.jinjaThis template should only be included if you are using Vega's Enterprise Parking functionalitylink

Resources​

Each of these Deployment Manager Templates deploys resources to your Project as described below.

Source TemplateResource NameResource IDDescriptionNotes
v-inform.jinjavega-service-accountvega-saThe Service Account Vega will use to connect to your environment.
v-inform.jinjavega-inform-custom-rolevegaInformRoleA Custom GCP Role that provides read only access to GCP Resource Metadata.
v-inform.jinjavega-inform-custom-role-bindingBinds vega-inform-custom-role to the Service Account created at vega-service-account.
billing-export.jinjavega-billing-sa-bq-jobuser-bindingBinds the GCP Role roles/bigquery.jobUser (link) to vega-service-account at the Project level.This GCP Role provides Vega the ability to create BigQuery Jobs within the Project, but does not provide access to data.
billing-export.jinjavega-billing-datasetVegaBillingDatasetCreates a BigQuery Dataset in the Project and binds the GCP Role roles/bigquery.dataViewer (link) to vega-service-account, limiting access to this BigQuery Dataset.Billing Export will be configured to load data into this BigQuery Dataset.
billing-export.jinjavega-billing-export-{{ PROJECT_NUMBER }}vega-billing-export-{{ PROJECT_NUMBER }}A GCP Cloud Storage Bucket that will be used to export Billing Data from the BigQuery Dataset. Additionally binds two roles, both scoped to the Bucket: roles/storage.objectUser (link) to vega-service-account and roles/storage.admin (link) to the GCP API Service Account (allows Deployment Manager to manage this resource).The template includes a Lifecycle Policy on the Bucket to delete objects after 30 days.
v-operate.jinjavega-operate-actions-rolevegaOperateRoleA Custom GCP Role that provides the permissions necessary for Vega's Enterprise Parking
v-operate.jinjavega-operate-custom-role-bindingBinds vega-operate-actions-role to the Service Account created at vega-service-account

Either download the Template files or reference the files by their Github Raw URL in your Deployment Manager Resource Configuration when creating your configuration, which we will walk through below.

Deployment​

Step 1: Gather Environment Info​

First we need to gather some information about the environment in order to execute CLI commands to prepare the environment for deployment.

  1. If you do not know what the Project ID is (see screenshot below) then log in to your Google Cloud account and select the Project that we will be configuring for use with Vega. Copy the Project ID and the Project number to a text editor or another location you will be able to reference. img.png

  2. If you are deploying the GCP Role to the Organization (recommended) then execute this command, substituting in PROJECT_ID.

    gcloud projects get-ancestors <PROJECT_ID>

    For example with output:

    gcp_user@cloudshell:~$ gcloud projects get-ancestors circular-hybrid-431319-d7
    ID: circular-hybrid-431319-d7
    TYPE: project

    ID: 2###########
    TYPE: folder

    ID: 8##########
    TYPE: organization

    Copy the Organization ID from the output to the same location you copied the Project ID and Project Number.

Step 2: Assign GCP API Service Account Role​

Execute the below commands to bind the roles/owner GCP Role to the GCP API Service Account. Substitute in values for PROJECT_ID and PROJECT_NUMBER.

  1. Bind roles/owner at the Project level to the Project's GCP API Service Account

    gcloud projects add-iam-policy-binding <PROJECT_ID> \
        --member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
        --role roles/owner

  2. If deploying Roles to the GCP Organization then bind both roles/iam.organizationRoleAdmin and roles/resourcemanager.organizationAdmin to the Project's GCP API Service Account, substituting in the ORGANIZATION_ID and PROJECT_NUMBER as necessary.

    gcloud organizations add-iam-policy-binding <ORGANIZATION_ID> \
        --member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
        --role roles/resourcemanager.organizationAdmin

    gcloud organizations add-iam-policy-binding <ORGANIZATION_ID> \
        --member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
        --role roles/iam.organizationRoleAdmin

Note: if a binding condition is required (organization bindings frequently require one) it will be indicated in the console with a list of selections - choose None by entering the corresponding number next to that condition and pressing Enter.

Step 3: Deployment Manager Resource Configuration​

While you are able to manage GCP Deployment Manager Configurations using the command line we do not recommend doing so. We prefer to create a GCP Deployment Manager Configuration that defines our templates and the resources we will be creating. Configuration files (and templates) can be included in Infrastructure as Code solutions and will allow for repeatable deployments and simpler paths to updates in the future.

There are a number of GCP Deployment Manager Resource Configuration samples in that can be found in the Vega Cloud customer-samples GCP directory. These can be referenced to build the configuration appropriate to your use-case.

To create your GCP Deployment Manager Configuration, create a yaml file in the directory you are using for this process, give the file a name that is meaningful to you and your organization. A basic sample configuration is available below that is sufficient for the vInform Vega Product.

Your exact configuration may vary depending on your use-case and any GCP Organization Policies that may be in place. If you have any questions reach out to your Customer Success Manager or Onboarding Team.

  1. Create a yaml file wherever you are executing gcloud cli commands from, the name of the file should be meaningful to you, e.g. vega-integration.yaml
  2. Populate the file with a configuration appropriate to your use-case
    1. The sample configuration below is an example of the typical customer integration

Note: If you are deploying the GCP Role at the Organization level then fill in the resources[0].properties.OrganizationId. If you are deploying at the Project level then set it to an empty string, e.g. resources[0].properties.OrganizationId: ""

Sample Configuration​

The below sample configuration has been copied from here with small modifications to Store Templates on an External Host.

To summarize: when templates are stored at a static url we do not need to import the templates, but instead we reference the template in the resource's type key as seen below.

resources:
  - name: vInform
    type: https://raw.githubusercontent.com/vegacloud/customer-samples/main/latest/gcp/templates/v-inform.jinja
    properties:
      OrganizationId: 8##########
  - name: Vega-Billing-Export
    type: https://raw.githubusercontent.com/vegacloud/customer-samples/main/latest/gcp/templates/billing-export.jinja
    properties:
      BindingType: $(ref.vInform.BindingType)
      VegaServiceAccountEmail: $(ref.vega-service-account.email)
    metadata:
      dependsOn:
        - vInform

A typical Resource Configuration for a Customer who is using the vInform offering from Vega is above. Using the tables at Vega Products and the Resources Deployed we expect this configuration to:

  • deploy the base Vega template v-inform.jinja, which will create:
    • a Service Account named vega-service-account
    • a GCP Custom IAM Role at the Organization level named vegaInformRole
    • a binding of the Organization Role vegaInformRole to the Service Account vega-service-account
  • deploy the Billing Export template billing-export.jinja, which will create:
    • a binding of the GCP defined Role roles/bigquery.jobUser to the Service Account vega-service-account
    • a BigQuery Dataset named VegaBillingDataset
    • a GCP Cloud Storage Bucket named vega-billing-export-<PROJECT_NUMBER>

Step 4: Deploy the Resource Configuration​

Substitute into the command below DEPLOYMENT_NAME (lower-case only) and RESOURCE_CONFIG_YAML (the yaml created in the previous step).

In the same directory as your Deployment Manager Resource Configuration, execute the following gcloud CLI command:

gcloud deployment-manager deployments create <DEPLOYMENT_NAME> --config <RESOURCE_CONFIG_YAML>

Step 5: Cloud Billing Export​

Enable Cloud Billing Export to BigQuery​

  1. Sign into the Google Cloud Console (if not already signed in).

  2. Open the Console's Navigation menu > Billing and verify that your desired Billing Account is selected.

  3. Under Cost Management select Billing export.

  4. Depending on your scenario, you will see one of the following:

    1. Projects with No Linked Billing Account

      Enable Billing on the Project by either selecting Link a Billing Account or Manage Billing Account Options.

    2. Projects with Linked Billing Account(s)

      a. If Billing is enabled on the Project and you have only one Cloud Billing Account then the Billing Overview Page will display.

      b. If Billing is enabled on the Project and you have MORE THAN one Cloud Billing Account a pop-up window will display: "Billing Account <YOUR_BILLING_ACCOUNT> is linked to this Project."

  5. Within the BigQuery export tab, under "Detailed usage cost" click "EDIT SETTINGS"

  6. For the Projects drop-down menu, select the project where you created the BigQuery Dataset previously.

  7. For the Dataset drop-down menu, select the Dataset you created in the Create a BigQuery Dataset section.

  8. Click "SAVE" You will see the report has been enabled. Note: You must enable BigQuery API for the project you set up to export data to BigQuery.

Billing Data Tables​

Once you enable Cloud Billing export to BigQuery, billing data tables are automatically created in the BigQuery Dataset. Make note of the billing data table name. You will need this information during Billing Connector creation.

  1. Navigate to your deployment project within BigQuery.
  2. Click the dropdown next to your project to expose the VegaDataset created previously.
  3. Click the dropdown next to VegaDataset to expose the table name.
  4. Click the action menu next to the table, and select "Copy ID"
  5. Provide the Table ID in your clipboard to your Vega CSM or Onboarding Engineer. Example: testproj-123456.VegaBillingDataset.gcp_billing_export_resource_v1_01C321_654321_321A1D

Step 6: Create Service Account Credentials​

Now we need to create a Google Credentials File so Vega can authenticate to the Service Account created previously.

This could be included with the Deployment Manager Resource Templates but we prefer not to as this would include the Service Account credentials in the output in plain text.

How you perform this task is up to you and may vary depending on your environment. Just ensure you send the Google Credentials File to your Customer Success Manager or Onboarding Team.

This author prefers to do this through the CLI, though it can be accomplished through the Google Cloud Console as well.

gcloud CLI

Substitute SERVICE_ACCOUNT_EMAIL for the Vega Service Account's email, which by default is vega-sa@<PROJECT_ID>.iam.gserviceaccount.com

gcloud iam service-accounts keys create ~/vega-sa.json --iam-account <SERVICE_ACCOUNT_EMAIL>

Please provide the resulting file, vega-sa.json in the above example, to your Vega Customer Success Manager or Onboarding Team.

Google Cloud Console (web UI)
  1. In the Cloud Console navigate to IAM & Admin > Service Accounts, select the Project from the drop-down menu at the top.
  2. Find the Service Account created by Deployment Manager, default email is vega-sa@<PROJECT_ID>.iam.gserviceaccount.com with the name Vega Inform Service Account
  3. On the right side of the page in the Actions column, click the ellipsis and select Manage Keys
  4. Click the ADD KEY drop down and select Create new key
  5. Leave JSON as the key type and click CREATE, this will cause the key to be generated and the file downloaded.
  6. Provide the downloaded JSON Google Credentials File to your Customer Success Manager or Onboarding Team.

The resulting file will look something like this:

{  
    "type": "service_account",  
    "project_id": "project-id",  
    "private_key_id": "key-id",  
    "private_key": "-----BEGIN PRIVATE KEY ---- \nprivate-key\n-----END PRIVATE KEY-----\n",  
    "client_email": "service-account-email",  
    "client_id": "client-id",  
    "auth_uri": "https://accounts.google.com/o/oauth2/auth", 
    "token_uri": "https://accounts.google.com/o/oauth2/token", 
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", 
    "client_x509_cert_url":"https://www.googleapis.com/robot/ v1/metadata/x509/service-account-email" 
}

Step 7: Provide Information to Vega​

Vega will require the Service Account Name and generated Key for each of the Service Accounts created during onboarding in addition to the BigQuery Dataset Name and Table Name used for the Billing Export.

Step 8: Remove Elevated Permissions (optional)​

If you do not want the GCP API Service Account to retain the GCP Roles we bound earlier then those bindings may be removed, however it is important to note that Deployment Manager will not have the ability to update or delete these Roles or Bindings without re-binding the Roles to the GCP API Service Account.

Which of the below commands you execute is dependent on your environment, configuration, organizational requirements, and the commands you executed previously as part of this setup.

gcloud organizations remove-iam-policy-binding <ORGANIZATION_ID> \
     --member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
     --role roles/resourcemanager.organizationAdmin
     
gcloud organizations remove-iam-policy-binding <ORGANIZATION_ID> \
     --member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
     --role roles/iam.organizationRoleAdmin
     
gcloud projects remove-iam-policy-binding <PROJECT_ID> \
     --member serviceAccount:<PROJECT_NUMBER>@cloudservices.gserviceaccount.com \
     --role roles/owner

Adding GCP Projects to the Vega Platform​

Use the process below to add your Google Cloud Projects to the Vega Platform. Provider Accounts will use the Service Account named vega-sa that was created by GCP Deployment Manager in Step 3.

Step 1: Navigate to settings -> provider accounts and click 'Link Account, then select GCP'​

GCP Account Create

Step 2: Paste the service account JSON into the Service Account field, and optionally enter an alias​

  • Vega account alias: A friendly name for the account

If you aren't ready for your accounts to be ingested by Vega yet, you can toggle the 'Enabled' switch to 'Disabled' to prevent data ingestion.

NOTE: If you have multiple projects, repeat the above steps for each project or alternatively use the 'Bulk Import' feature to add multiple subscriptions at once.