Google Object Storage as a Source

Google Cloud Storage (GCS) is an object storage service offered by Google Cloud Platform (GCP). This page contains the setup guide and reference information for the Google Cloud Storage source connector.

Pre-requisites

  • Bucket Name: Name of the GCS bucket where the file(s) exist.

  • Service Account Information: Google Cloud service account key in JSON format.

Setup guide

Step 1: Create a New Service account

  1. Open the Service Accounts page in your Google Cloud console.

  2. Select an existing project, or create a new project.

  3. At the top of the page, click + Create service account.

  4. Enter a name and description for the service account, then click Create and Continue.

  5. Under Service account permissions, select the roles to grant to the service account, then click Continue.

Step 2: Generate a key

  1. Go to the API Console/Credentials page and click on the email address of the service account you just created.

  2. In the Keys tab, click + Add key, then click Create new key.

  3. Select JSON as the Key type. This will generate and download the JSON key file that you’ll use for authentication. Click Continue.

Step 3: Enable the Google Drive API

  1. Go to the API Console/Library page.

  2. Make sure you have selected the correct project from the top.

  3. Find and select the Google Drive API.

  4. Click ENABLE.

Step 4: Grant Permissions

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

  2. In the list of buckets, click the name of the bucket for which you want to grant a principal a role.

  3. Select the Permissions tab near the top of the page.

  4. Click the + Grant access button.

  5. In the New principals field, enter your service account address.

  6. Select the role (or roles) from the Select a role drop-down menu to give access to storage account. The roles you select appear in the pane with a short description of the permissions they grant.

  7. Click Save.

Step 5: Setting Up Google Cloud Storage Source on TIR

  1. Go to TIR

  2. Go to Data Syncer > Sources > Create Source.

  3. Choose Google Cloud Storage.

  4. Add your Bucket Name from where you want your data to be ingested.

  5. Add the JSON file that was downloaded in Step 2 in the Service Account Information.

  6. Click CREATE.