Access Google Drive from an instance

Sign a notebook on your instance in to your Google Drive and download files from it into the workspace. You create an OAuth client in the Google Cloud console, then sign in once from the notebook by copying a code from your browser. The sign-in token is kept in /home/jovyan, so later runs reuse it. The same sign-in also lets you search, list, upload, and create folders in Drive.

TIR has a tutorial notebook with the same steps. It is public, so you can read it in your browser at the TIR tutorial notebook for Google Drive. To run it on your instance, download it into the workspace from a JupyterLab terminal:

cd /home/jovyan
wget https://github.com/tire2e/notebooks/raw/main/notebooks/tir-tutorials/drive_access_in_notebook.ipynb

Before you begin

• A Running instance whose image includes JupyterLab, such as a TIR pre-built image. See Create an instance.
• A Google account with access to the Google Cloud console, and a Google Cloud project to hold the OAuth client.
• The Google Drive API turned on for that project. Open Google Drive API in the Cloud console and select Enable.

Set up the app in Google Auth platform

Google asks you to describe the app that signs in to Drive on your behalf. In the Google Cloud console, choose your project, then:

1. Open Google Auth platform > Branding. If this is the project’s first app, select Get started.
2. Enter an App name and a User support email, and select Next.
3. For the audience, choose External, and select Next. Choose Internal instead only when everyone who signs in belongs to your own Google Workspace organization.
4. Enter an email address for Google’s notices, select Next, agree to the Google API Services User Data Policy, and select Continue and then Create.
5. For an External app, open Audience and select Publish app, so that the publishing status is In production.

While an external app is in Testing, the sign-in it gives expires seven days after you allow access, and you have to sign in again. Publishing the app removes that seven-day limit.

Full access to Drive is a restricted permission, so when you sign in, Google warns that it has not verified the app. Because the app is yours and you sign in to your own account, you can continue past the warning.

Create the OAuth client

1. Open Google Auth platform > Clients and select Create Client.

2. For Application type, choose Web application. The copy-the-code sign-in sends Google’s reply to a TIR page, and a web application client lets you add that page as its redirect address.

3. Enter a Name that tells you what the client is for, such as Instance Client.

4. Under Authorized redirect URIs, add this address exactly, with the slash at the end:

   https://tir.e2enetworks.com/oauth2callback/

5. Select Create, and download the client’s JSON file. It holds the client ID and the client secret, so keep it private.

6. In JupyterLab, upload the file to /home/jovyan with the file browser’s upload button, and name it credentials.json.

Keep the file in /home/jovyan. Everything outside the workspace, including /root and /tmp, is emptied whenever the instance restarts.

Install the Google libraries

Run this in a JupyterLab terminal, or in a notebook cell with ! in front:

pip3 install --upgrade google-auth google-api-python-client google-auth-oauthlib google-auth-httplib2

Packages you install this way are lost when the instance restarts. Install them again after a restart, or from a start script; see Run start scripts on an instance.

Sign in to Google Drive

Run this in a notebook cell. It uses the saved token when there is one, renews it when it has expired, and otherwise asks you to sign in. It then saves the token to /home/jovyan/token.json and connects to Drive.

import os
import io
from googleapiclient.discovery import build
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request

creds_file_path = "/home/jovyan/credentials.json"
token_path = "/home/jovyan/token.json"

SCOPES = ['https://www.googleapis.com/auth/drive']
creds = None

# Load existing credentials if available
if os.path.exists(token_path):
    creds = Credentials.from_authorized_user_file(token_path, SCOPES)

# If there are no valid credentials, start OAuth flow
if not creds or not creds.valid:
    if creds and creds.expired and creds.refresh_token:
        creds.refresh(Request())
    else:
        flow = InstalledAppFlow.from_client_secrets_file(
            creds_file_path,
            SCOPES,
            redirect_uri='https://tir.e2enetworks.com/oauth2callback/'
        )
        auth_url, _ = flow.authorization_url(prompt='consent')
        print('Go to the following URL in your browser:')
        print(auth_url)
        code = input('Enter the authorization code: ')
        flow.fetch_token(code=code)
        creds = flow.credentials

    # Save the credentials for the next run
    with open(token_path, 'w') as token:
        token.write(creds.to_json())

# Build the Drive API client
service = build('drive', 'v3', credentials=creds)

The first time, the cell prints a Google address and waits for a code:

1. Open the address in your browser, choose your Google account, and allow access.
2. Google sends you to a TIR page that shows Google Authorization Code. Select Copy Code.
3. Paste the code into the box under the cell and press Enter.

The cell saves the token and connects to Drive. Later runs read the token from /home/jovyan/token.json. You sign in again only when Google ends the sign-in, for example after seven days for an app still in Testing.

This copy-the-code sign-in works on an instance because your browser receives Google’s reply. Google’s own Python quickstart starts a sign-in server on the machine that runs the code, and on an instance that server is reachable only from the instance itself.

Download a file

Find the file’s ID in its Drive link: it is the long string after /d/. Then run this in a notebook cell, with the file ID and the path to save to:

from googleapiclient.http import MediaIoBaseDownload

file_id = 'your_file_id_here'                       # replace with the Drive file ID
file_path = '/home/jovyan/downloaded_file.ext'      # where to save the file in the workspace

request = service.files().get_media(fileId=file_id)
fh = io.FileIO(file_path, mode='wb')
downloader = MediaIoBaseDownload(fh, request)
done = False

while not done:
    status, done = downloader.next_chunk()
    if status:
        print(f"Download {int(status.progress() * 100)}%.")
fh.close()
print('Download complete:', file_path)

The cell prints the progress in percent and then the path of the downloaded file. Save files under /home/jovyan, or on a mounted dataset or file system, so they are kept when the instance restarts.

This works for files you uploaded to Drive, such as a CSV, a ZIP, or a model file. Google Docs, Sheets, and Slides are online documents in Drive, so export them to a format of your choice. To save a Google Doc as a PDF, replace the request = ... line with this, and end the path you save to in .pdf:

request = service.files().export_media(
    fileId=file_id,
    mimeType='application/pdf',
)

For the other export formats, and for searching, uploading, and creating folders, see the tutorial notebook and the Google Drive API guides.

If the sign-in fails

The redirect address does not match

The address under Authorized redirect URIs must be exactly https://tir.e2enetworks.com/oauth2callback/, with the slash at the end, the same as in the sign-in code. Correct it in Clients and run the cell again.

Authorization code not found

The TIR page opened without a code from Google, so it takes you to the TIR home page instead. Run the cell again, and open the new address it prints.

The code is refused

Copy the code exactly, trimming any space or line break around it. Each code works once, so run the cell again for a new one.

The sign-in expires every week

The app is still in Testing. Publish it as described in Set up the app in Google Auth platform.

The credentials file is not found

Check that credentials.json is in /home/jovyan and that the path in the code matches it.

The sign-in does not finish from the notebook

Sign in on your own computer instead, with Google’s Python quickstart for Drive and the Desktop app client it describes. Then upload the token.json it writes to /home/jovyan.

For other ways to move files in and out, such as over SSH, see Copy files to and from an instance. For other problems on the instance, see Troubleshooting TIR Instances. Google’s guides to configure the OAuth consent and create credentials cover each console screen in full.