In this third part of our series on storage options for Google Cloud VMware Engine, we will discuss Google Cloud Storage, an object-based storage architecture designed to work with large amounts of unstructured data.
Previous posts in the series:
- Storage Options with Google Cloud VMware Engine – Part 1
- Google Cloud Filestore with Google Cloud VMware Engine – Part 2
Google Cloud Storage – Overview
If you are unfamiliar with Object Storage, a good primer to read is “What is Object Storage?” available on Google’s learning site. As mentioned earlier, Google Cloud Storage is object storage, which differs from file storage or block storage mainly because it is accessed using RESTful APIs over HTTP/HTTPS. In addition, object storage uses a construct called “buckets,” in which items are stored individually in a highly scalable and elastic manner and can be accessed by metadata or its unique identifier. Although this storage pool is highly scalable, one of the potential downsides to object storage is that to modify an object, it needs to be rewritten entirely.
Table 1: Main Google Cloud Storage Bucket Options
| Option | Choices |
|---|---|
| Bucket Name | Must be globally unique and follow naming guidelines |
| Location Type | Multi-region, Dual Region, Single Region |
| Storage Class | Autoclass, Standard, Nearline, Coldline, Archive |
| Access control | Prevent Public Access (Enabled/Disabled), Uniform, Fine-grained |
| Protection | None, Object Versioning OR Retention Policy |
| Data Encryption | Google Managed encryption key, Customer Managed encryption key (CMEK) |
Configuration examples
The main uses for object storage are in developing cloud-native applications, big data analytics, or rich media storage. However, there are other practical uses, so I will share a few.
Archiving server files
In this first configuration example, I will create a new bucket to archive my intranet web server’s Apache DocumentRoot directory. This will be done from one of my web servers running Ubuntu 22.04.2 LTS in my Google Cloud VMware Engine web farm.
Note: If you haven’t already, you will need to install the gcloud CLI, which will allow you to use the gcloud, gsutil, and bq commands.
If this is the first time running the Google Cloud CLI commands from a host machine, you will have to authenticate and set the preferred project name.
dschmitz@web01:~$ gcloud auth login
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=*****big-long-url*****
Enter authorization code: ***********************Paste-the-auth-code-here**************************
You are now logged in as [************@**********.com].
Your current project is [None]. You can change this setting by running:
$ gcloud config set project PROJECT_ID
dschmitz@web01:~$ gcloud projects list
PROJECT_ID NAME PROJECT_NUMBER
cibg-tmm-ds CIBG-TMM-DS 1066123456789
dschmitz@web01:~$ gcloud config set project cibg-tmm-ds
Updated property [core/project].Once you are logged in and the preferred project is set, you can start creating buckets. Remember that bucket names must be unique and follow the naming guidelines. If you are creating one or more buckets programmatically with a script, you can even use a tool like “uuidgen” to generate unique bucket names for you.
dschmitz@web01:~$ uuidgen
9092f1bb-58c0-4172-8e20-4589866618a2
dschmitz@web01:~$ gsutil mb -b on -l us-central1 -c standard gs://9092f1bb-58c0-4172-8e20-4589866618a2/
Creating gs://9092f1bb-58c0-4172-8e20-4589866618a2/...Scripting tip: Within shell scripts, you can embed the uuidgen command into the bucket creation command like this:
gsutil mb -b on -l us-central1 -c standard gs://`uuidgen`/With the bucket created, I can easily archive the host’s web server html directory in case I need it later.
Note: Object storage has a flat namespace, so it doesn’t actually have any concept of directories or folders. In other words, copying an object in the format “foldername/filename.txt” to the bucket just appends the folder to the filename. Because of this, the Google Cloud console and the gsutil will visually represent a file and folder tree to help you keep everything organized.
Archiving old virtual machines or templates
Another thing that object storage can be used for is archiving old virtual machines or templates. Instead of storing that old virtual machine data on primary storage, such as within the vSAN cluster or Compute Engine Instances, sometimes it makes more sense to migrate the VM to a lower-cost Google Cloud Storage bucket. In some cases, and depending on the storage class and location assigned, the costs can be as low as a fraction of a penny per gigabyte.
From my Google Compute Engine management instance, I can export old virtual machines to OVA and subsequently copy them to my bucket.
Note: If this type of operation is something you might be doing frequently, Google Cloud Storage FUSE allows an admin to mount a bucket to a folder on a Debian or Ubuntu host.
Google Cloud Storage as a backup target
There are countless backup software vendors that can use Google Cloud Storage as a backup target. In this final example, I have installed a free and open-source backup software named Duplicati on one of my Windows servers in my Google Cloud VMware Engine private cloud. When I create a new backup job, the storage type dropdown allows me to specify the bucket I created earlier as the storage target.
After the job has been run, you can see the encrypted backup segments Duplicati has uploaded to the bucket.
Data Protection
To protect the data within each bucket, an admin can assign their desired options for controlling the data lifecycle. In a nutshell, the options either allow you to specify rules for how many versions of each object you would like to keep, or a static retention period.
Conclusion
Using Google Cloud Storage is a simple and cost-effective way to store data, either for use with cloud-native applications, or even administrative use cases. There are still a few more storage options to explore in future posts, so be sure to check back again!
For the latest technical guidance, blogs, or demo videos for Google Cloud VMware Engine, check out VMware Cloud Tech Zone.
Update: Check out the next post in the series — NetApp Cloud Volumes Service with Google Cloud VMware Engine – Part 4