Google Cloud Storage FUSE
Cloud Storage FUSE is an open source FUSE adapter that allows you to mount Cloud Storage buckets as file systems on Linux or macOS systems. It also provides a way for applications to upload and download Cloud Storage objects using standard file system semantics. Cloud Storage FUSE can be run anywhere with connectivity to Cloud Storage, including Google Compute Engine VMs or on-premises systems1. This allows you to interact with Google Cloud Storage Objects as if they were files on a local file system.
This can be useful for number of purposes, such as:
Accessing Google Cloud Storage objects from an application that does not have native support for Google Cloud Storage.
Performing batch operations (such as moving, copying, or deleting large numbers of objects) directly from the file system, without the need to use a Google Cloud Storage client library.
Using tools like rsync or backup software to back up data to Google Cloud Storage.
This demo will show you how to mount storage bucket as filesystem on Compute Engine VMs and on your local machine(Windows Subsystem for Linux)
Google Compute Engine VMs:
Following are the steps to be followed while mounting your storage bucket to Compute Engine VMs:
Provision your VM- Create a VM with a service account having Storage Admin role.
Create a storage bucket with an appropriate name that you want to mount later.
SSH into the VM you just created and execute the following commands.
Add the gcsfuse distribution URL as a package source and import its public key:
export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb https://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
Update the list of packages available and install gcsfuse.
sudo apt-get update
sudo apt-get install gcsfuse
(Ubuntu before wily only) Add yourself to the fuse group, then log out and back in:
sudo usermod -a -G fuse $USER
exit
4. Create a folder in the home directory, where it has to be mounted. Eg: mkdir linux-mount. Use df -h to check if anything else is mounted before executing the gcsfuse command.
5.Now execute the gcsfuse command accordingly- gcsfuse (your bucketname) (folder you want to mount on).
Gcsfuse buckt-task1 linux-mount
As you can see below the bucket (buckt-task1) is successfully mounted on /home/adithisathish/linux-mount.
You can check this using df -h
As you can see,1.0P size disk is mounted. By this you can easily transfer a good amount of data.
6. cd into the linux-mount folder and use ls command to list all the content available.
7.Now create a folder named text.txt- touch text.txt
8. Now you can add anything into this folder:
9.Go to the storage bucket in the web console and cross check if you can see the folder and message you just created.
If you click on text.txt and run the URL in a new tab you can see the message you just created.
Local machine-Windows subsystem for Linux(WSL): (using Ubuntu machine)
Prerequisites:
-Before starting gcloud CLI needs to be installed in your local machine. Follow these steps to install gcloud CLI: (for Ubuntu Machine)
curl https://sdk.cloud.google.com | bash
exec -l $SHELL
gcloud init
Steps to be followed:
Once gcloud CLI is installed in your Ubuntu machine, login into it using the below command.
● gcloud auth application-default login or gcloud auth login
- Once this command is run, you will get a URL which is needed to authenticate to your gcp account. Copy paste the URL in a new tab which will ask you to choose the gcloud account.Once you choose the gcloud account and proceed you will get a token/code which you need to paste below in order to log in.
2.After you are logged in, create a directory where you will be mounting your storage bucket.Eg: mkdir windows-mount.Now you use the gcsfuse command to mount your storage bucket.
● gcsfuse buckt-task1 mkdir windows-mount/
As you can see 1.0P size disk is available and file system is successfully mounted.
3.Once the bucket is mounted you can cd into the windows-mount folder and add objects.
4.You can re-check this in the web console. As you can see windows.txt file is created.
Conclusion:
In conclusion, the GCS Fuse adapter is a useful tool for those who want to interact with Google Cloud Storage as if it were a local file system. It provides a simple and convenient way to access, manipulate, and backup data stored in Google Cloud Storage. With the ability to mount Google Cloud Storage buckets as a file system, users can take advantage of familiar file system operations and tools to manage their data in the cloud. Whether you are a developer looking to integrate with Google Cloud Storage in your application, or simply someone who wants to backup data to the cloud, GCS Fuse is a valuable tool to have in your arsenal.
