Google Cloud Storage | Documentation

The Google Cloud Storage component is a data component that allows users to upload data to Google's Cloud Storage. It can carry out the following tasks:

Upload
Read Objects
Create Bucket

#Release Stage

Alpha

#Configuration

The component definition and tasks are defined in the definition.yaml and tasks.yaml files respectively.

#Setup

In order to communicate with Google, the following connection details need to be provided. You may specify them directly in a pipeline recipe as key-value pairs within the component's setup block, or you can create a Connection from the Integration Settings page and reference the whole setup as setup: ${connection.<my-connection-id>}.

Field	Field ID	Type	Note
JSON Key File Contents (required)	`json-key`	string	Contents of the JSON key file with access to the bucket.

#Supported Tasks

#Upload

Upload data to Google Cloud Storage.

Input	Field ID	Type	Description
Task ID (required)	`task`	string	`TASK_UPLOAD`
Bucket Name (required)	`bucket-name`	string	Name of the bucket to be used for object storage.
Object Name (required)	`object-name`	string	The name of the object to be created.
Data (required)	`data`	string	The data to be saved in the object.

Output	Field ID	Type	Description
Authenticated URL (optional)	`authenticated-url`	string	Only users granted permission can access the object with this link.
Gsutil URI (optional)	`gsutil-uri`	string	File path to this resource in Cloud Storage.
Public Access (optional)	`public-access`	boolean	Whether the object is publicly accessible.
Public URL (optional)	`public-url`	string	Anyone with this link can access the object on the public Internet.
Upload Status	`status`	string	Status of the upload operation.

#Read Objects

Read objects from Google Cloud Storage.

Input	Field ID	Type	Description
Task ID (required)	`task`	string	`TASK_READ_OBJECTS`
Bucket Name (required)	`bucket-name`	string	Name of the bucket to be used for object storage.
Delimiter	`delimiter`	string	The delimiter to use when listing objects.
Prefix	`prefix`	string	The prefix to use when listing objects.
Versions	`versions`	boolean	Whether to list all versions of an object.
Start Offset	`start-offset`	string	The offset to start listing objects from.
End Offset	`end-offset`	string	The offset to stop listing objects at.
Include Trailing Delimiter	`include-trailing-delimiter`	boolean	Whether to include the delimiter in the object name.
Match Glob	`match-glob`	string	Whether to match objects using a glob pattern.
Include Folders As Prefixes	`include-folders-as-prefixes`	boolean	Whether to include folders as prefixes.

Output	Field ID	Type	Description
Text Objects (optional)	`text-objects`	array[object]	The text objects in the bucket.
Image Objects (optional)	`image-objects`	array[object]	The image objects in the bucket.
Document Objects (optional)	`document-objects`	array[object]	The document objects in the bucket.
Audio Objects (optional)	`audio-objects`	array[object]	The audio objects in the bucket.
Video Objects (optional)	`video-objects`	array[object]	The video objects in the bucket.

Output Objects in Read Objects

Text Objects

Field	Field ID	Type	Note
Attributes	`attributes`	object	The attributes of the object.
Data	`data`	string	The data of the object.

Attributes

Field	Field ID	Type	Note
Content Disposition	`content-disposition`	string	The content disposition of the object.
Content Encoding	`content-encoding`	string	The content encoding of the object.
Content Language	`content-language`	string	The content language of the object.
Content Type	`content-type`	string	The content type of the object.
Media Link	`media-link`	string	The media link of the object.
Metadata	`metadata`	object	The object metadata.
Object Name	`name`	string	The name of the object.
Owner	`owner`	string	The owner of the object.
Size	`size`	integer	The size of the object.
Storage Class	`storage-class`	string	The storage class of the object.

Image Objects

Field	Field ID	Type	Note
Attributes	`attributes`	object	The attributes of the object.
Data	`data`	string	The data of the object.

Attributes

Field	Field ID	Type	Note
Content Disposition	`content-disposition`	string	The content disposition of the object.
Content Encoding	`content-encoding`	string	The content encoding of the object.
Content Language	`content-language`	string	The content language of the object.
Content Type	`content-type`	string	The content type of the object.
Media Link	`media-link`	string	The media link of the object.
Metadata	`metadata`	object	The object metadata.
Object Name	`name`	string	The name of the object.
Owner	`owner`	string	The owner of the object.
Size	`size`	integer	The size of the object.
Storage Class	`storage-class`	string	The storage class of the object.

Document Objects

Field	Field ID	Type	Note
Attributes	`attributes`	object	The attributes of the object.
Data	`data`	string	The data of the object.

Attributes

Field	Field ID	Type	Note
Content Disposition	`content-disposition`	string	The content disposition of the object.
Content Encoding	`content-encoding`	string	The content encoding of the object.
Content Language	`content-language`	string	The content language of the object.
Content Type	`content-type`	string	The content type of the object.
Media Link	`media-link`	string	The media link of the object.
Metadata	`metadata`	object	The object metadata.
Object Name	`name`	string	The name of the object.
Owner	`owner`	string	The owner of the object.
Size	`size`	integer	The size of the object.
Storage Class	`storage-class`	string	The storage class of the object.

Audio Objects

Field	Field ID	Type	Note
Attributes	`attributes`	object	The attributes of the object.
Data	`data`	string	The data of the object.

Attributes

Field	Field ID	Type	Note
Content Disposition	`content-disposition`	string	The content disposition of the object.
Content Encoding	`content-encoding`	string	The content encoding of the object.
Content Language	`content-language`	string	The content language of the object.
Content Type	`content-type`	string	The content type of the object.
Media Link	`media-link`	string	The media link of the object.
Metadata	`metadata`	object	The object metadata.
Object Name	`name`	string	The name of the object.
Owner	`owner`	string	The owner of the object.
Size	`size`	integer	The size of the object.
Storage Class	`storage-class`	string	The storage class of the object.

Video Objects

Field	Field ID	Type	Note
Attributes	`attributes`	object	The attributes of the object.
Data	`data`	string	The data of the object.

Attributes

Field	Field ID	Type	Note
Content Disposition	`content-disposition`	string	The content disposition of the object.
Content Encoding	`content-encoding`	string	The content encoding of the object.
Content Language	`content-language`	string	The content language of the object.
Content Type	`content-type`	string	The content type of the object.
Media Link	`media-link`	string	The media link of the object.
Metadata	`metadata`	object	The object metadata.
Object Name	`name`	string	The name of the object.
Owner	`owner`	string	The owner of the object.
Size	`size`	integer	The size of the object.
Storage Class	`storage-class`	string	The storage class of the object.

#Create Bucket

Read objects from Google Cloud Storage.

Input	Field ID	Type	Description
Task ID (required)	`task`	string	`TASK_CREATE_BUCKET`
Bucket Name (required)	`bucket-name`	string	Name of the bucket to be used for object storage.
Project ID (required)	`project-id`	string	The project ID to create the bucket in.
Location	`location`	string	The location to create the bucket in.

Output	Field ID	Type	Description
Result (optional)	`result`	string	The result of the operation.
Bucket Name (optional)	`name`	string	The name of the bucket.
Location (optional)	`location`	string	The location of the bucket.