Google Cloud Storage files

Google Cloud Storage is Google Cloud's object store. @mayurajs/filestorage-gcs plugs a bucket into mayura/files, so you get the same createFileStore API, limits, per-tenant views and file tools as with the built-in S3 backend. It talks to the Cloud Storage JSON API with fetch: no Google SDK, no other dependencies, and it runs on every runtime Mayura runs on.

#Install

npm install mayura @mayurajs/filestorage-gcs

#Credentials

The backend takes an OAuth 2.0 access token source, a function called before each request. A service account is the usual source: create one under IAM and admin, grant it the Storage Object User role (roles/storage.objectUser) on the bucket, which covers creating, reading, listing and deleting objects, and mint tokens for it with a Cloud Storage scope, for example with google-auth-library.

The package reads nothing from the environment, neither GOOGLE_APPLICATION_CREDENTIALS nor STORAGE_EMULATOR_HOST: your token source decides. Keep the bucket name in GCS_BUCKET, for example.

#Use it

import { createFileStore, fileTools } from 'mayura/files';
import { gcsFiles } from '@mayurajs/filestorage-gcs';

// An OAuth access token source, such as google-auth-library's () => auth.getAccessToken() (unwrapped to the string).
const token = async (): Promise<string> => process.env.GCS_ACCESS_TOKEN ?? '';

const files = createFileStore(gcsFiles({ bucket: process.env.GCS_BUCKET ?? 'acme-reports', token }), {
  maxFileBytes: 10 * 1024 * 1024,
});

// Create a file only if none has this key, then change it only if nobody else did since.
const created = await files.put('reports/q3.csv', new TextEncoder().encode('order,total\n'), { contentType: 'text/csv', ifNoneMatch: '*' });
await files.put('reports/q3.csv', new TextEncoder().encode('order,total\nord-1001,42.00\n'), { contentType: 'text/csv', ifMatch: created.etag });

// Or give an agent tools over one tenant's files; grant tool:reports.read, tool:reports.list and files:reports:read.
const tools = fileTools(files.within('tenants/acme'), { name: 'reports' });

See Files for every store method, the store's own options (maxFileBytes, maxListLimit, timeoutMs, prefix) and the file tools.

#Options

#Versions and preconditions

A file's etag is its Cloud Storage generation, a number that changes on every write. The store's conditionalWrites and conditionalDelete are both true:

• ifNoneMatch: '*' uploads with ifGenerationMatch=0: the write succeeds only if no object has the key.
• ifMatch on a write, a read or a delete sends that generation as ifGenerationMatch.
• An etag that is not a generation this backend issued fails with CONFLICT before anything is sent.

A read fetches the object's information, then downloads exactly that generation, so the bytes and the size, etag and metadata you get describe one version. If the file is replaced in between, the read starts again, up to three times.

#Writes, metadata and limits

• Each write is one multipart upload: the object's information and its bytes in one request.
• The media type is stored as the object's contentType; metadata is stored as Cloud Storage custom metadata and comes back from get and head. Listings return each file's size, etag, media type and time, but not its metadata.
• Files are as large as the store's maxFileBytes, up to 5 GiB. Each file is held in memory while it is written or read.
• Ranges are read with an HTTP Range request.

#Errors

Failures reach you as a FileStoreError (code STORAGE_UNAVAILABLE) with a reason, mapped from Cloud Storage's answers. Nothing Cloud Storage wrote reaches the message.

Configuration mistakes, such as a bucket name with capitals or an http endpoint on another host, fail with INVALID_CONFIG when you call gcsFiles().

#How it is tested

• Unit tests run the file store conformance suite against a fake JSON API that checks authorization, multipart uploads, generations and preconditions: all 13 cases pass. Other tests check the requests sent, that a file replaced mid-read is read at one version, that errors carry no service text, and that nothing is read from the environment.
• In CI, the suite runs against fake-gcs-server, an independent Cloud Storage emulator: 12 cases pass and "deletes only the version given with ifMatch" is skipped, because the emulator ignores ifGenerationMatch on deletes.
• It has not been run against a real Google Cloud account yet.

#Related

• Files: stores, views, preconditions and file tools.
• Artifacts: keep agent artifacts in this bucket.
• Google Drive files: files in a Drive folder instead.
• Azure Blob Storage files: the same store on Azure.