@brighter/storage

A cloud agnostic JavaScript object storage library that's built with simplicity and local development in mind.

It offers:

• a unified storage interface (for seamless switching between providers),
• a local storage provider implementation (enabling local development),
• an S3 compatible storage provider (AWS S3, Cloudflare R2, Hetzner Object Storage, Backblaze B2, ...),
• a simple, concise API that's tested against real infrastructure and
• comprehensive documentation.

Why

Most of today's software talks directly to the cloud, even in local environments. This extends the feedback loop and creates a storage provider dependency.

This library takes a different approach.

It introduces a unified storage interface that enables seamless switching between providers and a local storage provider implementation that shortens the feedback loop and maximizes velocity during development.

It's API is easy to use and remember. It removes away the complexity of manually passing around continuation tokens, promise throttling, dealing with various different content encodings and presigning requests.

Quick Start

Instead of manually installing and injecting the dependencies, you'll most likely want to use one of the following storage adapters that come pre-bundled with everything required:

• @brighter/storage-adapter-local and
• @brighter/storage-adapter-s3 (AWS S3, Cloudflare R2, Hetzner Object Storage, Backblaze B2, ...).

Note: Before installing, Node.js 18 or higher is required.

Installation, using npm:

npm i @brighter/storage-adapter-local
npm i @brighter/storage-adapter-s3

Usage:

import { Storage } from '@brighter/storage-adapter-s3'

const storage = Storage({ path: 'my-bucket' }, { region: 'eu-central-1' })

const main = async () => {
  await storage.write('file', 'hello')
  const msg = await storage.read('file')
  console.log(msg)
}

main().catch(console.error)

For more information:

• have a look at the demo folder or
• dive straight into the documentation.

API

Here's a quick API overview:

await storage.read(file, options)
await storage.write(file, stringOrBuffer, options)
await storage.remove(fileOrDir, options)
await storage.list(dir, options)
await storage.copy(fileOrDir, fileOrDir)
await storage.presign(file, options)
await storage.exists(fileOrDir)
await storage.stat(file)

See StorageInterface for more information.

Reading From Storage

To retrieve data from object storage, use the read() function, providing the file path of the desired object as the first argument and optionally specifying an encoding like ascii or binary in the second argument.

let data = await storage.read('file')
let data = await storage.read('file', { encoding: 'ascii' })
let data = await storage.read('image.bin', { encoding: 'binary' })

See the ReadFunction for more information.

Writing To Storage

To write data to object storage, use the write() function, specifying the desired file path as the first argument, the content to write (which can be a string or a Buffer) as the second, and optionally an encoding option like utf8 or binary in the third argument.

await storage.write('file', 'hello')
await storage.write('file', 'Ω', { encoding: 'utf8' })
await storage.write('file', Buffer.alloc(4), { encoding: 'binary' })

See the WriteFunction for more information.

Removing From Storage

To delete objects from storage, use the remove() function, providing the file or directory path to be deleted as the first argument. For directory removal, include the option { recursive: true } in the second argument. Additionally, add the concurrency option, such as { concurrency: 10 }, to control the parallelism.

await storage.remove('file')
await storage.remove('dir/', { recursive: true })
await storage.remove('dir/', { recursive: true, concurrency: 10 })

See the RemoveFunction for more information.

Listing Objects

To retrieve a list of objects within a specified path, use the list() function. The function requires the directory path as the first argument. Optional configuration can be provided as the second argument in an options object. It can include parameters such as recursive to enable listing of subdirectories, absolute to return absolute paths, and concurrency to adjust the parallelism of the listing operation.

let data = await storage.list('/')
let data = await storage.list('/', { recursive: true })
let data = await storage.list('/', { recursive: true, absolute: true })
let data = await storage.list('/', { recursive: true, absolute: true, concurrency: 10 })

See the ListFunction for more information.

Copying Objects

To copy objects within object storage, use the copy() function. This function takes two arguments: the source path to copy from and the destination path to copy to. It recursively copies files and directories from the source to the destination.

await storage.copy('file', 'file_copy')
await storage.copy('dir/', 'dir_copy/')

See the CopyFunction for more information.

Presigning URLs

To create a shareable link, use the presign() function. This generates a pre-signed URL for a specific file path. By default, these URLs are valid for a limited time, but you can customize the expiration using the expiresIn option to control how long the URL remains active. This is particularly useful when you need to grant temporary access to users for file operations without requiring them to authenticate through your application.

let data = await storage.presign('file')
let data = await storage.presign('file', { expiresIn: 3600 })

See the PreSignFunction for more information.

Verifying Object Existence

To determine if an object exists at a specific path within object storage, use the exists() function. This function takes the path to the object as its sole argument and returns a Promise that resolves to a boolean value, indicating the presence (true) or absence (false) of an object at the specified location.

let data = await storage.exists('file')
let data = await storage.exists('dir/')

See the ExistsFunction for more information.

Retrieving Object Metadata

To retrieve metadata for an object in object storage, use the stat() function. This function takes the path to the object and returns an object containing metadata about the object.

let data = await storage.stat('file')

See the StatFunction for more information.

Local Development

Storage can be created so that the code automatically switches between the providers depending on the environment.

import { Storage as StorageLocal } from '@brighter/storage-adapter-local'
import { Storage as StorageS3 } from '@brighter/storage-adapter-s3'

const createStorage = () => {
  if (process.env.NODE_ENV === 'local') {
    return StorageLocal({ path: '/tmp/storage' })
  } else {
    return StorageS3({ path: 'my-bucket' }, { region: 'eu-central-1' })
  }
}

const main = async () => {
  const storage = createStorage()
  await storage.read('file')
}

main().catch(console.error)

Compatibility

The library is actively tested against the following object storage providers:

• Local Provider
• AWS S3
• Cloudflare R2
• Hetzner Object Storage
• Backblaze B2

Sponsors

@brighter/storage is an MIT-licensed open source project with its ongoing development made possible by the support of the following sponsors:

• Hetzner

If you'd like to join them, please consider sponsoring the development. You can view detailed spending reports and financial transparency under bookkeeping.

Roadmap

• v2.0.0 Azure Blob Storage implementation.
• v3.0.0 Google Storage implementation.

Testing

npm run test                      # run tests
npm run test -- --provider=local  # run tests for a specific provider

Releasing

npm run release             # set version by bumping the patch number, tag the commit
npm run release -- 1.0.0    # set specific version, tag the commit