Working with storage

Every Ikomia SCALE project gets its own storage space that can be used to store inputs and outputs of your deployments. This is particularly useful for handling large file like videos.

The storage client is accessible through client.storage or as a "standalone" object:

import {StorageClient} from '@ikomia/ikclient/storage';

const storage = new StorageClient({
  url: 'https://your.scale.endpoint.url',
  token: 'your-api-token', // default to process.env.IKOMIA_TOKEN
});

Uploading files

import fs from 'fs/promises';

const data = await fs.readFile('path/to/video.mp4');
const obj = await client.storage.put(data, {path: "path/in/storage.mp4", contentType: "video/mp4"});

// Or upload as temporary (path-less) object
const obj = await client.storage.put(data, {contentType: "video/mp4"});

You can upload files as temporary objects or as regular objects with a specific path. Objects with a path will be stored permanently while temporary objects will be deleted after a certain period (~ 1 hour).

The latter are well-suited for uploading inputs that you want to process immediately without having to manage their lifecycle.

info

StorageClient.put() accepts string, ArrayBuffer, Buffer and Blob. If we can't infer the mime-type from these inputs, we will default to application/octet-stream or text/plain. You can also specify the mime-type explicitly using the contentType parameter, which we recommend you to do.

Running deployment on stored file

We provide Client.createStorageInput() and Client.createStorageInputFromUid() methods to easily work with stored files.

// Upload temporary video
const data = await fs.readFile('path/to/video.mp4');
const obj = await client.storage.put(data, {contentType: 'video/mp4'});

// Create inputs and run
const videoInput = await client.createStorageInputFromUid(obj.uid);
const videoInput2 = await client.createStorageInput('path/in/storage.mp4'); // This video was already uploaded to project's storage

const results = await client.run({inputs: [videoInput, videoInput2]});

Reading files

StorageClient.get() can be used to retrieve file metadata, while StorageClient.read() can be used to read the contents of a file:

// Get file metadata
const obj = await client.storage.get('path/in/storage.mp4');

const response = await client.storage.read(obj);

fs.writeFile(
  'path/to/local/file.mp4',
  Buffer.from(await response.arrayBuffer())
);

StorageClient.read() returns a Promises that resolves into a standard Fetch Response object for the file content on our CDN.

Handling StorageObjectIO outputs

Your deployment may produce outputs that are instances of StorageObjectIO. These outputs represent files stored in your project's storage.

When deployments returns StorageObjectIO outputs, they provide object metadata (like the one you would get from storage.get()) in the data.metadata field, then, you can use storage.read() to read the contents of these outputs, just like you would with any other file in storage:

import {StorageObjectIO} from '@ikomia/ikclient';

const results = await client.run({input: videoInput});

const videoOutput = results.getOutput(0, StorageObjectIO);

const response = await client.storage.read(videoOutput.data.metadata);
fs.writeFile(
  'path/to/local/file.mp4',
  Buffer.from(await response.arrayBuffer())
);

Managing objects

We provide some methods to manage objects in your project's storage:

// List all objects in a directory
const objects = await client.storage.list('directory');

// List by sha256
const objects = await client.storage.listBySha256('your_file_checksum');

// Delete all objects in a directory
await client.storage.delete('directory');

// Delete a specific object
await client.storage.delete('directory/file.mp4', { exact: true });

// Get obj metadata from uid
const obj = await client.storage.getByUid('your_file_uid');

// Copy an object
const copyObj = await client.storage.copy(obj, {path: 'copy_of_file.mp4'});

// Create a presigned download URL
const presignedUrl = await client.storage.getPresignedDownloadUrl(obj, { expiresIn: 3600 });  // URL valid for 1 hour