Working with storage

Every Ikomia SCALE project gets its own storage space that can be used to store inputs and outputs of your deployments. This is particularly useful for handling large file like videos.

The storage client is accessible through Client.storage or as a "standalone" object:

from ikclient.storage.client import StorageClient
# from ikclient.storage.client import AsyncStorageClient (if you want async)

Uploading files

with open("path/to/video.mp4", "rb") as f:
    obj = client.storage.put(f.read(), "path/in/storage.mp4")

    # Or upload as temporary (path-less) object
    obj = client.storage.put(f.read(), content_type="video/mp4")

You can upload files as temporary objects or as regular objects with a specific path. Objects with a path will be stored permanently while temporary objects will be deleted after a certain period (~ 1 hour).

The latter are well-suited for uploading inputs that you want to process immediately without having to manage their lifecycle.

info

We use the path file extension to infer the mime-type of the uploaded file. If we cannot infer the mime-type, we will default to application/octet-stream or text/plain. You can also specify the mime-type explicitly using the content_type parameter, which we recommend for temporary objects.

Running deployment on stored file

We provide Client.create_storage_input() and Client.create_storage_input_from_uid() methods to easily work with stored files.

Videos inputs

The following example is for demonstration purposes. Videos are always transmitted using storage inputs. As such, if you pass a video file path directly to Client.run() or Client.create_input() it will be automatically uploaded to storage as a temporary object.

with Client(url="https://your.scale.endpoint.url") as client:
    # Upload temporary video
    with open("path/to/video.mp4", "rb") as f:
        obj = client.storage.put(f.read(), content_type="video/mp4")

    # Create inputs and run
    video_input = client.create_storage_input_from_uid(obj["uid"])
    video_input2 = client.create_storage_input("path/in/storage.mp4")  # This video was already uploaded to project's storage

    results = client.run(video_input, video_input2)

Reading files

StorageClient.get() can be used to retrieve file metadata, while StorageClient.read() can be used to read the contents of a file:

with Client(url="https://your.scale.endpoint.url") as client:
    # Get file metadata
    obj = client.storage.get("path/in/storage.mp4")

    # Retrieve file contents and save to local file
    with client.storage.read(obj, streaming=True) as response, \
         open("path/to/local/file.mp4", "wb") as f:
        for chunk in response.iter_bytes():
            f.write(chunk)

StorageClient.read() returns a context manager that provides the HTTPX response object for the file content on our CDN. You can use streaming=True to get a streaming response.

Handling StorageObjectIO outputs

Your deployment may produce outputs that are instances of StorageObjectIO. These outputs represent files stored in your project's storage.

When deployments returns StorageObjectIO outputs, they provide object metadata (like the one you would get from StorageClient.get()) in the metadata field, then, you can use StorageClient.read() to read the contents of these outputs, just like you would with any other file in storage:

from ikclient.core.io import StorageObjectIO

results = client.run(video_input)

video_output = results.get_output(assert_type=StorageObjectIO)

with client.storage.read(video_output["metadata"], streaming=True) as response, \
     open("path/to/local/file.mp4", "wb") as f:
    for chunk in response.iter_bytes():
        f.write(chunk)

Managing objects

We provide some methods to manage objects in your project's storage:

# List all objects in a directory
objects = client.storage.list('directory')

# List by sha256
objects = client.storage.list_by_sha256('your_file_checksum')

# Delete all objects in a directory
client.storage.delete('directory')

# Delete a specific object
client.storage.delete('directory/file.mp4', exact=True)

# Get obj metadata from uid
obj = client.storage.get_by_uid('your_file_uid')

# Copy an object
copy_obj = client.storage.copy(obj, 'copy_of_file.mp4')

# Create a presigned download URL
presigned_url = client.storage.get_presigned_download_url(obj, expires_in=3600)  # URL valid for 1 hour