Beta documentation

This is the documentation of the Python SDK Beta (0.15.0 and earlier), which is no longer updated. For 1.0 and later, see the LucidLink Developer Portal; to upgrade, follow the migration guide.

connect

LucidLink Connect - external file management.

Provides the ability to attach existing S3 datasets to a LucidLink filespace as read-only external files (1:1 file-to-S3-object mapping).

Also supports linking publicly accessible HTTP URLs directly as external files, without the need for an S3 data store.

class lucidlink.connect.ConnectManager[source]

Bases: object

Manages LucidLink Connect data stores and external file linking.

Obtained via filespace.connect after linking to a filespace. Not intended to be constructed directly.

Example

connect = filespace.connect
connect.add_data_store("my-store", S3DataStoreConfig(
    access_key="AKIA...",
    secret_key="...",
    bucket_name="my-bucket",
    region="us-east-1",
))
connect.link_file("/data/file.csv", "my-store", "path/to/file.csv")

add_data_store(name: str, config: S3DataStoreConfig) → DataStoreInfo[source]

Add a new S3 data store.

The new data store is registered in this client’s local metadata and appears in list_data_stores() immediately. It is not propagated to LucidLink services until the change is synced: call filespace.sync_all() before list_external_files(), count_external_files(), remove_data_store(), or rekey_data_store() — those are served by LucidLink services and fail while the data store has not reached them yet.

Parameters:

name – Unique name for the data store
config – S3 data store configuration

Returns:

DataStoreInfo for the newly created data store

Raises:

RuntimeError – If data store creation fails
ValueError – If name is empty or config is invalid

add_http_link_data_store(name: str) → DataStoreInfo[source]

Add a new HTTP-link data store. Available on filespace format 3.8+

Parameters:

name – Unique name for the data store

Returns:

DataStoreInfo for the newly created data store

Raises:

RuntimeError – If data store creation fails
ValueError – If name is empty or invalid

are_data_stores_available() → bool[source]

Check if data stores can be managed on this filespace.

Verifies both that the filespace version supports external files (V9+) and that the feature has been explicitly configured.

Returns:: True if data stores are available, False otherwise

count_external_files(data_store_name: str) → int[source]

Count external files linked to a data store (no path resolution).

Much faster than iterating list_external_files() when only the count is needed, as it skips per-file path lookups in the metadata store.

Like list_external_files(), the count is served by LucidLink services: call filespace.sync_all() first so locally made changes are reflected. The same caveat about an in-progress credential rotation applies.

Parameters:

data_store_name – Name of the data store

Returns:

Number of linked files

Raises:

KeyError – If the data store is not known to LucidLink services — including a data store added on this client but not yet synced
RuntimeError – If the operation fails

get_data_store(name: str) → DataStoreInfo | None[source]

Get a data store by name.

Parameters:: name – Name of the data store
Returns:: DataStoreInfo if found (includes decrypted secret_key), None otherwise

link_file(file_path: str, data_store_name: str, object_id: str, size: int | None = None, checksum: str = '') → None[source]

Link an S3 object as an external file in the filespace.

Creates a read-only file entry that maps to an S3 object via presigned URL.

The new link is written to this client’s local metadata; it is not reflected by list_external_files() / count_external_files() — which are served by LucidLink services — until the change is synced — call filespace.sync_all() before listing.

Parameters:

file_path – Path in the filespace for the new file
data_store_name – Name of the data store containing the object
object_id – S3 object key
size – Optional object size in bytes. When provided together with checksum, skips the S3 HeadObject call (much faster for bulk ops).
checksum – Optional object ETag/checksum. When provided together with size, skips the S3 HeadObject call.

Raises:

RuntimeError – If data store not found or link fails
ValueError – If only one of size/checksum is provided

link_http_file(file_path: str, url: str, *, data_store_name: str | None = None, size: int | None = None) → None[source]

Link an HTTP URL as an external file in the filespace.

Creates a read-only file entry that maps to content at the given URL.

Parameters:

file_path – Path in the filespace for the new file
url – HTTP URL of the file content
data_store_name – Name of an http link data store Required on filespace format 3.8+, should be None otherwise
size – Optional file size in bytes. When provided, skips the access check and size validation (useful for bulk operations where size is known in advance).

Raises:

ValueError – If size is negative
RuntimeError – If the URL is inaccessible or expired, the data store is not found, or the link fails

list_data_stores() → List[DataStoreInfo][source]

List all registered data stores.

Served from this client’s local metadata: data stores added or removed on this client are visible here immediately, without sync_all().

Returns:: List of DataStoreInfo objects (secret_key will be empty)
Raises:: RuntimeError – If operation fails

list_external_files(data_store_name: str, limit: int = 100, cursor: str = '') → LinkedFilesResult[source]

List external files linked to a data store.

The listing is served by LucidLink services: the index of linked files is maintained there, not in this client’s local metadata. Locally made changes — a newly added data store, newly linked or unlinked files — are not reflected until they are synced to LucidLink services. Call filespace.sync_all() after add_data_store(), link_file(), or unlink_file() before listing.

Parameters:

data_store_name – Name of the data store
limit – Maximum number of results (default: 100)
cursor – Opaque pagination token from a previous result (default: start)

Returns:

LinkedFilesResult with the linked entries (path, entry ID, inode ID per file), pagination flag, and cursor

Raises:

KeyError – If the data store is not known to LucidLink services — including a data store added on this client but not yet synced
RuntimeError – If the operation fails

Note

While a credential rotation started by rekey_data_store() is in progress (DataStoreInfo.rekey_state is IN_PROGRESS), results — in particular paginated ones — may be incomplete or unstable, because LucidLink services migrate linked files to the new key in the background. Prefer listing after the rotation completes.

rekey_data_store(name: str, credentials: S3Credentials | None = None, *, new_access_key: str | None = None, new_secret_key: str | None = None) → None[source]

Rotate credentials for a data store.

Parameters:

name – Name of the data store
credentials – Typed credentials object
new_access_key – New S3 access key ID
new_secret_key – New S3 secret access key

Raises:

RuntimeError – If data store not found or rekey fails
ValueError – If credentials are missing or empty

Note

The rotation is asynchronous: this call initiates it, and LucidLink services then migrate the store’s linked files to the new credentials in the background. DataStoreInfo.rekey_state reports IN_PROGRESS until the migration completes; while it is in progress, list_external_files() / count_external_files() results may be incomplete or unstable. The data store must already be synced (filespace.sync_all()) before it can be rekeyed.

remove_data_store(name: str) → None[source]

Remove a data store by name.

Parameters:: name – Name of the data store to remove
Raises:: RuntimeError – If data store not found or removal fails

unlink_file(file_path: str) → None[source]

Remove an external file link from the filespace.

The removal is written to this client’s local metadata; list_external_files() / count_external_files() — which are served by LucidLink services — keep reporting the file until the change is synced — call filespace.sync_all() after unlinking.

Parameters:: file_path – Path of the external file to remove
Raises:: RuntimeError – If file not found or unlink fails

update_http_file_url(file_path: str, new_url: str, *, size: int | None = None) → None[source]

Update the URL of an existing HTTP link file.

Parameters:

file_path – Path of the HTTP link file in the filespace
new_url – New HTTP URL for the file content
size – Optional file size in bytes. When provided, skips the access check and size validation (useful for bulk operations where size is known in advance).

Raises:

ValueError – If size is negative
RuntimeError – If the file is not found or is not an HTTP link file