connect

LucidLink Connect - external file management.

Provides the ability to attach existing S3 datasets to a LucidLink filespace as read-only external files (1:1 file-to-S3-object mapping).

class lucidlink.connect.ConnectManager[source]

Bases: object

Manages LucidLink Connect data stores and external file linking.

Obtained via filespace.connect after linking to a filespace. Not intended to be constructed directly.

Example

connect = filespace.connect
connect.add_data_store("my-store", S3DataStoreConfig(
    access_key="AKIA...",
    secret_key="...",
    bucket_name="my-bucket",
    region="us-east-1",
))
connect.link_file("/data/file.csv", "my-store", "path/to/file.csv")
add_data_store(name: str, config: S3DataStoreConfig) → DataStoreInfo[source]

Add a new S3 data store.

Parameters:

  • name – Unique name for the data store

  • config – S3 data store configuration

Returns:

DataStoreInfo for the newly created data store

Raises:

  • RuntimeError – If data store creation fails

  • ValueError – If name is empty or config is invalid

are_data_stores_available() → bool[source]

Check if data stores can be managed on this filespace.

Verifies both that the filespace version supports external files (V9+) and that the feature has been explicitly configured.

Returns:

True if data stores are available, False otherwise

count_external_files(data_store_name: str) → int[source]

Count external files linked to a data store (no path resolution).

Much faster than iterating list_external_files() when only the count is needed, as it skips per-file path lookups in the metadata store.

Parameters:

data_store_name – Name of the data store

Returns:

Number of linked files

get_data_store(name: str) → DataStoreInfo | None[source]

Get a data store by name.

Parameters:

name – Name of the data store

Returns:

DataStoreInfo if found (includes decrypted secret_key), None otherwise

Link an S3 object as an external file in the filespace.

Creates a read-only file entry that maps to an S3 object via presigned URL.

Parameters:

  • file_path – Path in the filespace for the new file

  • data_store_name – Name of the data store containing the object

  • object_id – S3 object key

  • size – Optional object size in bytes. When provided together with checksum, skips the S3 HeadObject call (much faster for bulk ops).

  • checksum – Optional object ETag/checksum. When provided together with size, skips the S3 HeadObject call.

Raises:

  • RuntimeError – If data store not found or link fails

  • ValueError – If only one of size/checksum is provided

list_data_stores() → List[DataStoreInfo][source]

List all registered data stores.

Returns:

List of DataStoreInfo objects (secret_key will be empty)

Raises:

RuntimeError – If operation fails

list_external_files(data_store_name: str, limit: int = 100, cursor: str = '') → LinkedFilesResult[source]

List external files linked to a data store.

Parameters:

  • data_store_name – Name of the data store

  • limit – Maximum number of results (default: 100)

  • cursor – Opaque pagination token from a previous result (default: start)

Returns:

LinkedFilesResult with file paths, pagination flag, and cursor

rekey_data_store(name: str, credentials: S3Credentials | None = None, *, new_access_key: str | None = None, new_secret_key: str | None = None) → None[source]

Rotate credentials for a data store.

Parameters:

  • name – Name of the data store

  • credentials – Typed credentials object

  • new_access_key – New S3 access key ID

  • new_secret_key – New S3 secret access key

Raises:

  • RuntimeError – If data store not found or rekey fails

  • ValueError – If credentials are missing or empty

remove_data_store(name: str) → None[source]

Remove a data store by name.

Parameters:

name – Name of the data store to remove

Raises:

RuntimeError – If data store not found or removal fails

Remove an external file link from the filespace.

Parameters:

file_path – Path of the external file to remove

Raises:

RuntimeError – If file not found or unlink fails