Beta documentation

This is the documentation of the Python SDK Beta (0.15.0 and earlier), which is no longer updated. For 1.0 and later, see the LucidLink Developer Portal; to upgrade, follow the migration guide.

fsspec

fsspec integration for LucidLink filesystem.

Provides fsspec.AbstractFileSystem implementation that allows standard Python data libraries (Pandas, Dask, PyArrow, etc.) to access LucidLink filespaces using URL protocols like lucidlink://workspace/filespace/path/to/file.csv

Features:

Standard file operations: open, read, write, ls, mkdir, rm
Native move/rename operations (much faster than copy+delete fallback)
Directory operations: mkdir, makedirs, rmdir
Sync control: automatic or manual sync to LucidLink services

class lucidlink.fsspec.AbstractFileSystem[source]

Bases: object

Dummy base class when fsspec is not available.

class lucidlink.fsspec.LucidLinkFileSystem(token: str | None = None, sandboxed: bool = True, persist_files: bool = False, root_path: str | Path | None = None, sync_mode: SyncMode = SyncMode.SYNC_ALL, **kwargs)[source]

Bases: AbstractFileSystem

fsspec filesystem implementation for LucidLink.

Enables standard Python data libraries to access LucidLink filespaces using URL protocols like: lucidlink://workspace/filespace/path/to/file.csv

Usage with Pandas:

df = pd.read_csv('lucidlink://my-workspace/production-data/data.csv',
                 storage_options={'token': 'sa_live:...'})

Usage with Dask:

df = dd.read_parquet('lucidlink://workspace/filespace/dataset/*.parquet',
                     storage_options={'token': 'sa_live:...'})

Usage with PyArrow:

table = pq.read_table('lucidlink://workspace/filespace/data.parquet',
                      filesystem=LucidLinkFileSystem(token='sa_live:...'))

Direct usage:

fs = LucidLinkFileSystem(token='sa_live:...')
with fs.open('lucidlink://workspace/filespace/file.txt', 'r') as f:
    content = f.read()

cat(path: str, start: int | None = None, end: int | None = None, **kwargs) → bytes[source]

Read entire file or byte range.

Parameters:

path – LucidLink URL
start – Start byte position (optional)
end – End byte position (optional)
**kwargs – Additional parameters

Returns:

File contents as bytes

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs) → bytes[source]: Alias for cat() for compatibility.

close() → None[source]: Close all connections and release resources.

exists(path: str, **kwargs) → bool[source]

Check if path exists.

Raises:

PermissionError – If token doesn’t have access to the workspace or filespace
FilespaceError – If the filespace link itself failed (e.g. a filespace internal error) — a failed link must not read as “missing”

get(rpath: str, lpath: str, recursive: bool = False, **kwargs) → None[source]

Download file from LucidLink to local filesystem.

Parameters:

rpath – Remote LucidLink URL
lpath – Local filesystem path
recursive – Download directory recursively
**kwargs – Additional parameters

info(path: str, **kwargs) → Dict[str, Any][source]

Get file or directory information.

Parameters:

path – LucidLink URL
**kwargs – Additional parameters

Returns:

Dictionary with file/directory metadata

isdir(path: str) → bool[source]

Check if path is a directory.

Raises:

PermissionError – If token doesn’t have access to the workspace or filespace
FilespaceError – If the filespace link itself failed (e.g. a filespace internal error) — a failed link must not read as “missing”

isfile(path: str) → bool[source]

Check if path is a file.

Raises:

PermissionError – If token doesn’t have access to the workspace or filespace
FilespaceError – If the filespace link itself failed (e.g. a filespace internal error) — a failed link must not read as “missing”

ls(path: str, detail: bool = True, **kwargs) → List[str] | List[Dict[str, Any]][source]

List directory contents.

Parameters:

path – LucidLink URL
detail – Return detailed info (True) or just names (False)
**kwargs – Additional parameters

Returns:

List of full paths (detail=False) or detailed info dicts (detail=True)

makedirs(path: str, exist_ok: bool = False) → None[source]

Create directory recursively.

Parameters:

path – LucidLink URL
exist_ok – Don’t raise error if directory exists

mkdir(path: str, create_parents: bool = True, **kwargs) → None[source]

Create a directory.

Parameters:

path – LucidLink URL
create_parents – Create parent directories if needed
**kwargs – Additional parameters

mv(path1: str, path2: str, recursive: bool = False, maxdepth: int | None = None, **kwargs) → None[source]

Move/rename a file or directory.

Uses native rename operation which is much faster than copy+delete.

Parameters:

path1 – Source LucidLink URL
path2 – Destination LucidLink URL
recursive – Ignored (move is always recursive for directories)
maxdepth – Ignored
**kwargs – Additional parameters (ignored)

Raises:

ValueError – If paths are in different filespaces

property options: LucidLinkOptions: Read-only access to filesystem configuration options.

protocol = 'lucidlink'

put(lpath: str, rpath: str, recursive: bool = False, **kwargs) → None[source]

Upload file from local filesystem to LucidLink.

Parameters:

lpath – Local filesystem path
rpath – Remote LucidLink URL
recursive – Upload directory recursively
**kwargs – Additional parameters

rename(path1: str, path2: str, **kwargs) → None[source]

Rename a file or directory (alias for mv).

Parameters:

path1 – Source LucidLink URL
path2 – Destination LucidLink URL
**kwargs – Additional parameters (passed to mv)

rm(path: str, recursive: bool = False, maxdepth: int | None = None) → None[source]

Remove file or directory.

Parameters:

path – LucidLink URL
recursive – Remove directory recursively
maxdepth – Maximum recursion depth (ignored, always full recursion)

rmdir(path: str) → None[source]

Remove an empty directory.

Parameters:: path – LucidLink URL

sync_all() → None[source]

Synchronize all pending changes to LucidLink services.

Calls sync_all() on all connected filespaces to ensure all metadata and data changes are propagated to LucidLink services.

Raises:: FilespaceInternalError – If one or more filespaces hit an internal error — their pending changes cannot be synced (they are lost), which callers must not mistake for a successful sync.