fsspec

fsspec integration for LucidLink filesystem.

Provides fsspec.AbstractFileSystem implementation that allows standard Python data libraries (Pandas, Dask, PyArrow, etc.) to access LucidLink filespaces using URL protocols like lucidlink://workspace/filespace/path/to/file.csv

Features:

  • Standard file operations: open, read, write, ls, mkdir, rm

  • Native move/rename operations (much faster than copy+delete fallback)

  • Directory operations: mkdir, makedirs, rmdir

  • Sync control: automatic or manual sync to hub

class lucidlink.fsspec.AbstractFileSystem[source]

Bases: object

Dummy base class when fsspec is not available.

class lucidlink.fsspec.LucidLinkFileSystem(token: str | None = None, sandboxed: bool = True, persist_files: bool = False, root_path: str | Path | None = None, sync_mode: SyncMode = SyncMode.SYNC_ALL, **kwargs)[source]

Bases: AbstractFileSystem

fsspec filesystem implementation for LucidLink.

Enables standard Python data libraries to access LucidLink filespaces using URL protocols like: lucidlink://workspace/filespace/path/to/file.csv

Usage with Pandas:

df = pd.read_csv('lucidlink://my-workspace/production-data/data.csv',
                 storage_options={'token': 'sa_live:...'})

Usage with Dask:

df = dd.read_parquet('lucidlink://workspace/filespace/dataset/*.parquet',
                     storage_options={'token': 'sa_live:...'})

Usage with PyArrow:

table = pq.read_table('lucidlink://workspace/filespace/data.parquet',
                      filesystem=LucidLinkFileSystem(token='sa_live:...'))

Direct usage:

fs = LucidLinkFileSystem(token='sa_live:...')
with fs.open('lucidlink://workspace/filespace/file.txt', 'r') as f:
    content = f.read()
cat(path: str, start: int | None = None, end: int | None = None, **kwargs) → bytes[source]

Read entire file or byte range.

Parameters:

  • path – LucidLink URL

  • start – Start byte position (optional)

  • end – End byte position (optional)

  • **kwargs – Additional parameters

Returns:

File contents as bytes

cat_file(path: str, start: int | None = None, end: int | None = None, **kwargs) → bytes[source]

Alias for cat() for compatibility.

close() → None[source]

Close all connections and stop daemon.

exists(path: str, **kwargs) → bool[source]

Check if path exists.

Raises:

PermissionError – If token doesn’t have access to the workspace or filespace

get(rpath: str, lpath: str, recursive: bool = False, **kwargs) → None[source]

Download file from LucidLink to local filesystem.

Parameters:

  • rpath – Remote LucidLink URL

  • lpath – Local filesystem path

  • recursive – Download directory recursively

  • **kwargs – Additional parameters

info(path: str, **kwargs) → Dict[str, Any][source]

Get file or directory information.

Parameters:

  • path – LucidLink URL

  • **kwargs – Additional parameters

Returns:

Dictionary with file/directory metadata

isdir(path: str) → bool[source]

Check if path is a directory.

Raises:

PermissionError – If token doesn’t have access to the workspace or filespace

isfile(path: str) → bool[source]

Check if path is a file.

Raises:

PermissionError – If token doesn’t have access to the workspace or filespace

ls(path: str, detail: bool = True, **kwargs) → List[str] | List[Dict[str, Any]][source]

List directory contents.

Parameters:

  • path – LucidLink URL

  • detail – Return detailed info (True) or just names (False)

  • **kwargs – Additional parameters

Returns:

List of full paths (detail=False) or detailed info dicts (detail=True)

makedirs(path: str, exist_ok: bool = False) → None[source]

Create directory recursively.

Parameters:

  • path – LucidLink URL

  • exist_ok – Don’t raise error if directory exists

mkdir(path: str, create_parents: bool = True, **kwargs) → None[source]

Create a directory.

Parameters:

  • path – LucidLink URL

  • create_parents – Create parent directories if needed

  • **kwargs – Additional parameters

mv(path1: str, path2: str, recursive: bool = False, maxdepth: int | None = None, **kwargs) → None[source]

Move/rename a file or directory.

Uses native rename operation which is much faster than copy+delete.

Parameters:

  • path1 – Source LucidLink URL

  • path2 – Destination LucidLink URL

  • recursive – Ignored (move is always recursive for directories)

  • maxdepth – Ignored

  • **kwargs – Additional parameters (ignored)

Raises:

ValueError – If paths are in different filespaces

property options: LucidLinkOptions

Read-only access to filesystem configuration options.

protocol = 'lucidlink'
put(lpath: str, rpath: str, recursive: bool = False, **kwargs) → None[source]

Upload file from local filesystem to LucidLink.

Parameters:

  • lpath – Local filesystem path

  • rpath – Remote LucidLink URL

  • recursive – Upload directory recursively

  • **kwargs – Additional parameters

rename(path1: str, path2: str, **kwargs) → None[source]

Rename a file or directory (alias for mv).

Parameters:

  • path1 – Source LucidLink URL

  • path2 – Destination LucidLink URL

  • **kwargs – Additional parameters (passed to mv)

rm(path: str, recursive: bool = False, maxdepth: int | None = None) → None[source]

Remove file or directory.

Parameters:

  • path – LucidLink URL

  • recursive – Remove directory recursively

  • maxdepth – Maximum recursion depth (ignored, always full recursion)

rmdir(path: str) → None[source]

Remove an empty directory.

Parameters:

path – LucidLink URL

sync_all() → None[source]

Synchronize all pending changes to the hub.

Calls sync_all() on all connected filespaces to ensure all metadata and data changes are propagated to LucidHub.