Migrating from Beta (0.15) to 1.0
Summary
Installation and imports
Beta |
1.0 |
|
|---|---|---|
Python 3.10 |
Python 3.11 |
|
|
|
|
|
|
|
no |
|
|
|
— |
Client, authentication and workspaces
Beta |
1.0 |
|
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
— (raises |
|
a forked worker reused the parent’s client |
|
Filespaces and caching
Beta |
1.0 |
|
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
25 GiB data |
5 GiB data |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
— |
File system
Beta |
1.0 |
|
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
— |
|
|
|
|
|
— |
|
|
|
|
|
|
|
|
every write in |
|
|
|
|
|
|
|
|
— |
Connect
Beta |
1.0 |
|
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
format 3.8: |
|
|
format 3.8: |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
— (catch |
|
|
|
|
|
— |
fsspec
Beta |
1.0 |
|
|---|---|---|
|
|
|
|
— |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
copies into |
|
|
moves into the directory or replaces the file |
|
|
omits only a missing, non-directory or unreadable subtree |
|
|
|
|
pickled |
|
|
|
|
Exceptions
Beta |
1.0 |
|
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Installation and imports
Python 3.11 or newer required
The SDK no longer supports Python 3.10. Use Python 3.11 or newer before you install 1.0.
Beta
[project]
requires-python = ">=3.10"
dependencies = ["lucidlink==0.15.0"]
1.0
[project]
requires-python = ">=3.11"
dependencies = ["lucidlink>=1.0,<2"]
Classes imported from the lucidlink package
The modules lucidlink.client, lucidlink.credentials, lucidlink.storage, lucidlink.workspace, lucidlink.workspace_models, lucidlink.filespace, lucidlink.filespace_models, lucidlink.filesystem_models, lucidlink.daemon and lucidlink.daemon_models no longer exist. Import Client, ServiceAccountCredentials, StorageConfig, StorageMode, Workspace, Filespace, FilespaceInfo and CacheConfig from lucidlink, which already exported them in Beta. from lucidlink import * now imports only the names in lucidlink.__all__; import exceptions from lucidlink.exceptions.
Beta
from lucidlink.client import Client
from lucidlink.credentials import ServiceAccountCredentials
from lucidlink.filespace import Filespace
from lucidlink.storage import StorageConfig
from lucidlink.workspace import FilespaceInfo, Workspace
def describe(workspace: Workspace, info: FilespaceInfo, filespace: Filespace) -> str:
return f"{workspace.name}/{info.name}: {filespace.id}"
with Client(storage=StorageConfig()) as client:
client.login(ServiceAccountCredentials("sa_live:..."))
1.0
from lucidlink import (
Client,
Filespace,
FilespaceInfo,
ServiceAccountCredentials,
StorageConfig,
Workspace,
)
def describe(workspace: Workspace, info: FilespaceInfo, filespace: Filespace) -> str:
return f"{workspace.name}/{info.name}: {filespace.id}"
with Client(storage=StorageConfig()) as client:
client.login(ServiceAccountCredentials("sa_live:..."))
fsspec 2023.10.0 or newer required
The lucidlink[fsspec] extra now requires fsspec 2023.10.0 or newer. Raise any fsspec pin below that version.
Beta
[project]
dependencies = ["lucidlink[fsspec]==0.15.0", "fsspec==2023.6.0"]
1.0
[project]
dependencies = ["lucidlink[fsspec]>=1.0,<2", "fsspec>=2023.10.0"]
Type information included
The package now ships type information, so mypy and pyright check your calls into the SDK instead of skipping it. Code that still runs can report new type errors, which point at the changes on this page.
Beta
$ mypy app.py
app.py:1: error: Skipping analyzing "lucidlink": module is installed,
but missing library stubs or py.typed marker [import-untyped]
1.0
$ mypy app.py
app.py:9: error: "FsEntry" has no attribute "mtime" [attr-defined]
Internal client helpers removed
Beta exposed these undocumented helpers only because its modules were public, and they have no replacement: the native bindings module lucidlink.lucidlink_native (also reachable as lucidlink.client.lucidlink_native and lucidlink.daemon.lucidlink_native), whose FileSystem class is unrelated to the new lucidlink.FileSystem; lucidlink.const.MiB; VALID_TOKEN_PREFIXES and redact_service_account_token() from lucidlink.credentials; and Workspace.unlink_filespace(). Define constants such as MiB yourself, print the ServiceAccountCredentials object (its repr() redacts the token), and call filespace.unlink() to unlink a filespace.
Beta
from lucidlink.const import MiB
from lucidlink.credentials import redact_service_account_token
chunk_size = 8 * MiB
print("using token", redact_service_account_token(credentials.token))
1.0
MiB = 1024**2
chunk_size = 8 * MiB
print("using", credentials) # the repr redacts the token
Client, authentication and workspaces
Workspace access through client.workspace
A service account token gives access to exactly one workspace, so list_workspaces(), get_workspace() and WorkspaceInfo are removed. Use the client.workspace property; its id and name replace the WorkspaceInfo fields. Before login() it raises NotLoggedInError, where Beta raised AuthenticationError.
Beta
from lucidlink import WorkspaceInfo
client.login(credentials)
info: WorkspaceInfo = client.list_workspaces()[0]
workspace = client.get_workspace(info.id)
print(info.id, info.name)
1.0
client.login(credentials)
workspace = client.workspace
print(workspace.id, workspace.name)
Filespaces linked by ID only
link_filespace() no longer accepts a filespace name: its first argument is the filespace ID, so a name passed there is looked up as an ID and the link fails with FilespaceNotFoundError. The id keyword is renamed filespace_id, and root_path, which had no effect, is removed. Look up IDs with workspace.list_filespaces().
Beta
production = workspace.link_filespace("production-data") # by name
archive = workspace.link_filespace(id=archive_id, root_path="/")
1.0
ids_by_name = {info.name: info.id for info in workspace.list_filespaces()}
production = workspace.link_filespace(ids_by_name["production-data"])
archive = workspace.link_filespace(archive_id)
Daemon and create_daemon() removed
The deprecated Daemon class, create_daemon() and DaemonStatus are removed. Use Client: client.login(credentials) replaces start() plus authenticate(), client.close() (or leaving the with block) replaces stop(), and client.is_logged_in and client.workspace.linked_filespaces replace get_status(). Pass storage options as Client(storage=StorageConfig(...)) and cache sizes to link_filespace(cache=...); the free-form config dict has no replacement. Its fs.cache.size key had no effect in Beta: set the data cache size with CacheConfig(data_size=...).
Beta
from lucidlink import Daemon, ServiceAccountCredentials
with Daemon() as daemon: # start() on enter, stop() on exit
workspace = daemon.authenticate(ServiceAccountCredentials(token))
filespace = workspace.link_filespace(id=filespace_id)
print(daemon.get_status().is_authenticated)
print([entry.name for entry in filespace.fs.read_dir("/")])
1.0
from lucidlink import Client, ServiceAccountCredentials
with Client() as client: # close() on exit
client.login(ServiceAccountCredentials(token))
filespace = client.workspace.link_filespace(filespace_id)
print(client.is_logged_in)
print([entry.name for entry in filespace.file_system.read_dir("/")])
Storage modes renamed to EPHEMERAL and PERSISTENT
SANDBOXED is now EPHEMERAL (still the default) and PHYSICAL is now PERSISTENT, with the values "ephemeral" and "persistent". PERSISTENT always keeps its files, has no persist_on_exit, and requires root_path instead of defaulting to the current directory; EPHEMERAL rejects a root_path. StorageConfig takes keyword arguments only and is immutable, and get_root_path() and should_cleanup() are removed: read root_path (the files go to its .lucid subdirectory). PHYSICAL without persist_on_exit=True deleted its files on close: if the location did not matter, use EPHEMERAL; to keep the location, use PERSISTENT and remove <root_path>/.lucid after the client closes.
Beta
from pathlib import Path
from lucidlink import Client, StorageConfig, StorageMode
temporary = StorageConfig(StorageMode.SANDBOXED)
kept = StorageConfig(
StorageMode.PHYSICAL, persist_on_exit=True, root_path=Path("/data/lucid")
)
print(kept.mode, kept.get_root_path()) # /data/lucid/.lucid
with Client(storage=kept) as client:
...
1.0
from pathlib import Path
from lucidlink import Client, StorageConfig, StorageMode
temporary = StorageConfig(mode=StorageMode.EPHEMERAL)
kept = StorageConfig(mode=StorageMode.PERSISTENT, root_path=Path("/data/lucid"))
print(kept.mode, kept.root_path) # files go to /data/lucid/.lucid
with Client(storage=kept) as client:
...
Specific errors from Client.login()
login() used to raise AuthenticationError for every authentication or service failure. It now raises AuthenticationError only for an invalid, expired or revoked token; denied access, rate limiting and service failures raise other LucidLinkError subclasses, or a plain LucidLinkError for a service status the SDK does not classify, and logging in again with different credentials raises ClientError. The first login() also starts the client: a start-up failure raised ClientError and now raises the class for its cause. Catch LucidLinkError to keep handling every login failure.
Beta
from lucidlink.exceptions import AuthenticationError
try:
client.login(credentials)
except AuthenticationError as error: # any login failure
print("login failed:", error)
1.0
from lucidlink.exceptions import AuthenticationError, LucidLinkError
try:
client.login(credentials)
except AuthenticationError as error: # token invalid, expired or revoked
print("bad token:", error)
except LucidLinkError as error: # access denied, rate limited, unreachable, ...
print("login failed:", error)
Client.close() reports failures and is final
close() used to ignore a failed final sync or unlink. It now finishes the teardown and then raises a LucidLinkErrorGroup, for example with a FilespaceSyncError for changes that were not saved; leaving a with block normally raises it too. Call filespace.sync() before closing when the changes must be saved. A closed client cannot be reused, where Beta allowed login() again after close(): it and every object obtained through it raise ClientClosedError, so create a new Client. Workspace.stop() likewise raises a LucidLinkErrorGroup when unlinking a filespace fails; Beta ignored those errors.
Beta
client.login(credentials)
workspace = client.get_workspace(client.list_workspaces()[0].id)
filespace = workspace.link_filespace(id=filespace_id)
... # write files
client.close() # a failed final sync is ignored
1.0
from lucidlink.exceptions import FilespaceSyncError
client.login(credentials)
filespace = client.workspace.link_filespace(filespace_id)
... # write files
try:
client.close()
except* FilespaceSyncError as failed:
print("changes not saved:", failed.exceptions)
SDK objects cannot be copied
copy.copy() of a Client, Workspace, Filespace, FileSystem, FileStream, FsUtils or a Connect object now raises TypeError. In Beta it made a copy that shared the native state of the original, so closing or unlinking one broke the other; a buffered file from open() could not be copied at all. Use the object itself. copy.deepcopy() and pickling already failed in Beta; in another process, create a new Client.
Beta
import copy
report_fs = copy.copy(filespace) # shares the link with filespace
report_fs.fs.create_dir("/reports")
1.0
report_fs = filespace # copy.copy(filespace) raises TypeError
report_fs.file_system.create_dir("/reports")
No LucidLink calls in a forked process
In a process forked while a LucidLink client was running, every LucidLink call now raises ClientError, including calls on a LucidLinkFileSystem. In Beta a forked child reused the parent’s client: some calls worked and data reads could hang. On Linux, multiprocessing and ProcessPoolExecutor fork by default up to Python 3.13. Start workers with the spawn or forkserver start method and create a client in each worker, or close every client before forking.
Beta
from concurrent.futures import ProcessPoolExecutor
def count(path: str) -> int:
# On Linux the worker is forked and reuses the parent's client
return len(filespace.fs.read_dir(path))
if __name__ == "__main__":
with ProcessPoolExecutor() as pool:
print(list(pool.map(count, ["/a", "/b"])))
1.0
import multiprocessing
import os
from concurrent.futures import ProcessPoolExecutor
from lucidlink import Client, ServiceAccountCredentials
def count(path: str) -> int:
# Each worker logs in with a client of its own
with Client() as client:
client.login(ServiceAccountCredentials(token=os.environ["LUCIDLINK_TOKEN"]))
filespace = client.workspace.link_filespace("fs-uuid-12345")
return len(filespace.file_system.read_dir(path))
if __name__ == "__main__":
spawn = multiprocessing.get_context("spawn")
with ProcessPoolExecutor(mp_context=spawn) as pool:
print(list(pool.map(count, ["/a", "/b"])))
Filespaces and caching
Filespace.fs renamed to file_system
The property that returns the file system of a linked filespace is renamed from fs to file_system. Replace every filespace.fs with filespace.file_system.
Beta
filespace.fs.create_dir("/reports")
for entry in filespace.fs.read_dir("/"):
print(entry.name)
1.0
filespace.file_system.create_dir("/reports")
for entry in filespace.file_system.read_dir("/"):
print(entry.name)
Filespace.sync_all() renamed to sync()
sync_all() is renamed to sync(). It still blocks until all pending changes have reached LucidLink.
Beta
filespace.fs.create_dir("/reports")
filespace.sync_all()
1.0
filespace.file_system.create_dir("/reports")
filespace.sync()
SyncMode replaced by Filespace.sync_barrier
SyncMode and the sync_mode argument of link_filespace() are removed. Set the sync_barrier property of the linked filespace instead: SyncMode.SYNC_NONE becomes SyncBarrier.NONE, and SyncMode.SYNC_ALL becomes SyncBarrier.ON_UNLINK, still the default. The values are now "none" and "on_unlink" (Beta had "none" and "all").
Beta
from lucidlink import SyncMode
filespace = workspace.link_filespace(id="fs-uuid-12345", sync_mode=SyncMode.SYNC_NONE)
filespace.fs.create_dir("/batch")
filespace.sync_all()
filespace.unlink()
1.0
from lucidlink import SyncBarrier
filespace = workspace.link_filespace("fs-uuid-12345")
filespace.sync_barrier = SyncBarrier.NONE
filespace.file_system.create_dir("/batch")
filespace.sync()
filespace.unlink()
ClientConfig removed; cache sizes set per link
ClientConfig and the config argument of Client() are removed: pass a CacheConfig to each link_filespace() call instead. Its fields data_bytes and metadata_bytes are renamed to data_size and metadata_size (still in bytes), and it is now immutable and accepts only integers. Linking a filespace that is already linked raises FilespaceAlreadyLinkedError only when the new CacheConfig sets a size that differs from the one in use; with no CacheConfig, a None field or the same size it returns the linked filespace. To change the sizes, unlink first. A size you set stays in effect when the same client links the filespace again, even with no CacheConfig. In Beta, FilespaceAlreadyLinkedError meant that the filespace was already linked under another identifier.
Beta
from lucidlink import CacheConfig, Client, ClientConfig
cache = CacheConfig(data_bytes=2 * 1024**3, metadata_bytes=512 * 1024**2)
with Client(config=ClientConfig(cache=cache)) as client:
client.login(credentials)
workspace = client.get_workspace(client.list_workspaces()[0].id)
filespace = workspace.link_filespace(id="fs-uuid-12345")
1.0
from lucidlink import CacheConfig, Client
cache = CacheConfig(data_size=2 * 1024**3, metadata_size=512 * 1024**2)
with Client() as client:
client.login(credentials)
filespace = client.workspace.link_filespace("fs-uuid-12345", cache=cache)
Smaller default cache sizes
Silent: old code still runs but behaves differently.
When a link gives no cache sizes and none are set for the filespace, on this machine, filespace-wide or by an earlier link of the same client, the data cache is now capped at 5 GiB instead of 25 GiB and the metadata cache at 1 GiB instead of 5 GiB. If your workload needs the larger caches, pass a CacheConfig with the sizes you need.
Beta
# No cache sizes given: data cache up to 25 GiB, metadata cache up to 5 GiB
filespace = workspace.link_filespace(id="fs-uuid-12345")
1.0
from lucidlink import CacheConfig
# With no size given the caps are 5 GiB data and 1 GiB metadata
filespace = workspace.link_filespace(
"fs-uuid-12345",
cache=CacheConfig(data_size=25 * 1024**3, metadata_size=5 * 1024**3),
)
Workspace details through Filespace.workspace
Filespace no longer has workspace_id, workspace_name or full_name. Read id and name from the new workspace property, the Workspace the filespace was linked from. full_name has no replacement; use name for the filespace name.
Beta
print(f"{filespace.workspace_name} ({filespace.workspace_id}): {filespace.full_name}")
1.0
print(f"{filespace.workspace.name} ({filespace.workspace.id}): {filespace.name}")
FilespaceInfo.created replaced by created_at
FilespaceInfo.created was an ISO 8601 string. It is now created_at, a timezone-aware datetime in UTC. Drop your own parsing. .isoformat() does not give back the Beta string: 2026-10-18T22:56:03.743Z becomes 2026-10-18T22:56:03.743000+00:00, so code that compares or stores the text sees a different value.
Beta
for info in workspace.list_filespaces():
print(f"{info.name}: created {info.created}")
1.0
for info in workspace.list_filespaces():
print(f"{info.name}: created {info.created_at.isoformat()}")
Filespace.unlink() reports a failed final sync
If the final sync fails, unlink() and leaving a with block normally now raise FilespaceSyncError, where Beta ignored the failure; the same happens on the first unlink() after the SDK unlinked the filespace because of an internal error. The filespace is unlinked either way. unlink() also closes files still open through file_system, and raises a FilespaceError if that fails, so catch FilespaceError for both; if the close or the sync fails and unlinking fails too, it raises a LucidLinkErrorGroup. Later use of the filespace raises FilespaceNotLinkedError instead of RuntimeError. Call sync() before you unlink when your changes must be saved.
Beta
filespace = workspace.link_filespace(id="fs-uuid-12345")
filespace.fs.create_dir("/out")
filespace.unlink() # a failed final sync was ignored
1.0
from lucidlink.exceptions import FilespaceError
filespace = workspace.link_filespace("fs-uuid-12345")
filespace.file_system.create_dir("/out")
try:
filespace.unlink()
except FilespaceError as error:
print(f"Unlinked, but closing a file or the final sync failed: {error}")
Filespace constructor changed
The Filespace constructor was documented as internal and now takes different parameters. Do not construct a Filespace; get one from workspace.link_filespace().
Beta
from lucidlink import Filespace
filespace = Filespace(
native_fs,
native_daemon,
workspace_id="ws-uuid-1",
workspace_name="acme",
id="fs-uuid-12345",
full_name="data.acme",
on_unlink=lambda filespace_id: None,
)
1.0
filespace = workspace.link_filespace("fs-uuid-12345")
Internal filespace helpers removed
The undocumented lucidlink.filespace_cache module (FilespaceCache and the names it imported) is removed, with no replacement. To link a filespace by name, map names to IDs with list_filespaces() and call link_filespace(), which returns the live Filespace when the filespace is already linked. FilespaceCache linked without a final sync; a filespace linked this way syncs when it is unlinked, so unlink() can wait and raise FilespaceSyncError. An unknown name is a KeyError from your mapping, where FilespaceCache raised PermissionError.
Beta
from lucidlink.filespace_cache import FilespaceCache
from lucidlink.url_parser import ParsedUrl
links = FilespaceCache()
links.set_workspace(workspace)
filespace = links.get_or_connect(ParsedUrl(workspace.name, "media", "/"))
1.0
ids = {info.name: info.id for info in workspace.list_filespaces()}
# Returns the live Filespace when it is already linked
filespace = workspace.link_filespace(ids["media"])
File system
Filesystem renamed to FileSystem
The file system class is now FileSystem, imported from lucidlink only; the lucidlink.filesystem and lucidlink.filespace modules no longer exist. Update imports and type annotations; a filespace returns it as filespace.file_system.
Beta
def count_entries(fs: Filesystem, path: str) -> int:
return len(fs.read_dir(path))
print(count_entries(filespace.fs, "/projects"))
1.0
def count_entries(fs: FileSystem, path: str) -> int:
return len(fs.read_dir(path))
print(count_entries(filespace.file_system, "/projects"))
DirEntry replaced by FsEntry
read_dir() and get_entry() return FsEntry instead of DirEntry. entry_id and the deprecated file_id are now id, and file_id_external is removed with no replacement. FsEntry is immutable and its constructor takes keyword arguments only, all required (id, name, type, size, modified_at, created_at, inode_id, num_hard_links): test doubles must pass all of them and use dataclasses.replace() to change one.
Beta
entry: DirEntry = filespace.fs.get_entry("/projects/report.pdf")
print(entry.entry_id, entry.file_id_external)
# A test double
stub = DirEntry(name="report.pdf", size=3, type="file", file_id="e1",
file_id_external=1, ctime=0, mtime=0)
stub.size = 4
1.0
entry: FsEntry = filespace.file_system.get_entry("/projects/report.pdf")
print(entry.id)
# A test double
stub = FsEntry(id="e1", name="report.pdf", type=EntryType.FILE, size=3,
modified_at=0, created_at=0, inode_id="i1", num_hard_links=1)
stub = replace(stub, size=4)
is_file, is_dir and is_symlink as properties
is_file and is_dir are now read-only properties, and the is_link() method is now the is_symlink property. Drop the parentheses: calling them raises TypeError: 'bool' object is not callable.
Beta
for entry in filespace.fs.read_dir("/projects"):
if entry.is_dir():
print("dir ", entry.name)
elif entry.is_link():
print("link", entry.name)
elif entry.is_file():
print("file", entry.name)
1.0
for entry in filespace.file_system.read_dir("/projects"):
if entry.is_dir:
print("dir ", entry.name)
elif entry.is_symlink:
print("link", entry.name)
elif entry.is_file:
print("file", entry.name)
ctime and mtime renamed to created_at and modified_at
The timestamp fields are renamed and their values are unchanged: integer Unix timestamps in nanoseconds, as in Beta, whose documentation said only “Unix timestamp”. Divide by 1e9 for seconds.
Beta
entry = filespace.fs.get_entry("/projects/report.pdf")
modified = datetime.fromtimestamp(entry.mtime / 1e9, tz=timezone.utc)
created = datetime.fromtimestamp(entry.ctime / 1e9, tz=timezone.utc)
print(created, modified)
1.0
entry = filespace.file_system.get_entry("/projects/report.pdf")
modified = datetime.fromtimestamp(entry.modified_at / 1e9, tz=timezone.utc)
created = datetime.fromtimestamp(entry.created_at / 1e9, tz=timezone.utc)
print(created, modified)
EntryType for entry types; “symlink” for symbolic links
Silent: old code still runs but behaves differently.
FsEntry.type is now an EntryType (FILE, DIR, SYMLINK), a string enum that still compares equal to "file" and "dir". A symbolic link’s value changed from "link" to "symlink", so a comparison with "link" is silently always false: compare with EntryType.SYMLINK or use entry.is_symlink. str(entry) starts with symlink: for a link, and get_entry() on an entry of unknown type raises InternalError instead of reporting "unknown".
Beta
entries = filespace.fs.read_dir("/projects")
links = [entry.name for entry in entries if entry.type == "link"]
print(links)
1.0
entries = filespace.file_system.read_dir("/projects")
links = [entry.name for entry in entries if entry.type == EntryType.SYMLINK]
print(links)
read_file(), write_file() and list_dir() moved or removed
The whole-file helpers moved to FsUtils, a wrapper around a FileSystem: read_file() is now read_bytes() and write_file() is now write_bytes(). list_dir() is removed; take the names from read_dir(). write_bytes() accepts any bytes-like object, raises IsADirectoryError for a directory, and raises TypeError for a str, which Beta wrote as garbage bytes: encode text first.
Beta
filespace.fs.write_file("/notes/hello.txt", b"hello")
data = filespace.fs.read_file("/notes/hello.txt")
names = filespace.fs.list_dir("/notes")
1.0
utils = FsUtils(filespace.file_system)
utils.write_bytes("/notes/hello.txt", b"hello")
data = utils.read_bytes("/notes/hello.txt")
names = [entry.name for entry in filespace.file_system.read_dir("/notes")]
dir_exists() and file_exists() removed
Call get_entry(), catch FileNotFoundError for a missing path and check is_dir or is_file, or attempt the operation and catch its error. get_entry() describes a symbolic link itself and does not follow it.
Beta
if not filespace.fs.dir_exists("/exports"):
filespace.fs.create_dir("/exports")
if filespace.fs.file_exists("/exports/out.csv"):
filespace.fs.delete("/exports/out.csv")
1.0
fs = filespace.file_system
try:
exports_is_dir = fs.get_entry("/exports").is_dir
except FileNotFoundError:
exports_is_dir = False
if not exports_is_dir:
fs.create_dir("/exports")
with suppress(FileNotFoundError):
fs.delete("/exports/out.csv")
delete_dir(recursive=True) replaced by FsUtils.delete_tree()
delete_dir() deletes only an empty directory and no longer takes recursive. To delete a directory with its contents, call FsUtils(fs).delete_tree(path); it deletes symbolic links below the directory, never their targets.
Beta
filespace.fs.delete_dir("/tmp/build", recursive=True)
filespace.fs.delete_dir("/tmp/empty")
1.0
FsUtils(filespace.file_system).delete_tree("/tmp/build")
filespace.file_system.delete_dir("/tmp/empty")
get_size() and get_statistics() replaced by statistics()
Both methods are replaced by statistics(), which returns a FileSystemStatistics; FilespaceSize and FilespaceStatistics are removed. entries / entries_size become entries_count (they always held a count, although Beta documented them as bytes), storage becomes storage_size, and the external_files_* fields keep their names. data / data_size and the file_count, directory_count and symlink_count counters have no replacement. The new class is immutable, takes keyword arguments only and has no custom str().
Beta
size = filespace.fs.get_size()
print(f"{size.entries} entries, {size.storage} bytes stored")
stats = filespace.fs.get_statistics()
print(stats.file_count, stats.directory_count, stats.symlink_count)
print(stats.external_files_count, stats.external_files_size)
1.0
stats = filespace.file_system.statistics()
print(f"{stats.entries_count} entries, {stats.storage_size} bytes stored")
print(stats.external_files_count, stats.external_files_size)
LucidFileStream renamed to FileStream
The raw stream that open(..., buffering=0) returns is now FileStream, imported from lucidlink; the lucidlink.stream and lucidlink.filesystem paths no longer exist. Open files with fs.open() rather than constructing the class. With a literal binary mode and buffering=0, open() is typed to return FileStream, so no isinstance check is needed.
Beta
raw = filespace.fs.open("/data/blob.bin", "rb", buffering=0)
assert isinstance(raw, LucidFileStream)
with raw:
header = raw.read(16)
1.0
raw: FileStream = filespace.file_system.open("/data/blob.bin", "rb", buffering=0)
with raw:
header = raw.read(16)
open_buffered() and open_text() removed
These module-level helpers, which needed the file system’s internal native object, are removed. fs.open() covers both: pass buffering= for the buffer size, and a text mode with encoding=, errors= and newline= for text.
Beta
native = filespace.fs._native
with open_text(native, "/logs/app.log", "r", encoding="latin-1") as f:
lines = f.readlines()
with open_buffered(native, "/data/big.bin", "rb", buffer_size=1 << 20) as b:
chunk = b.read(4096)
1.0
fs = filespace.file_system
with fs.open("/logs/app.log", "r", encoding="latin-1") as f:
lines = f.readlines()
with fs.open("/data/big.bin", "rb", buffering=1 << 20) as b:
chunk = b.read(4096)
No name attribute on open files
Objects returned by open() no longer have a name: the raw FileStream dropped it, so f.name on the buffered and text wrappers raises AttributeError too. Keep the path you opened. Libraries such as pyarrow treat a file object with a name as a local path, which is why it was removed; code that tests hasattr(f, "name") now takes the other branch.
Beta
with filespace.fs.open("/reports/q3.csv", "rb") as f:
print("reading", f.name)
data = f.read()
1.0
path = "/reports/q3.csv"
with filespace.file_system.open(path, "rb") as f:
print("reading", path)
data = f.read()
FileHandle, open_legacy() and create() removed
The legacy handle API is removed: open_legacy() with its FileHandle class (offset-based read and write), and create(), which returned a bare handle ID. Use open(), and seek() instead of the offset arguments. open(path, "wb") creates a file but, unlike create(), truncates an existing one instead of failing; there is no exclusive-create mode.
Beta
with filespace.fs.open_legacy("/logs/today.log", "r") as fh:
tail = fh.read(1024, offset=4096)
handle_id = filespace.fs.create("/logs/tomorrow.log")
1.0
with filespace.file_system.open("/logs/today.log", "rb") as f:
f.seek(4096)
tail = f.read(1024)
with filespace.file_system.open("/logs/tomorrow.log", "wb"):
pass
File locking removed
open() no longer takes lock_type (passing it raises TypeError), and lock_byte_range(), unlock_byte_range(), unlock_all_byte_ranges(), LOCK_NONE, LOCK_SHARED, LOCK_EXCLUSIVE, VALID_LOCK_TYPES and validate_lock_type are removed with no replacement: coordinate concurrent writers in your application. 1.0 neither takes nor checks locks, so a lock that another client holds does not block it.
Beta
with filespace.fs.open("/db/app.sqlite", "r+b", lock_type="exclusive") as f:
page = f.read(4096)
1.0
# No SDK lock: coordinate writers of this file in the application
with filespace.file_system.open("/db/app.sqlite", "r+b") as f:
page = f.read(4096)
ELOOP when opening or truncating a symbolic link
open() in every mode, truncate(), and FsUtils.read_bytes() and write_bytes() never follow a symbolic link given as the path: they raise FsError with errno.ELOOP. Beta opened or resized the link entry itself, so a read did not return the target’s content and a write or truncate changed the link, not its target. Resolve the link with read_symlink() and use the target path; links among the directories of a path are still followed.
Beta
filespace.fs.create_symlink("/data/current.csv", "/data/latest.csv")
with filespace.fs.open("/data/latest.csv", "rb") as f:
data = f.read() # reads the link entry itself, not current.csv
1.0
fs = filespace.file_system
fs.create_symlink("/data/current.csv", "/data/latest.csv")
target = fs.read_symlink("/data/latest.csv")
with fs.open(target, "rb") as f:
data = f.read()
Errors instead of empty or wrong results from read_dir(), get_entry() and truncate()
Three calls that quietly succeeded in Beta now raise. read_dir() on a file or a symbolic link returned [], so listing through a link to a directory gave an empty result; it now raises FsNotADirectoryError. To list through a link, list the target from read_symlink(). get_entry() with a trailing slash after a file name returned the file and now raises FsNotADirectoryError. truncate() and write_file() on a directory changed the size of the directory entry itself; truncate() and FsUtils.write_bytes() now raise FsIsADirectoryError. fsspec ls() changed the other way: see “Paths without the protocol from ls() and info()”.
Beta
fs = filespace.fs
# /projects/current is a link to a directory: Beta returned []
names = [e.name for e in fs.read_dir("/projects/current")]
entry = fs.get_entry("/reports/q3.csv/") # trailing slash: Beta returned the file
1.0
fs = filespace.file_system
path = "/projects/current"
if fs.get_entry(path).is_symlink:
path = fs.read_symlink(path) # list the link's target (an absolute path here)
names = [e.name for e in fs.read_dir(path)]
entry = fs.get_entry("/reports/q3.csv")
Append mode always writes at the end
Silent: old code still runs but behaves differently.
In the append modes every write goes to the current end of the file, even after seek(), as with the built-in open(). Beta wrote at the position sought to, overwriting existing data. To write at a position, open the file with "r+b".
Beta
with filespace.fs.open("/logs/app.log", "ab") as f:
f.write(b"new entry\n")
f.seek(0)
f.write(b"HEADER v2\n") # overwrites the start of the file
1.0
fs = filespace.file_system
with fs.open("/logs/app.log", "ab") as f:
f.write(b"new entry\n")
with fs.open("/logs/app.log", "r+b") as f:
f.write(b"HEADER v2\n")
Errors from closing a file
Closing a file writes its pending data. Beta ignored a failed native close; close(), and so leaving a with block, now raises the error, and the file is closed anyway. When writing the buffered data failed, Beta raised too, but a builtin such as KeyError (for example after the path was deleted); that is now InternalError, which is not a KeyError. Catch LucidLinkError where a lost write must be noticed.
Beta
payload = b"..." * 1000
with filespace.fs.open("/exports/out.bin", "wb") as f:
f.write(payload)
print("saved")
1.0
payload = b"..." * 1000
try:
with filespace.file_system.open("/exports/out.bin", "wb") as f:
f.write(payload)
except LucidLinkError as e:
print("not saved:", e)
else:
print("saved")
ValueError for a negative seek position
In binary mode, seek() to a negative absolute position now raises InvalidArgumentError (a ValueError) instead of moving to position 0, as Python’s own files do; a relative seek before the start still stops at 0. On a stream opened with buffering=0, readable(), seekable(), tell() and the other methods raise ValueError once the stream is closed.
Beta
with filespace.fs.open("/logs/app.log", "rb") as f:
size = f.seek(0, 2)
f.seek(size - 4096)
tail = f.read()
1.0
with filespace.file_system.open("/logs/app.log", "rb") as f:
size = f.seek(0, 2)
f.seek(max(0, size - 4096))
tail = f.read()
Internal file system helpers removed
Beta exposed undocumented helpers through its public modules lucidlink.file_modes, lucidlink.filesystem and lucidlink.stream: parse_mode, ParsedMode, is_text_mode, ensure_binary_mode, get_buffered_wrapper_type, normalize_mode_for_cpp, MiB and READ_CHUNK_SIZE. They have no replacement; an open file’s readable(), writable() and mode describe how it was opened.
Beta
mode = parse_mode("r+b")
print(mode.readable, mode.writable)
with filespace.fs.open("/data/blob.bin", "r+b") as f:
head = f.read(16)
1.0
with filespace.file_system.open("/data/blob.bin", "r+b") as f:
print(f.readable(), f.writable())
head = f.read(16)
Connect
Connect classes moved to lucidlink.connect
The lucidlink package, lucidlink.connect_models and lucidlink.filespace no longer export the Connect classes. Import ConnectManager, DataStoreInfo, DataStoreKind, DataStoreRekeyState, S3Credentials and S3DataStoreConfig from lucidlink.connect, which also holds the data store and external file classes that are new in GA.
Beta
from lucidlink import ConnectManager, DataStoreRekeyState, S3DataStoreConfig
connect: ConnectManager = filespace.connect
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
waiting = DataStoreRekeyState.IN_PROGRESS
1.0
from lucidlink.connect import ConnectManager, DataStoreRekeyState, S3DataStoreConfig
connect: ConnectManager = filespace.connect
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
waiting = DataStoreRekeyState.IN_PROGRESS
Adding data stores
Add data stores through connect.data_stores; the config type selects the kind: S3DataStoreConfig for S3, HttpLinkDataStoreConfig() for HTTP links (filespace format 3.8 or newer; on format 3.6 it raises UnsupportedOperationError). add() returns a handle (S3DataStore or HttpLinkDataStore) instead of a DataStoreInfo: call info() on it to read the details.
Beta
from lucidlink import S3DataStoreConfig
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
info = filespace.connect.add_data_store("my-store", config)
print(info.name, info.bucket_name)
links = filespace.connect.add_http_link_data_store("my-links")
filespace.sync_all()
1.0
from lucidlink.connect import HttpLinkDataStoreConfig, S3DataStoreConfig
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
store = filespace.connect.data_stores.add("my-store", config)
print(store.name, store.info().bucket_name)
links = filespace.connect.data_stores.add("my-links", HttpLinkDataStoreConfig())
filespace.sync()
Getting, listing and removing data stores
Use get(), list() and remove() on connect.data_stores. get() raises DataStoreNotFoundError (a KeyError) for an unknown name instead of returning None; pass DataStoreKind.S3 or DataStoreKind.HTTP_LINK as kind to get a handle of that type. get() and list() return handles: call info() on one to read its details.
Beta
connect = filespace.connect
info = connect.get_data_store("my-store")
if info is None:
print("my-store does not exist")
else:
print(info.bucket_name)
for store in connect.list_data_stores():
print(store.name, store.kind)
connect.remove_data_store("old-store")
1.0
from lucidlink.connect import DataStoreKind
from lucidlink.exceptions import DataStoreNotFoundError
data_stores = filespace.connect.data_stores
try:
store = data_stores.get("my-store", DataStoreKind.S3)
except DataStoreNotFoundError:
print("my-store does not exist")
else:
print(store.info().bucket_name)
for handle in data_stores.list():
print(handle.name, handle.kind)
data_stores.remove("old-store")
DataStoreInfo split into DataStoreInfo and S3DataStoreInfo
DataStoreInfo now has only name and kind. info() on an S3 data store returns the subclass S3DataStoreInfo with bucket_name, region, endpoint, access_key, url_expiration_minutes, use_virtual_addressing and rekey_state. secret_key and key_id are removed, so the secret key can no longer be read back. Both classes are immutable, take keyword arguments only, require kind, and have no from_dict().
Beta
from lucidlink import DataStoreInfo, DataStoreKind, DataStoreRekeyState
info = filespace.connect.get_data_store("my-store")
assert info is not None
if info.kind == DataStoreKind.S3:
print(info.bucket_name, info.region, info.key_id)
secret = info.secret_key
busy = info.rekey_state == DataStoreRekeyState.IN_PROGRESS
placeholder = DataStoreInfo("my-links", kind=DataStoreKind.HTTP_LINK)
1.0
from lucidlink.connect import (
DataStoreInfo,
DataStoreKind,
DataStoreRekeyState,
S3DataStoreInfo,
)
info = filespace.connect.data_stores.get("my-store").info()
if isinstance(info, S3DataStoreInfo):
print(info.bucket_name, info.region)
busy = info.rekey_state == DataStoreRekeyState.IN_PROGRESS
placeholder = DataStoreInfo(name="my-links", kind=DataStoreKind.HTTP_LINK)
S3DataStoreConfig keyword-only and immutable
Pass every S3DataStoreConfig field by keyword. The config is now immutable: assigning a field raises FrozenInstanceError, so use dataclasses.replace() to get a changed copy.
Beta
from lucidlink import S3DataStoreConfig
config = S3DataStoreConfig("AKIA...", "secret", "my-bucket", "us-east-1")
config.url_expiration_minutes = 4320
filespace.connect.add_data_store("my-store", config)
1.0
import dataclasses
from lucidlink.connect import S3DataStoreConfig
config = S3DataStoreConfig(
access_key="AKIA...",
secret_key="secret",
bucket_name="my-bucket",
region="us-east-1",
)
config = dataclasses.replace(config, url_expiration_minutes=4320)
filespace.connect.data_stores.add("my-store", config)
Linking S3 objects
Link S3 objects through the S3 data store handle: the store name argument is removed, and size and checksum must be passed by keyword. link() returns an S3ExternalFile handle for the new file instead of None.
Beta
connect = filespace.connect
connect.link_file("/data/file.csv", "my-store", "path/to/file.csv")
connect.link_file(
"/data/big.bin",
"my-store",
"path/to/big.bin",
1048576,
"9b2cf535f27731c974343645a3985328",
)
1.0
from lucidlink.connect import DataStoreKind
store = filespace.connect.data_stores.get("my-store", DataStoreKind.S3)
store.link("/data/file.csv", "path/to/file.csv")
big = store.link(
"/data/big.bin",
"path/to/big.bin",
# exact object size and ETag (MD5, no quotes): a wrong value breaks reads
size=1048576,
checksum="9b2cf535f27731c974343645a3985328",
)
print(big.entry_id, big.get_path())
Linking HTTP URLs
On filespace format 3.8 or newer, link URLs through the HTTP-link data store handle; its link() returns an HttpLinkExternalFile handle. Format 3.6 has no HTTP-link data stores: call link() on HttpConnect_3_6(filespace) from lucidlink.connect, which returns None. The HttpConnect_3_6 constructor raises UnsupportedOperationError on format 3.8 or newer.
Beta
filespace_3_8.connect.link_http_file(
"/web/page.html", "https://example.com/page.html", data_store_name="my-links"
)
filespace_3_6.connect.link_http_file("/web/old.html", "https://example.com/old.html")
1.0
from lucidlink.connect import DataStoreKind, HttpConnect_3_6
links = filespace_3_8.connect.data_stores.get("my-links", DataStoreKind.HTTP_LINK)
page = links.link("/web/page.html", "https://example.com/page.html")
HttpConnect_3_6(filespace_3_6).link("/web/old.html", "https://example.com/old.html")
Changing the URL of an HTTP link file
On format 3.8 or newer, call update_url() on the file’s handle, from get_file(path), iter_files() or link() of the HTTP-link data store; the handle keeps working after the file is renamed or moved. On format 3.6, call HttpConnect_3_6(filespace).update_url(path, url). A path that is not an HTTP link file now raises InvalidArgumentError (a ValueError) instead of an OSError, and a symbolic link at the path is no longer followed.
Beta
filespace_3_8.connect.update_http_file_url(
"/web/page.html", "https://cdn.example.com/page.html"
)
filespace_3_6.connect.update_http_file_url(
"/web/old.html", "https://cdn.example.com/old.html"
)
1.0
from lucidlink.connect import DataStoreKind, HttpConnect_3_6
links = filespace_3_8.connect.data_stores.get("my-links", DataStoreKind.HTTP_LINK)
links.get_file("/web/page.html").update_url("https://cdn.example.com/page.html")
HttpConnect_3_6(filespace_3_6).update_url(
"/web/old.html", "https://cdn.example.com/old.html"
)
Unlinking an external file
unlink_file() is removed. Delete the external file with filespace.file_system.delete(path), which has the same effect.
Beta
filespace.connect.unlink_file("/data/file.csv")
filespace.sync_all()
1.0
filespace.file_system.delete("/data/file.csv")
filespace.sync()
Listing and counting linked files
iter_files() on the data store handle iterates over all linked files and fetches the pages itself, so limit, cursor and has_more are gone. It yields ExternalFile handles instead of LinkedFile records: read the path with get_path() (one call per file); entry_id and inode_id are unchanged. LinkedFile and LinkedFilesResult, with their deprecated file_paths and file_ids, are removed, and count_external_files(name) becomes store.count().
Beta
connect = filespace.connect
filespace.sync_all()
cursor = ""
while True:
page = connect.list_external_files("my-store", limit=500, cursor=cursor)
for linked in page.entries:
print(linked.path, linked.entry_id)
if not page.has_more:
break
cursor = page.cursor
total = connect.count_external_files("my-store")
1.0
store = filespace.connect.data_stores.get("my-store")
filespace.sync()
for linked in store.iter_files():
print(linked.get_path(), linked.entry_id)
total = store.count()
Rotating S3 credentials
Rekey through the S3 data store handle with an S3Credentials. The new_access_key and new_secret_key keywords and the DataStoreCredentials alias are removed, and S3Credentials is immutable, takes keyword arguments only and has no kind field. Poll store.info().rekey_state to follow the rotation.
Beta
from lucidlink import DataStoreCredentials
connect = filespace.connect
connect.rekey_data_store("my-store", DataStoreCredentials("AKIA-NEW", "new-secret"))
connect.rekey_data_store(
"my-store", new_access_key="AKIA-NEW", new_secret_key="new-secret"
)
1.0
from lucidlink.connect import DataStoreKind, S3Credentials
store = filespace.connect.data_stores.get("my-store", DataStoreKind.S3)
store.rekey(S3Credentials(access_key="AKIA-NEW", secret_key="new-secret"))
New DataStoreKind values
Silent: old code still runs but behaves differently.
The string values of DataStoreKind changed, so a comparison with the old strings is silently False, and DataStoreKind("S3DataStore") on a saved value raises ValueError. Compare with the members (kind is DataStoreKind.S3). DataStoreKind and DataStoreRekeyState are now StrEnums, so str() and f-strings give the value ("s3", "in_progress") instead of "DataStoreKind.S3".
Beta
info = filespace.connect.get_data_store("my-store")
assert info is not None
if info.kind == "S3DataStore":
print("S3 data store")
saved = {"name": info.name, "kind": info.kind.value}
1.0
from lucidlink.connect import DataStoreKind
store = filespace.connect.data_stores.get("my-store")
if store.kind is DataStoreKind.S3:
print("S3 data store")
saved = {"name": store.name, "kind": store.kind.value}
are_data_stores_available() removed
There is no separate availability check: call the operation and catch UnsupportedOperationError, which every Connect call raises when the filespace format is older than 3.6 or Connect is not configured. Beta raised RuntimeError there, and UnsupportedOperationError is not a RuntimeError.
Beta
connect = filespace.connect
if connect.are_data_stores_available():
names = [info.name for info in connect.list_data_stores()]
else:
names = []
1.0
from lucidlink.exceptions import UnsupportedOperationError
try:
names = [store.name for store in filespace.connect.data_stores.list()]
except UnsupportedOperationError:
names = []
LucidLink exception classes for Connect errors
Silent: old code still runs but behaves differently.
Connect failures now raise classes from lucidlink.exceptions, and many no longer match the builtin Beta raised, so an existing except clause silently stops catching them. Instead of RuntimeError, GA raises AccessDeniedError when the service account is not a filespace admin, UnsupportedOperationError when Connect is not available, DataStoreInUseError when the store still has linked files, DataStoreCredentialsError when rekey credentials are rejected, ConnectError when another client holds the data store lock, ExternalStoreError (an OSError) when a linked S3 object fails its checksum, and InvalidArgumentError (a ValueError) for a url_expiration_minutes outside 4320 to 10080 or an endpoint without http:// or https://. A duplicate name raises DataStoreExistsError instead of FileExistsError, and the data store limit ConnectError instead of ValueError. Reading a linked file whose S3 object is missing or shorter than its linked size raised ConnectionError; it now raises ExternalObjectNotFoundError or ExternalStoreError, which are OSError but not ConnectionError. Any other failure of the HTTP store client also raised ConnectionError; now only a 429 or 5xx answer stays a ConnectionError (ExternalStoreUnavailableError), and a 4xx answer raises another ConnectError subclass, so except ConnectionError misses it. Catch the new classes, or ConnectError for any data store failure.
Beta
from lucidlink import S3DataStoreConfig
connect = filespace.connect
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
try:
connect.add_data_store("my-store", config)
except FileExistsError:
print("my-store already exists")
try:
connect.remove_data_store("old-store")
except RuntimeError as e:
print("cannot remove old-store:", e)
1.0
from lucidlink.connect import S3DataStoreConfig
from lucidlink.exceptions import DataStoreExistsError, DataStoreInUseError
data_stores = filespace.connect.data_stores
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
try:
data_stores.add("my-store", config)
except DataStoreExistsError:
print("my-store already exists")
try:
data_stores.remove("old-store")
except DataStoreInUseError as e:
print("cannot remove old-store:", e)
Internal Connect helpers removed
Beta’s public modules exposed undocumented helpers: the Connect validators and normalizers, redacted_dataclass_repr and the ConnectManager constructor. They have no replacement: connect.data_stores.add() and the link calls validate their arguments and raise InvalidArgumentError (a ValueError), and ConnectManager comes from filespace.connect.
Beta
from lucidlink.connect import validate_data_store_name
name = "my-links"
validate_data_store_name(name)
filespace.connect.add_http_link_data_store(name)
1.0
from lucidlink.connect import HttpLinkDataStoreConfig
from lucidlink.exceptions import InvalidArgumentError
name = "my-links"
try:
filespace.connect.data_stores.add(name, HttpLinkDataStoreConfig())
except InvalidArgumentError as e:
print("invalid data store name:", e)
fsspec
LucidLinkFileSystem moved to lucidlink.fsspec
The lucidlink package no longer exports LucidLinkFileSystem. Import it from lucidlink.fsspec; that import raises ImportError when fsspec is not installed, so use it to detect fsspec. lucidlink.fsspec.FSSPEC_AVAILABLE is removed. lucidlink.fsspec no longer re-exports Client or ServiceAccountCredentials: import them from lucidlink.
Beta
import lucidlink
from lucidlink.fsspec import FSSPEC_AVAILABLE, Client, ServiceAccountCredentials
if not FSSPEC_AVAILABLE:
raise SystemExit('fsspec is missing: pip install "lucidlink[fsspec]"')
fs = lucidlink.LucidLinkFileSystem(token="sa_live:...")
client = Client()
client.login(ServiceAccountCredentials(token="sa_live:..."))
1.0
from lucidlink import Client, ServiceAccountCredentials
try:
from lucidlink.fsspec import LucidLinkFileSystem
except ImportError:
raise SystemExit('fsspec is missing: pip install "lucidlink[fsspec]"')
fs = LucidLinkFileSystem(token="sa_live:...")
client = Client()
client.login(ServiceAccountCredentials(token="sa_live:..."))
Storage options and the options property removed
LucidLinkFileSystem always uses ephemeral local storage, removed on close(). Passing sandboxed or root_path raises InvalidArgumentError, persist_files and other unknown options are ignored with a LucidLinkWarning (a UserWarning), and the options property is removed. Remove these arguments.
Beta
from lucidlink.fsspec import LucidLinkFileSystem
fs = LucidLinkFileSystem(
token="sa_live:...",
sandboxed=False,
persist_files=True,
root_path="/var/cache/lucid",
)
print(fs.options.root_path)
1.0
from lucidlink.fsspec import LucidLinkFileSystem
# Always ephemeral storage, removed on close();
# there is no persistent or custom location
fs = LucidLinkFileSystem(token="sa_live:...")
sync_mode option replaced by sync_barrier
The sync_mode option is now sync_barrier. It takes "on_unlink" (the default: close() waits for pending changes), "none", or a SyncBarrier. SyncMode is no longer importable from lucidlink.fsspec. A leftover sync_mode, for example in storage_options, is ignored with a UserWarning, and close() then waits for the upload.
Beta
from lucidlink.fsspec import LucidLinkFileSystem, SyncMode
fs = LucidLinkFileSystem(token="sa_live:...", sync_mode=SyncMode.SYNC_NONE)
with fs.open("lucidlink://workspace/filespace/out/part-0.csv", "wb") as f:
f.write(b"id,value\n")
fs.sync_all()
fs.close()
1.0
from lucidlink.fsspec import LucidLinkFileSystem
fs = LucidLinkFileSystem(token="sa_live:...", sync_barrier="none")
with fs.open("lucidlink://workspace/filespace/out/part-0.csv", "wb") as f:
f.write(b"id,value\n")
fs.sync_all()
fs.close()
Paths without the protocol from ls() and info()
Silent: old code still runs but behaves differently.
ls() returned names that started with lucidlink:// when you passed a URL, and info() returned only the base name. Both now return the full path without the protocol, as find(), glob() and walk() already did and as other fsspec file systems do. These paths still work with this file system; use fs.unstrip_protocol(name) to get a URL. ls() on a file or link now returns that one entry, where Beta returned [], so code that took an empty result to mean “not a directory” behaves differently.
Beta
url = "lucidlink://workspace/filespace/reports"
for name in fs.ls(url, detail=False):
# "lucidlink://workspace/filespace/reports/q3.csv"
with fs.open(name, "rb") as f:
print(name, len(f.read()))
print(fs.info(f"{url}/q3.csv")["name"]) # "q3.csv"
1.0
url = "lucidlink://workspace/filespace/reports"
for name in fs.ls(url, detail=False):
# "workspace/filespace/reports/q3.csv": no protocol,
# still a valid path for this file system
with fs.open(name, "rb") as f:
print(fs.unstrip_protocol(name), len(f.read()))
print(fs.info(f"{url}/q3.csv")["name"]) # "workspace/filespace/reports/q3.csv"
mtime and created in seconds
Silent: old code still runs but behaves differently.
The mtime and created values from ls() and info() were integer nanoseconds since the epoch, with 0 for a filespace root. They are now float seconds, as fsspec expects, and None when unknown: remove any division by 1e9. modified() and created() now return a UTC datetime.
Beta
from datetime import datetime, timezone
info = fs.info("lucidlink://workspace/filespace/reports/q3.csv")
# Nanoseconds since the epoch (an int)
modified = datetime.fromtimestamp(info["mtime"] / 1e9, tz=timezone.utc)
print(modified.isoformat())
1.0
from datetime import datetime, timezone
info = fs.info("lucidlink://workspace/filespace/reports/q3.csv")
# Seconds since the epoch (a float), None when unknown
if info["mtime"] is not None:
modified = datetime.fromtimestamp(info["mtime"], tz=timezone.utc)
print(modified.isoformat())
# Or: fs.modified(url) returns a UTC datetime
Symbolic links reported as “link”
Silent: old code still runs but behaves differently.
ls() and info() reported a symbolic link as "file". They now report "link" and add an islink key. A link at the end of a path is never followed, so exists(), isfile() and isdir() return False for it; use lexists() to test whether a link exists. open() and cat_file() on a link now raise FsError with errno.ELOOP, where Beta returned b'', so a loop that opens each path from find(), ls() or walk() fails on the first link.
Beta
url = "lucidlink://workspace/filespace/current" # a symbolic link
info = fs.info(url)
print(info["type"]) # "file"
print(fs.exists(url), fs.isfile(url)) # True True
1.0
url = "lucidlink://workspace/filespace/current" # a symbolic link
info = fs.info(url)
print(info["type"], info["islink"]) # "link" True
print(fs.exists(url), fs.isfile(url)) # False False
print(fs.lexists(url)) # True
FileExistsError from makedirs() for an existing directory
Without exist_ok=True, makedirs() returned silently when the directory already existed. It now raises FsExistsError (a FileExistsError), as os.makedirs() does: pass exist_ok=True. With exist_ok=True, a file at the path, or a create that fails, now raises instead of being ignored.
Beta
out = "lucidlink://workspace/filespace/out"
fs.makedirs(out) # also fine when out/ already exists
with fs.open(f"{out}/part-0.csv", "wb") as f:
f.write(b"id,value\n")
1.0
out = "lucidlink://workspace/filespace/out"
fs.makedirs(out, exist_ok=True)
with fs.open(f"{out}/part-0.csv", "wb") as f:
f.write(b"id,value\n")
Byte ranges through cat_file(), not positional cat() arguments
Silent: old code still runs but behaves differently.
cat() is now fsspec’s own, cat(path, recursive=False, on_error="raise", **kwargs), and also accepts globs and lists. A byte range passed by position now lands in recursive and on_error, and the whole file is returned without an error. Use cat_file(url, start=..., end=...) for a byte range.
Beta
header = fs.cat("lucidlink://workspace/filespace/video.mov", 0, 1024)
print(len(header))
1.0
header = fs.cat_file("lucidlink://workspace/filespace/video.mov", start=0, end=1024)
print(len(header))
fsspec copy rules for get() and put()
Silent: old code still runs but behaves differently.
get() and put() are now fsspec’s own. When the destination directory already exists, a source directory is copied into it as a subdirectory, like cp -r; add a trailing / to the source to copy only its contents. A directory without recursive=True is skipped silently instead of raising ValueError. Globs and lists of paths now work, and put() creates missing remote directories.
Beta
import os
os.makedirs("local/reports", exist_ok=True)
# Copies the contents of reports/ into local/reports/
fs.get("lucidlink://workspace/filespace/reports", "local/reports", recursive=True)
# Copies the contents of local/upload/ into the existing incoming/
fs.put("local/upload", "lucidlink://workspace/filespace/incoming", recursive=True)
1.0
import os
os.makedirs("local/reports", exist_ok=True)
# A trailing "/" on the source copies its contents, not the directory itself
fs.get("lucidlink://workspace/filespace/reports/", "local/reports", recursive=True)
fs.put("local/upload/", "lucidlink://workspace/filespace/incoming", recursive=True)
mv() into directories, over files and across filespaces
Silent: old code still runs but behaves differently.
mv() onto an existing directory now moves the source into it, and onto an existing file it replaces that file, where Beta raised FileExistsError. Moving into a directory never replaces anything: when the directory already holds an entry of that name, mv() raises FsExistsError (a FileExistsError). A move between two filespaces copies the data and then deletes the source, where Beta raised ValueError. Check the destination first if you relied on the error. mv() takes one path at a time: a list raises TypeError.
Beta
src = "lucidlink://workspace/filespace/inbox/q3.csv"
try:
fs.mv(src, "lucidlink://workspace/filespace/archive") # an existing directory
except FileExistsError:
fs.mv(src, "lucidlink://workspace/filespace/archive/q3.csv")
try:
fs.mv(
"lucidlink://workspace/filespace/archive/q3.csv",
"lucidlink://workspace/other-filespace/q3.csv",
)
except ValueError:
print("cannot move across filespaces")
1.0
src = "lucidlink://workspace/filespace/inbox/q3.csv"
# Into an existing directory: becomes archive/q3.csv
fs.mv(src, "lucidlink://workspace/filespace/archive")
# Across filespaces: the data is copied, then the source deleted
fs.mv(
"lucidlink://workspace/filespace/archive/q3.csv",
"lucidlink://workspace/other-filespace/q3.csv",
)
Listing errors raised by walk(), find(), glob() and du()
Silent: old code still runs but behaves differently.
walk(), and so find(), glob() and du(), dropped any subtree that failed to list with an OSError, including timeouts and network failures, so they could return partial results. For an unknown or denied filespace, walk() returned nothing and the other three raised PermissionError. Now only a missing, non-directory or unreadable subtree is left out and every other error is raised. An unknown filespace name counts as missing: all four now return an empty result (0 for du()), so a mistyped name no longer fails. An explicit on_error applies to the listing of the given path; Beta ignored it with fsspec older than 2023.9.0. Catch the errors you want to retry.
Beta
# Any listing error (timeout, network, access) silently dropped that subtree
files = fs.find("lucidlink://workspace/filespace/datasets")
total = fs.du("lucidlink://workspace/filespace/datasets")
for root, dirs, names in fs.walk("lucidlink://workspace/filespace/datasets"):
print(root, len(names))
1.0
from lucidlink.exceptions import OperationTimeoutError, ServiceUnavailableError
# Only a missing, non-directory or unreadable subtree is omitted;
# other errors are raised
try:
files = fs.find("lucidlink://workspace/filespace/datasets")
total = fs.du("lucidlink://workspace/filespace/datasets")
for root, dirs, names in fs.walk("lucidlink://workspace/filespace/datasets"):
print(root, len(names))
except (ServiceUnavailableError, OperationTimeoutError):
print("listing incomplete; retry later")
Failures reported by close() and sync_all(); closed instances stay closed
sync_all() raised only FilespaceInternalError and dropped every other failure. It now raises a LucidLinkErrorGroup (an ExceptionGroup) with every filespace that failed: use except* or group.exceptions. close(), and leaving a with block normally, ignored errors and now raise a LucidLinkErrorGroup after closing everything; when the block is already raising, the failure is added as a note to that exception. A closed instance logged in again when used; now every call on it raises ClientClosedError, so create a new instance. fsspec reuses instances, so closing one, for example by leaving with LucidLinkFileSystem(token=...), also closes the instance that fsspec.filesystem("lucidlink", token=...) returns elsewhere with the same arguments.
Beta
from lucidlink import FilespaceInternalError
try:
fs.sync_all()
except FilespaceInternalError as e:
print(f"changes lost in filespace {e.filespace_id}")
fs.close() # never raised
print(fs.ls("lucidlink://workspace/filespace")) # logged in again
1.0
from lucidlink.exceptions import FilespaceInternalError, LucidLinkErrorGroup
try:
fs.sync_all()
except* FilespaceInternalError as group:
for e in group.exceptions:
if isinstance(e, FilespaceInternalError):
print(f"changes lost in filespace {e.filespace_id}")
try:
fs.close()
except LucidLinkErrorGroup as group:
print(f"close failed: {group.exceptions}")
# A call on the closed fs raises ClientClosedError: create a new instance
fs = LucidLinkFileSystem(token="sa_live:...")
print(fs.ls("lucidlink://workspace/filespace"))
No token in a pickled LucidLinkFileSystem
storage_options, to_json(), to_dict() and pickling no longer include the service account token. An instance rebuilt from them has no token, even in the same process, and neither has a copy from copy.copy() or copy.deepcopy(); its first call raises InvalidArgumentError. Set the FSSPEC_LUCIDLINK_TOKEN environment variable, which a worker process inherits: an instance rebuilt there reads the token from it, so passing a pickled instance to a worker keeps working. Or create the instance in the worker. Start such workers with the spawn method (see “No LucidLink calls in a forked process”).
Beta
from concurrent.futures import ProcessPoolExecutor
def size(fs: LucidLinkFileSystem, url: str) -> int:
return fs.size(url) # the pickled fs carried its token
if __name__ == "__main__":
with ProcessPoolExecutor() as pool:
print(pool.submit(size, fs, "lucidlink://workspace/filespace/a.bin").result())
1.0
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
# A pickled fs carries no token: set FSSPEC_LUCIDLINK_TOKEN, which the worker
# inherits, and the instance rebuilt there reads the token from it
def size(fs: LucidLinkFileSystem, url: str) -> int | None:
return fs.size(url)
if __name__ == "__main__":
# spawn: see "No LucidLink calls in a forked process"
spawn = multiprocessing.get_context("spawn")
with ProcessPoolExecutor(mp_context=spawn) as pool:
print(pool.submit(size, fs, "lucidlink://workspace/filespace/a.bin").result())
Internal fsspec helpers removed
Beta exposed internal fsspec helpers because their modules were public: the lucidlink.url_parser module (UrlParser, ParsedUrl, PROTOCOL, PROTOCOL_PREFIX, stringify_path, FSSPEC_AVAILABLE) and UrlParser, FilespaceCache, LucidLinkOptions, PROTOCOL and redacted_dataclass_repr in lucidlink.fsspec. They have no replacement. For the protocol name, read LucidLinkFileSystem.protocol (still "lucidlink"); to split a URL, remove the lucidlink:// prefix and split the rest at / (urllib.parse cuts a path at # or ?).
Beta
from lucidlink.url_parser import PROTOCOL, UrlParser
url = "lucidlink://workspace/filespace/reports/q3.csv"
assert url.startswith(f"{PROTOCOL}://")
parsed = UrlParser().parse(url)
print(parsed.workspace, parsed.filespace, parsed.path)
1.0
from lucidlink.fsspec import LucidLinkFileSystem
url = "lucidlink://workspace/filespace/reports/q3.csv"
prefix = f"{LucidLinkFileSystem.protocol}://"
assert url.startswith(prefix)
workspace, filespace, path = url.removeprefix(prefix).split("/", 2)
print(workspace, filespace, "/" + path)
Exceptions
Exceptions imported from lucidlink.exceptions only
The lucidlink package and its other modules no longer re-export the exception classes. Import every exception from lucidlink.exceptions, which already held them in Beta; the class names are unchanged.
Beta
from lucidlink import FilespaceInternalError, LucidLinkError
try:
data = filespace.fs.read_file("/reports/q3.csv")
except FilespaceInternalError as e:
print(f"filespace {e.filespace_id} was unlinked: link it again")
except LucidLinkError as e:
print(f"LucidLink error: {e}")
1.0
from lucidlink import FsUtils
from lucidlink.exceptions import FilespaceInternalError, LucidLinkError
try:
data = FsUtils(filespace.file_system).read_bytes("/reports/q3.csv")
except FilespaceInternalError as e:
print(f"filespace {e.filespace_id} was unlinked: link it again")
except LucidLinkError as e:
print(f"LucidLink error: {e}")
LucidLinkError subclasses instead of RuntimeError and other builtins
Beta raised many errors as plain builtins: RuntimeError for most service, version, corruption and internal failures and for any error it could not classify, plus NotImplementedError, AssertionError and SystemExit. They now raise LucidLinkError or a subclass that is not that builtin, such as InternalError, IncompatibleVersionError, UnsupportedOperationError or ClientError: catch LucidLinkError or the specific class. File system errors keep their builtin base (FileNotFoundError, PermissionError, TimeoutError, ConnectionError, ValueError, …) and now also set errno and filename, with three exceptions: a KeyError, for example from listing after an unsynced delete, is now InternalError; reading a linked S3 object raised ConnectionError and now raises ExternalStoreError; and moving a directory into its own subdirectory raised FileNotFoundError and now raises FsError with errno.EINVAL.
Beta
try:
filespace = workspace.link_filespace(id="fs-uuid-12345")
except NotImplementedError as e:
print(f"not supported here: {e}")
except RuntimeError as e:
# version mismatch, corrupted metadata, cancelled link, ...
print(f"link failed: {e}")
1.0
from lucidlink.exceptions import (
IncompatibleVersionError,
LucidLinkError,
UnsupportedOperationError,
)
try:
filespace = workspace.link_filespace("fs-uuid-12345")
except UnsupportedOperationError as e:
print(f"not supported here: {e}")
except IncompatibleVersionError as e:
print(f"upgrade the SDK: {e}")
except LucidLinkError as e:
print(f"link failed: {e}")
Authentication and access errors no longer PermissionError
In fsspec, every URL whose workspace or filespace could not be used raised PermissionError, including rejected credentials and denied access; so did a rejected session or denied access reported by a later call, such as a file operation (login() and link_filespace() raised other classes). These now raise AuthenticationError or AccessDeniedError, neither an OSError, so data libraries do not mistake them for a missing path. An unknown filespace name now raises FsNotFoundError (a FileNotFoundError), a malformed URL InvalidArgumentError, and other link failures their own class. A denied path is still FsPermissionError, which is also a PermissionError.
Beta
try:
names = fs.ls("lucidlink://workspace/filespace/reports")
except PermissionError:
# the service account has no access to this filespace
print("ask an admin for access")
1.0
from lucidlink.exceptions import AccessDeniedError, AuthenticationError
try:
names = fs.ls("lucidlink://workspace/filespace/reports")
except (AuthenticationError, AccessDeniedError):
# the service account has no access to this filespace
print("ask an admin for access")
None for an unknown filespace_id
Silent: old code still runs but behaves differently.
FilespaceInternalError.filespace_id and FilespaceAlreadyLinkedError.filespace_id were an empty string when the filespace was unknown. Every FilespaceError now has filespace_id, and it is None when the filespace is unknown: compare with None instead of "".
Beta
from lucidlink import FilespaceInternalError
try:
filespace.sync_all()
except FilespaceInternalError as e:
failed = e.filespace_id if e.filespace_id != "" else "<unknown>"
print(f"filespace {failed} must be linked again")
1.0
from lucidlink.exceptions import FilespaceInternalError
try:
filespace.sync()
except FilespaceInternalError as e:
failed = e.filespace_id if e.filespace_id is not None else "<unknown>"
print(f"filespace {failed} must be linked again")
ConfigurationError and DaemonError removed
ConfigurationError is removed; Beta never raised it, and an invalid configuration raised a plain ValueError, now InvalidArgumentError, which is also a ValueError. DaemonError came only from the removed Daemon class. A Client start-up failure, such as an unusable storage root, raised ClientError; it now raises the LucidLinkError subclass for the cause, often a plain LucidLinkError. ClientError now means misuse, such as logging in again with other credentials or using a client in a forked process.
Beta
from lucidlink import (
Client,
ClientError,
ConfigurationError,
DaemonError,
ServiceAccountCredentials,
)
def start(token: str) -> Client:
try:
client = Client()
client.login(ServiceAccountCredentials(token=token))
except ConfigurationError as e:
raise SystemExit(f"invalid configuration: {e}")
except (DaemonError, ClientError) as e:
raise SystemExit(f"client failed to start: {e}")
return client
1.0
from lucidlink import Client, ServiceAccountCredentials
from lucidlink.exceptions import InvalidArgumentError, LucidLinkError
def start(token: str) -> Client:
try:
client = Client()
client.login(ServiceAccountCredentials(token=token))
except InvalidArgumentError as e:
raise SystemExit(f"invalid configuration: {e}")
except LucidLinkError as e:
raise SystemExit(f"client failed to start or log in: {e}")
return client