# Migrating from Beta (0.15) to 1.0
## Summary
### Installation and imports
| Beta | 1.0 | |
|---|---|---|
| Python 3.10 | Python 3.11 | [details](#client-python-311-required) |
| `from lucidlink.client import Client` | `from lucidlink import Client` | [details](#client-import-from-package) |
| `fsspec>=2023.1.0` | `fsspec>=2023.10.0` | [details](#fsspec-min-version) |
| no `py.typed` | `py.typed` included | [details](#package-type-information) |
| `lucidlink.lucidlink_native`
`lucidlink.const.MiB`
`redact_service_account_token()`
`Workspace.unlink_filespace()` | — | [details](#client-internal-helpers-removed) |
### Client, authentication and workspaces
| Beta | 1.0 | |
|---|---|---|
| `client.get_workspace(client.list_workspaces()[0].id)` | `client.workspace` | [details](#client-workspace-property) |
| `workspace.link_filespace("production-data")`
`link_filespace(id=...)` | `workspace.link_filespace(filespace_id)` | [details](#client-link-filespace-by-id) |
| `Daemon()`
`create_daemon()`
`daemon.get_status()` | `Client()`
`client.is_logged_in` | [details](#client-daemon-removed) |
| `StorageMode.SANDBOXED`
`StorageMode.PHYSICAL`
`persist_on_exit=` | `StorageMode.EPHEMERAL`
`StorageMode.PERSISTENT`
`root_path=` required | [details](#client-storage-modes) |
| `except AuthenticationError` (any authentication or service failure) | `except AuthenticationError` (bad token)
`except LucidLinkError` | [details](#client-login-errors) |
| `client.close()` ignores errors | `client.close()` raises `LucidLinkErrorGroup` | [details](#client-close-raises-and-final) |
| `copy.copy(filespace)` | — (raises `TypeError`) | [details](#client-objects-not-copyable) |
| a forked worker reused the parent's client | `ClientError` in the forked worker
start workers with `spawn` | [details](#client-fork-restricted) |
### Filespaces and caching
| Beta | 1.0 | |
|---|---|---|
| `filespace.fs` | `filespace.file_system` | [details](#filespace-fs-renamed) |
| `filespace.sync_all()` | `filespace.sync()` | [details](#filespace-sync-renamed) |
| `link_filespace(..., sync_mode=SyncMode.SYNC_NONE)` | `filespace.sync_barrier = SyncBarrier.NONE` | [details](#sync-mode-to-sync-barrier) |
| `Client(config=ClientConfig(cache=CacheConfig(data_bytes=n)))` | `workspace.link_filespace(filespace_id, cache=CacheConfig(data_size=n))` | [details](#cache-config-per-link) |
| 25 GiB data
5 GiB metadata | 5 GiB data
1 GiB metadata | [details](#default-cache-caps) |
| `filespace.workspace_id`
`filespace.workspace_name`
`filespace.full_name` | `filespace.workspace.id`
`filespace.workspace.name`
— | [details](#filespace-workspace-identity) |
| `info.created` (ISO 8601 `str`) | `info.created_at` (`datetime`, UTC) | [details](#filespace-info-created-at) |
| `filespace.unlink()` ignores a failed final sync | `filespace.unlink()` raises `FilespaceSyncError` | [details](#filespace-unlink-reports-failures) |
| `Filespace(native_fs, native_daemon, workspace_id, ...)` | `workspace.link_filespace(filespace_id)` | [details](#filespace-constructor-internal) |
| `lucidlink.filespace_cache.FilespaceCache` | — | [details](#filespace-cache-internals) |
### File system
| Beta | 1.0 | |
|---|---|---|
| `lucidlink.Filesystem` | `lucidlink.FileSystem` | [details](#filesystem-class-renamed) |
| `DirEntry`
`entry.entry_id`
`entry.file_id` | `FsEntry`
`entry.id` | [details](#fsentry-renamed) |
| `entry.is_dir()`
`entry.is_link()` | `entry.is_dir`
`entry.is_symlink` | [details](#fsentry-predicates-properties) |
| `entry.ctime`
`entry.mtime` | `entry.created_at`
`entry.modified_at` | [details](#fsentry-timestamps-renamed) |
| `entry.type == "link"` | `entry.type == EntryType.SYMLINK` | [details](#fsentry-type-symlink) |
| `fs.read_file(p)`
`fs.write_file(p, data)`
`fs.list_dir(p)` | `FsUtils(fs).read_bytes(p)`
`FsUtils(fs).write_bytes(p, data)`
`[e.name for e in fs.read_dir(p)]` | [details](#whole-file-helpers-moved) |
| `fs.dir_exists(p)`
`fs.file_exists(p)` | `fs.get_entry(p).is_dir` (catch `FileNotFoundError`) | [details](#exists-checks-removed) |
| `fs.delete_dir(p, recursive=True)` | `FsUtils(fs).delete_tree(p)` | [details](#delete-dir-recursive-removed) |
| `fs.get_size()`
`fs.get_statistics()` | `fs.statistics()` | [details](#filesystem-statistics) |
| `LucidFileStream` | `FileStream` | [details](#filestream-renamed) |
| `open_buffered(...)`
`open_text(...)` | `fs.open(path, "rb", buffering=n)`
`fs.open(path, "r", encoding=...)` | [details](#open-buffered-text-removed) |
| `f.name` | — | [details](#open-name-removed) |
| `fs.open_legacy(path)`
`fs.create(path)` | `fs.open(path, "rb")`
`fs.open(path, "wb")` | [details](#file-handle-api-removed) |
| `fs.open(path, mode, lock_type="exclusive")`
`fs.lock_byte_range(...)` | — | [details](#file-locking-removed) |
| `fs.open(link)` opened the link entry itself | `FsError` with `errno.ELOOP` | [details](#symlink-open-raises-eloop) |
| `read_dir(link)` returned `[]`
`get_entry("/a/file.txt/")` returned the file | `FsNotADirectoryError`
`FsIsADirectoryError` for a directory | [details](#filesystem-silent-successes-raise) |
| `seek()` then `write()` in `"ab"` wrote at that position | every write in `"a"` / `"ab"` appends | [details](#append-mode-writes-at-end) |
| `f.close()` ignored a failed native close | `f.close()` raises `LucidLinkError` | [details](#stream-close-raises) |
| `f.seek(-1)` moved to 0 | `f.seek(-1)` raises `ValueError` | [details](#stream-seek-stricter) |
| `lucidlink.file_modes.parse_mode`
`lucidlink.filesystem.READ_CHUNK_SIZE`, ... | — | [details](#filesystem-internal-helpers-removed) |
### Connect
| Beta | 1.0 | |
|---|---|---|
| `from lucidlink import ConnectManager, S3DataStoreConfig, ...` | `from lucidlink.connect import ConnectManager, S3DataStoreConfig, ...` | [details](#connect-import-path) |
| `connect.add_data_store(name, config)`
`connect.add_http_link_data_store(name)` | `connect.data_stores.add(name, config)`
`connect.data_stores.add(name, HttpLinkDataStoreConfig())` | [details](#connect-add-data-store) |
| `connect.get_data_store(name)`
`connect.list_data_stores()`
`connect.remove_data_store(name)` | `connect.data_stores.get(name)`
`connect.data_stores.list()`
`connect.data_stores.remove(name)` | [details](#connect-get-list-remove-data-store) |
| `info.bucket_name`
`info.secret_key`
`DataStoreInfo(name, ...)` | `S3DataStoreInfo.bucket_name`, no `secret_key`
`DataStoreInfo(name=..., kind=...)` | [details](#connect-data-store-info) |
| `S3DataStoreConfig("AKIA...", "secret", "bucket", "us-east-1")` | `S3DataStoreConfig(access_key=..., secret_key=..., bucket_name=..., region=...)` | [details](#connect-s3-config-keyword-only) |
| `connect.link_file(path, store_name, object_id, size, checksum)` | `store.link(path, object_id, *, size=..., checksum=...)` | [details](#connect-link-s3-file) |
| `connect.link_http_file(path, url, data_store_name=...)` | format 3.8: `links.link(path, url)`
format 3.6: `HttpConnect_3_6(filespace).link(path, url)` | [details](#connect-link-http-file) |
| `connect.update_http_file_url(path, new_url, size=...)` | format 3.8: `links.get_file(path).update_url(url, size=...)`
format 3.6: `HttpConnect_3_6(filespace).update_url(path, url, size=...)` | [details](#connect-update-http-url) |
| `connect.unlink_file(path)` | `filespace.file_system.delete(path)` | [details](#connect-unlink-file) |
| `connect.list_external_files(name, limit, cursor)`
`connect.count_external_files(name)` | `store.iter_files()`
`store.count()` | [details](#connect-list-count-external-files) |
| `connect.rekey_data_store(name, DataStoreCredentials(...))`
`rekey_data_store(name, new_access_key=..., new_secret_key=...)` | `store.rekey(S3Credentials(access_key=..., secret_key=...))` | [details](#connect-rekey) |
| `DataStoreKind.S3 == "S3DataStore"`
`DataStoreKind.HTTP_LINK == "HttpLinkDataStore"` | `DataStoreKind.S3 == "s3"`
`DataStoreKind.HTTP_LINK == "http_link"` | [details](#connect-data-store-kind-values) |
| `connect.are_data_stores_available()` | — (catch `UnsupportedOperationError`) | [details](#connect-are-data-stores-available) |
| `FileExistsError`
`RuntimeError`
`ValueError` (data store limit)
`ConnectionError` (reading a linked S3 object) | `DataStoreExistsError`
`DataStoreInUseError`
`ConnectError`, ... | [details](#connect-errors) |
| `lucidlink.connect.validate_data_store_name`
`lucidlink.connect_validation.*`
`redacted_dataclass_repr` | — | [details](#connect-internal-helpers) |
### fsspec
| Beta | 1.0 | |
|---|---|---|
| `lucidlink.LucidLinkFileSystem` | `lucidlink.fsspec.LucidLinkFileSystem` | [details](#fsspec-import-path) |
| `LucidLinkFileSystem(sandboxed=False, root_path=...)`
`fs.options` | — | [details](#fsspec-storage-options-removed) |
| `sync_mode=SyncMode.SYNC_NONE` | `sync_barrier="none"` | [details](#fsspec-sync-mode) |
| `ls(url)` names `lucidlink://workspace/filespace/a`
`info(url)["name"]` is `"a"` | `"workspace/filespace/a"` for both | [details](#fsspec-info-names) |
| `info["mtime"]` in nanoseconds (int) | `info["mtime"]` in seconds (float or `None`) | [details](#fsspec-timestamps-seconds) |
| `info["type"] == "file"`
`fs.exists(link)` is `True` | `info["type"] == "link"`
`fs.exists(link)` is `False` | [details](#fsspec-link-type) |
| `fs.exists(url)` is `False` on a network error | `fs.exists(url)` raises the error | [details](#fsspec-existence-checks-raise) |
| `fs.makedirs(existing_dir)` returns | `fs.makedirs(existing_dir, exist_ok=True)` | [details](#fsspec-makedirs-existing) |
| `fs.cat(url, start, end)` | `fs.cat_file(url, start=..., end=...)` | [details](#fsspec-cat-range) |
| `fs.get(src_dir, existing_dir, recursive=True)` copies the contents | copies into `existing_dir/`
add a trailing `/` to `src_dir` | [details](#fsspec-get-put) |
| `fs.mv(src, existing)` raises `FileExistsError`
across filespaces `ValueError` | moves into the directory or replaces the file
copies across filespaces | [details](#fsspec-mv-semantics) |
| `fs.find(url)` omits every subtree that fails to list | omits only a missing, non-directory or unreadable subtree | [details](#fsspec-walk-errors) |
| `except FilespaceInternalError` around `fs.sync_all()`
`fs.close()` never raised | `except* FilespaceInternalError`
`close()` raises `LucidLinkErrorGroup` | [details](#fsspec-close-sync-errors) |
| pickled `fs` carries the token | `FSSPEC_LUCIDLINK_TOKEN` set for the worker | [details](#fsspec-pickle-token) |
| `lucidlink.url_parser.UrlParser`
`PROTOCOL`, ... | `LucidLinkFileSystem.protocol`
otherwise — | [details](#fsspec-internal-helpers) |
### Exceptions
| Beta | 1.0 | |
|---|---|---|
| `from lucidlink import LucidLinkError` | `from lucidlink.exceptions import LucidLinkError` | [details](#exceptions-import-location) |
| `except RuntimeError`
`NotImplementedError` | `except LucidLinkError`
`UnsupportedOperationError` | [details](#exceptions-builtin-classes) |
| `except PermissionError` | `except (AuthenticationError, AccessDeniedError)` | [details](#exceptions-access-not-permissionerror) |
| `e.filespace_id == ""` | `e.filespace_id is None` | [details](#exceptions-filespace-id-none) |
| `ConfigurationError`
`DaemonError` | `InvalidArgumentError`
`LucidLinkError` | [details](#exceptions-configuration-daemon-removed) |
## Installation and imports
(client-python-311-required)=
### Python 3.11 or newer required
The SDK no longer supports Python 3.10. Use Python 3.11 or newer before you install 1.0.
**Beta**
```toml
[project]
requires-python = ">=3.10"
dependencies = ["lucidlink==0.15.0"]
```
**1.0**
```toml
[project]
requires-python = ">=3.11"
dependencies = ["lucidlink>=1.0,<2"]
```
(client-import-from-package)=
### Classes imported from the lucidlink package
The modules `lucidlink.client`, `lucidlink.credentials`, `lucidlink.storage`, `lucidlink.workspace`, `lucidlink.workspace_models`, `lucidlink.filespace`, `lucidlink.filespace_models`, `lucidlink.filesystem_models`, `lucidlink.daemon` and `lucidlink.daemon_models` no longer exist. Import `Client`, `ServiceAccountCredentials`, `StorageConfig`, `StorageMode`, `Workspace`, `Filespace`, `FilespaceInfo` and `CacheConfig` from `lucidlink`, which already exported them in Beta. `from lucidlink import *` now imports only the names in `lucidlink.__all__`; import exceptions from `lucidlink.exceptions`.
**Beta**
```python
from lucidlink.client import Client
from lucidlink.credentials import ServiceAccountCredentials
from lucidlink.filespace import Filespace
from lucidlink.storage import StorageConfig
from lucidlink.workspace import FilespaceInfo, Workspace
def describe(workspace: Workspace, info: FilespaceInfo, filespace: Filespace) -> str:
return f"{workspace.name}/{info.name}: {filespace.id}"
with Client(storage=StorageConfig()) as client:
client.login(ServiceAccountCredentials("sa_live:..."))
```
**1.0**
```python
from lucidlink import (
Client,
Filespace,
FilespaceInfo,
ServiceAccountCredentials,
StorageConfig,
Workspace,
)
def describe(workspace: Workspace, info: FilespaceInfo, filespace: Filespace) -> str:
return f"{workspace.name}/{info.name}: {filespace.id}"
with Client(storage=StorageConfig()) as client:
client.login(ServiceAccountCredentials("sa_live:..."))
```
(fsspec-min-version)=
### fsspec 2023.10.0 or newer required
The `lucidlink[fsspec]` extra now requires fsspec 2023.10.0 or newer. Raise any fsspec pin below that version.
**Beta**
```toml
[project]
dependencies = ["lucidlink[fsspec]==0.15.0", "fsspec==2023.6.0"]
```
**1.0**
```toml
[project]
dependencies = ["lucidlink[fsspec]>=1.0,<2", "fsspec>=2023.10.0"]
```
(package-type-information)=
### Type information included
The package now ships type information, so mypy and pyright check your calls into the SDK instead of skipping it. Code that still runs can report new type errors, which point at the changes on this page.
**Beta**
```text
$ mypy app.py
app.py:1: error: Skipping analyzing "lucidlink": module is installed,
but missing library stubs or py.typed marker [import-untyped]
```
**1.0**
```text
$ mypy app.py
app.py:9: error: "FsEntry" has no attribute "mtime" [attr-defined]
```
(client-internal-helpers-removed)=
### Internal client helpers removed
Beta exposed these undocumented helpers only because its modules were public, and they have no replacement: the native bindings module `lucidlink.lucidlink_native` (also reachable as `lucidlink.client.lucidlink_native` and `lucidlink.daemon.lucidlink_native`), whose `FileSystem` class is unrelated to the new `lucidlink.FileSystem`; `lucidlink.const.MiB`; `VALID_TOKEN_PREFIXES` and `redact_service_account_token()` from `lucidlink.credentials`; and `Workspace.unlink_filespace()`. Define constants such as `MiB` yourself, print the `ServiceAccountCredentials` object (its `repr()` redacts the token), and call `filespace.unlink()` to unlink a filespace.
**Beta**
```python
from lucidlink.const import MiB
from lucidlink.credentials import redact_service_account_token
chunk_size = 8 * MiB
print("using token", redact_service_account_token(credentials.token))
```
**1.0**
```python
MiB = 1024**2
chunk_size = 8 * MiB
print("using", credentials) # the repr redacts the token
```
## Client, authentication and workspaces
(client-workspace-property)=
### Workspace access through client.workspace
A service account token gives access to exactly one workspace, so `list_workspaces()`, `get_workspace()` and `WorkspaceInfo` are removed. Use the `client.workspace` property; its `id` and `name` replace the `WorkspaceInfo` fields. Before `login()` it raises `NotLoggedInError`, where Beta raised `AuthenticationError`.
**Beta**
```python
from lucidlink import WorkspaceInfo
client.login(credentials)
info: WorkspaceInfo = client.list_workspaces()[0]
workspace = client.get_workspace(info.id)
print(info.id, info.name)
```
**1.0**
```python
client.login(credentials)
workspace = client.workspace
print(workspace.id, workspace.name)
```
(client-link-filespace-by-id)=
### Filespaces linked by ID only
`link_filespace()` no longer accepts a filespace name: its first argument is the filespace ID, so a name passed there is looked up as an ID and the link fails with `FilespaceNotFoundError`. The `id` keyword is renamed `filespace_id`, and `root_path`, which had no effect, is removed. Look up IDs with `workspace.list_filespaces()`.
**Beta**
```python
production = workspace.link_filespace("production-data") # by name
archive = workspace.link_filespace(id=archive_id, root_path="/")
```
**1.0**
```python
ids_by_name = {info.name: info.id for info in workspace.list_filespaces()}
production = workspace.link_filespace(ids_by_name["production-data"])
archive = workspace.link_filespace(archive_id)
```
(client-daemon-removed)=
### Daemon and create_daemon() removed
The deprecated `Daemon` class, `create_daemon()` and `DaemonStatus` are removed. Use `Client`: `client.login(credentials)` replaces `start()` plus `authenticate()`, `client.close()` (or leaving the `with` block) replaces `stop()`, and `client.is_logged_in` and `client.workspace.linked_filespaces` replace `get_status()`. Pass storage options as `Client(storage=StorageConfig(...))` and cache sizes to `link_filespace(cache=...)`; the free-form `config` dict has no replacement. Its `fs.cache.size` key had no effect in Beta: set the data cache size with `CacheConfig(data_size=...)`.
**Beta**
```python
from lucidlink import Daemon, ServiceAccountCredentials
with Daemon() as daemon: # start() on enter, stop() on exit
workspace = daemon.authenticate(ServiceAccountCredentials(token))
filespace = workspace.link_filespace(id=filespace_id)
print(daemon.get_status().is_authenticated)
print([entry.name for entry in filespace.fs.read_dir("/")])
```
**1.0**
```python
from lucidlink import Client, ServiceAccountCredentials
with Client() as client: # close() on exit
client.login(ServiceAccountCredentials(token))
filespace = client.workspace.link_filespace(filespace_id)
print(client.is_logged_in)
print([entry.name for entry in filespace.file_system.read_dir("/")])
```
(client-storage-modes)=
### Storage modes renamed to EPHEMERAL and PERSISTENT
`SANDBOXED` is now `EPHEMERAL` (still the default) and `PHYSICAL` is now `PERSISTENT`, with the values `"ephemeral"` and `"persistent"`. `PERSISTENT` always keeps its files, has no `persist_on_exit`, and requires `root_path` instead of defaulting to the current directory; `EPHEMERAL` rejects a `root_path`. `StorageConfig` takes keyword arguments only and is immutable, and `get_root_path()` and `should_cleanup()` are removed: read `root_path` (the files go to its `.lucid` subdirectory). `PHYSICAL` without `persist_on_exit=True` deleted its files on close: if the location did not matter, use `EPHEMERAL`; to keep the location, use `PERSISTENT` and remove `/.lucid` after the client closes.
**Beta**
```python
from pathlib import Path
from lucidlink import Client, StorageConfig, StorageMode
temporary = StorageConfig(StorageMode.SANDBOXED)
kept = StorageConfig(
StorageMode.PHYSICAL, persist_on_exit=True, root_path=Path("/data/lucid")
)
print(kept.mode, kept.get_root_path()) # /data/lucid/.lucid
with Client(storage=kept) as client:
...
```
**1.0**
```python
from pathlib import Path
from lucidlink import Client, StorageConfig, StorageMode
temporary = StorageConfig(mode=StorageMode.EPHEMERAL)
kept = StorageConfig(mode=StorageMode.PERSISTENT, root_path=Path("/data/lucid"))
print(kept.mode, kept.root_path) # files go to /data/lucid/.lucid
with Client(storage=kept) as client:
...
```
(client-login-errors)=
### Specific errors from Client.login()
`login()` used to raise `AuthenticationError` for every authentication or service failure. It now raises `AuthenticationError` only for an invalid, expired or revoked token; denied access, rate limiting and service failures raise other `LucidLinkError` subclasses, or a plain `LucidLinkError` for a service status the SDK does not classify, and logging in again with different credentials raises `ClientError`. The first `login()` also starts the client: a start-up failure raised `ClientError` and now raises the class for its cause. Catch `LucidLinkError` to keep handling every login failure.
**Beta**
```python
from lucidlink.exceptions import AuthenticationError
try:
client.login(credentials)
except AuthenticationError as error: # any login failure
print("login failed:", error)
```
**1.0**
```python
from lucidlink.exceptions import AuthenticationError, LucidLinkError
try:
client.login(credentials)
except AuthenticationError as error: # token invalid, expired or revoked
print("bad token:", error)
except LucidLinkError as error: # access denied, rate limited, unreachable, ...
print("login failed:", error)
```
(client-close-raises-and-final)=
### Client.close() reports failures and is final
`close()` used to ignore a failed final sync or unlink. It now finishes the teardown and then raises a `LucidLinkErrorGroup`, for example with a `FilespaceSyncError` for changes that were not saved; leaving a `with` block normally raises it too. Call `filespace.sync()` before closing when the changes must be saved. A closed client cannot be reused, where Beta allowed `login()` again after `close()`: it and every object obtained through it raise `ClientClosedError`, so create a new `Client`. `Workspace.stop()` likewise raises a `LucidLinkErrorGroup` when unlinking a filespace fails; Beta ignored those errors.
**Beta**
```python
client.login(credentials)
workspace = client.get_workspace(client.list_workspaces()[0].id)
filespace = workspace.link_filespace(id=filespace_id)
... # write files
client.close() # a failed final sync is ignored
```
**1.0**
```python
from lucidlink.exceptions import FilespaceSyncError
client.login(credentials)
filespace = client.workspace.link_filespace(filespace_id)
... # write files
try:
client.close()
except* FilespaceSyncError as failed:
print("changes not saved:", failed.exceptions)
```
(client-objects-not-copyable)=
### SDK objects cannot be copied
`copy.copy()` of a `Client`, `Workspace`, `Filespace`, `FileSystem`, `FileStream`, `FsUtils` or a Connect object now raises `TypeError`. In Beta it made a copy that shared the native state of the original, so closing or unlinking one broke the other; a buffered file from `open()` could not be copied at all. Use the object itself. `copy.deepcopy()` and pickling already failed in Beta; in another process, create a new `Client`.
**Beta**
```python
import copy
report_fs = copy.copy(filespace) # shares the link with filespace
report_fs.fs.create_dir("/reports")
```
**1.0**
```python
report_fs = filespace # copy.copy(filespace) raises TypeError
report_fs.file_system.create_dir("/reports")
```
(client-fork-restricted)=
### No LucidLink calls in a forked process
In a process forked while a LucidLink client was running, every LucidLink call now raises `ClientError`, including calls on a `LucidLinkFileSystem`. In Beta a forked child reused the parent's client: some calls worked and data reads could hang. On Linux, `multiprocessing` and `ProcessPoolExecutor` fork by default up to Python 3.13. Start workers with the `spawn` or `forkserver` start method and create a client in each worker, or close every client before forking.
**Beta**
```python
from concurrent.futures import ProcessPoolExecutor
def count(path: str) -> int:
# On Linux the worker is forked and reuses the parent's client
return len(filespace.fs.read_dir(path))
if __name__ == "__main__":
with ProcessPoolExecutor() as pool:
print(list(pool.map(count, ["/a", "/b"])))
```
**1.0**
```python
import multiprocessing
import os
from concurrent.futures import ProcessPoolExecutor
from lucidlink import Client, ServiceAccountCredentials
def count(path: str) -> int:
# Each worker logs in with a client of its own
with Client() as client:
client.login(ServiceAccountCredentials(token=os.environ["LUCIDLINK_TOKEN"]))
filespace = client.workspace.link_filespace("fs-uuid-12345")
return len(filespace.file_system.read_dir(path))
if __name__ == "__main__":
spawn = multiprocessing.get_context("spawn")
with ProcessPoolExecutor(mp_context=spawn) as pool:
print(list(pool.map(count, ["/a", "/b"])))
```
## Filespaces and caching
(filespace-fs-renamed)=
### Filespace.fs renamed to file_system
The property that returns the file system of a linked filespace is renamed from `fs` to `file_system`. Replace every `filespace.fs` with `filespace.file_system`.
**Beta**
```python
filespace.fs.create_dir("/reports")
for entry in filespace.fs.read_dir("/"):
print(entry.name)
```
**1.0**
```python
filespace.file_system.create_dir("/reports")
for entry in filespace.file_system.read_dir("/"):
print(entry.name)
```
(filespace-sync-renamed)=
### Filespace.sync_all() renamed to sync()
`sync_all()` is renamed to `sync()`. It still blocks until all pending changes have reached LucidLink.
**Beta**
```python
filespace.fs.create_dir("/reports")
filespace.sync_all()
```
**1.0**
```python
filespace.file_system.create_dir("/reports")
filespace.sync()
```
(sync-mode-to-sync-barrier)=
### SyncMode replaced by Filespace.sync_barrier
`SyncMode` and the `sync_mode` argument of `link_filespace()` are removed. Set the `sync_barrier` property of the linked filespace instead: `SyncMode.SYNC_NONE` becomes `SyncBarrier.NONE`, and `SyncMode.SYNC_ALL` becomes `SyncBarrier.ON_UNLINK`, still the default. The values are now `"none"` and `"on_unlink"` (Beta had `"none"` and `"all"`).
**Beta**
```python
from lucidlink import SyncMode
filespace = workspace.link_filespace(id="fs-uuid-12345", sync_mode=SyncMode.SYNC_NONE)
filespace.fs.create_dir("/batch")
filespace.sync_all()
filespace.unlink()
```
**1.0**
```python
from lucidlink import SyncBarrier
filespace = workspace.link_filespace("fs-uuid-12345")
filespace.sync_barrier = SyncBarrier.NONE
filespace.file_system.create_dir("/batch")
filespace.sync()
filespace.unlink()
```
(cache-config-per-link)=
### ClientConfig removed; cache sizes set per link
`ClientConfig` and the `config` argument of `Client()` are removed: pass a `CacheConfig` to each `link_filespace()` call instead. Its fields `data_bytes` and `metadata_bytes` are renamed to `data_size` and `metadata_size` (still in bytes), and it is now immutable and accepts only integers. Linking a filespace that is already linked raises `FilespaceAlreadyLinkedError` only when the new `CacheConfig` sets a size that differs from the one in use; with no `CacheConfig`, a `None` field or the same size it returns the linked filespace. To change the sizes, unlink first. A size you set stays in effect when the same client links the filespace again, even with no `CacheConfig`. In Beta, `FilespaceAlreadyLinkedError` meant that the filespace was already linked under another identifier.
**Beta**
```python
from lucidlink import CacheConfig, Client, ClientConfig
cache = CacheConfig(data_bytes=2 * 1024**3, metadata_bytes=512 * 1024**2)
with Client(config=ClientConfig(cache=cache)) as client:
client.login(credentials)
workspace = client.get_workspace(client.list_workspaces()[0].id)
filespace = workspace.link_filespace(id="fs-uuid-12345")
```
**1.0**
```python
from lucidlink import CacheConfig, Client
cache = CacheConfig(data_size=2 * 1024**3, metadata_size=512 * 1024**2)
with Client() as client:
client.login(credentials)
filespace = client.workspace.link_filespace("fs-uuid-12345", cache=cache)
```
(default-cache-caps)=
### Smaller default cache sizes
> **Silent:** old code still runs but behaves differently.
When a link gives no cache sizes and none are set for the filespace, on this machine, filespace-wide or by an earlier link of the same client, the data cache is now capped at 5 GiB instead of 25 GiB and the metadata cache at 1 GiB instead of 5 GiB. If your workload needs the larger caches, pass a `CacheConfig` with the sizes you need.
**Beta**
```python
# No cache sizes given: data cache up to 25 GiB, metadata cache up to 5 GiB
filespace = workspace.link_filespace(id="fs-uuid-12345")
```
**1.0**
```python
from lucidlink import CacheConfig
# With no size given the caps are 5 GiB data and 1 GiB metadata
filespace = workspace.link_filespace(
"fs-uuid-12345",
cache=CacheConfig(data_size=25 * 1024**3, metadata_size=5 * 1024**3),
)
```
(filespace-workspace-identity)=
### Workspace details through Filespace.workspace
`Filespace` no longer has `workspace_id`, `workspace_name` or `full_name`. Read `id` and `name` from the new `workspace` property, the `Workspace` the filespace was linked from. `full_name` has no replacement; use `name` for the filespace name.
**Beta**
```python
print(f"{filespace.workspace_name} ({filespace.workspace_id}): {filespace.full_name}")
```
**1.0**
```python
print(f"{filespace.workspace.name} ({filespace.workspace.id}): {filespace.name}")
```
(filespace-info-created-at)=
### FilespaceInfo.created replaced by created_at
`FilespaceInfo.created` was an ISO 8601 string. It is now `created_at`, a timezone-aware `datetime` in UTC. Drop your own parsing. `.isoformat()` does not give back the Beta string: `2026-10-18T22:56:03.743Z` becomes `2026-10-18T22:56:03.743000+00:00`, so code that compares or stores the text sees a different value.
**Beta**
```python
for info in workspace.list_filespaces():
print(f"{info.name}: created {info.created}")
```
**1.0**
```python
for info in workspace.list_filespaces():
print(f"{info.name}: created {info.created_at.isoformat()}")
```
(filespace-unlink-reports-failures)=
### Filespace.unlink() reports a failed final sync
If the final sync fails, `unlink()` and leaving a `with` block normally now raise `FilespaceSyncError`, where Beta ignored the failure; the same happens on the first `unlink()` after the SDK unlinked the filespace because of an internal error. The filespace is unlinked either way. `unlink()` also closes files still open through `file_system`, and raises a `FilespaceError` if that fails, so catch `FilespaceError` for both; if the close or the sync fails and unlinking fails too, it raises a `LucidLinkErrorGroup`. Later use of the filespace raises `FilespaceNotLinkedError` instead of `RuntimeError`. Call `sync()` before you unlink when your changes must be saved.
**Beta**
```python
filespace = workspace.link_filespace(id="fs-uuid-12345")
filespace.fs.create_dir("/out")
filespace.unlink() # a failed final sync was ignored
```
**1.0**
```python
from lucidlink.exceptions import FilespaceError
filespace = workspace.link_filespace("fs-uuid-12345")
filespace.file_system.create_dir("/out")
try:
filespace.unlink()
except FilespaceError as error:
print(f"Unlinked, but closing a file or the final sync failed: {error}")
```
(filespace-constructor-internal)=
### Filespace constructor changed
The `Filespace` constructor was documented as internal and now takes different parameters. Do not construct a `Filespace`; get one from `workspace.link_filespace()`.
**Beta**
```python
from lucidlink import Filespace
filespace = Filespace(
native_fs,
native_daemon,
workspace_id="ws-uuid-1",
workspace_name="acme",
id="fs-uuid-12345",
full_name="data.acme",
on_unlink=lambda filespace_id: None,
)
```
**1.0**
```python
filespace = workspace.link_filespace("fs-uuid-12345")
```
(filespace-cache-internals)=
### Internal filespace helpers removed
The undocumented `lucidlink.filespace_cache` module (`FilespaceCache` and the names it imported) is removed, with no replacement. To link a filespace by name, map names to IDs with `list_filespaces()` and call `link_filespace()`, which returns the live `Filespace` when the filespace is already linked. `FilespaceCache` linked without a final sync; a filespace linked this way syncs when it is unlinked, so `unlink()` can wait and raise `FilespaceSyncError`. An unknown name is a `KeyError` from your mapping, where `FilespaceCache` raised `PermissionError`.
**Beta**
```python
from lucidlink.filespace_cache import FilespaceCache
from lucidlink.url_parser import ParsedUrl
links = FilespaceCache()
links.set_workspace(workspace)
filespace = links.get_or_connect(ParsedUrl(workspace.name, "media", "/"))
```
**1.0**
```python
ids = {info.name: info.id for info in workspace.list_filespaces()}
# Returns the live Filespace when it is already linked
filespace = workspace.link_filespace(ids["media"])
```
## File system
(filesystem-class-renamed)=
### Filesystem renamed to FileSystem
The file system class is now `FileSystem`, imported from `lucidlink` only; the `lucidlink.filesystem` and `lucidlink.filespace` modules no longer exist. Update imports and type annotations; a filespace returns it as `filespace.file_system`.
**Beta**
```python
def count_entries(fs: Filesystem, path: str) -> int:
return len(fs.read_dir(path))
print(count_entries(filespace.fs, "/projects"))
```
**1.0**
```python
def count_entries(fs: FileSystem, path: str) -> int:
return len(fs.read_dir(path))
print(count_entries(filespace.file_system, "/projects"))
```
(fsentry-renamed)=
### DirEntry replaced by FsEntry
`read_dir()` and `get_entry()` return `FsEntry` instead of `DirEntry`. `entry_id` and the deprecated `file_id` are now `id`, and `file_id_external` is removed with no replacement. `FsEntry` is immutable and its constructor takes keyword arguments only, all required (`id`, `name`, `type`, `size`, `modified_at`, `created_at`, `inode_id`, `num_hard_links`): test doubles must pass all of them and use `dataclasses.replace()` to change one.
**Beta**
```python
entry: DirEntry = filespace.fs.get_entry("/projects/report.pdf")
print(entry.entry_id, entry.file_id_external)
# A test double
stub = DirEntry(name="report.pdf", size=3, type="file", file_id="e1",
file_id_external=1, ctime=0, mtime=0)
stub.size = 4
```
**1.0**
```python
entry: FsEntry = filespace.file_system.get_entry("/projects/report.pdf")
print(entry.id)
# A test double
stub = FsEntry(id="e1", name="report.pdf", type=EntryType.FILE, size=3,
modified_at=0, created_at=0, inode_id="i1", num_hard_links=1)
stub = replace(stub, size=4)
```
(fsentry-predicates-properties)=
### is_file, is_dir and is_symlink as properties
`is_file` and `is_dir` are now read-only properties, and the `is_link()` method is now the `is_symlink` property. Drop the parentheses: calling them raises `TypeError: 'bool' object is not callable`.
**Beta**
```python
for entry in filespace.fs.read_dir("/projects"):
if entry.is_dir():
print("dir ", entry.name)
elif entry.is_link():
print("link", entry.name)
elif entry.is_file():
print("file", entry.name)
```
**1.0**
```python
for entry in filespace.file_system.read_dir("/projects"):
if entry.is_dir:
print("dir ", entry.name)
elif entry.is_symlink:
print("link", entry.name)
elif entry.is_file:
print("file", entry.name)
```
(fsentry-timestamps-renamed)=
### ctime and mtime renamed to created_at and modified_at
The timestamp fields are renamed and their values are unchanged: integer Unix timestamps in nanoseconds, as in Beta, whose documentation said only "Unix timestamp". Divide by 1e9 for seconds.
**Beta**
```python
entry = filespace.fs.get_entry("/projects/report.pdf")
modified = datetime.fromtimestamp(entry.mtime / 1e9, tz=timezone.utc)
created = datetime.fromtimestamp(entry.ctime / 1e9, tz=timezone.utc)
print(created, modified)
```
**1.0**
```python
entry = filespace.file_system.get_entry("/projects/report.pdf")
modified = datetime.fromtimestamp(entry.modified_at / 1e9, tz=timezone.utc)
created = datetime.fromtimestamp(entry.created_at / 1e9, tz=timezone.utc)
print(created, modified)
```
(fsentry-type-symlink)=
### EntryType for entry types; "symlink" for symbolic links
> **Silent:** old code still runs but behaves differently.
`FsEntry.type` is now an `EntryType` (`FILE`, `DIR`, `SYMLINK`), a string enum that still compares equal to `"file"` and `"dir"`. A symbolic link's value changed from `"link"` to `"symlink"`, so a comparison with `"link"` is silently always false: compare with `EntryType.SYMLINK` or use `entry.is_symlink`. `str(entry)` starts with `symlink:` for a link, and `get_entry()` on an entry of unknown type raises `InternalError` instead of reporting `"unknown"`.
**Beta**
```python
entries = filespace.fs.read_dir("/projects")
links = [entry.name for entry in entries if entry.type == "link"]
print(links)
```
**1.0**
```python
entries = filespace.file_system.read_dir("/projects")
links = [entry.name for entry in entries if entry.type == EntryType.SYMLINK]
print(links)
```
(whole-file-helpers-moved)=
### read_file(), write_file() and list_dir() moved or removed
The whole-file helpers moved to `FsUtils`, a wrapper around a `FileSystem`: `read_file()` is now `read_bytes()` and `write_file()` is now `write_bytes()`. `list_dir()` is removed; take the names from `read_dir()`. `write_bytes()` accepts any bytes-like object, raises `IsADirectoryError` for a directory, and raises `TypeError` for a `str`, which Beta wrote as garbage bytes: encode text first.
**Beta**
```python
filespace.fs.write_file("/notes/hello.txt", b"hello")
data = filespace.fs.read_file("/notes/hello.txt")
names = filespace.fs.list_dir("/notes")
```
**1.0**
```python
utils = FsUtils(filespace.file_system)
utils.write_bytes("/notes/hello.txt", b"hello")
data = utils.read_bytes("/notes/hello.txt")
names = [entry.name for entry in filespace.file_system.read_dir("/notes")]
```
(exists-checks-removed)=
### dir_exists() and file_exists() removed
Call `get_entry()`, catch `FileNotFoundError` for a missing path and check `is_dir` or `is_file`, or attempt the operation and catch its error. `get_entry()` describes a symbolic link itself and does not follow it.
**Beta**
```python
if not filespace.fs.dir_exists("/exports"):
filespace.fs.create_dir("/exports")
if filespace.fs.file_exists("/exports/out.csv"):
filespace.fs.delete("/exports/out.csv")
```
**1.0**
```python
fs = filespace.file_system
try:
exports_is_dir = fs.get_entry("/exports").is_dir
except FileNotFoundError:
exports_is_dir = False
if not exports_is_dir:
fs.create_dir("/exports")
with suppress(FileNotFoundError):
fs.delete("/exports/out.csv")
```
(delete-dir-recursive-removed)=
### delete_dir(recursive=True) replaced by FsUtils.delete_tree()
`delete_dir()` deletes only an empty directory and no longer takes `recursive`. To delete a directory with its contents, call `FsUtils(fs).delete_tree(path)`; it deletes symbolic links below the directory, never their targets.
**Beta**
```python
filespace.fs.delete_dir("/tmp/build", recursive=True)
filespace.fs.delete_dir("/tmp/empty")
```
**1.0**
```python
FsUtils(filespace.file_system).delete_tree("/tmp/build")
filespace.file_system.delete_dir("/tmp/empty")
```
(filesystem-statistics)=
### get_size() and get_statistics() replaced by statistics()
Both methods are replaced by `statistics()`, which returns a `FileSystemStatistics`; `FilespaceSize` and `FilespaceStatistics` are removed. `entries` / `entries_size` become `entries_count` (they always held a count, although Beta documented them as bytes), `storage` becomes `storage_size`, and the `external_files_*` fields keep their names. `data` / `data_size` and the `file_count`, `directory_count` and `symlink_count` counters have no replacement. The new class is immutable, takes keyword arguments only and has no custom `str()`.
**Beta**
```python
size = filespace.fs.get_size()
print(f"{size.entries} entries, {size.storage} bytes stored")
stats = filespace.fs.get_statistics()
print(stats.file_count, stats.directory_count, stats.symlink_count)
print(stats.external_files_count, stats.external_files_size)
```
**1.0**
```python
stats = filespace.file_system.statistics()
print(f"{stats.entries_count} entries, {stats.storage_size} bytes stored")
print(stats.external_files_count, stats.external_files_size)
```
(filestream-renamed)=
### LucidFileStream renamed to FileStream
The raw stream that `open(..., buffering=0)` returns is now `FileStream`, imported from `lucidlink`; the `lucidlink.stream` and `lucidlink.filesystem` paths no longer exist. Open files with `fs.open()` rather than constructing the class. With a literal binary mode and `buffering=0`, `open()` is typed to return `FileStream`, so no `isinstance` check is needed.
**Beta**
```python
raw = filespace.fs.open("/data/blob.bin", "rb", buffering=0)
assert isinstance(raw, LucidFileStream)
with raw:
header = raw.read(16)
```
**1.0**
```python
raw: FileStream = filespace.file_system.open("/data/blob.bin", "rb", buffering=0)
with raw:
header = raw.read(16)
```
(open-buffered-text-removed)=
### open_buffered() and open_text() removed
These module-level helpers, which needed the file system's internal native object, are removed. `fs.open()` covers both: pass `buffering=` for the buffer size, and a text mode with `encoding=`, `errors=` and `newline=` for text.
**Beta**
```python
native = filespace.fs._native
with open_text(native, "/logs/app.log", "r", encoding="latin-1") as f:
lines = f.readlines()
with open_buffered(native, "/data/big.bin", "rb", buffer_size=1 << 20) as b:
chunk = b.read(4096)
```
**1.0**
```python
fs = filespace.file_system
with fs.open("/logs/app.log", "r", encoding="latin-1") as f:
lines = f.readlines()
with fs.open("/data/big.bin", "rb", buffering=1 << 20) as b:
chunk = b.read(4096)
```
(open-name-removed)=
### No name attribute on open files
Objects returned by `open()` no longer have a `name`: the raw `FileStream` dropped it, so `f.name` on the buffered and text wrappers raises `AttributeError` too. Keep the path you opened. Libraries such as pyarrow treat a file object with a `name` as a local path, which is why it was removed; code that tests `hasattr(f, "name")` now takes the other branch.
**Beta**
```python
with filespace.fs.open("/reports/q3.csv", "rb") as f:
print("reading", f.name)
data = f.read()
```
**1.0**
```python
path = "/reports/q3.csv"
with filespace.file_system.open(path, "rb") as f:
print("reading", path)
data = f.read()
```
(file-handle-api-removed)=
### FileHandle, open_legacy() and create() removed
The legacy handle API is removed: `open_legacy()` with its `FileHandle` class (offset-based `read` and `write`), and `create()`, which returned a bare handle ID. Use `open()`, and `seek()` instead of the `offset` arguments. `open(path, "wb")` creates a file but, unlike `create()`, truncates an existing one instead of failing; there is no exclusive-create mode.
**Beta**
```python
with filespace.fs.open_legacy("/logs/today.log", "r") as fh:
tail = fh.read(1024, offset=4096)
handle_id = filespace.fs.create("/logs/tomorrow.log")
```
**1.0**
```python
with filespace.file_system.open("/logs/today.log", "rb") as f:
f.seek(4096)
tail = f.read(1024)
with filespace.file_system.open("/logs/tomorrow.log", "wb"):
pass
```
(file-locking-removed)=
### File locking removed
`open()` no longer takes `lock_type` (passing it raises `TypeError`), and `lock_byte_range()`, `unlock_byte_range()`, `unlock_all_byte_ranges()`, `LOCK_NONE`, `LOCK_SHARED`, `LOCK_EXCLUSIVE`, `VALID_LOCK_TYPES` and `validate_lock_type` are removed with no replacement: coordinate concurrent writers in your application. 1.0 neither takes nor checks locks, so a lock that another client holds does not block it.
**Beta**
```python
with filespace.fs.open("/db/app.sqlite", "r+b", lock_type="exclusive") as f:
page = f.read(4096)
```
**1.0**
```python
# No SDK lock: coordinate writers of this file in the application
with filespace.file_system.open("/db/app.sqlite", "r+b") as f:
page = f.read(4096)
```
(symlink-open-raises-eloop)=
### ELOOP when opening or truncating a symbolic link
`open()` in every mode, `truncate()`, and `FsUtils.read_bytes()` and `write_bytes()` never follow a symbolic link given as the path: they raise `FsError` with `errno.ELOOP`. Beta opened or resized the link entry itself, so a read did not return the target's content and a write or truncate changed the link, not its target. Resolve the link with `read_symlink()` and use the target path; links among the directories of a path are still followed.
**Beta**
```python
filespace.fs.create_symlink("/data/current.csv", "/data/latest.csv")
with filespace.fs.open("/data/latest.csv", "rb") as f:
data = f.read() # reads the link entry itself, not current.csv
```
**1.0**
```python
fs = filespace.file_system
fs.create_symlink("/data/current.csv", "/data/latest.csv")
target = fs.read_symlink("/data/latest.csv")
with fs.open(target, "rb") as f:
data = f.read()
```
(filesystem-silent-successes-raise)=
### Errors instead of empty or wrong results from read_dir(), get_entry() and truncate()
Three calls that quietly succeeded in Beta now raise. `read_dir()` on a file or a symbolic link returned `[]`, so listing through a link to a directory gave an empty result; it now raises `FsNotADirectoryError`. To list through a link, list the target from `read_symlink()`. `get_entry()` with a trailing slash after a file name returned the file and now raises `FsNotADirectoryError`. `truncate()` and `write_file()` on a directory changed the size of the directory entry itself; `truncate()` and `FsUtils.write_bytes()` now raise `FsIsADirectoryError`. fsspec `ls()` changed the other way: see "Paths without the protocol from ls() and info()".
**Beta**
```python
fs = filespace.fs
# /projects/current is a link to a directory: Beta returned []
names = [e.name for e in fs.read_dir("/projects/current")]
entry = fs.get_entry("/reports/q3.csv/") # trailing slash: Beta returned the file
```
**1.0**
```python
fs = filespace.file_system
path = "/projects/current"
if fs.get_entry(path).is_symlink:
path = fs.read_symlink(path) # list the link's target (an absolute path here)
names = [e.name for e in fs.read_dir(path)]
entry = fs.get_entry("/reports/q3.csv")
```
(append-mode-writes-at-end)=
### Append mode always writes at the end
> **Silent:** old code still runs but behaves differently.
In the append modes every write goes to the current end of the file, even after `seek()`, as with the built-in `open()`. Beta wrote at the position sought to, overwriting existing data. To write at a position, open the file with `"r+b"`.
**Beta**
```python
with filespace.fs.open("/logs/app.log", "ab") as f:
f.write(b"new entry\n")
f.seek(0)
f.write(b"HEADER v2\n") # overwrites the start of the file
```
**1.0**
```python
fs = filespace.file_system
with fs.open("/logs/app.log", "ab") as f:
f.write(b"new entry\n")
with fs.open("/logs/app.log", "r+b") as f:
f.write(b"HEADER v2\n")
```
(stream-close-raises)=
### Errors from closing a file
Closing a file writes its pending data. Beta ignored a failed native close; `close()`, and so leaving a `with` block, now raises the error, and the file is closed anyway. When writing the buffered data failed, Beta raised too, but a builtin such as `KeyError` (for example after the path was deleted); that is now `InternalError`, which is not a `KeyError`. Catch `LucidLinkError` where a lost write must be noticed.
**Beta**
```python
payload = b"..." * 1000
with filespace.fs.open("/exports/out.bin", "wb") as f:
f.write(payload)
print("saved")
```
**1.0**
```python
payload = b"..." * 1000
try:
with filespace.file_system.open("/exports/out.bin", "wb") as f:
f.write(payload)
except LucidLinkError as e:
print("not saved:", e)
else:
print("saved")
```
(stream-seek-stricter)=
### ValueError for a negative seek position
In binary mode, `seek()` to a negative absolute position now raises `InvalidArgumentError` (a `ValueError`) instead of moving to position 0, as Python's own files do; a relative seek before the start still stops at 0. On a stream opened with `buffering=0`, `readable()`, `seekable()`, `tell()` and the other methods raise `ValueError` once the stream is closed.
**Beta**
```python
with filespace.fs.open("/logs/app.log", "rb") as f:
size = f.seek(0, 2)
f.seek(size - 4096)
tail = f.read()
```
**1.0**
```python
with filespace.file_system.open("/logs/app.log", "rb") as f:
size = f.seek(0, 2)
f.seek(max(0, size - 4096))
tail = f.read()
```
(filesystem-internal-helpers-removed)=
### Internal file system helpers removed
Beta exposed undocumented helpers through its public modules `lucidlink.file_modes`, `lucidlink.filesystem` and `lucidlink.stream`: `parse_mode`, `ParsedMode`, `is_text_mode`, `ensure_binary_mode`, `get_buffered_wrapper_type`, `normalize_mode_for_cpp`, `MiB` and `READ_CHUNK_SIZE`. They have no replacement; an open file's `readable()`, `writable()` and `mode` describe how it was opened.
**Beta**
```python
mode = parse_mode("r+b")
print(mode.readable, mode.writable)
with filespace.fs.open("/data/blob.bin", "r+b") as f:
head = f.read(16)
```
**1.0**
```python
with filespace.file_system.open("/data/blob.bin", "r+b") as f:
print(f.readable(), f.writable())
head = f.read(16)
```
## Connect
(connect-import-path)=
### Connect classes moved to lucidlink.connect
The `lucidlink` package, `lucidlink.connect_models` and `lucidlink.filespace` no longer export the Connect classes. Import `ConnectManager`, `DataStoreInfo`, `DataStoreKind`, `DataStoreRekeyState`, `S3Credentials` and `S3DataStoreConfig` from `lucidlink.connect`, which also holds the data store and external file classes that are new in GA.
**Beta**
```python
from lucidlink import ConnectManager, DataStoreRekeyState, S3DataStoreConfig
connect: ConnectManager = filespace.connect
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
waiting = DataStoreRekeyState.IN_PROGRESS
```
**1.0**
```python
from lucidlink.connect import ConnectManager, DataStoreRekeyState, S3DataStoreConfig
connect: ConnectManager = filespace.connect
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
waiting = DataStoreRekeyState.IN_PROGRESS
```
(connect-add-data-store)=
### Adding data stores
Add data stores through `connect.data_stores`; the config type selects the kind: `S3DataStoreConfig` for S3, `HttpLinkDataStoreConfig()` for HTTP links (filespace format 3.8 or newer; on format 3.6 it raises `UnsupportedOperationError`). `add()` returns a handle (`S3DataStore` or `HttpLinkDataStore`) instead of a `DataStoreInfo`: call `info()` on it to read the details.
**Beta**
```python
from lucidlink import S3DataStoreConfig
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
info = filespace.connect.add_data_store("my-store", config)
print(info.name, info.bucket_name)
links = filespace.connect.add_http_link_data_store("my-links")
filespace.sync_all()
```
**1.0**
```python
from lucidlink.connect import HttpLinkDataStoreConfig, S3DataStoreConfig
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
store = filespace.connect.data_stores.add("my-store", config)
print(store.name, store.info().bucket_name)
links = filespace.connect.data_stores.add("my-links", HttpLinkDataStoreConfig())
filespace.sync()
```
(connect-get-list-remove-data-store)=
### Getting, listing and removing data stores
Use `get()`, `list()` and `remove()` on `connect.data_stores`. `get()` raises `DataStoreNotFoundError` (a `KeyError`) for an unknown name instead of returning `None`; pass `DataStoreKind.S3` or `DataStoreKind.HTTP_LINK` as `kind` to get a handle of that type. `get()` and `list()` return handles: call `info()` on one to read its details.
**Beta**
```python
connect = filespace.connect
info = connect.get_data_store("my-store")
if info is None:
print("my-store does not exist")
else:
print(info.bucket_name)
for store in connect.list_data_stores():
print(store.name, store.kind)
connect.remove_data_store("old-store")
```
**1.0**
```python
from lucidlink.connect import DataStoreKind
from lucidlink.exceptions import DataStoreNotFoundError
data_stores = filespace.connect.data_stores
try:
store = data_stores.get("my-store", DataStoreKind.S3)
except DataStoreNotFoundError:
print("my-store does not exist")
else:
print(store.info().bucket_name)
for handle in data_stores.list():
print(handle.name, handle.kind)
data_stores.remove("old-store")
```
(connect-data-store-info)=
### DataStoreInfo split into DataStoreInfo and S3DataStoreInfo
`DataStoreInfo` now has only `name` and `kind`. `info()` on an S3 data store returns the subclass `S3DataStoreInfo` with `bucket_name`, `region`, `endpoint`, `access_key`, `url_expiration_minutes`, `use_virtual_addressing` and `rekey_state`. `secret_key` and `key_id` are removed, so the secret key can no longer be read back. Both classes are immutable, take keyword arguments only, require `kind`, and have no `from_dict()`.
**Beta**
```python
from lucidlink import DataStoreInfo, DataStoreKind, DataStoreRekeyState
info = filespace.connect.get_data_store("my-store")
assert info is not None
if info.kind == DataStoreKind.S3:
print(info.bucket_name, info.region, info.key_id)
secret = info.secret_key
busy = info.rekey_state == DataStoreRekeyState.IN_PROGRESS
placeholder = DataStoreInfo("my-links", kind=DataStoreKind.HTTP_LINK)
```
**1.0**
```python
from lucidlink.connect import (
DataStoreInfo,
DataStoreKind,
DataStoreRekeyState,
S3DataStoreInfo,
)
info = filespace.connect.data_stores.get("my-store").info()
if isinstance(info, S3DataStoreInfo):
print(info.bucket_name, info.region)
busy = info.rekey_state == DataStoreRekeyState.IN_PROGRESS
placeholder = DataStoreInfo(name="my-links", kind=DataStoreKind.HTTP_LINK)
```
(connect-s3-config-keyword-only)=
### S3DataStoreConfig keyword-only and immutable
Pass every `S3DataStoreConfig` field by keyword. The config is now immutable: assigning a field raises `FrozenInstanceError`, so use `dataclasses.replace()` to get a changed copy.
**Beta**
```python
from lucidlink import S3DataStoreConfig
config = S3DataStoreConfig("AKIA...", "secret", "my-bucket", "us-east-1")
config.url_expiration_minutes = 4320
filespace.connect.add_data_store("my-store", config)
```
**1.0**
```python
import dataclasses
from lucidlink.connect import S3DataStoreConfig
config = S3DataStoreConfig(
access_key="AKIA...",
secret_key="secret",
bucket_name="my-bucket",
region="us-east-1",
)
config = dataclasses.replace(config, url_expiration_minutes=4320)
filespace.connect.data_stores.add("my-store", config)
```
(connect-link-s3-file)=
### Linking S3 objects
Link S3 objects through the S3 data store handle: the store name argument is removed, and `size` and `checksum` must be passed by keyword. `link()` returns an `S3ExternalFile` handle for the new file instead of `None`.
**Beta**
```python
connect = filespace.connect
connect.link_file("/data/file.csv", "my-store", "path/to/file.csv")
connect.link_file(
"/data/big.bin",
"my-store",
"path/to/big.bin",
1048576,
"9b2cf535f27731c974343645a3985328",
)
```
**1.0**
```python
from lucidlink.connect import DataStoreKind
store = filespace.connect.data_stores.get("my-store", DataStoreKind.S3)
store.link("/data/file.csv", "path/to/file.csv")
big = store.link(
"/data/big.bin",
"path/to/big.bin",
# exact object size and ETag (MD5, no quotes): a wrong value breaks reads
size=1048576,
checksum="9b2cf535f27731c974343645a3985328",
)
print(big.entry_id, big.get_path())
```
(connect-link-http-file)=
### Linking HTTP URLs
On filespace format 3.8 or newer, link URLs through the HTTP-link data store handle; its `link()` returns an `HttpLinkExternalFile` handle. Format 3.6 has no HTTP-link data stores: call `link()` on `HttpConnect_3_6(filespace)` from `lucidlink.connect`, which returns `None`. The `HttpConnect_3_6` constructor raises `UnsupportedOperationError` on format 3.8 or newer.
**Beta**
```python
filespace_3_8.connect.link_http_file(
"/web/page.html", "https://example.com/page.html", data_store_name="my-links"
)
filespace_3_6.connect.link_http_file("/web/old.html", "https://example.com/old.html")
```
**1.0**
```python
from lucidlink.connect import DataStoreKind, HttpConnect_3_6
links = filespace_3_8.connect.data_stores.get("my-links", DataStoreKind.HTTP_LINK)
page = links.link("/web/page.html", "https://example.com/page.html")
HttpConnect_3_6(filespace_3_6).link("/web/old.html", "https://example.com/old.html")
```
(connect-update-http-url)=
### Changing the URL of an HTTP link file
On format 3.8 or newer, call `update_url()` on the file's handle, from `get_file(path)`, `iter_files()` or `link()` of the HTTP-link data store; the handle keeps working after the file is renamed or moved. On format 3.6, call `HttpConnect_3_6(filespace).update_url(path, url)`. A path that is not an HTTP link file now raises `InvalidArgumentError` (a `ValueError`) instead of an `OSError`, and a symbolic link at the path is no longer followed.
**Beta**
```python
filespace_3_8.connect.update_http_file_url(
"/web/page.html", "https://cdn.example.com/page.html"
)
filespace_3_6.connect.update_http_file_url(
"/web/old.html", "https://cdn.example.com/old.html"
)
```
**1.0**
```python
from lucidlink.connect import DataStoreKind, HttpConnect_3_6
links = filespace_3_8.connect.data_stores.get("my-links", DataStoreKind.HTTP_LINK)
links.get_file("/web/page.html").update_url("https://cdn.example.com/page.html")
HttpConnect_3_6(filespace_3_6).update_url(
"/web/old.html", "https://cdn.example.com/old.html"
)
```
(connect-unlink-file)=
### Unlinking an external file
`unlink_file()` is removed. Delete the external file with `filespace.file_system.delete(path)`, which has the same effect.
**Beta**
```python
filespace.connect.unlink_file("/data/file.csv")
filespace.sync_all()
```
**1.0**
```python
filespace.file_system.delete("/data/file.csv")
filespace.sync()
```
(connect-list-count-external-files)=
### Listing and counting linked files
`iter_files()` on the data store handle iterates over all linked files and fetches the pages itself, so `limit`, `cursor` and `has_more` are gone. It yields `ExternalFile` handles instead of `LinkedFile` records: read the path with `get_path()` (one call per file); `entry_id` and `inode_id` are unchanged. `LinkedFile` and `LinkedFilesResult`, with their deprecated `file_paths` and `file_ids`, are removed, and `count_external_files(name)` becomes `store.count()`.
**Beta**
```python
connect = filespace.connect
filespace.sync_all()
cursor = ""
while True:
page = connect.list_external_files("my-store", limit=500, cursor=cursor)
for linked in page.entries:
print(linked.path, linked.entry_id)
if not page.has_more:
break
cursor = page.cursor
total = connect.count_external_files("my-store")
```
**1.0**
```python
store = filespace.connect.data_stores.get("my-store")
filespace.sync()
for linked in store.iter_files():
print(linked.get_path(), linked.entry_id)
total = store.count()
```
(connect-rekey)=
### Rotating S3 credentials
Rekey through the S3 data store handle with an `S3Credentials`. The `new_access_key` and `new_secret_key` keywords and the `DataStoreCredentials` alias are removed, and `S3Credentials` is immutable, takes keyword arguments only and has no `kind` field. Poll `store.info().rekey_state` to follow the rotation.
**Beta**
```python
from lucidlink import DataStoreCredentials
connect = filespace.connect
connect.rekey_data_store("my-store", DataStoreCredentials("AKIA-NEW", "new-secret"))
connect.rekey_data_store(
"my-store", new_access_key="AKIA-NEW", new_secret_key="new-secret"
)
```
**1.0**
```python
from lucidlink.connect import DataStoreKind, S3Credentials
store = filespace.connect.data_stores.get("my-store", DataStoreKind.S3)
store.rekey(S3Credentials(access_key="AKIA-NEW", secret_key="new-secret"))
```
(connect-data-store-kind-values)=
### New DataStoreKind values
> **Silent:** old code still runs but behaves differently.
The string values of `DataStoreKind` changed, so a comparison with the old strings is silently `False`, and `DataStoreKind("S3DataStore")` on a saved value raises `ValueError`. Compare with the members (`kind is DataStoreKind.S3`). `DataStoreKind` and `DataStoreRekeyState` are now `StrEnum`s, so `str()` and f-strings give the value (`"s3"`, `"in_progress"`) instead of `"DataStoreKind.S3"`.
**Beta**
```python
info = filespace.connect.get_data_store("my-store")
assert info is not None
if info.kind == "S3DataStore":
print("S3 data store")
saved = {"name": info.name, "kind": info.kind.value}
```
**1.0**
```python
from lucidlink.connect import DataStoreKind
store = filespace.connect.data_stores.get("my-store")
if store.kind is DataStoreKind.S3:
print("S3 data store")
saved = {"name": store.name, "kind": store.kind.value}
```
(connect-are-data-stores-available)=
### are_data_stores_available() removed
There is no separate availability check: call the operation and catch `UnsupportedOperationError`, which every Connect call raises when the filespace format is older than 3.6 or Connect is not configured. Beta raised `RuntimeError` there, and `UnsupportedOperationError` is not a `RuntimeError`.
**Beta**
```python
connect = filespace.connect
if connect.are_data_stores_available():
names = [info.name for info in connect.list_data_stores()]
else:
names = []
```
**1.0**
```python
from lucidlink.exceptions import UnsupportedOperationError
try:
names = [store.name for store in filespace.connect.data_stores.list()]
except UnsupportedOperationError:
names = []
```
(connect-errors)=
### LucidLink exception classes for Connect errors
> **Silent:** old code still runs but behaves differently.
Connect failures now raise classes from `lucidlink.exceptions`, and many no longer match the builtin Beta raised, so an existing `except` clause silently stops catching them. Instead of `RuntimeError`, GA raises `AccessDeniedError` when the service account is not a filespace admin, `UnsupportedOperationError` when Connect is not available, `DataStoreInUseError` when the store still has linked files, `DataStoreCredentialsError` when rekey credentials are rejected, `ConnectError` when another client holds the data store lock, `ExternalStoreError` (an `OSError`) when a linked S3 object fails its checksum, and `InvalidArgumentError` (a `ValueError`) for a `url_expiration_minutes` outside 4320 to 10080 or an endpoint without `http://` or `https://`. A duplicate name raises `DataStoreExistsError` instead of `FileExistsError`, and the data store limit `ConnectError` instead of `ValueError`. Reading a linked file whose S3 object is missing or shorter than its linked size raised `ConnectionError`; it now raises `ExternalObjectNotFoundError` or `ExternalStoreError`, which are `OSError` but not `ConnectionError`. Any other failure of the HTTP store client also raised `ConnectionError`; now only a 429 or 5xx answer stays a `ConnectionError` (`ExternalStoreUnavailableError`), and a 4xx answer raises another `ConnectError` subclass, so `except ConnectionError` misses it. Catch the new classes, or `ConnectError` for any data store failure.
**Beta**
```python
from lucidlink import S3DataStoreConfig
connect = filespace.connect
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
try:
connect.add_data_store("my-store", config)
except FileExistsError:
print("my-store already exists")
try:
connect.remove_data_store("old-store")
except RuntimeError as e:
print("cannot remove old-store:", e)
```
**1.0**
```python
from lucidlink.connect import S3DataStoreConfig
from lucidlink.exceptions import DataStoreExistsError, DataStoreInUseError
data_stores = filespace.connect.data_stores
config = S3DataStoreConfig(
access_key="AKIA...", secret_key="...", bucket_name="my-bucket", region="us-east-1"
)
try:
data_stores.add("my-store", config)
except DataStoreExistsError:
print("my-store already exists")
try:
data_stores.remove("old-store")
except DataStoreInUseError as e:
print("cannot remove old-store:", e)
```
(connect-internal-helpers)=
### Internal Connect helpers removed
Beta's public modules exposed undocumented helpers: the Connect validators and normalizers, `redacted_dataclass_repr` and the `ConnectManager` constructor. They have no replacement: `connect.data_stores.add()` and the link calls validate their arguments and raise `InvalidArgumentError` (a `ValueError`), and `ConnectManager` comes from `filespace.connect`.
**Beta**
```python
from lucidlink.connect import validate_data_store_name
name = "my-links"
validate_data_store_name(name)
filespace.connect.add_http_link_data_store(name)
```
**1.0**
```python
from lucidlink.connect import HttpLinkDataStoreConfig
from lucidlink.exceptions import InvalidArgumentError
name = "my-links"
try:
filespace.connect.data_stores.add(name, HttpLinkDataStoreConfig())
except InvalidArgumentError as e:
print("invalid data store name:", e)
```
## fsspec
(fsspec-import-path)=
### LucidLinkFileSystem moved to lucidlink.fsspec
The `lucidlink` package no longer exports `LucidLinkFileSystem`. Import it from `lucidlink.fsspec`; that import raises `ImportError` when fsspec is not installed, so use it to detect fsspec. `lucidlink.fsspec.FSSPEC_AVAILABLE` is removed. `lucidlink.fsspec` no longer re-exports `Client` or `ServiceAccountCredentials`: import them from `lucidlink`.
**Beta**
```python
import lucidlink
from lucidlink.fsspec import FSSPEC_AVAILABLE, Client, ServiceAccountCredentials
if not FSSPEC_AVAILABLE:
raise SystemExit('fsspec is missing: pip install "lucidlink[fsspec]"')
fs = lucidlink.LucidLinkFileSystem(token="sa_live:...")
client = Client()
client.login(ServiceAccountCredentials(token="sa_live:..."))
```
**1.0**
```python
from lucidlink import Client, ServiceAccountCredentials
try:
from lucidlink.fsspec import LucidLinkFileSystem
except ImportError:
raise SystemExit('fsspec is missing: pip install "lucidlink[fsspec]"')
fs = LucidLinkFileSystem(token="sa_live:...")
client = Client()
client.login(ServiceAccountCredentials(token="sa_live:..."))
```
(fsspec-storage-options-removed)=
### Storage options and the options property removed
`LucidLinkFileSystem` always uses ephemeral local storage, removed on `close()`. Passing `sandboxed` or `root_path` raises `InvalidArgumentError`, `persist_files` and other unknown options are ignored with a `LucidLinkWarning` (a `UserWarning`), and the `options` property is removed. Remove these arguments.
**Beta**
```python
from lucidlink.fsspec import LucidLinkFileSystem
fs = LucidLinkFileSystem(
token="sa_live:...",
sandboxed=False,
persist_files=True,
root_path="/var/cache/lucid",
)
print(fs.options.root_path)
```
**1.0**
```python
from lucidlink.fsspec import LucidLinkFileSystem
# Always ephemeral storage, removed on close();
# there is no persistent or custom location
fs = LucidLinkFileSystem(token="sa_live:...")
```
(fsspec-sync-mode)=
### sync_mode option replaced by sync_barrier
The `sync_mode` option is now `sync_barrier`. It takes `"on_unlink"` (the default: `close()` waits for pending changes), `"none"`, or a `SyncBarrier`. `SyncMode` is no longer importable from `lucidlink.fsspec`. A leftover `sync_mode`, for example in `storage_options`, is ignored with a `UserWarning`, and `close()` then waits for the upload.
**Beta**
```python
from lucidlink.fsspec import LucidLinkFileSystem, SyncMode
fs = LucidLinkFileSystem(token="sa_live:...", sync_mode=SyncMode.SYNC_NONE)
with fs.open("lucidlink://workspace/filespace/out/part-0.csv", "wb") as f:
f.write(b"id,value\n")
fs.sync_all()
fs.close()
```
**1.0**
```python
from lucidlink.fsspec import LucidLinkFileSystem
fs = LucidLinkFileSystem(token="sa_live:...", sync_barrier="none")
with fs.open("lucidlink://workspace/filespace/out/part-0.csv", "wb") as f:
f.write(b"id,value\n")
fs.sync_all()
fs.close()
```
(fsspec-info-names)=
### Paths without the protocol from ls() and info()
> **Silent:** old code still runs but behaves differently.
`ls()` returned names that started with `lucidlink://` when you passed a URL, and `info()` returned only the base name. Both now return the full path without the protocol, as `find()`, `glob()` and `walk()` already did and as other fsspec file systems do. These paths still work with this file system; use `fs.unstrip_protocol(name)` to get a URL. `ls()` on a file or link now returns that one entry, where Beta returned `[]`, so code that took an empty result to mean "not a directory" behaves differently.
**Beta**
```python
url = "lucidlink://workspace/filespace/reports"
for name in fs.ls(url, detail=False):
# "lucidlink://workspace/filespace/reports/q3.csv"
with fs.open(name, "rb") as f:
print(name, len(f.read()))
print(fs.info(f"{url}/q3.csv")["name"]) # "q3.csv"
```
**1.0**
```python
url = "lucidlink://workspace/filespace/reports"
for name in fs.ls(url, detail=False):
# "workspace/filespace/reports/q3.csv": no protocol,
# still a valid path for this file system
with fs.open(name, "rb") as f:
print(fs.unstrip_protocol(name), len(f.read()))
print(fs.info(f"{url}/q3.csv")["name"]) # "workspace/filespace/reports/q3.csv"
```
(fsspec-timestamps-seconds)=
### mtime and created in seconds
> **Silent:** old code still runs but behaves differently.
The `mtime` and `created` values from `ls()` and `info()` were integer nanoseconds since the epoch, with `0` for a filespace root. They are now float seconds, as fsspec expects, and `None` when unknown: remove any division by 1e9. `modified()` and `created()` now return a UTC `datetime`.
**Beta**
```python
from datetime import datetime, timezone
info = fs.info("lucidlink://workspace/filespace/reports/q3.csv")
# Nanoseconds since the epoch (an int)
modified = datetime.fromtimestamp(info["mtime"] / 1e9, tz=timezone.utc)
print(modified.isoformat())
```
**1.0**
```python
from datetime import datetime, timezone
info = fs.info("lucidlink://workspace/filespace/reports/q3.csv")
# Seconds since the epoch (a float), None when unknown
if info["mtime"] is not None:
modified = datetime.fromtimestamp(info["mtime"], tz=timezone.utc)
print(modified.isoformat())
# Or: fs.modified(url) returns a UTC datetime
```
(fsspec-link-type)=
### Symbolic links reported as "link"
> **Silent:** old code still runs but behaves differently.
`ls()` and `info()` reported a symbolic link as `"file"`. They now report `"link"` and add an `islink` key. A link at the end of a path is never followed, so `exists()`, `isfile()` and `isdir()` return `False` for it; use `lexists()` to test whether a link exists. `open()` and `cat_file()` on a link now raise `FsError` with `errno.ELOOP`, where Beta returned `b''`, so a loop that opens each path from `find()`, `ls()` or `walk()` fails on the first link.
**Beta**
```python
url = "lucidlink://workspace/filespace/current" # a symbolic link
info = fs.info(url)
print(info["type"]) # "file"
print(fs.exists(url), fs.isfile(url)) # True True
```
**1.0**
```python
url = "lucidlink://workspace/filespace/current" # a symbolic link
info = fs.info(url)
print(info["type"], info["islink"]) # "link" True
print(fs.exists(url), fs.isfile(url)) # False False
print(fs.lexists(url)) # True
```
(fsspec-existence-checks-raise)=
### Errors no longer hidden by exists(), isdir() and isfile()
`exists()`, `isdir()` and `isfile()` returned `False` for almost any error, such as a timeout, a network failure or a malformed URL. They now return `False` only for a missing path or a loop of symbolic links, and raise every other error. An unknown filespace name now gives `False` instead of `PermissionError`. An instance without a token now raises `InvalidArgumentError` instead of returning `False`, and a URL for another workspace raises `AccessDeniedError` instead of `PermissionError`; it is not an `OSError`, so neither `except PermissionError` nor `except OSError` catches it.
**Beta**
```python
url = "lucidlink://workspace/filespace/out/run-42"
# A network error or timeout also answered False
if not fs.exists(url):
fs.mkdir(url)
```
**1.0**
```python
from lucidlink.exceptions import OperationTimeoutError, ServiceUnavailableError
url = "lucidlink://workspace/filespace/out/run-42"
try:
if not fs.exists(url):
fs.mkdir(url)
except (ServiceUnavailableError, OperationTimeoutError):
print("LucidLink unreachable; retry later")
```
(fsspec-makedirs-existing)=
### FileExistsError from makedirs() for an existing directory
Without `exist_ok=True`, `makedirs()` returned silently when the directory already existed. It now raises `FsExistsError` (a `FileExistsError`), as `os.makedirs()` does: pass `exist_ok=True`. With `exist_ok=True`, a file at the path, or a create that fails, now raises instead of being ignored.
**Beta**
```python
out = "lucidlink://workspace/filespace/out"
fs.makedirs(out) # also fine when out/ already exists
with fs.open(f"{out}/part-0.csv", "wb") as f:
f.write(b"id,value\n")
```
**1.0**
```python
out = "lucidlink://workspace/filespace/out"
fs.makedirs(out, exist_ok=True)
with fs.open(f"{out}/part-0.csv", "wb") as f:
f.write(b"id,value\n")
```
(fsspec-cat-range)=
### Byte ranges through cat_file(), not positional cat() arguments
> **Silent:** old code still runs but behaves differently.
`cat()` is now fsspec's own, `cat(path, recursive=False, on_error="raise", **kwargs)`, and also accepts globs and lists. A byte range passed by position now lands in `recursive` and `on_error`, and the whole file is returned without an error. Use `cat_file(url, start=..., end=...)` for a byte range.
**Beta**
```python
header = fs.cat("lucidlink://workspace/filespace/video.mov", 0, 1024)
print(len(header))
```
**1.0**
```python
header = fs.cat_file("lucidlink://workspace/filespace/video.mov", start=0, end=1024)
print(len(header))
```
(fsspec-get-put)=
### fsspec copy rules for get() and put()
> **Silent:** old code still runs but behaves differently.
`get()` and `put()` are now fsspec's own. When the destination directory already exists, a source directory is copied into it as a subdirectory, like `cp -r`; add a trailing `/` to the source to copy only its contents. A directory without `recursive=True` is skipped silently instead of raising `ValueError`. Globs and lists of paths now work, and `put()` creates missing remote directories.
**Beta**
```python
import os
os.makedirs("local/reports", exist_ok=True)
# Copies the contents of reports/ into local/reports/
fs.get("lucidlink://workspace/filespace/reports", "local/reports", recursive=True)
# Copies the contents of local/upload/ into the existing incoming/
fs.put("local/upload", "lucidlink://workspace/filespace/incoming", recursive=True)
```
**1.0**
```python
import os
os.makedirs("local/reports", exist_ok=True)
# A trailing "/" on the source copies its contents, not the directory itself
fs.get("lucidlink://workspace/filespace/reports/", "local/reports", recursive=True)
fs.put("local/upload/", "lucidlink://workspace/filespace/incoming", recursive=True)
```
(fsspec-mv-semantics)=
### mv() into directories, over files and across filespaces
> **Silent:** old code still runs but behaves differently.
`mv()` onto an existing directory now moves the source into it, and onto an existing file it replaces that file, where Beta raised `FileExistsError`. Moving into a directory never replaces anything: when the directory already holds an entry of that name, `mv()` raises `FsExistsError` (a `FileExistsError`). A move between two filespaces copies the data and then deletes the source, where Beta raised `ValueError`. Check the destination first if you relied on the error. `mv()` takes one path at a time: a list raises `TypeError`.
**Beta**
```python
src = "lucidlink://workspace/filespace/inbox/q3.csv"
try:
fs.mv(src, "lucidlink://workspace/filespace/archive") # an existing directory
except FileExistsError:
fs.mv(src, "lucidlink://workspace/filespace/archive/q3.csv")
try:
fs.mv(
"lucidlink://workspace/filespace/archive/q3.csv",
"lucidlink://workspace/other-filespace/q3.csv",
)
except ValueError:
print("cannot move across filespaces")
```
**1.0**
```python
src = "lucidlink://workspace/filespace/inbox/q3.csv"
# Into an existing directory: becomes archive/q3.csv
fs.mv(src, "lucidlink://workspace/filespace/archive")
# Across filespaces: the data is copied, then the source deleted
fs.mv(
"lucidlink://workspace/filespace/archive/q3.csv",
"lucidlink://workspace/other-filespace/q3.csv",
)
```
(fsspec-walk-errors)=
### Listing errors raised by walk(), find(), glob() and du()
> **Silent:** old code still runs but behaves differently.
`walk()`, and so `find()`, `glob()` and `du()`, dropped any subtree that failed to list with an `OSError`, including timeouts and network failures, so they could return partial results. For an unknown or denied filespace, `walk()` returned nothing and the other three raised `PermissionError`. Now only a missing, non-directory or unreadable subtree is left out and every other error is raised. An unknown filespace name counts as missing: all four now return an empty result (`0` for `du()`), so a mistyped name no longer fails. An explicit `on_error` applies to the listing of the given path; Beta ignored it with fsspec older than 2023.9.0. Catch the errors you want to retry.
**Beta**
```python
# Any listing error (timeout, network, access) silently dropped that subtree
files = fs.find("lucidlink://workspace/filespace/datasets")
total = fs.du("lucidlink://workspace/filespace/datasets")
for root, dirs, names in fs.walk("lucidlink://workspace/filespace/datasets"):
print(root, len(names))
```
**1.0**
```python
from lucidlink.exceptions import OperationTimeoutError, ServiceUnavailableError
# Only a missing, non-directory or unreadable subtree is omitted;
# other errors are raised
try:
files = fs.find("lucidlink://workspace/filespace/datasets")
total = fs.du("lucidlink://workspace/filespace/datasets")
for root, dirs, names in fs.walk("lucidlink://workspace/filespace/datasets"):
print(root, len(names))
except (ServiceUnavailableError, OperationTimeoutError):
print("listing incomplete; retry later")
```
(fsspec-close-sync-errors)=
### Failures reported by close() and sync_all(); closed instances stay closed
`sync_all()` raised only `FilespaceInternalError` and dropped every other failure. It now raises a `LucidLinkErrorGroup` (an `ExceptionGroup`) with every filespace that failed: use `except*` or `group.exceptions`. `close()`, and leaving a `with` block normally, ignored errors and now raise a `LucidLinkErrorGroup` after closing everything; when the block is already raising, the failure is added as a note to that exception. A closed instance logged in again when used; now every call on it raises `ClientClosedError`, so create a new instance. fsspec reuses instances, so closing one, for example by leaving `with LucidLinkFileSystem(token=...)`, also closes the instance that `fsspec.filesystem("lucidlink", token=...)` returns elsewhere with the same arguments.
**Beta**
```python
from lucidlink import FilespaceInternalError
try:
fs.sync_all()
except FilespaceInternalError as e:
print(f"changes lost in filespace {e.filespace_id}")
fs.close() # never raised
print(fs.ls("lucidlink://workspace/filespace")) # logged in again
```
**1.0**
```python
from lucidlink.exceptions import FilespaceInternalError, LucidLinkErrorGroup
try:
fs.sync_all()
except* FilespaceInternalError as group:
for e in group.exceptions:
if isinstance(e, FilespaceInternalError):
print(f"changes lost in filespace {e.filespace_id}")
try:
fs.close()
except LucidLinkErrorGroup as group:
print(f"close failed: {group.exceptions}")
# A call on the closed fs raises ClientClosedError: create a new instance
fs = LucidLinkFileSystem(token="sa_live:...")
print(fs.ls("lucidlink://workspace/filespace"))
```
(fsspec-pickle-token)=
### No token in a pickled LucidLinkFileSystem
`storage_options`, `to_json()`, `to_dict()` and pickling no longer include the service account token. An instance rebuilt from them has no token, even in the same process, and neither has a copy from `copy.copy()` or `copy.deepcopy()`; its first call raises `InvalidArgumentError`. Set the `FSSPEC_LUCIDLINK_TOKEN` environment variable, which a worker process inherits: an instance rebuilt there reads the token from it, so passing a pickled instance to a worker keeps working. Or create the instance in the worker. Start such workers with the `spawn` method (see "No LucidLink calls in a forked process").
**Beta**
```python
from concurrent.futures import ProcessPoolExecutor
def size(fs: LucidLinkFileSystem, url: str) -> int:
return fs.size(url) # the pickled fs carried its token
if __name__ == "__main__":
with ProcessPoolExecutor() as pool:
print(pool.submit(size, fs, "lucidlink://workspace/filespace/a.bin").result())
```
**1.0**
```python
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
# A pickled fs carries no token: set FSSPEC_LUCIDLINK_TOKEN, which the worker
# inherits, and the instance rebuilt there reads the token from it
def size(fs: LucidLinkFileSystem, url: str) -> int | None:
return fs.size(url)
if __name__ == "__main__":
# spawn: see "No LucidLink calls in a forked process"
spawn = multiprocessing.get_context("spawn")
with ProcessPoolExecutor(mp_context=spawn) as pool:
print(pool.submit(size, fs, "lucidlink://workspace/filespace/a.bin").result())
```
(fsspec-internal-helpers)=
### Internal fsspec helpers removed
Beta exposed internal fsspec helpers because their modules were public: the `lucidlink.url_parser` module (`UrlParser`, `ParsedUrl`, `PROTOCOL`, `PROTOCOL_PREFIX`, `stringify_path`, `FSSPEC_AVAILABLE`) and `UrlParser`, `FilespaceCache`, `LucidLinkOptions`, `PROTOCOL` and `redacted_dataclass_repr` in `lucidlink.fsspec`. They have no replacement. For the protocol name, read `LucidLinkFileSystem.protocol` (still `"lucidlink"`); to split a URL, remove the `lucidlink://` prefix and split the rest at `/` (`urllib.parse` cuts a path at `#` or `?`).
**Beta**
```python
from lucidlink.url_parser import PROTOCOL, UrlParser
url = "lucidlink://workspace/filespace/reports/q3.csv"
assert url.startswith(f"{PROTOCOL}://")
parsed = UrlParser().parse(url)
print(parsed.workspace, parsed.filespace, parsed.path)
```
**1.0**
```python
from lucidlink.fsspec import LucidLinkFileSystem
url = "lucidlink://workspace/filespace/reports/q3.csv"
prefix = f"{LucidLinkFileSystem.protocol}://"
assert url.startswith(prefix)
workspace, filespace, path = url.removeprefix(prefix).split("/", 2)
print(workspace, filespace, "/" + path)
```
## Exceptions
(exceptions-import-location)=
### Exceptions imported from lucidlink.exceptions only
The `lucidlink` package and its other modules no longer re-export the exception classes. Import every exception from `lucidlink.exceptions`, which already held them in Beta; the class names are unchanged.
**Beta**
```python
from lucidlink import FilespaceInternalError, LucidLinkError
try:
data = filespace.fs.read_file("/reports/q3.csv")
except FilespaceInternalError as e:
print(f"filespace {e.filespace_id} was unlinked: link it again")
except LucidLinkError as e:
print(f"LucidLink error: {e}")
```
**1.0**
```python
from lucidlink import FsUtils
from lucidlink.exceptions import FilespaceInternalError, LucidLinkError
try:
data = FsUtils(filespace.file_system).read_bytes("/reports/q3.csv")
except FilespaceInternalError as e:
print(f"filespace {e.filespace_id} was unlinked: link it again")
except LucidLinkError as e:
print(f"LucidLink error: {e}")
```
(exceptions-builtin-classes)=
### LucidLinkError subclasses instead of RuntimeError and other builtins
Beta raised many errors as plain builtins: `RuntimeError` for most service, version, corruption and internal failures and for any error it could not classify, plus `NotImplementedError`, `AssertionError` and `SystemExit`. They now raise `LucidLinkError` or a subclass that is not that builtin, such as `InternalError`, `IncompatibleVersionError`, `UnsupportedOperationError` or `ClientError`: catch `LucidLinkError` or the specific class. File system errors keep their builtin base (`FileNotFoundError`, `PermissionError`, `TimeoutError`, `ConnectionError`, `ValueError`, ...) and now also set `errno` and `filename`, with three exceptions: a `KeyError`, for example from listing after an unsynced delete, is now `InternalError`; reading a linked S3 object raised `ConnectionError` and now raises `ExternalStoreError`; and moving a directory into its own subdirectory raised `FileNotFoundError` and now raises `FsError` with `errno.EINVAL`.
**Beta**
```python
try:
filespace = workspace.link_filespace(id="fs-uuid-12345")
except NotImplementedError as e:
print(f"not supported here: {e}")
except RuntimeError as e:
# version mismatch, corrupted metadata, cancelled link, ...
print(f"link failed: {e}")
```
**1.0**
```python
from lucidlink.exceptions import (
IncompatibleVersionError,
LucidLinkError,
UnsupportedOperationError,
)
try:
filespace = workspace.link_filespace("fs-uuid-12345")
except UnsupportedOperationError as e:
print(f"not supported here: {e}")
except IncompatibleVersionError as e:
print(f"upgrade the SDK: {e}")
except LucidLinkError as e:
print(f"link failed: {e}")
```
(exceptions-access-not-permissionerror)=
### Authentication and access errors no longer PermissionError
In fsspec, every URL whose workspace or filespace could not be used raised `PermissionError`, including rejected credentials and denied access; so did a rejected session or denied access reported by a later call, such as a file operation (`login()` and `link_filespace()` raised other classes). These now raise `AuthenticationError` or `AccessDeniedError`, neither an `OSError`, so data libraries do not mistake them for a missing path. An unknown filespace name now raises `FsNotFoundError` (a `FileNotFoundError`), a malformed URL `InvalidArgumentError`, and other link failures their own class. A denied path is still `FsPermissionError`, which is also a `PermissionError`.
**Beta**
```python
try:
names = fs.ls("lucidlink://workspace/filespace/reports")
except PermissionError:
# the service account has no access to this filespace
print("ask an admin for access")
```
**1.0**
```python
from lucidlink.exceptions import AccessDeniedError, AuthenticationError
try:
names = fs.ls("lucidlink://workspace/filespace/reports")
except (AuthenticationError, AccessDeniedError):
# the service account has no access to this filespace
print("ask an admin for access")
```
(exceptions-filespace-id-none)=
### None for an unknown filespace_id
> **Silent:** old code still runs but behaves differently.
`FilespaceInternalError.filespace_id` and `FilespaceAlreadyLinkedError.filespace_id` were an empty string when the filespace was unknown. Every `FilespaceError` now has `filespace_id`, and it is `None` when the filespace is unknown: compare with `None` instead of `""`.
**Beta**
```python
from lucidlink import FilespaceInternalError
try:
filespace.sync_all()
except FilespaceInternalError as e:
failed = e.filespace_id if e.filespace_id != "" else ""
print(f"filespace {failed} must be linked again")
```
**1.0**
```python
from lucidlink.exceptions import FilespaceInternalError
try:
filespace.sync()
except FilespaceInternalError as e:
failed = e.filespace_id if e.filespace_id is not None else ""
print(f"filespace {failed} must be linked again")
```
(exceptions-configuration-daemon-removed)=
### ConfigurationError and DaemonError removed
`ConfigurationError` is removed; Beta never raised it, and an invalid configuration raised a plain `ValueError`, now `InvalidArgumentError`, which is also a `ValueError`. `DaemonError` came only from the removed `Daemon` class. A `Client` start-up failure, such as an unusable storage root, raised `ClientError`; it now raises the `LucidLinkError` subclass for the cause, often a plain `LucidLinkError`. `ClientError` now means misuse, such as logging in again with other credentials or using a client in a forked process.
**Beta**
```python
from lucidlink import (
Client,
ClientError,
ConfigurationError,
DaemonError,
ServiceAccountCredentials,
)
def start(token: str) -> Client:
try:
client = Client()
client.login(ServiceAccountCredentials(token=token))
except ConfigurationError as e:
raise SystemExit(f"invalid configuration: {e}")
except (DaemonError, ClientError) as e:
raise SystemExit(f"client failed to start: {e}")
return client
```
**1.0**
```python
from lucidlink import Client, ServiceAccountCredentials
from lucidlink.exceptions import InvalidArgumentError, LucidLinkError
def start(token: str) -> Client:
try:
client = Client()
client.login(ServiceAccountCredentials(token=token))
except InvalidArgumentError as e:
raise SystemExit(f"invalid configuration: {e}")
except LucidLinkError as e:
raise SystemExit(f"client failed to start or log in: {e}")
return client
```