Skip to content

bagof.paths.Path

Path(path: Any, *, driver: Any = None, storage_options: Mapping[str, Any] | None = None)

Bases: PurePathMixin, LocalCopyMixin, BaseWrapper

A path that works like pathlib.Path, local or in the cloud.

Make one from a path string, a URL, or any path object:

>>> from bagof.paths import Path
>>> p = Path("/data/sets/train.zarr")
>>> p.name
'train.zarr'
>>> p.parent
Path('/data/sets')
>>> (p / "chunks").suffix
''

Each method uses the underlying library when it can, builds the method from simpler ones when it cannot, or raises UnsupportedPathOperation when neither is possible.

Methods:

exists

exists(*, follow_symlinks: bool = True) -> bool

Whether the path exists.

On a local path this follows Python 3.14 on every version: an error returns False rather than raising. A name too long for the file system is not there, and neither is one whose parent you may not search. To see the error, call stat(), which still raises. The other status checks (is_file, is_dir, is_symlink, ...) do the same.

follow_symlinks=False (also on is_file and is_dir) asks about a symbolic link itself, on every Python version.

>>> Path("a" * 300).exists()
False

is_file

is_file(*, follow_symlinks: bool = True) -> bool

Whether the path is a regular file.

Like exists, it returns False on an error on a local path.

is_dir

is_dir(*, follow_symlinks: bool = True) -> bool

Whether the path is a directory.

Like exists, it returns False on an error on a local path.

is_symlink() -> bool

Whether the path is a symbolic link.

stat

stat(*, follow_symlinks: bool = True) -> stat_result

The result of stat on the path.

lstat

lstat() -> stat_result

Like stat, without following symbolic links.

samefile

samefile(other: Any, *, strict: bool = True) -> bool

Whether the path and other refer to the same file.

As in pathlib, both files must exist. With strict=False they need not: the answer is whether writing one would write the other, which is what a writer checks before it replaces the file it reads.

>>> import tempfile
>>> from bagof.paths import Path
>>> folder = Path(tempfile.mkdtemp())
>>> (folder / "sub").mkdir()
>>> new = folder / "new.bin"
>>> new.samefile(folder / "sub" / ".." / "new.bin", strict=False)
True
>>> new.samefile(folder / "other.bin", strict=False)
False

A local path resolves symbolic links and .., and a hard link to an existing file is the same file. A name that does not exist yet compares as it is written below its nearest existing directory, so on a case-insensitive file system (macOS) two spellings of a new name count as different. A remote path is the same file as another remote path when the two are equal, and never the same as a local one.

is_mount

is_mount() -> bool

Whether the path is a mount point.

is_socket

is_socket() -> bool

Whether the path is a Unix domain socket.

is_fifo

is_fifo() -> bool

Whether the path is a FIFO (named pipe).

is_block_device

is_block_device() -> bool

Whether the path is a block device.

is_char_device

is_char_device() -> bool

Whether the path is a character device.

is_junction

is_junction() -> bool

Whether the path is a junction (a Windows concept; else False).

open

open(mode: str = 'r', buffering: int = -1, encoding: str | None = None, errors: str | None = None, newline: str | None = None, *, atomic: bool = False) -> IO[Any]

Open the path and return a file object, like open.

With atomic=True (a writing mode only) the file is written in a staging area and moved onto the path when it is closed, so readers see the old content or the new, never a half-written file; leaving a with block with an error discards it. An appending or r+ mode starts from a copy of the current content. Like the plain open, it refuses a directory and its errors name this path; if it fails, closing included, it leaves nothing behind and raises the error as the plain open would. See staged.

read_bytes

read_bytes() -> bytes

Read the whole file as bytes.

read_text

read_text(encoding: str | None = None, errors: str | None = None, newline: str | None = None) -> str

Read the whole file as text.

write_bytes

write_bytes(data: Any, *, atomic: bool = False) -> int

Write data to the file as bytes, replacing any content.

With atomic=True the new content is written beside the file and moved over it in one step, so readers never see a partial file (see staged). Like the plain write, it refuses a directory and its errors name this path; if it fails, it leaves nothing behind and raises the error as the plain write would.

write_text

write_text(data: str, encoding: str | None = None, errors: str | None = None, newline: str | None = None, *, atomic: bool = False) -> int

Write data to the file as text, replacing any content.

atomic=True works as in write_bytes.

staged

staged(*, copy: bool = False, replace_type: bool = False, keep_on_failure: bool = False) -> Stage

A staging area for a new version of this path.

Create a file or a directory at the staged path; on leaving the with block it replaces this path. An error inside the block leaves this path as it was, and staging nothing changes nothing.

>>> import tempfile
>>> from bagof.paths import Path
>>> target = Path(tempfile.mkdtemp()) / "scan.txt"
>>> with target.staged() as tmp:
...     tmp.name == target.name
...     _ = tmp.write_text("new")
...     target.exists()
True
False
>>> target.read_text()
'new'

The staged path has the same name as this one, so a library that saves by file name (and reads the extension) works on it. It does not exist until your code creates it, as a file or a directory:

with Path("out/scan.nii.gz").staged() as tmp:
    nibabel.save(image, tmp)

with Path("s3://bucket/store.zarr").staged() as tmp:
    zarr.save(tmp, data)       # a local directory, uploaded on exit

Staging nothing is not an error. When a library may fail quietly or write somewhere else, check that it made something before leaving the block; raising there leaves this path as it was:

with Path("out/scan.nii.gz").staged() as tmp:
    nibabel.save(image, tmp)
    if not tmp.exists():
        raise FileNotFoundError(f"nothing was written to {tmp.name}")

copy=True starts the staged version as a copy of this path (a file, or a whole directory), for jobs that update it in place: zarr in "r+" mode, h5py in "a" mode, appending. On a remote path that is a full download. If this path does not exist, nothing is copied.

Outside a with block, call commit() or discard() on the returned Stage; its path is the staged path.

A file never replaces a directory, nor a directory a file, unless you pass replace_type=True: as with pathlib, committing raises CommitError with the errno of IsADirectoryError or NotADirectoryError, and this path is left as it was. Pass it when the change is the new version you mean, such as a zarr store moving from a zip file to a directory.

What committing guarantees:

  • A file replaces a local file in one atomic step on POSIX (readers that have the old file open, even memory-mapped, keep reading it), and a remote object in one upload.
  • A directory, or a change between file and directory (with replace_type=True), is all-or-nothing but not atomic: the old version is moved aside and the new one moved in, so it is briefly missing. On a remote path it is uploaded file by file, and readers can see a mixture meanwhile.
  • An error inside the with block leaves this path as it was and removes the staged data.
  • If committing fails, this path is left as it was (or restored), the staged data is removed, and CommitError is raised. With keep_on_failure=True the data is kept instead, for work that is costly to redo: the error says where, and the returned Stage can commit() again or discard(). The atomic=True writers raise the original error rather than CommitError, as the plain writers would, and refuse a directory at this path, as the plain writers do.
  • A failure that leaves this path changed is the exception: a remote directory uploaded in part, or an old directory that could not be moved back. Then nothing is removed, whatever the options: the staged data is kept, and the old directory is left in a hidden directory beside this path, never removed; CommitError says where both are.

An error that leaves the with block names this path, not the staged one, which is gone by then: a path below or beside the staged path is named below or beside this one. It is the same error, with its filename changed. Inside the block, before it leaves, it still names the staged path your code wrote.

>>> import tempfile
>>> from bagof.paths import Path
>>> target = Path(tempfile.mkdtemp()) / "store.zarr"
>>> try:
...     with target.staged() as tmp:
...         (tmp / "missing" / "chunk").write_bytes(b"x")
... except FileNotFoundError as error:
...     error.filename == str(target / "missing" / "chunk")
True

The staging area is a hidden directory beside this path (beside the file a symbolic link names), so the move stays on one file system; a remote path is staged in a local temporary directory. While a directory is replaced, the old one waits in a second hidden directory beside this path. A crash can leave either behind. Replacing a file gives it a new inode, so other hard links keep the old content; the old file's permissions are kept. On Windows, the replacement fails while another process has the file open.

iterdir

iterdir() -> Iterator[Self]

Yield the paths of the directory's entries.

glob

glob(pattern: str, *, case_sensitive: bool | None = None, recurse_symlinks: bool = False) -> Iterator[Self]

Yield the paths matching pattern under this directory.

rglob

rglob(pattern: str, *, case_sensitive: bool | None = None, recurse_symlinks: bool = False) -> Iterator[Self]

Like glob, recursively.

mkdir

mkdir(mode: int = 511, parents: bool = False, exist_ok: bool = False) -> None

Create a directory at the path.

touch

touch(mode: int = 438, exist_ok: bool = True) -> None

Create the file at the path, or update its modification time.

unlink(*, missing_ok: bool = False) -> None

Remove the file at the path.

rmdir

rmdir(*, recursive: bool = False) -> None

Remove the directory at the path.

With recursive=True the whole tree is removed. The default is non-recursive and stays safe even on drivers whose own rmdir recurses by default (universal-pathlib).

copy

copy(target: Any, *, follow_symlinks: bool = True, preserve_metadata: bool = False) -> Self

Copy this file or directory to target; return the new path.

copy_into

copy_into(target_dir: Any, *, follow_symlinks: bool = True, preserve_metadata: bool = False) -> Self

Copy into target_dir, keeping this path's name.

Within the same store the copy keeps the store's own name for the object, query and all, so two objects whose names differ only after a ? stay apart. Into another store it takes name, which leaves the query out.

move

move(target: Any) -> Self

Move this path to target; return the new path.

move_into

move_into(target_dir: Any) -> Self

Move into target_dir, keeping this path's name.

The name follows the same rule as in copy_into.

walk

walk(top_down: bool = True, on_error: Callable | None = None, follow_symlinks: bool = False) -> Iterator[tuple[Self, list[str], list[str]]]

Walk the tree, yielding (path, dirnames, filenames) per dir.

resolve

resolve(strict: bool = False) -> Self

The absolute path, with symlinks resolved.

absolute

absolute() -> Self

The absolute path, without resolving symlinks.

expanduser

expanduser() -> Self

The path with a leading ~ expanded.

readlink() -> Self

The path a symbolic link points to.

rename

rename(target: Any) -> Self

Rename the path to target and return the new path.

replace

replace(target: Any) -> Self

Rename the path to target, replacing any existing file.

chmod

chmod(mode: int, *, follow_symlinks: bool = True) -> None

Change the file mode and permission bits.

lchmod

lchmod(mode: int) -> None

Like chmod, without following symbolic links.

owner

owner(*, follow_symlinks: bool = True) -> str

The login name of the file's owner.

group

group(*, follow_symlinks: bool = True) -> str

The group name of the file.

symlink_to(target: Any, target_is_directory: bool = False) -> None

Make this path a symbolic link to target.

hardlink_to(target: Any) -> None

Make this path a hard link to target.

link_to(target: Any) -> None

Make target a hard link to this path.

.. deprecated:: link_to takes the reverse argument order of hardlink_to and was removed from pathlib in Python 3.12. Prefer hardlink_to; this is kept, and synthesized where the driver dropped it, only for backward compatibility.

as_url

as_url(**kwargs: Any) -> str

A URL for the path; keyword arguments pass to the driver.

download_to

download_to(destination: Any) -> Any

Download the path's contents to a local destination.

upload_from

upload_from(source: Any, **kwargs: Any) -> Any

Upload a local source to the path.

clear_cache

clear_cache() -> None

Discard any locally cached copy of the path (cloudpathlib).

rmtree

rmtree() -> None

Remove the directory tree at the path. Alias of rmdir.

copytree

copytree(target: Any, *, follow_symlinks: bool = True, preserve_metadata: bool = False) -> Self

Copy a directory tree to target. Alias of copy.

bagof.paths.Stage

Stage(target: Any, *, copy: bool = False, replace_type: bool = False, keep_on_failure: bool = False)

Bases: _StageBase

A staging area for a new version of a path.

Returned by Path.staged; use it as a context manager, or call commit or discard yourself. The staging directory exists from the moment the stage is made.

Attributes

path property

path: Any

Where to create the new version: named like the target.

A local path, even for a remote target. It does not exist until your code creates a file or a directory there (or copy=True started it as a copy of the target).

Methods:

commit

commit() -> None

Move what was created at path onto the target.

Nothing created means nothing changes. If the move or the upload fails, the target is left as it was (or restored), the staged data is removed, and CommitError is raised. A stage made with keep_on_failure=True keeps the data instead and the error says where; commit may then be called again, or discard removes it.

discard

discard() -> None

Remove the staged version; the target is never touched.

Discarding after a commit does nothing.