Megfile

Overview

Guide to using megfile’s unified APIs, SmartPath, configuration, and CLI for file operations across various backends, like local FS, S3/OSS-compatible object storage, SFTP, WebDAV, HTTP, HDFS, and stdio.

Quick Start

• Install base: pip install megfile; add extras per backend (megfile[cli], megfile[hdfs], megfile[webdav]).
• Configure credentials/endpoints (env vars > config files). S3 examples: AWSACCESSKEYID, AWSSECRETACCESSKEY, optional AWSENDPOINTURL / OSSENDPOINT / AWSENDPOINTURLS3, AWSS3ADDRESSING_STYLE.
• Path format: protocol://bucket/key or alias (e.g., tos://); bare POSIX paths are treated as file://.
• Import functional APIs (from megfile import smartopen, smartsync, ...) or SmartPath (from megfile.smart_path import SmartPath).

Supported Protocols & Extras

• Local FS (file:// or bare paths) — base install.
• S3/OSS-compatible (s3://, plus aliases) — base install.
• SFTP (sftp://) — install megfile[cli] or megfile with SFTP deps.
• HTTP/HTTPS (http://, https://) — base install.
• Stdio (stdio://) — base install.
• HDFS (hdfs://) — install megfile[hdfs].
• WebDAV (webdav://) — install megfile[webdav].
• Full Protocol path format reference: references/path_format.md.

Core Tasks

File IO

• smart_open(path, mode='r', encoding=None, **options): open binary/text handles.
• Convenience loaders/savers: smartloadcontent / smartsavecontent (bytes), smartloadtext / smartsavetext (str), smartsaveas(fileobj, path), smartload_from(path) returns BinaryIO.
• smartcombineopen(glob, mode='rb', openfunc=smartopen): sequentially reads multiple files as one stream.

Existence & Metadata

• smartexists, smartisfile, smartisdir, smartislink, smart_isabs.
• smart_access(path, mode=Access.READ/WRITE).
• smartstat / smartlstat (via SmartPath), smartgetsize, smartgetmtime, smart_getmd5(recalculate=False, followlinks=False).

Listing & Globbing

• Directory traversal: smartlistdir, smartscandir, smartwalk, smartscan, smartscanstat (returns FileEntry/StatResult).
• Pattern matching: smartglob, smartiglob, smartglobstat for stat-rich globbing.

Data Transfer & Lifecycle

• Copy/sync: smartcopy, smartsync, smartsyncwithprogress (progress-friendly wrapper), smartconcat to merge multiple sources.
• Moves/deletes: smartmove, smartrename, smartremove, smartunlink, smarttouch, smartmakedirs.
• Links: smartsymlink, smartreadlink.

Path Utilities

• smartpathjoin, smartabspath, smartrealpath, smartrelpath, smartisabs.
• SmartPath mirrors pathlib semantics but routes to the right backend: path = SmartPath("s3://bucket/key"); path.exists(); path.open(mode="rb").

Caching

• smart_cache(path, cacher=SmartCacher, **options): cache remote resources locally for tools that only support local files.

Configuration

• Use CLI helpers to persist credentials/endpoints; environment variables take precedence.
• Profiles enable multiple endpoints (e.g., s3+prod://...). See references/configuration/ for protocol-specific flags and env vars.
• Full config reference: references/configuration/.

CLI Essentials

• Install CLI extras: pip install 'megfile[cli]'.
• Common commands (ls/cp/sync/stat/md5sum/mkdir/rm/touch) mirror POSIX semantics across backends.
• Completion scripts: megfile completion zsh.
• Full command list and flags: references/cli.md.

Usage Notes

• Prefer smart_* for protocol-agnostic code paths; avoid branching per backend.
• Ensure required extras are installed for target protocols before invoking APIs.
• For high-volume sync/copy, supply map_func (e.g., ThreadPoolExecutor.map) and callback to report progress.
• Use aliases via megfile config alias <alias> <protocol> to shorten paths (e.g., tos://).

References

• API surface: references/megfile.smart.md and references/megfile.smart_path.md.
• Configuration flags, env vars, and profiles: references/configuration/
• CLI commands and flags: references/cli.md
• Full Protocol path format reference: references/path_format.md.
• Glob patterns reference: references/advanced/glob.md.