Command Palette
Search for a command to run
GuideCLI referenceAPI reference

Managing Large Files

How Outpost handles large binary files natively — no LFS, no plugins, no file size limits.

Outpost handles files of any size natively. There are no file size limits, no LFS configuration, and no external plugins required. A 50GB model checkpoint is versioned the same way as a 2KB Python script.

This is possible because the Outpost SCM engine uses content-addressed storage with SHA2 hashing and deduplication, rather than the diff-based approach of traditional Git.

How large files work

When you push a file to Outpost:

  1. The file is hashed with SHA2 and broken into content-addressed chunks.
  2. Only chunks that don't already exist on the server are uploaded.
  3. A pointer in the Merkle tree references the file by its hash.

This means:

  • No file size limits — push model checkpoints, video datasets, sensor data of any size.
  • Deduplication — if the same file appears in multiple commits, branches, or repositories, it's stored once.
  • Delta transfers — change one layer in a 50GB model, push again, and only the changed chunks are transferred.

Working with large files

There's nothing special you need to do. The standard workflow handles everything:

# Add a large model checkpoint outpost add model.safetensors # 47 GB # Add a dataset outpost add dataset/ # 30 GB of images # Commit and push outpost commit -m "add trained model and dataset" outpost push
Tip
Use `outpost status` to see which files are staged, modified, or untracked — including their sizes.

Partial and selective clones

For repositories with very large data, you don't need to download everything:

# Clone only the latest commit (skip full history) outpost clone --depth 1 https://outpost.run/my-namespace/my-repo # Clone only a specific directory outpost clone --filter data/processed https://outpost.run/my-namespace/my-repo # Clone in remote mode (metadata only, fetch files on demand) outpost clone --remote https://outpost.run/my-namespace/my-repo

Downloading specific files

You can download individual files without cloning the entire repository:

# Download a specific file outpost download my-namespace/my-repo model.safetensors # Download from a specific branch or commit outpost download my-namespace/my-repo model.safetensors --revision experiment-v2 # Save to a specific location outpost download my-namespace/my-repo model.safetensors --output ./models/

Storage backends

By default, Outpost stores content on its managed object store. You can configure alternative storage backends:

# Use local storage outpost config --storage-backend local --storage-backend-path /data/outpost # Use S3 outpost config --storage-backend s3 --storage-backend-bucket my-bucket --storage-backend-path prefix/

Storage backends can be configured per-repository during outpost init or globally via outpost config.

Best practices

  • Use outpost clone --depth 1 in CI/CD — download only the latest commit to avoid fetching full history.
  • Use outpost clone --remote for exploration — browse metadata and download files on demand without pulling everything.
  • Use outpost clone --filter for selective access — only download the directories you need.
  • Organize consistently — use directories like models/, data/, code/ so team members can clone selectively.

Next steps

Previous → Branching & Merging

Next Overview →