JFrog Artifactory Adds Xet Protocol Support: How Enterprise Model Distribution Finally Scales

Transferring a fine-tuned 50GB model that shares 95% of its weights with the base has always required pulling all 50GB again. Xet changes that. JFrog Artifactory now implements it natively.

What Happened

JFrog Artifactory now natively supports the Xet protocol — a content-addressable storage (CAS) system developed by Hugging Face to replace Git LFS for AI model distribution at scale. The integration extends Artifactory’s existing checksum-based deduplication to Hugging Face repositories through chunk-level deduplication and intelligent caching.

Xet’s core mechanism splits model files into content-defined chunks grouped into xorb bundles of up to 64 MB each. Rather than treating a model as a monolithic file, Artifactory stores only the chunks that represent genuinely new content. When a fine-tuned variant of a base model is pulled, only the modified chunks — the 5% that changed — transfer. The 95% that’s identical to what’s already cached never moves.

For enterprises where Artifactory proxies upstream Hugging Face for 500+ engineers, the practical effect is that the upstream sees Artifactory as a single rate-limited client rather than 500 simultaneous pull requests — a significant operational benefit for gated models like Llama, Gemma, and Mistral.

Why It Matters

The scale problem in enterprise AI distribution is different from open-source package management. A JavaScript package is kilobytes; a frontier model is tens or hundreds of gigabytes. Fine-tuning produces a derivative model that’s structurally similar to its parent — but Git LFS, the tool most teams inherited for large file management, treats every version as a complete object. Pull a fine-tuned Llama 3 variant: Git LFS fetches the full 70B parameter file again, even if 95% of the weights are identical.

Xet’s content-defined chunking is a fundamentally different approach. It looks at what actually changed, not what file changed. JFrog’s implementation extends this with usage-shaped caching: Artifactory stores only the xorb ranges clients actually request, not complete bundles. Storage growth becomes proportional to genuinely new content rather than to the total volume of model variants deployed.

For platform teams that have already adopted JFrog’s artifact governance for coding agent plugins and supply chain security governance, native Xet support closes the model-distribution gap in the AI artifact pipeline. Model weights now have the same governance, caching, and deduplication infrastructure that packages have had for years.

What This Means for Platform Engineers

Three immediate operational wins for platform teams managing enterprise ML infrastructure:

Storage efficiency: Fine-tuning programs that iterate through model variants no longer accumulate linear storage costs. Only genuinely new weights are stored; variant counts decouple from storage growth.

Production continuity: Teams can maintain local Artifactory caches that provide production continuity independent of upstream Hugging Face availability — relevant for any team with SLA requirements on model-serving infrastructure.

Access governance for gated models: Gated models (Llama, Gemma, Mistral) require individual developer authentication at Hugging Face. Proxying through Artifactory with centralized credentials and access controls removes that authentication burden from individual engineers and centralizes audit logging.

Source: JFrog Blog

Content created with AI assistance and reviewed for accuracy.