File Classification at Scale: Our Accuracy Benchmark
40,000 files. Here's what we found.
Methodology
We assembled a benchmark dataset of 40,000 files drawn from a variety of real-world sources: personal file collections donated by beta testers, public document datasets, open-source code repositories, Creative Commons media archives, and synthetic data generated to fill gaps in rare categories. Each file was manually reviewed and assigned a ground-truth label from our taxonomy before being run through the classifier. The classifier had no access to the labels during inference. We measured top-1 accuracy (the classifier's first prediction matches the ground truth) and top-3 accuracy (the ground truth appears in the classifier's top 3 predictions).
Dataset breakdown
The 40,000 files span the following broad categories. Numbers are approximate — the dataset was assembled to reflect realistic file collection distributions rather than equal class balance.
Results
The classifier correctly identified the top-level category in 91% of cases on the first prediction. In 97% of cases, the correct category appeared somewhere in the top 3 predictions. These numbers hold across file types — the lowest-performing categories were ambiguous by design (e.g., a plain text file that could be either a README or a personal note), while structured file types like CSV and JSON achieved near-perfect scores.
Error analysis
The 9% of top-1 misclassifications break into three broad patterns. First, genuine ambiguity: a Markdown file could be documentation, a personal journal entry, or a technical note — these are legitimately multi-category, and the classifier's second prediction usually captures the correct label. Second, sparse content: very short files (under 100 tokens) give the model little signal, and classification accuracy drops significantly for files under this threshold. Third, domain-specific vocabulary: highly specialized documents (e.g., legal contracts in unusual jurisdictions, scientific papers in niche fields) occasionally map to adjacent but incorrect categories. We are addressing the third class with additional fine-tuning data in specialist domains.
How users can improve it
embly exposes corrections as a first-class feature. When the classifier assigns the wrong category, you can correct it — and that correction is stored locally and used to weight future predictions for similar files. Over time, the system adapts to your specific file collection and domain vocabulary. We also aggregate anonymised correction signals (with user consent) to improve the global model. The benchmark numbers above represent the out-of-box baseline; in practice, active users see higher effective accuracy as the model adapts to their corpus.