06RESEARCH PAPERPreprint
Per-Channel Quantile Fullgrid Probing for Small Language Model Observation
發表日期
摘要
ABSTRACTZenodo 原文We describe a dense activation probing protocol for small to mid-sized language models that operates within a fixed cell budget while maintaining high observation coverage. The protocol combines per-channel quantile binning, an expanded calibration probe corpus (P=1200), and an additive shard merge step that allows heterogeneous compute resources to contribute. Across thirteen runs spanning seven model families and parameter counts from 124M to 7.6B, the protocol reaches at least 90.6 percent coverage on every run, with a median of 94.95 percent and a maximum of 97.05 percent. Per-channel binning combined with a larger probe budget was associated with higher coverage compared to earlier pooled-bin, smaller-corpus configurations on the same models, but the isolated contribution of each factor remains untested in this draft. A controlled 2x2 ablation is the recommended next experiment. The pipeline is organized for reproducibility and can be released alongside the run artifacts. It runs on commodity hardware.
全文 PDF
FULL TEXT1 個檔- 下載 PDFpaper-3-perchannel-quantile.pdf・68 KB・本站
PDF 與 Zenodo 上的檔案相同(大小與 md5 已核對)。
論文資料
DETAILSZenodo 紀錄 20448674- 作者
- Chen, Ho Yiing
- 發表日期
- 資源類型
- Preprint
- DOI
- 10.5281/zenodo.20448674
- Zenodo
- zenodo.org/records/20448674
- 授權
- Creative Commons Attribution 4.0 International
- 版本
- v1
作者 Chen, Ho Yiing 是織田紀香的本名(陳禾穎)英文拼寫。標題、摘要、關鍵字與授權照 Zenodo 原文。
關鍵字
KEYWORDS23 個- LLM interpretability
- mechanistic interpretability
- dense activation probing
- per-channel quantiles
- fullgrid
- shard merge
- additive heat
- distributed observation
- heterogeneous compute
- coverage metric
- Tier-A
- Tier-B
- BLOOM
- Qwen2.5
- Phi-2
- Pythia
- InternLM-2
- GPT-2
- Mercury
- consumer hardware
- open source
- independent research
- reproducibility