NORIKA ODA

06RESEARCH PAPERPreprint

Per-Channel Quantile Fullgrid Probing for Small Language Model Observation

Chen, Ho Yiing

發表日期

DOI 10.5281/zenodo.20448674 Zenodo 紀錄頁 PDF(68 KB)

01

摘要

ABSTRACTZenodo 原文
We describe a dense activation probing protocol for small to mid-sized language models that operates within a fixed cell budget while maintaining high observation coverage. The protocol combines per-channel quantile binning, an expanded calibration probe corpus (P=1200), and an additive shard merge step that allows heterogeneous compute resources to contribute. Across thirteen runs spanning seven model families and parameter counts from 124M to 7.6B, the protocol reaches at least 90.6 percent coverage on every run, with a median of 94.95 percent and a maximum of 97.05 percent. Per-channel binning combined with a larger probe budget was associated with higher coverage compared to earlier pooled-bin, smaller-corpus configurations on the same models, but the isolated contribution of each factor remains untested in this draft. A controlled 2x2 ablation is the recommended next experiment. The pipeline is organized for reproducibility and can be released alongside the run artifacts. It runs on commodity hardware.
02

全文 PDF

FULL TEXT1 個檔
  • 下載 PDFpaper-3-perchannel-quantile.pdf・68 KB・本站

PDF 與 Zenodo 上的檔案相同(大小與 md5 已核對)。

03

論文資料

DETAILSZenodo 紀錄 20448674
作者
Chen, Ho Yiing
發表日期
資源類型
Preprint
DOI
10.5281/zenodo.20448674
Zenodo
zenodo.org/records/20448674
授權
Creative Commons Attribution 4.0 International
版本
v1

作者 Chen, Ho Yiing 是織田紀香的本名(陳禾穎)英文拼寫。標題、摘要、關鍵字與授權照 Zenodo 原文。

04

關鍵字

KEYWORDS23 個
  • LLM interpretability
  • mechanistic interpretability
  • dense activation probing
  • per-channel quantiles
  • fullgrid
  • shard merge
  • additive heat
  • distributed observation
  • heterogeneous compute
  • coverage metric
  • Tier-A
  • Tier-B
  • BLOOM
  • Qwen2.5
  • Phi-2
  • Pythia
  • InternLM-2
  • GPT-2
  • Mercury
  • consumer hardware
  • open source
  • independent research
  • reproducibility
05

其他研究論文

MORE