Notion LLM Config Table Management

Database Information

IDs are stored in ~/.config/notion/config.json:

• Database ID: database_id key
• Data Source ID: datasourceid key
• Location: Family Notes workspace → "LLM config table"
• Purpose: Track LLM model configurations across multiple providers (Llama, GPT-2, Qwen, others)

Data Sources

Llama Models

Primary source: ~/projects/github/llama-models/models/sku_list.py

• Contains all registered Llama models with architecture parameters
• Constants: LLAMA2VOCABSIZE = 32000, LLAMA3VOCABSIZE = 128256
• WARNING: skulist.py has incorrect ffndim_multiplier values for Llama 2 7B and 13B models

- skulist.py shows 1.3 for all Llama 2 models, but actual checkpoints differ - Always verify dff values against HuggingFace configs, not just sku_list.py

GPT-2 Models

Primary source: HuggingFace config.json files

• Base repos: openai-community/gpt2, gpt2-medium, gpt2-large, gpt2-xl
• All variants: vocabsize=50257, maxcontext=1024, uses BPE tokenizer
• Uses learned positional embeddings (not RoPE)

User Preferences

1. No emojis in any updates or content
2. Batch operations preferred over sequential updates
3. Focus on practical utility - add "Main" column for filtering to representative models
4. Verify calculations - user will catch errors, so double-check formulas

Database Schema Key Columns

Identity

• Model Name (title), Model Family, Model Type, Core Model ID
• HuggingFace Repo, Variant

Architecture Parameters

• dhidden = model hidden dimension (dim from Llama archargs)
• d_ff = feed-forward hidden dimension (see calculation formulas below)
• dff / dhidden Ratio = dff / dhidden, rounded to 3 decimals

- Llama 2: 2.688 (7B), 2.7 (13B), 3.5 (70B) - Llama 3+: 2.667 (3B), 3.125 (Guard INT4), 3.25 (405B), 3.5 (most common), 4.0 (1B) - GPT-2: 4.0 (all variants - standard transformer) - Null for models without dff (Llama 4 has empty archargs)

• Gated MLP (checkbox) = gated MLP/activation (SwiGLU for Llama, standard GELU for GPT-2)
• nlayers, nheads, nkvheads, head_dim
• Multiple Of, Norm Eps

Tokenization

• Tokenizer: Llama 2 Tokenizer (32k vocab) or Llama 3 Tokenizer (128k vocab)
• Vocab Size: 32000 (Llama 2), 128256 (Llama 3+)

MoE Architecture

• Is MoE (checkbox)
• Num Experts: Llama 4: 16 (Scout), 128 (Maverick)
• Top K Experts: Number of routed experts per token
• MoE Routing: "Token Choice" (Llama 4) vs "Expert Choice"
• Has Shared Expert: Llama 4 has 1 shared + 1 routed per token
• Activated Params (B): Llama 4: 17B
• Total Params (B): Llama 4: 109B (Scout), 400B (Maverick)

RoPE Configuration

• RoPE Theta, RoPE Freq Base
• Use Scaled RoPE (checkbox)

Other

• Quantization Format, PTH File Count, Max Context Size
• Main (checkbox) - marks representative models for filtering

Update Patterns

Property Updates (Preferred)

mcp__notion__notion-update-page({
    "page_id": "page-id-here",
    "command": "update_properties",
    "properties": {
        "Tokenizer": "Llama 3 Tokenizer",
        "Vocab Size": 128256,
        "Is MoE": "__YES__",  # Checkboxes use __YES__/__NO__
        "Num Experts": 16
    }
})

Batch Updates

• Use parallel tool calls when updating multiple independent pages
• Group by model family for logical batching (e.g., all Scout models together)

Search & Fetch

# Search within database
mcp__notion__notion-search({
    "query": "Llama 4",
    "query_type": "internal",
    "data_source_url": "collection://<DATA_SOURCE_ID>"
})

# Fetch page details
mcp__notion__notion-fetch({"id": "page-id-or-url"})

Common Tasks

Adding New Models

1. Parse ~/projects/github/llama-models/models/sku_list.py to get model data
2. Extract archargs and calculate derived fields (headdim, dff using Llama formula, dff/d_hidden ratio)
3. Determine tokenizer based on model family (see Llama Model Family Mappings)
4. Set MoE fields for Llama 4 models
5. Set "Gated MLP" to Yes (all Llama models use SwiGLU)
6. Create pages in batches using mcpnotionnotion-create-pages

Marking Representative Models

Representative "Main" models (user preference):

• Llama 2: 7b chat, 70b chat
• Llama 3.1: 8b instruct, 70b instruct, 405b instruct (FP8)
• Llama 3.2: 1b instruct, 3b instruct, 11b vision instruct (user trains small models)
• Llama 3.3: 70b instruct
• Llama 4: Scout instruct, Maverick instruct
• GPT-2: All variants (117M, 345M, 774M, 1.5B) marked as Main

Schema Updates

mcp__notion__notion-update-database({
    "database_id": "<DATABASE_ID>",
    "properties": {
        "New Column": {"type": "number", "number": {}}
    }
})

Important Notes

Pitfalls to Avoid

1. Don't clear existing fields - Only specify properties you're updating
2. Column ordering - Cannot be changed via API (view-level setting in UI)
3. Checkbox format - Must use "YES" or "NO", not boolean
4. Llama 4 models - Have empty archargs={} in skulist.py, no dhidden/dff available

Model Family Mappings

Llama

• llama2: Llama 2 Tokenizer, 32k vocab

- Actual dff values (verified from HuggingFace): - 7B: dff=11008 (NOT 14336 from skulist.py formula) - 13B: dff=13824 (NOT 17920 from skulist.py formula) - 70B: dff=28672 (correct in sku_list.py)

• llama3, llama31, llama32, llama3_3, llama4, safety: Llama 3 Tokenizer, 128k vocab

GPT-2

• gpt2: BPE tokenizer, 50257 vocab, 1024 max context
• Naming: "GPT-2", "GPT-2 Medium", "GPT-2 Large", "GPT-2 XL"
• Model Type: "base" (all are base models)
• Parameter counts: 117M (base), 345M (medium), 774M (large), 1.5B (XL)
• Config parameter mappings: nembd → dhidden, nlayer → nlayers

Llama MoE Architecture (Llama 4 only)

• Only Llama 4 models are MoE
• Architecture: 1 shared expert (always active) + 1 routed expert (top_k=1)
• Effective: 2 experts per token (shared + routed)
• Routing: Token Choice (each token selects expert via router scores)

Llama Gated MLP / Activation Function

• All Llama models use gated MLP with SwiGLU activation
• Implementation: w2(F.silu(w1(x)) * w3(x)) (from models/llama3/model.py)
• Uses 3 weight matrices (w1, w2, w3) instead of standard 2-matrix FFN
• SwiGLU = Swish-Gated Linear Unit (Swish is same as SiLU)

Calculation Formulas

General Derived Fields

head_dim = d_hidden / n_heads
d_ff_d_hidden_ratio = round(d_ff / d_hidden, 3)

Llama d_ff Calculation

From actual Llama code (models/llama3/model.py):

# Initial: hidden_dim = 4 * dim
hidden_dim = int(2 * hidden_dim / 3)  # = int(8 * dim / 3)
if ffn_dim_multiplier is not None:
    hidden_dim = int(ffn_dim_multiplier * hidden_dim)
# Round up to multiple_of
hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of)
# Result is d_ff

IMPORTANT: Llama 2 Model-Specific Behavior

Llama 2 models use different ffndimmultiplier values than what's in sku_list.py:

• 7B & 13B: Use ffndimmultiplier=None (pure 8d/3 formula)

- 7B: dff=11008, ratio=2.688 - 13B: dff=13824, ratio=2.7

• 70B: Uses ffndimmultiplier=1.3 (as specified in sku_list.py)

- 70B: d_ff=28672, ratio=3.5

This was verified from actual HuggingFace checkpoint configs. The skulist.py incorrectly shows ffndim_multiplier=1.3 for all Llama 2 models.

Alternative formula from HuggingFace transformers (equivalent for models with multiple_of=256):

# For Llama 2 7B, 13B (no multiplier)
def compute_intermediate_size(n, multiple_of=256):
    return int(math.ceil(n * 8 / 3) + multiple_of - 1) // multiple_of * multiple_of

GPT-2 d_ff Calculation

Standard transformer architecture:

d_ff = 4 * d_hidden  # Always 4x for all GPT-2 variants

• Not explicitly in config.json, but defined in model architecture
• All GPT-2 models have dff/dhidden ratio of exactly 4.0

Helpful Scripts & References

Llama

Working directory: ~/projects/github/llama-models/

Scripts:

• Parse models: parsellamamodels.py
• Prepare Notion data: preparenotiondata.py
• Update tokenizers: update_tokenizers.py
• Bulk updates: bulkupdatellama3.py
• Calculate dff: fixdff.py (correct dff calculation using actual Llama formula)
• Calculate ratios: calculateratios.py (dff/d_hidden ratios and gated MLP status)
• Data files: dffcorrections.json, completenotionupdates.json, ratio_updates.json
• Documentation: COMPLETIONSUMMARY.md (dff update history), STATUS.md (current status)

References:

• Llama models repo: ~/projects/github/llama-models/
• Model definitions: models/sku_list.py
• Architecture files: models/llama{2,3,4}/args.py
• MoE implementation: models/llama4/moe.py

GPT-2

HuggingFace References:

• Base: https://huggingface.co/openai-community/gpt2
• Medium: https://huggingface.co/gpt2-medium
• Large: https://huggingface.co/gpt2-large
• XL: https://huggingface.co/gpt2-xl
• Config location: /raw/main/config.json (append to repo URL)

Architecture Notes:

• Uses learned positional embeddings (not RoPE) - add to Notes field
• Activation: GELU (gelu_new variant) - not gated MLP
• Standard attention (nkvheads = n_heads, no GQA)
• All variants have head_dim = 64
• Norm epsilon: 1e-05 (layernormepsilon in config)