# MK3: Continuous Autoregressive Language Model with Complete Salience Scoring

MK3 is a from-scratch implementation combining two key innovations:
1. **Complete Salience Formula with Normalization Invariant** - Improved information scoring that properly handles scale and energy conservation
2. **Continuous Autoregressive Language Models (CALM)** - Paradigm shift from token-by-token to vector-by-vector prediction for K times faster generation

## Key Innovation: The Complete Salience Invariant

**MK2 Formula (Incomplete):**
```
S' = (w₁·ΔA + w₂·R + w₃·M) × C × e^(-λt) × (1 - kφ)
```

**MK3 Formula (Complete with Invariant):**
```
S' = σ((w₁·ΔA + w₂·R + w₃·M) × C × e^(-λt) × (1 - kφ) / (√d · τ))
```

**What Was Missing:**
- **σ**: Normalization (softmax/layernorm) for energy conservation
- **√d**: Dimensional scaling factor for scale invariance
- **τ**: Learnable temperature for distribution control

This ensures salience is **invariant** to:
- Input scale transformations
- Context size changes
- Discrete ↔ Continuous representation transitions

## CALM Integration

Based on arxiv:2510.27688, MK3 implements:
- **High-fidelity autoencoder**: Compresses K=8 tokens into 1 continuous vector with >99.9% reconstruction accuracy
- **Continuous prediction**: Model predicts next vector instead of next token
- **K times speedup**: Generating 1 vector = generating K tokens, reducing steps by factor of K
- **Likelihood-free training**: MSE + cosine + reconstruction + contrastive losses

## Architecture

```
MK3/
├── core/                           # Core salience system
│   ├── salience_formula.py        # Complete formula with normalization
│   ├── salience_layers.py         # Attention and selective layers
│   └── continuous_embeddings.py   # Discrete ↔ Continuous embeddings
│
├── calm/                           # CALM implementation
│   ├── autoencoder.py             # High-fidelity K-token compression
│   ├── continuous_model.py        # Continuous autoregressive model
│   └── likelihood_free.py         # Training framework
│
├── training/                       # Training infrastructure
│   ├── config.py                  # Configuration classes
│   └── trainer.py                 # Complete training pipeline
│
├── utils/                          # Utilities
│   ├── tokenizer.py               # Text tokenization
│   ├── data_loader.py             # Data loading
│   └── reproducibility.py         # Reproducibility guarantees
│
├── train.py                        # Main training script
├── generate.py                     # Text generation script
└── requirements.txt                # Dependencies
```

## Installation

```bash
# Clone repository and navigate to MK3
cd MK3

# Install dependencies
pip install -r requirements.txt
```

Requirements:
- Python 3.8+
- PyTorch 2.0+
- CUDA (optional, for GPU training)

## Quick Start

### Test Mode (Synthetic Data)

```bash
# Train on synthetic data (for testing)
python train.py --test_mode --num_epochs 3
```

### Training on Your Data

```bash
# Train with your text data
python train.py \
    --train_data data/train.txt \
    --val_data data/val.txt \
    --num_epochs 10 \
    --batch_size 32 \
    --checkpoint_dir ./checkpoints
```

### Text Generation

```bash
# Generate text with trained model
python generate.py \
    --checkpoint checkpoints/best_model.pt \
    --prompt "Once upon a time" \
    --max_new_vectors 20 \
    --temperature 0.8
```

## Training Pipeline

MK3 uses a **two-stage training process**:

### Stage 1: Autoencoder Pretraining
- Target: >99.9% token reconstruction accuracy
- Trains encoder and decoder to compress K tokens ↔ 1 vector
- Typically takes 5,000-10,000 steps
- Can be frozen or fine-tuned in Stage 2

### Stage 2: Continuous Model Training
- Trains autoregressive model to predict continuous vectors
- Uses salience-based attention for information selection
- Employs curriculum learning (gradually increase sequence length)
- Likelihood-free training with multiple loss components

## Configuration

### Model Configuration

Key hyperparameters in `training/config.py`:

```python
ModelConfig(
    vocab_size=10000,           # Vocabulary size
    embedding_dim=768,          # Embedding dimension
    vector_dim=1024,            # Continuous vector dimension
    chunk_size=8,               # Tokens per vector (K)
    num_layers=12,              # Transformer layers
    num_heads=8,                # Attention heads

    # Salience parameters (learnable)
    salience_w1=0.4,            # Novelty weight
    salience_w2=0.3,            # Retention weight
    salience_w3=0.3,            # Payoff weight
    salience_temperature=1.0,   # Temperature τ
    salience_normalization='softmax',  # σ function
)
```

### Training Configuration

```python
TrainingConfig(
    batch_size=32,
    learning_rate=1e-4,
    num_epochs=10,

    # Stage 1: Autoencoder pretraining
    pretrain_autoencoder=True,
    autoencoder_steps=10000,
    autoencoder_target_accuracy=0.999,

    # Optimization
    use_amp=True,               # Mixed precision
    max_grad_norm=1.0,          # Gradient clipping

    # Reproducibility
    seed=42,
)
```

## Features

### ✅ Complete Implementations (NO Placeholders)
Every component is fully implemented with:
- Complete forward/backward passes
- Proper gradient flow
- Comprehensive error handling
- Scientific reproducibility

### ✅ Scientific Reproducibility
- Deterministic seeding (Python, NumPy, PyTorch, CUDA)
- Reproducible data ordering
- Version tracking
- Configuration saving
- Checkpoint management

### ✅ Salience Invariant Properties
- **Scale Invariance**: Normalized by √d
- **Energy Conservation**: Softmax ensures scores sum to 1
- **Temperature Control**: Learnable τ controls distribution sharpness
- **Stable Training**: Layer normalization prevents exploding/vanishing values

### ✅ CALM Benefits
- **K times faster generation**: Predict K tokens at once
- **High fidelity**: >99.9% reconstruction accuracy
- **Continuous representations**: Smoother interpolation
- **Flexible K**: Adjustable compression ratio

## Performance Targets

| Metric | Target | Notes |
|--------|--------|-------|
| Autoencoder Reconstruction | >99.9% | Exact token accuracy |
| Generation Speedup | K times | K=8 → 8x faster |
| Salience Stability | σ ∈ [0,1] | After normalization |
| Training Reproducibility | 100% | Same seed → same results |

## Understanding the Salience Formula

### Components

1. **Novelty (ΔA)**: Information gain - how new is this information?
2. **Retention (R)**: Long-term value - will this be important later?
3. **Payoff (M)**: Immediate utility - is this useful right now?
4. **Continuity (C)**: Coherence - does this fit with context?
5. **Fatigue (φ)**: Redundancy - have we seen this recently?

### Temporal Dynamics
- **e^(-λt)**: Time decay - recent items weighted higher
- **λ**: Decay rate (learnable)

### Normalization Invariant
- **√d**: Prevents scale issues with different embedding dimensions
- **τ**: Temperature controls distribution sharpness (low τ = sharp, high τ = smooth)
- **σ**: Ensures proper probability distribution (energy conservation)

## Comparison with MK2

| Aspect | MK2 | MK3 |
|--------|-----|-----|
| Salience Formula | Incomplete | Complete with invariant |
| Prediction | Token-by-token | Vector-by-vector |
| Speed | 1x | K times faster |
| Normalization | Manual | Built-in (√d, τ, σ) |
| Reconstruction | N/A | >99.9% accuracy |
| Training | Single stage | Two-stage (AE + Model) |
| Continuous Space | No | Yes (CALM) |

## Reproducibility Guarantees

MK3 ensures scientific reproducibility through:

1. **Deterministic Operations**: All random operations seeded
2. **Configuration Tracking**: All hyperparameters saved
3. **Version Tracking**: PyTorch, CUDA versions recorded
4. **Checkpoint Management**: Model weights, optimizer state, training progress
5. **Data Ordering**: Reproducible data shuffling and batching

To reproduce a training run:
```python
from training.config import TrainingConfig, ModelConfig
from utils.reproducibility import make_deterministic

# Load saved configs
model_config = ModelConfig.load('checkpoints/model_config.json')
training_config = TrainingConfig.load('checkpoints/training_config.json')

# Apply reproducibility settings
make_deterministic(training_config.seed)

# Train with same configs → same results
```

## Advanced Usage

### Custom Tokenizer
```python
from utils.tokenizer import SimpleTokenizer

tokenizer = SimpleTokenizer(vocab_size=30000, min_frequency=5)
tokenizer.train(texts)
tokenizer.save('my_tokenizer.json')
```

### Curriculum Learning
```python
from calm.likelihood_free import CurriculumScheduler

curriculum = CurriculumScheduler(
    initial_seq_length=64,
    max_seq_length=2048,
    warmup_steps=10000,
    schedule='cosine'
)
```

### Custom Salience Configuration
```python
salience_config = {
    'w1': 0.5,  # More weight on novelty
    'normalization': 'layernorm',  # Use layer norm instead of softmax
    'use_dimensional_scaling': True,  # Enable √d scaling
    'learnable_temperature': True,  # Learn τ during training
}
```

## Troubleshooting

### OOM (Out of Memory)
- Reduce `batch_size`
- Reduce `max_seq_length`
- Reduce `num_layers` or `embedding_dim`
- Enable `use_amp=True` (mixed precision)

### Low Autoencoder Accuracy
- Increase `autoencoder_steps`
- Increase `vector_dim` (more capacity)
- Increase `num_encoder_layers` / `num_decoder_layers`
- Check `chunk_size` (smaller = easier to reconstruct)

### Training Instability
- Check `max_grad_norm` (try 0.5)
- Reduce `learning_rate`
- Ensure salience normalization is enabled
- Check for NaN values in loss

## Citation

If you use MK3 in your research, please cite:

```
MK3: Continuous Autoregressive Language Model with Complete Salience Scoring
Formula: S' = σ((w₁·ΔA + w₂·R + w₃·M) × C × e^(-λt) × (1 - kφ) / (√d · τ))
Integrates CALM (arxiv:2510.27688) for K times faster generation
```

## License

Open source for research and educational purposes.

## Acknowledgments

- CALM paper (arxiv:2510.27688) for continuous autoregressive paradigm
- MK2 for the foundation salience scoring concept

## Project Status

**✅ COMPLETE AND READY FOR TRAINING**

All components fully implemented:
- ✅ Complete salience formula with normalization invariant
- ✅ High-fidelity CALM autoencoder
- ✅ Continuous autoregressive model
- ✅ Likelihood-free training framework
- ✅ Full reproducibility guarantees
- ✅ Training and generation scripts
- ✅ Comprehensive configuration system

**NO PLACEHOLDERS. NO TODOs. READY TO RUN.**