Fine-Tune Config Generator

Generate production-ready configurations for LoRA, QLoRA, PEFT, and more

Select Training Method

Training Parameters

Base Model

Learning Rate: 2.0e-4

Batch Size: 4

Epochs: 3

Max Sequence Length: 512

LoRA Configuration

LoRA Rank (r): 8

Higher = more parameters

LoRA Alpha: 16

Scaling factor (usually 2×r)

Dropout: 0.05

Regularization

Generated Configuration

{
  "model_name_or_path": "meta-llama/Llama-2-7b-hf",
  "task_type": "CAUSAL_LM",
  "lora_config": {
    "r": 8,
    "lora_alpha": 16,
    "lora_dropout": 0.05,
    "target_modules": [
      "q_proj",
      "v_proj",
      "k_proj",
      "o_proj"
    ],
    "bias": "none",
    "task_type": "CAUSAL_LM"
  },
  "training_arguments": {
    "output_dir": "./lora-output",
    "num_train_epochs": 3,
    "per_device_train_batch_size": 4,
    "gradient_accumulation_steps": 4,
    "learning_rate": 0.0002,
    "fp16": true,
    "logging_steps": 10,
    "save_strategy": "epoch",
    "optim": "adamw_torch",
    "warmup_ratio": 0.03,
    "lr_scheduler_type": "cosine",
    "max_seq_length": 512
  }
}

LoRA

Low-Rank Adaptation for efficient fine-tuning

Best For:

• Consumer GPUs (8-16GB)
• Fast iteration cycles
• Task-specific adaptation

Quick Tips

LoRA Rank: Start with r=8 for most tasks. Increase to 16-32 for complex adaptations.

Learning Rate: 2e-4 to 5e-4 works well for most LoRA training.

Batch Size: Use gradient accumulation if your GPU can't fit larger batches.

Epochs: 3-5 epochs usually sufficient. Monitor validation loss to avoid overfitting.

Estimated VRAM

7B Model:

~12-16GB VRAM required