4

Source document

[2025-04-06 01:15] lisamegawatts: This is one of the things i wanted to test, need to pick base model then run it against different methods of training and do evaluation [2025-04-06 03:14] traves_theberge: the new llama 4 model is a pretty interesting model , with 17b Core generalized parameters and 16 domain specific "expert" paramaters. still to large if you ask me its like 105b total. but its getting interesting [2025-04-06 03:17] traves_theberge: ## Model Adaptation Techniques ### 1. Fine-Tuning **What it is:** Involves retraining a pre-trained model on a specific task or dataset, updating all or a significant portion of its parameters. **Pros:** - Can achieve high performance, especially with large datasets. **Cons:** - Requires significant computational resources and memory. --- ### 2. Distillation **What it is:** Trains a smaller **"student"** model to mimic the outputs of a larger **"teacher"** model, aiming for a trade-off between performance and efficiency. **Pros:** - Allows for smaller, faster models without sacrificing too much performance. **Cons:** - Can result in some knowledge loss compared to fine-tuning the original model. --- ### 3. LoRA (Low-Rank Adaptation) **What it is:** A PEFT (Parameter-Efficient Fine-Tuning) technique that modifies only a small subset of a model's parameters, typically by adding low-rank matrices to the existing weights. **How it works:** - LoRA freezes the original model's weights and only trains the low-rank matrices, reducing the number of trainable parameters significantly. **Benefits:** - Reduces computational cost and memory requirements during fine-tuning. - Enables fine-tuning of large models on resource-constrained hardware. - Can achieve comparable performance to full fine-tuning. --- ### 4. QLoRA (Quantized LoRA) **What it is:** An extension of LoRA that incorporates quantization, further reducing the model's size, memory footprint, and computational requirements. **How it works:** - QLoRA quantizes the model's weights (e.g., to 4-bit or 8-bit precision) while also using LoRA's low-rank adaptation. **Benefits:** - Even greater efficiency than LoRA, allowing for fine-tuning of very large models on consumer-grade hardware. - Can support higher maximum sequence lengths due to reduced GPU memory consumption. - Maintains similar performance levels to LoRA. [2025-04-06 09:49] lisamegawatts: https://huggingface.co/blog/smollm really interesting breakdown of their training methods for their smol models, so it initially was trained on some python samples [2025-04-06 10:44] lisamegawatts: https://huggingface.co/learn/cookbook/fine_tuning_code_llm_on_single_gpu [2025-04-06 11:50] lisamegawatts: https://tufalabs.ai/blog/_site/jekyll/update/2025/03/25/textbooks-to-rl.html [2025-04-06 12:13] ajaxdavis: llhama spm v1 resumed_cli https://thomasalwyndavis--example-axolotl-inference-web.modal.run?input=%5BINST%5Dsay%20hello%20in%20SQL%5B%2FINST%5D