ReduceLROnPlateau

- **Strategy**: Use a learning rate scheduler to adjust the learning rate during training. 2. **Batch Size (`per_device_train_batch_size`)**: - **Description**: Number of samples processed before the model is updated. - **Range**:…

optimizer = torch.optim.SGD(model.parameters(), lr=0.01) # Train the model for epoch in range(100): optimizer.zero_grad() outputs = model(input_data) loss = criterion(outputs, labels) loss.backward() optimizer.step() ``…

running_loss = 0.0 for inputs, targets in dataloader: optimizer.zero_grad() outputs = model(inputs) loss = criterion(outputs, targets) loss.backward() optimizer.step() running_loss += …

inputs, labels = inputs.to(device), labels.to(device) optimizer.zero_grad() outputs = model(inputs) loss = criterion(outputs, labels) loss.backward() optimizer.step() running_loss +…

- **Margin**: Adjust the margin in contrastive loss functions to penalize incorrect predictions more heavily. ### 5. **Evaluation Metrics** - **Precision@k**: Monitor Precision@k metrics during training to ensure the model is improvi…

torch.save(model.state_dict(), 'dense_retrieval_model.pth') ``` ### Explanation 1. **Optimizer and Learning Rate Scheduler**: - Use `AdamW` optimizer with weight decay. - Implement a learning rate scheduler to adjust the learning ra…

- Added a `Dropout` layer with a dropout rate of 0.1. - Applied dropout to the embeddings before computing the similarity scores. 2. **Weight Decay**: - Included weight decay (L2 regularization) in the `AdamW` optimizer with a val…

- Use `torch.cuda.amp` to enable mixed precision training with `GradScaler` and `autocast`. ### Additional Considerations - **Batch Size**: Adjust the batch size based on the available VRAM. For example, if your GPU has 16 GB of VRAM, …

# Calculate average loss for the epoch avg_loss = running_loss / len(data_loader) print(f'Epoch [{epoch + 1}/100], Loss: {avg_loss:.4f}, LR: {optimizer.param_groups[0]["lr"]}') # Step the scheduler s…