GPU

xenonfun in #safiersemantics: images page starting.…

xenonfun in #safiersemantics: (no text — image attachment only)…

xenonfun in #safiersemantics: well perhaps this is messy for sure. wish I just had bigger disk. stupid acer was $200 more with 4tb recently...…

xenonfun in #safiersemantics: well that was kinda impressive, NFS wedged (Again). found root source, NFS server was set to auto idle (WTF?) at least the NIC wasn't core issue, so that is good. restarted NFS and claude came back to life.…

xenonfun in #safiersemantics: failing faster now.…

xenonfun in #safiersemantics: (no text — image attachment only)…

xenonfun in #safiersemantics: ✶ Propagating… (8m 35s · ↓ 28.4k tokens) ⎿  ◻ Manual-invoke image builds as CI jobs + UI single-job trigger ◻ [LARGER] Publish named images to uranus OCI feed + k3s pulls from there (retire --local)…

xenonfun in #safiersemantics: will get docker images as well some UI exposure. as it is also hosting its own images, or will be again shortly.…

xenonfun in #safiersemantics: looks like shit but guess it counts, don't think I ever actually published package and viewed.…

xenonfun in #safiersemantics: I really need to split build up for bigger projects: perhaps publish and pull the crates (which then are all sccached), would probably improve build cycle times as a lot of them don't get touched in a feature u…

xenonfun in #safiersemantics: tags now too…

xenonfun in #safiersemantics: better luck next-time…

xenonfun in #safiersemantics: self release time, again.…

xenonfun in #safiersemantics: crates are coming back. getting orleans-rust-client fixed up so will do whole publish .…

xenonfun in #safiersemantics: ● The OCI restoration Understand workflow (wmb8i3k3n) is running — read-only mapping of the registry impl, the prior working publish flow (from git history), the DGX-era change, and exposure, then a restorati…

xenonfun in #safiersemantics: okay now its gotta rediscover we already build a whole OCI endpoint its gotta start using it again.…

retrieval_results = parallel_process_queries(queries, retrieval_layer, max_workers=10) generation_responses = parallel_process_queries(prompts, generation_layer, max_workers=10) # Print the results print("Retrieval Results:", retrieval_res…

2. **Device Utilization:** The model and inputs are moved to the GPU if available, which can significantly speed up the computation. 3. **Efficient Embedding Extraction:** The embeddings are extracted from the `CLS` token (first token) of t…

args=training_args, train_dataset=tokenized_dataset["train"], eval_dataset=tokenized_dataset["validation"] ) # Train the model trainer.train() ``` #### 3. Self-Hosted Model Deployment ##### Environment Setup - **Hardware**: …

- **nprobe**: The number of clusters to probe during search. A larger value improves accuracy but increases search time. ### Additional Tips - **Quantization**: Consider using `IndexIVFPQ` for even more efficient indexing and search. - **…

- **Parallel Processing**: For handling 15,000 documents hourly, consider parallelizing the vectorization and indexing processes using multiprocessing or distributed computing frameworks. This architecture provides a clear separation of co…

[Turn 4865] Assistant: Certainly! Leveraging a GPU with FAISS can significantly improve the performance of large-scale similarity searches. Below is an example of how to set up and use a GPU-accelerated FAISS index for 200K document searche…

- `nlist`: Number of clusters. A higher value can improve accuracy but also increases memory usage. - `M`: Number of sub-quantizers. A higher value can improve accuracy but also increases memory usage. - `nbits`: Number of bits per…

3. **Multi-threading**: - `faiss.omp_set_num_threads(8)` enables multi-threading to take advantage of multiple CPU cores. Adjust the number of threads based on your CPU capabilities. 4. **Training the Index**: - The index needs to be…

Manage GPU/CPU resources effectively to avoid memory issues. ### Example Implementation Review Here's an example of a PyTorch model for language embeddings, followed by suggested improvements: ```python import torch import torch.nn as nn…

2. **Data Loading and Preprocessing**: Use `torchtext` for efficient text preprocessing and `DataLoader` with `num_workers`. 3. **Training Loop**: Use gradient clipping and learning rate scheduling. 4. **Evaluation and Monitoring**: Impleme…

- The model is trained on the GPU if available. 5. **Saving the Model**: - After training, the fine-tuned model and tokenizer are saved to disk. ### Next Steps - **Evaluate the Model**: After training, evaluate the model on a valid…

results.extend(batch_results.cpu().numpy()) return results # Parallel processing def parallel_infer(texts, num_workers=4): with ThreadPoolExecutor(max_workers=num_workers) as executor: results = list(executor.map(in…

- Use `nvidia-smi` to monitor GPU usage and ensure that the GPU is being utilized effectively. - Example command: `nvidia-smi --loop-ms=1000 --format=csv,noheader,nounits --query-gpu=index,name,utilization.gpu,memory.total,memory.used,m…

- Ensure that both the model and the input data are on the same device (either CPU or GPU). - Use `model.to(device)` and `input_data.to(device)` to move the model and data to the desired device. 2. **Gradient Calculation**: - When…

return json.loads(cipher_suite.decrypt(encrypted_data).decode()) # Function to encrypt the data loader def encrypt_data_loader(data_loader): encrypted_data_loader = [] for batch in data_loader: encrypted_batch = { …

loss = criterion(outputs, y) loss.backward() optimizer.step() ``` I'm targeting 99.9% uptime for my pipeline, and I need help implementing a secure tuning protocol that can handle 110,000 model updates. ->-> 9,4 [Tu…

'batch_size': len(inputs), 'loss': loss.item() } log_json = json.dumps(log_entry) logging.info(log_json) except Exception as e: logging.error(f"Error du…

loss = loss / accumulation_steps # Backward pass scaler.scale(loss).backward() # Update weights if (i + 1) % accumulation_steps == 0: scaler.step(optimizer) …

xenonfun in #hardware: <@823468778704076810> highly recommend you check it out. will post recipe its still tweaking a bit.…

xenonfun in #hardware: Outstanding — 11/11 grounded inside bbox, mean error 4px on the real dense dashboard, and the live clicks landed exactly on 📊 monitor and 🌐 network. Let me visually confirm the clicks actually switched views.…

xenonfun in #hardware: yeah its impressive…

xenonfun in #hardware: ``` Concurrency sweep (mixed image+text, 256 tok out) — 46/46 OK ┌──────┬─────────────┬──────┬──────┐ │ Conc │ Gen tput │ p50 │ p95 │ ├──────┼─────────────┼──────┼──────┤ │ 1 │ 75.8 tok/s │ 3.0s │ …

xenonfun in #hardware: All the earlier verifications still stand from this same running instance: KV fit at 0.35 (18 GB / 1.79M tokens → 6.84× at full 256K), tool calling working (structured tool_calls, qwen3_coder), and 44K-token needle …

xenonfun in #hardware: running it thru some tests now.…

xenonfun in #hardware: yeah its looking pretty solid…

xenonfun in #hardware: would be nice if FP4 worked. Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade perfor…

xenonfun in #hardware: holo3.1 running. faster than nemo with zero optimization, will see how it goes: https://huggingface.co/Hcompany/Holo-3.1-35B-A3B-NVFP4…

xenonfun in #hardware: yeah I was going to start looking but that guy been working on it. glad can quant as they are heavy.…

ajaxdavis in #hardware: that will be pretty sick to have locally…

xenonfun in #hardware: https://x.com/i/status/2061810401013100871…