975

Source document

[2026-01-28 12:13] uncloseai [bot]: Based on the fetched content from uncloseai.com, here are the key points to run inference with Dry (using the original Urho3D names): 1. Use the vLLM library to run models, typically with full f16 safetensors. Ensure to use a virtualenv to hold the dependencies. 2. For Hermes, use an FP8 quant model by adamo1139 (adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic), which is optimized for 4090 and 3090 GPUs. Enable --enable-auto-tool-choice and --tool-call-parser hermes flags for OpenAI-compatible API function calling. 3. Consider supporting ollama for better quant support. 4. Stand up a replica cluster on a new domain: - Install gcc and python3.12-dev/python3.12-venv - Create a new virtualenv - Install vllm 5. Configure the proxy in /etc/caddy/Caddyfile to reverse proxy to the inference server. 6. Discover available models and explore the full API using the /docs endpoint (e.g. hermes.ai.unturf.com/docs). 7. To get the current model ID being hosted, query the /v1/models endpoint (e.g. hermes.ai.unturf.com/v1/models or qwen.ai.unturf.com/v1/models). 8. Rate limiting is configured based on client IP address: 3 requests per second per IP per endpoint. In summary, use vLLM with an FP8 quant Hermes model on a replica cluster, configure a proxy, and query the /v1/models endpoint to discover available models. Enable --enable-auto-tool-choice and --tool-call-parser hermes flags for OpenAI compatibility. [2026-01-28 12:13] uncloseai [bot]: **Sources:** - **Open-Source Text-to-Speech | uncloseai-speech | uncloseai.com**: <https://uncloseai.com/text-to-speech.html> - **C Examples | uncloseai.com**: <https://uncloseai.com/c-examples.html> - **C# Examples | uncloseai.com**: <https://uncloseai.com/csharp-examples.html> - **Java Examples | uncloseai.com**: <https://uncloseai.com/java-examples.html> - **Go Examples | uncloseai.com**: <https://uncloseai.com/go-examples.html> - **Kotlin Examples | uncloseai.com**: <https://uncloseai.com/kotlin-examples.html> - **Dart Examples | uncloseai.com**: <https://uncloseai.com/dart-examples.html> - **Node.js Examples | uncloseai.com**: <https://uncloseai.com/nodejs-examples.html> - **Elixir Examples | uncloseai.com**: <https://uncloseai.com/elixir-examples.html> - **How We Run Inference | uncloseai.com**: <https://uncloseai.com/inference.html> [2026-01-28 12:13] uncloseai [bot]: Code execution error: Error code: 500 - {'error': {'message': 'do load request: Post "http://127.0.0.1:33503/load": EOF', 'type': 'api_error', 'param': None, 'code': None}} [2026-01-28 12:13] uncloseai [bot]: **Attempting Fix:** ```json { "success": true, "issueNumber": 954, "issueUrl": "https://github.com/thomasdavis/omega/issues/954", "message": "Created issue #954: Integrate New Qwen Model and Voices from UncloseAI for Omega Blog TTS", "requiresDatabase": true, "labels": [ "enhancement", "database" ] } ```