Inference infrastructure, minus the infrastructure
Every feature below is live in the TIR console today.
One-click catalog deploys
Serve state-of-the-art open models from the TIR catalog, or hit the shared GenAI API (GPT-OSS 120B, Llama 3.1 8B, Qwen3 32B, Sarvam and more) without deploying anything yourself.
Hugging Face & Model Repository
Point an endpoint at any HF model or a TIR Model Repository on E2E Object Storage — files download to the container automatically at start. Validate model and config before deploy.
Custom containers
Bring your own Docker image from a public or private registry, optionally attached to a TIR Model for automatic weight downloads. Full control over the serving stack.
Metric-driven autoscaling
Scale on concurrent requests, requests/sec, or runtime metrics like vllm:num_requests_running, sglang:gen_throughput and Triton queue depth. Tune idle timeout and cooldown to balance latency vs cost.
Async & WebSocket APIs
Queue long-running jobs with async invocation — get a request_id, results land as JSON in your EOS dataset, workers bill only while the queue is non-empty. Streaming? Connect over wss://.
Private, observable serving
Security Groups and VPC attach keep endpoints off the public internet. Centralized inference and per-replica logs, hardware + service metrics, access logs and email alerts come standard.
From model to production API in four steps
The actual deploy flow in the TIR console.
Pick a source
Catalog model, Hugging Face repo, TIR Model Repository, or your own container image.
Choose runtime & GPU
vLLM, SGLang, Triton or PyTorch; select GPU plan and replicas. Tune engine args — dtype, quantization, context length — if you need to.
Set scaling
Active Workers, Max Workers, metric type and target value. Different min/max = autoscaling on. Zero minimum = scale-to-zero.
Call it
Create an API token, copy the sample cURL/Python from the console, and test in the built-in playground before wiring up your app.
OpenAI-compatible out of the box
Every endpoint gets a root URL; append standard routes like /v1/chat/completions.
# illustrative — copy the exact URL from your endpoint's API Request tab
curl https://infer.e2enetworks.net/project/p-XXXX/endpoint/is-XXXX/v1/chat/completions \
-H "Authorization: Bearer $TIR_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-r1",
"messages": [{"role": "user", "content": "Summarise this contract clause:"}],
"max_tokens": 400
}'Tokens live under API Tokens in your TIR project. The playground supports chat models with temperature, top-p, top-k and repetition-penalty controls.
Autoscaling signals, compared
Choose the metric that matches your traffic shape.
| Metric type | Scales on | Best for | Example target |
|---|---|---|---|
| Concurrent Request Count | In-flight requests per worker | Latency-sensitive LLM chat, where each request holds the GPU for the full generation | 10 concurrent requests |
| Request Rate per Second | Requests/sec arriving per worker | High-throughput, short-lived calls — embeddings, classification | 50 req/s |
| Custom (vLLM) | vllm:num_requests_running, vllm:gpu_cache_usage_perc… | Preventing KV-cache OOM, queue-depth-aware scaling | 32 running requests |
| Custom (SGLang) | sglang:num_queue_reqs, sglang:gen_throughput… | Token-throughput-driven serving | Queue > 8 |
| Triton | nv_inference_pending_request_count, nv_gpu_utilization… | Multi-model servers | GPU util 80% |
| TorchServe | GPUUtilization, ts_queue_latency_microseconds… | Classic PyTorch serving | Queue latency |
Idle Timeout keeps warm workers available (billed); Initial Cooldown Period prevents flapping before scale-to-zero.
Fine-tune first, then serve it here
Models trained or fine-tuned on TIR deploy to endpoints in a few clicks — same project, same storage.
Frequently Asked Questions
Everything you need to know about deploying inference endpoints on TIR.
Inference & Endpoints
Your model could be an API by lunch
OpenAI-style routes, Indian GPUs, INR billing.