Inference infrastructure, minus the infrastructure
Every feature below is live in the TIR console today.
One-click catalog deploys
Serve state-of-the-art open models from the TIR catalog, or hit the shared GenAI API (GPT-OSS 120B, Llama 3.1 8B, Qwen3 32B, Sarvam and more) without deploying anything yourself.
Hugging Face & Model Repository
Point an endpoint at any HF model or a TIR Model Repository on E2E Object Storage — files download to the container automatically at start. Validate model and config before deploy.
Custom containers
Bring your own Docker image from a public or private registry, optionally attached to a TIR Model for automatic weight downloads. Full control over the serving stack.
Metric-driven autoscaling
Scale on concurrent requests, requests/sec, or runtime metrics like vllm:num_requests_running, sglang:gen_throughput and Triton queue depth. Tune idle timeout and cooldown to balance latency vs cost.
Async & WebSocket APIs
Queue long-running jobs with async invocation — get a request_id, results land as JSON in your EOS dataset, workers bill only while the queue is non-empty. Streaming? Connect over wss://.
Private, observable serving
Security Groups and VPC attach keep endpoints off the public internet. Centralized inference and per-replica logs, hardware + service metrics, access logs and email alerts come standard.
From model to production API in four steps
The actual deploy flow in the TIR console.
Pick a source
Catalog model, Hugging Face repo, TIR Model Repository, or your own container image.
Choose runtime & GPU
vLLM, SGLang, Triton or PyTorch; select GPU plan and replicas. Tune engine args — dtype, quantization, context length — if you need to.
Set scaling
Active Workers, Max Workers, metric type and target value. Different min/max = autoscaling on. Zero minimum = scale-to-zero.
Call it
Create an API token, copy the sample cURL/Python from the console, and test in the built-in playground before wiring up your app.
OpenAI-compatible out of the box
Every endpoint gets a root URL; append standard routes like /v1/chat/completions.
# illustrative — copy the exact URL from your endpoint's API Request tab
curl https://infer.e2enetworks.net/project/p-XXXX/endpoint/is-XXXX/v1/chat/completions \
-H "Authorization: Bearer $TIR_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-r1",
"messages": [{"role": "user", "content": "Summarise this contract clause:"}],
"max_tokens": 400
}'Tokens live under API Tokens in your TIR project. The playground supports chat models with temperature, top-p, top-k and repetition-penalty controls.
Autoscaling signals, compared
Choose the metric that matches your traffic shape.
| Metric type | Scales on | Best for | Example target |
|---|---|---|---|
| Concurrent Request Count | In-flight requests per worker | Latency-sensitive LLM chat, where each request holds the GPU for the full generation | 10 concurrent requests |
| Request Rate per Second | Requests/sec arriving per worker | High-throughput, short-lived calls — embeddings, classification | 50 req/s |
| Custom (vLLM) | vllm:num_requests_running, vllm:gpu_cache_usage_perc… | Preventing KV-cache OOM, queue-depth-aware scaling | 32 running requests |
| Custom (SGLang) | sglang:num_queue_reqs, sglang:gen_throughput… | Token-throughput-driven serving | Queue > 8 |
| Triton | nv_inference_pending_request_count, nv_gpu_utilization… | Multi-model servers | GPU util 80% |
| TorchServe | GPUUtilization, ts_queue_latency_microseconds… | Classic PyTorch serving | Queue latency |
Idle Timeout keeps warm workers available (billed); Initial Cooldown Period prevents flapping before scale-to-zero.
Fine-tune first, then serve it here
Models trained or fine-tuned on TIR deploy to endpoints in a few clicks — same project, same storage.
Frequently Asked Questions
Everything you need to know about deploying inference endpoints on TIR.
Inference & Endpoints
Set Active Workers to zero and the endpoint scales to zero when idle; you pay only while workers run (including any configured idle timeout). Async workers bill only while the request queue is non-empty.
LLMs, embeddings, speech (ASR/TTS) and diffusion models across vLLM, SGLang, Triton and PyTorch runtimes — or anything that runs in your own container.
Attach the endpoint to your E2E VPC to get a private IP reachable only from your VPC nodes, then lock ports down with Security Groups. Note: VPC Connect requires a VPC created with E2E-provided CIDR blocks.
Yes — centralized inference logs, per-replica logs, deployment/lifecycle events, hardware and service metrics, per-hour access logs, and threshold-based email alerts are built into every endpoint.
Yes — reserve capacity at committed rates for steady traffic (committed endpoints can't be stopped during the commitment period). See the rate card for current hourly vs committed pricing.
Your model could be an API by lunch
OpenAI-style routes, Indian GPUs, INR billing.