TL;DR#

Ray Serve is a Python-native model serving framework built on the Ray distributed runtime. It excels at multi-model composition, GPU-aware scheduling, and low-latency inference — all within its own cluster abstraction. Asya is a Kubernetes-native actor mesh where each step is an independent pod scaling on queue depth via KEDA. Ray Serve owns the cluster; Asya delegates everything to Kubernetes.

Choose Ray Serve when you need tight GPU multiplexing and sub-50ms inter-model latency. Choose Asya when you need independent per-step scaling, scale-to-zero, dynamic routing, and want to stay on pure Kubernetes without a second cluster runtime.

At a Glance#

Architecture#

Ray Serve#

Ray Serve runs on top of a Ray cluster — a head node plus autoscaled worker nodes. Each deployment is a Ray actor class replicated across workers. Calls between deployments travel through Ray's object store (shared-memory plasma, or distributed via GCS), giving near-zero serialization overhead for in-cluster communication.

Client --> HTTP/gRPC Ingress
              |
         Ray Head Node (GCS, Dashboard, Autoscaler)
              |
     +--------+--------+
     |        |        |
  Worker 1  Worker 2  Worker 3
  (GPU)     (GPU)     (CPU)
  [ModelA]  [ModelA]  [PostProc]
  [ModelB]  [ModelB]

Deployment graph (or bind()) lets you compose models:

from ray import serve

@serve.deployment(num_replicas=2, ray_actor_options={"num_gpus": 1})
class Embedder:
    def __init__(self):
        self.model = load_model("bge-large")

    async def __call__(self, request):
        return self.model.encode(request.json()["text"])

@serve.deployment(num_replicas=1)
class Reranker:
    def __init__(self, embedder):
        self.embedder = embedder

    async def __call__(self, request):
        embedding = await self.embedder.remote(request)
        return self.rerank(embedding)

app = Reranker.bind(Embedder.bind())
serve.run(app)

Asya#

Asya has no cluster runtime. Each actor is a standard Kubernetes Deployment with an injected sidecar. Actors communicate exclusively through message queues. KEDA watches each queue independently and scales each actor from 0 to N.

Client --> Gateway (A2A / MCP / HTTP)
              |
         Queue A ──> Actor A (Embedder, GPU, 0-5 pods)
                         |
                    Queue B ──> Actor B (Reranker, CPU, 0-10 pods)
                                    |
                               Queue C ──> x-sink (persist result)

The same pipeline as two pure functions and one manifest:

# embedder/handler.py
def embed(payload: dict) -> dict:
    payload["embedding"] = model.encode(payload["text"])
    return payload

# reranker/handler.py
def rerank(payload: dict) -> dict:
    payload["ranked"] = do_rerank(payload["embedding"])
    return payload

# asyncactor.yaml
apiVersion: asya.sh/v1alpha1
kind: AsyncActor
metadata:
  name: embedder
spec:
  image: embedder:latest
  handler: handler.embed
  scaling:
    minReplicaCount: 0
    maxReplicaCount: 5
    queueLength: 1
  resources:
    limits:
      nvidia.com/gpu: "1"
  resiliency:
    actorTimeout: 120s
    policies:
      default:
        maxAttempts: 3
        backoff: exponential

Developer Experience#

When to Choose Ray Serve#

• Low-latency model composition — your pipeline chains 3-5 models and needs sub-50ms end-to-end; Ray's shared-memory object store avoids serialization overhead.

• GPU multiplexing — you want to pack multiple models onto a single GPU with fractional allocation (num_gpus=0.5); Ray handles placement and memory management.

• Python-centric team — your team is all Python, models are all PyTorch/TensorFlow, and you want everything in one language with minimal YAML.

• Batch inference pipelines — Ray's @serve.batch decorator handles dynamic batching natively, accumulating requests to fill GPU utilization.

• Tight coupling between models — models share tensors or embeddings in-memory; serializing to a queue would be wasteful.

When to Choose Asya#

• Independent per-step scaling — each pipeline stage has a different resource profile (CPU preprocessing, GPU inference, CPU postprocessing) and needs to scale on its own queue depth, not a shared autoscaler.

• Scale to zero — GPU pods cost nothing when idle. Ray Serve requires at least one head node running at all times; Asya actors scale to zero pods when their queue is empty.

• Dynamic routing — actors rewrite route.next at runtime based on LLM confidence, content type, or business rules. No static DAG to rebuild.

• Separation of concerns — data scientists write pure Python functions; platform engineers own the AsyncActor manifest (scaling, retries, timeouts). The handler has zero infrastructure imports.

• Agentic workflows — human-in-the-loop pause/resume, multi-turn conversations, A2A protocol support, MCP tool exposure. These are built into the Asya gateway.

• Multi-transport / multi-cloud — the same actor runs on SQS, RabbitMQ, or Pub/Sub by changing one field in the manifest. No code changes.

• K8s-native operations — you already have Kubernetes with RBAC, network policies, namespaces, GitOps (ArgoCD/Flux), and observability. Asya adds no new cluster runtime — it is Kubernetes.

• Queue-native resilience — if a pod dies mid-inference, the message goes back to the queue. No custom checkpointing, no object store reconstruction. The queue is the checkpoint.

Further Reading#

• Motivation — why async-first, queue-based AI pipelines
• Actor Mesh — choreography vs orchestration
• Scale to Zero — per-actor KEDA scaling