Twitxy Saved

Cloudthrill

@Cloud_Thrill

Welcome to CloudThrill, a dynamic consulting firm specializing in cloud automation, inference & DevOps services. Founded by our passionate @clouddude_ follow
cloudthrill.ca Joined July 2024
82 Following 13 Followers 146 Posts

Latest posts 5

Cloudthrill @Cloud_Thrill
Did you know how different GPUs are from CPUs?
Tech Beats Unplugged @TechBeatsShow 3d
๐—š๐—ฃ๐—จ ๐˜ƒ๐˜€ ๐—–๐—ฃ๐—จ

CPUs time-share thousands of times a second. GPUs can't: a running kernel holds the card until it's done.

That's why Kubernetes still hands out whole GPUs. Daniele Polencic & Mesut Oezdil explain. Full clip: youtu.be/uGuPkCuTqFc

#GPU #Kubernetes
View quoted post on X
Cloudthrill @Cloud_Thrill
We've done a full review of ๐—”๐—œ ๐—š๐˜‚๐—ฎ๐—ฟ๐—ฑ๐—ฟ๐—ฎ๐—ถ๐—น๐˜€. Check the full breakdown and 'When to pick which' below #LLMOps๐Ÿ‘‡๐Ÿฝ
๐Ÿ›ก๏ธ You fixed routing with @LiteLLM or @OpenRouter as one front door for your models. But this wonโ€™t stop a prompt injection, a jailbreak, or an SSN in a shared endpoint. โ€œPut a guardrail on it ...โ€, they say. Nobody says which, or where.

So I tested 12 AI guardrails, OSS vs commercial ๐Ÿงต
View quoted post on X
Cloudthrill @Cloud_Thrill
๐Ÿง  ๐—Ÿ๐—Ÿ๐—  guardrails...which one to choose? Jailbreak, PII ,secret masking
We dissected 12 ๐—”๐—œ prompt guardrails, and wrote down what they do and when to use them. Full article below ๐Ÿ‘‡ #enjoy
๐Ÿ›ก๏ธ๐—Ÿ๐—Ÿ๐—  ๐—ฃ๐—ฟ๐—ผ๐—บ๐—ฝ๐˜ ๐—š๐˜‚๐—ฎ๐—ฟ๐—ฑ๐—ฟ๐—ฎ๐—ถ๐—น๐˜€: OSS vs Commercial
๐Ÿ‘‰ cloudthrill.ca/llm-guardrail-โ€ฆ
So you fixed routing with LiteLLM or OpenRouter. But how do you stay out of trouble with your customers and users?

Guardrail's the thing: catch prompt injections, jailbreak, or SSN in prompts before it becomes your problem.

In this blog I put 12 guardrail tools head to head, open source vs commercial.

The list: Agentgateway, LLM Guard, Guardrails AI, NeMo, Presidio, Llama Guard, Giskard, Lakera, Aporia, Arthur, Patronus, HiddenLayer

Each with pros, cons, best-for, plus a free combo: Agentgateway + LLM Guard via a LiteLLM callback.

#LLM #AISecurity #PromptInjection
View quoted post on X
Cloudthrill @Cloud_Thrill
๐Ÿง  ๐—Ÿ๐—Ÿ๐—  ๐—ด๐—ฎ๐˜๐—ฒ๐˜„๐—ฎ๐˜†s...which one to choose?
We tested & dissected ๐Ÿต ๐—”๐—œ ๐—ด๐—ฎ๐˜๐—ฒ๐˜„๐—ฎ๐˜†๐˜€, and wrote down what actually fits which team. Full article below ๐Ÿ‘‡ #enjoy
๐Ÿšฆ ๐—Ÿ๐—Ÿ๐—  ๐—ฅ๐—ผ๐˜‚๐˜๐—ถ๐—ป๐—ด ๐—ฆ๐—ผ๐—น๐˜‚๐˜๐—ถ๐—ผ๐—ป๐˜€: OSS ๐˜ƒ๐˜€ Commercial
Everyone benchmarks models. Almost nobody questions the gateway in front of them.

In this blog I picked 9 LLM gateways head to head, commercial vs open source: @LiteLLM @helicone_ai @getmaximai @PortkeyAI @kong @truefoundry @merge_api @OpenRouterAI + @llmgateway.io

Pros, cons, strengths, the licensing + pricing traps, and who each is for with a final Head-to-head table๐Ÿ‘‡ cloudthrill.ca/llm-routing-soโ€ฆ
View quoted post on X
Cloudthrill @Cloud_Thrill
How do you shard a Deepseek whale across nodes and still serve it fast? with @raydistributed !
check out our founder's post๐Ÿ‘‡๐Ÿผ
๐Ÿ‹DeepSeek-V3.2 is a monster: 685B params, 643GB of weights that blow past any single node.

So how do you shard a whale this big across nodes and still serve it fast?

I did it with @raydistributed: 16ร— H100s across two @CoreWeave bare-metal nodes, #vLLM + KubeRay, tensor/pipeline parallel, all in one terraform apply.

Full code + demo ๐Ÿ”ฅ
cloudthrill.ca/vllm-deepseek-โ€ฆ
View quoted post on X