Nirav Patel

Software Engineer
talking about #Tech #finance #fintech

[Nirav Patel @nlathiya565]

@mindtensor_ @akshaymarch7 @deVamshidhar @Iam_DEv22 I mentioned that show any reliable previous work if not then list of promise which can be trustable. Private sector reservation? Cast census? Do you vote alternative for this ?

[Nirav Patel @nlathiya565]

@akshaymarch7 @deVamshidhar @Iam_DEv22 Can you tell us any other alternative than it ? Can you able to recall any leader which deliver promise to remove reservation, fix public and civic infra or provide any sample model (like Gujarat model) which we can rely on it ? Would like to know your take

[Nirav Patel @nlathiya565]

@myprasanna Honestly asking, who should be replace him in the power ? Would like to know your take here

[Nirav Patel @nlathiya565]

@divyaporwal_ @i_m_Ankit_001 @grok Sabko Jagna padega then only this bottom level power people get pressurised to work done

[Nirav Patel @nlathiya565]

@divyaporwal_ @i_m_Ankit_001 @grok We need to understand that we need to start the questioning from bottom for civic and public infra. Ask question to your area counsellor or MLA or MP and make then responsible for this regardless whichever party they are from and this is what right constitution give us

[Nirav Patel @nlathiya565]

@divyaporwal_ @i_m_Ankit_001 Forgot to include the work done energy and shipping sector. As per my understanding intent is always right but somehow they are missing on execution across the level which if they can sorted then it’ll be beast

[Nirav Patel @nlathiya565]

@divyaporwal_ @i_m_Ankit_001 Problem is that we just got the thing since last 7-8 years as we were kid pre 2015 era so we could not figured out the what has done in past and comparison with current era

[Nirav Patel @nlathiya565]

@akshaymarch7 By the way you forgot to rectify one thing which you posted yesterday without fact check so when can we expect ? Yes this is also wrong but here also you should come up with pure fact

[Nirav Patel @nlathiya565]

@Arpitaaa01 @skt18101999 Starlink is absolute beast no doubt about that but it has to follow security norms because security always come ahead of accessibility

[Muthukrishnan Dhandapani @dmuthuk]

I’ve touched upon this subject a few times before.

When the families of top politicians and bureaucrats obtain citizenship and settle in foreign countries like the United States, it creates an inherent conflict of interest.

The kith and kin of those running our country acquiring citizenship in a country like America is a huge red flag.

This does not apply to ordinary private citizens of our country.

Either their family members should renounce the foreign citizenship and return to India, or those who run the country, whether politicians or bureaucrats, should automatically lose their positions once members of their family acquire overseas citizenship.

[Nirav Patel @nlathiya565]

@shreyacasmalert @Shivam_2610_ Again, start questioning from the Bottom more effectively (along with Top of the power ) which is your area Corporators, MLA, MP which you sent them to legislative and parliament and make them responsible. people's are not aware or forgot about their area MLA or Corporator

[Nirav Patel @nlathiya565]

@heyyyyanand @divyaporwal_ Current era is not about "Good" vs "Bad" but It is "Less Evil" vs "More Evil". start the questioning to your area representatives which only can eventually put pressure at the top of power

[Nirav Patel @nlathiya565]

@divyaporwal_ @_yours_majesty The day people will start the questioning their area cooperator, MLA, MP regarding basic civic and public infra and making them responsible, we will not need to questioning at the top. start questioning from the Bottom which we sent them to legislative and parliament

[Nirav Patel @nlathiya565]

@himanshu170499 @vanshyadav1408 @AnyBodyCanFly @aravind There is thing name with data and statistics. However everyone is agreeing on the pace of this urban issue fixing but It'll take a time

[Nirav Patel @nlathiya565]

@vanshyadav1408 @AnyBodyCanFly @aravind If you don't have any logical answer then it does not mean to start the barking

[Nirav Patel @nlathiya565]

@malpani @DIWAKARKAUSHAL Then why US is not allowing Chinese EV which is clearly on interference of Elon. Don’t write done blindly

[Nirav Patel @nlathiya565]

@AnthonyGalli @Reshmanair356 😂 what a joke

[Tech with Mak @techNmak]

Inference Engineering is becoming one of the most important skill sets in AI, and I think a lot of engineers are still underestimating it.

Training gets most of the attention because that is where the model is created. But once a model has to serve real traffic, a different set of engineering problems appears.

Requests arrive with different prompt lengths, output lengths, cache reuse, arrival times and latency requirements. Now you are dealing with queueing, batching, KV-cache capacity, GPU memory, scheduling, kernel efficiency, communication and tail latency.

The first thing worth understanding is that inference is not one homogeneous workload.

Prefill processes the input prompt and creates the KV state that will be reused during generation. Many operations across the prompt can be executed in parallel, so prefill usually has much higher arithmetic intensity and can make good use of GPU compute.

Decode is different. Each sequence advances autoregressively, normally one new token per decoding step. The system repeatedly reads model weights and the growing KV cache while doing relatively little computation for each token. At practical serving batch sizes, decode can therefore become heavily constrained by memory bandwidth.

That prefill/decode split explains a lot of modern inference engineering.

FlashAttention reduces reads and writes between HBM and on-chip SRAM while computing exact attention.

PagedAttention solves a different problem. It manages KV-cache memory in blocks instead of requiring each sequence to occupy one large contiguous allocation, reducing fragmentation and making dynamically growing caches easier to manage.

MQA and GQA reduce the number of key/value heads relative to standard multi-head attention, which reduces KV-cache storage and the amount of KV data that must be moved during decoding.

Scheduling matters just as much.

Continuous batching lets requests enter and leave the active batch as generation progresses instead of forcing a fixed group of sequences to finish together.

Chunked prefill breaks a large prefill into smaller pieces that can be scheduled alongside decode work. This gives the scheduler much more control over the interference between long prompts and already-running generations.

Prefix caching goes after redundant work from another direction. If requests share an already processed prefix, the corresponding KV state can be reused instead of recomputing that part of the prompt. Importantly, this saves prefill computation. It does not make generation of the new output tokens themselves faster.

Then there is memory.

The model weights are only part of the GPU memory footprint. The KV cache grows as more tokens are kept for active sequences. For conventional full-context attention, its size depends on the number of layers, cached tokens, KV heads, head dimension and bytes used for each cached value.

That is one reason MQA and GQA matter so much for serving. Fewer KV heads mean less cache per token.

Quantization attacks the problem from another direction. Lower-precision weights can reduce model memory and memory traffic. KV-cache quantization can also reduce the memory occupied by cached state, although the exact performance and quality trade-offs depend on the format, hardware and implementation.

Speculative decoding tries to reduce the cost of autoregressive generation differently. A cheaper draft mechanism proposes several candidate tokens, then the target model verifies them together. The actual speedup depends on things such as drafting cost, acceptance rate, verification efficiency and workload characteristics. It is not automatically faster for every deployment.

Scaling across GPUs introduces another layer of engineering.

Tensor parallelism shards model operations across devices, reducing the amount of model state each GPU must hold but introducing communication between them.

Pipeline parallelism assigns different groups of layers to different stages, which brings pipeline scheduling and bubble problems.

Data parallelism keeps replicas of the model and distributes requests across them.

MoE serving adds expert parallelism, token routing, communication and possible load imbalance between experts.

Prefill and decode can even be placed on separate GPU pools. Systems such as DistServe do this because the two phases have different performance characteristics and latency objectives. The trade-off is that the KV state produced during prefill then has to move to the decode side.

This is also why tokens per second is an incomplete way to evaluate an inference system.

You need to watch TTFT, the time until the first token reaches the user.

You need ITL, the delay between streamed tokens.

You need throughput, queueing time, concurrency, KV-cache occupancy, cache hit rates, GPU utilization, memory bandwidth, communication cost and tail latency.

And increasingly, you need goodput: how much request load the system can actually serve while still meeting its latency objectives. A server can report excellent aggregate throughput while users are still waiting too long for the first token or seeing slow generation.

The metrics also tell you where to start looking.

High TTFT often points toward queueing, scheduling or expensive prefill.

Poor ITL or TPOT can point toward decode efficiency, memory bandwidth or KV-cache traffic.

OOMs that appear as concurrency rises often mean the combined memory pressure from weights, KV cache and runtime state has become too large.

Poor multi-GPU scaling should make you look closely at communication volume, collectives and topology.

There usually is no single optimization that fixes everything. Larger batches may improve throughput while increasing latency. More parallelism may make a model fit while adding communication. Prefix caching can be extremely useful for workloads with repeated prefixes and nearly irrelevant when every prompt is different.

This is what makes inference engineering interesting.

It sits between ML systems, distributed systems, GPU architecture, kernels, networking, scheduling and performance engineering.

The model decides what token should come next.

Inference engineering determines how much serving that token costs, how quickly it reaches the user, and how many users the hardware can support at once.

That is why I think inference engineering is going to matter far more over the next few years than most people currently realize.

[signüll @signulll]

i was in a random bookstore & came across a book that described achieving moksha as achieving “super intelligence”.

the hindu religion figured it out a while ago.