Early Look: Increasing potential for a major winter storm this weekend in NYC, but the details remain very complex and uncertain. https://t.co/buVnC3pt79
@AdvaitOnline How would it possibly know that though? Happy to send you an example of it returning three valid URLs that were not in the source prompt and don’t appear to be in the output.
@AdvaitOnline It’s akin to a batch workload (think of potentially 100s of parallel requests) that needs to happen in real-time. Appreciate the responsiveness! We would really love to make Vertex/Gemini work for us as we’re pretty impressed by its output quality.
@AdvaitOnline I would add that this issue is the one thing preventing us moving a decent amount of our inference workload from Anthropic to Vertex/Gemini
@AdvaitOnline Provisioned Throughput works maybe well for consumer-based chat applications. But for enterprise backend workloads like ours with very spiky use, provisioned throughput needed to make our app performant with 100s of requests per 5-min and then dormant for an hour is too costly.