Usage Panda

Open Source Security and Observability Proxy for OpenAI | Enabling Safe LLM Usage for Developers and Organizations

[Usage Panda @UsagePanda]

The Usage Panda proxy is now open source! Check us out on @GitHub.
#opensource #Security #LLM

[Matt Fuller @matthewdfuller]

🌟 Super excited to share that the Usage Panda proxy is now available 100% open source!

This is the first (to my knowledge) open source, security/controls-focused #LLM API proxy.
https://t.co/28hkfNrA9X

[Usage Panda @UsagePanda]

✨ Usage Panda now supports dynamic conversion of OpenAI-formatted API requests to Google PaLM. Easily toggle between OpenAI/PaLM with a one-line change to your codebase.

Coming soon: dynamic failover (fall back to PaLM when OpenAI is unresponsive).

https://t.co/UjhVFhR4eK

[Usage Panda @UsagePanda]

You can also now export your OpenAI API request logs as CSV.
https://t.co/QIFiJ7GUhU

[Usage Panda @UsagePanda]

We've made a number of UI enhancements over the past few weeks to help make it easier to manage your OpenAI requests in Usage Panda 🧵

[Usage Panda @UsagePanda]

The "Logs" page now includes a lot more detail about each OpenAI API request: token usage, cost, latency, model types, operation details, end user IDs, IP address, user agent, error codes, and more. You can also sort and filter across any of these metrics, or open more details.

[Usage Panda @UsagePanda]

📣 Introducing: Usage Panda Self-Hosted Proxy (Beta)

You can now run Usage Panda's OpenAI proxy locally or in your own cloud or datacenter for maximum control over your data.

Docs: https://t.co/JoZJ8JAMgO

[Usage Panda @UsagePanda]

💰 Costs: auto-replies don't consume tokens

🧪 Integration Testing: auto-replies match the OpenAI response format but maintain response consistency

💨 Latency: UsagePanda autoreplies in milliseconds

🟰 Consistency: always want your bot to reply the same way? Autoreply.

[Usage Panda @UsagePanda]

Introducing: LLM Auto-Replies ↩️

Define a default cached response for matching completion or chat prompts. Usage Panda will respond immediately with the defined response, in the correct API format, without calling #OpenAI. No code change required.

This is useful for: 🧵

[Usage Panda @UsagePanda]

✨ New feature: custom word list support.

Define custom words/phrases to audit/block/redact in your prompt or #OpenAI response. Want to know how many times your users are hearing the word "periwinkle" from your bot? Audit it. Never want your bot to talk about Florida? Block it.

[Usage Panda @UsagePanda]

📄 Read more in the docs here: https://t.co/WkJHTXQRbd

[Usage Panda @UsagePanda]

👀 Usage Panda now supports associating costs, errors, and other security/moderation flags w/ end-users of your application.

This dashboard helps quickly answer questions like "which users are driving cost spikes?" or "who is submitting requests with moderated keywords?"

[Usage Panda @UsagePanda]

A real-world example of LLM prompt injection: tricking automated resume screening tools.
https://t.co/0LlHGTBHeh

[Usage Panda @UsagePanda]

🐼 Usage Panda is open to anyone, completely free to use during our public beta! Sign up at: https://t.co/Yte1V9gnXJ. (8/8)

[Usage Panda @UsagePanda]

🔥 New features for Usage Panda!

We shipped a ton of new features this week including multi-user support, content and moderation controls, and more, furthering Usage Panda's mission to enable safe LLM usage for developers and organizations. 🧵 (1/8)
#OpenAI

[Usage Panda @UsagePanda]

🧩 Prompt Tampering Detection: we've enabled an experimental feature that monitors prompts for 100s of common phrases and patterns used to tamper with or reveal prompts. This isn't foolproof, but it will allow you to rate limit or flag users who may be misusing your app. (7/8)