exploguy

[Code With Antonio @codewithantonio]

dammit @joshtriedcoding I was so close to a perfect "nextjs clone" search results page😁

[Vaibhav (VB) Srivastav @reach_vb]

Want high-quality Audio embeddings? CLAP! 👏

We support the latest general, music and speech CLAP models in Transformers! Use it for Text-to-Speech/ Text-to-Music training and more.

What is CLAP?

CLAP (Contrastive Language-Audio Pretraining) is a neural network trained on various (audio, text) pairs. It uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features.

Combined CLAP gives you high-quality audio embeddings blazingly fast! ⚡️

How do you use CLAP?

from transformers import ClapModel, ClapProcessor

# initialise the model
model = ClapModel.from_pretrained(
"laion/larger_clap_music").to("cuda:0")

processor = ClapProcessor.from_pretrained(
"laion/larger_clap_music")

# process the inputs
inputs = processor(
audios=<PASS YOUR AUDIO SAMPLE HERE>,
return_tensors="pt").to("cuda:0")

# generate embeddings
audio_embed = model.get_audio_features(**inputs)

What would you like to embed first? :)

P.S. Remember to check out other CLAP checkpoints in the @laion_ai org on the hub!

That's it! 🤗

[Jack Morris @jxmnop]

cool research idea for someone: text diffusion in embedding space

solve any sequence-to-sequence task in three steps:
1. embed source sentence text
2. build diffusion model that maps input text embedding to target text embedding
3. invert to produce target text

[AK @_akhaliq]

PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics

paper page: https://t.co/c8wy4SIckw

introduce PhysGaussian, a new method that seamlessly integrates physically grounded Newtonian dynamics within 3D Gaussians to achieve high-quality novel motion synthesis. Employing a custom Material Point Method (MPM), our approach enriches 3D Gaussian kernels with physically meaningful kinematic deformation and mechanical stress attributes, all evolved in line with continuum mechanics principles. A defining characteristic of our method is the seamless integration between physical simulation and visual rendering: both components utilize the same 3D Gaussian kernels as their discrete representations. This negates the necessity for triangle/tetrahedron meshing, marching cubes, "cage meshes," or any other geometry embedding, highlighting the principle of "what you see is what you simulate (WS^2)." Our method demonstrates exceptional versatility across a wide variety of materials--including elastic entities, metals, non-Newtonian fluids, and granular materials--showcasing its strong capabilities in creating diverse visual content with novel viewpoints and movements.

[AK @_akhaliq]

ChatAnything: Facetime Chat with LLM-Enhanced Personas

paper page: https://t.co/sAsMTJwKh0

In this technical report, we target generating anthropomorphized personas for LLM-based characters in an online manner, including visual appearance, personality and tones, with only text descriptions. To achieve this, we first leverage the in-context learning capability of LLMs for personality generation by carefully designing a set of system prompts. We then propose two novel concepts: the mixture of voices (MoV) and the mixture of diffusers (MoD) for diverse voice and appearance generation. For MoV, we utilize the text-to-speech (TTS) algorithms with a variety of pre-defined tones and select the most matching one based on the user-provided text description automatically. For MoD, we combine the recent popular text-to-image generation techniques and talking head algorithms to streamline the process of generating talking objects. We termed the whole framework as ChatAnything. With it, users could be able to animate anything with any personas that are anthropomorphic using just a few text inputs. However, we have observed that the anthropomorphic objects produced by current generative models are often undetectable by pre-trained face landmark detectors, leading to failure of the face motion generation, even if these faces possess human-like appearances because those images are nearly seen during the training (e.g., OOD samples). To address this issue, we incorporate pixel-level guidance to infuse human face landmarks during the image generation phase. To benchmark these metrics, we have built an evaluation dataset. Based on it, we verify that the detection rate of the face landmark is significantly increased from 57.0% to 92.5% thus allowing automatic face animation based on generated speech content.

[Sylvain Filoni @fffiloni]

Introducing 🫴 Music To Zeroscope Video ✨
—
Similar to the Music to Image pipeline, but generates a zeroscope video instead of an image.

Follow me @fffiloni to keep track of next improvements

Try @huggingface demo 🤗: https://t.co/FMqWlIcarR

[camenduru @camenduru]

AudioGen v2 Colab 🥳 Thanks to @FelixKreuk ❤ @syhw ❤ Adam Polyak ❤ @urielsinger ❤ @honualx ❤ @jadecopet ❤ @deviparikh ❤ @taigman ❤ @adiyossLC ❤

📄paper: https://t.co/IoX60ktdbH
🧬code: https://t.co/YgoHqNgKDx
🦒colab: please try it 🐣 https://t.co/1yVklkOK5a

[Hirokazu Yokohara @Yokohara_h]

AIでおっさんを美少女化。晒していくスタイルのbefore after。元は可愛い動きや表情をそれほどやってない。画像生成が見る人に可愛い印象や画面全体の統一感を作りだしている

[Java Guides @GuidesJava]

𝗡𝗲𝘁𝗳𝗹𝗶𝘅'𝘀 𝗧𝗲𝗰𝗵 𝗦𝘁𝗮𝗰𝗸: 𝗔 𝗕𝗲𝗵𝗶𝗻𝗱-𝘁𝗵𝗲-𝗦𝗰𝗲𝗻𝗲𝘀 𝗟𝗼𝗼𝗸

[Shaun.AGI @agishaun]

Build a realtime AI character/companion is hard, but it shouldn’t be hard

RealChar. - all in one open source codebase🎙️🤖

Powered by your favorite AI tech
@llama_index, @langchain, @elevenlabs, @trychroma,@openai GPT3.5/4, @AnthropicAI Claude2

Demo first! Repo last!
🧵👇

[AK @_akhaliq]

HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models

paper page: https://t.co/xfDDJtc2a0

Personalization has emerged as a prominent aspect within the field of generative AI, enabling the synthesis of individuals in diverse contexts and styles, while retaining high-fidelity to their identities. However, the process of personalization presents inherent challenges in terms of time and memory requirements. Fine-tuning each personalized model needs considerable GPU time investment, and storing a personalized model per subject can be demanding in terms of storage capacity. To overcome these challenges, we propose HyperDreamBooth-a hypernetwork capable of efficiently generating a small set of personalized weights from a single image of a person. By composing these weights into the diffusion model, coupled with fast finetuning, HyperDreamBooth can generate a person's face in various contexts and styles, with high subject details while also preserving the model's crucial knowledge of diverse styles and semantic modifications. Our method achieves personalization on faces in roughly 20 seconds, 25x faster than DreamBooth and 125x faster than Textual Inversion, using as few as one reference image, with the same quality and style diversity as DreamBooth. Also our method yields a model that is 10000x smaller than a normal DreamBooth model.

[FloorYUKA/Kisaka@3DCG @22av0115]

@ratMGL これはUnreal EngineのCesiumっていうプラグインです。開発者ではないので詳しいことはわかりませんが画期的なシステムになると思います！https://t.co/lNaOHNp05R

[Hirokazu Yokohara @Yokohara_h]

リアルタイムでのおっさんのAI美少女生成2。頭に黒Tシャツ被って髪にした。このカクカクfpsなら配信できる。体含めて全体が生成されるのは画像生成ならでは。日曜午前に何やってるんだとは言わないでくれ

[Modest Mitkus @ModestMitkus]

I make $16,000/month selling digital products.

I started 2 years ago with $0.

Here's my 10-step framework to make money online:

[Madni Aghadi @hey_madni]

People are spending $20/month for ChatGPT.

But you can use it for free.

Here's How in Just 2 Simple Steps:

[younes @yb2698]

Great that Segment Anything from @Meta has been open sourced! You can now use SAM with @huggingface transformers with few lines of code and generate segmentation masks. How to use it? 🧵

More to come very soon in the 🤗 ecosystem!

[peter! 🥷 @pwang_szn]

🪄 INTRODUCING: Chat with any Github Repo.

1) Scrapes @github repo 🤖
2) 🗂️ Embeds codebase using @langchain, stores embeddings in @activeloop
3) Chat with the codebase w/ @streamlit 👨‍💻

Demo, Code explained &amp; Github Link↓

[𝚉𝚊𝚎𝚜𝚊𝚛 @zaesarius]

Meet Bark 🐶

Transformer based text-to-audio. FREE and OPEN SOURCE.

Can sing, laugh, sigh,...

"Bark is a transformer-based text-to-audio model created by Suno. Bark can generate highly realistic, multilingual speech as well as other audio - including music, background noise and simple sound effects. The model can also produce nonverbal communications like laughing, sighing and crying. To support the research community, we are providing access to pretrained model checkpoints ready for inference."

Leave you here the Github repo (also option on Google Colab)

https://t.co/tjbruRad63

[Sanyam Bhutani @bhutanisanyam1]

“Transformers from scratch” by Brandon Rohrer 🤖

This is one of the best write ups, that starts from 0 and explains every single detail of the model architecture.

Even if you need a refresher or don’t, I would still highly recommend reading it:

https://t.co/D25bs6TP5X

[Rowan Cheung @rowancheung]

How to turn long YouTube videos into 5-minute narrated summaries using AI:

(productivity cheat code)