Albert Zeyer

Developer

[Albert Zeyer @albertzeyer]

https://t.co/cSlwWLsecB Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs. Backprop the token probs to the input. Works for any differentiable model. Can beat the model's native alignment (CTC Viterbi, Whisper DTW).

[Albert Zeyer @albertzeyer]

https://t.co/IR6biuk6UH Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition. Two-segment piecewise linear relation in log-log space, joined at internal LM PPL. With @RobinSc71228876 @moezeineldeen

[Albert Zeyer @albertzeyer]

https://t.co/kLYsm190YM https://t.co/4M8jROs4Nx Fun side #Python project PyCPython, interpret CPython in Python. Now I get to the REPL, simple things work.

[Albert Zeyer @albertzeyer]

https://t.co/Xf0V7QNDME Text-Utilization for Encoder-dominated Speech Recognition Models. Comparison of approaches from literature like MAESTRO. Simpler variants like random durations seem to work better. TTS wins by far.

[Albert Zeyer @albertzeyer]

https://t.co/trO2mgV115 Diffusion Language Models for Speech Recognition. Masked and uniform-state diffusion models (MDLM, USDM). Shallow fusion, via rescoring and joint decoding. It is still behind the autoregressive LM, but joint decoding is much faster.

[Albert Zeyer @albertzeyer]

https://t.co/4mJEZMesuq LLMs and Speech: Integration vs. Combination. Comparison. Shallow fusion CTC + ext LLM with delayed fusion performs best. Proposed CTC + speech LLM (prefix LM), also with novel optimizations.

[Albert Zeyer @albertzeyer]

@giffmana I stumbled upon various weight init bugs in HF Transformers. E.g. https://t.co/1xaiICWFyL. I wouldn't trust that lib when starting from scratch. While the model code is usually verified (and easy to verify), I don't think the weight init code is so well tested for every model.

[Albert Zeyer @albertzeyer]

@francoisfleuret In our group, we log all shell commands into the current dir, as `$PWD/.history.$USER`. That way, it's easier to discover, also from other users, it also would be included in backups, or when dirs are copied around, etc. https://t.co/pRQOgDIScZ

[Albert Zeyer @albertzeyer]

@giffmana Related is the problem that beam search finds more OOD sequences and then then the prob becomes unreliable. We often have that for seq2seq and powerful models. Solution to that is seq disc training (max expected reward, min WER, ...), or global normalization, not left-to-right.

[Albert Zeyer @albertzeyer]

https://t.co/06R9ArO3Em Denoising Language Models for Speech Recognition. Outperforming standard LMs in data-constrained setting given enough compute, similar to diffusion LMs. More efficient decoding compared to std LMs. Public recipe, state-of-the-art, comprehensive studies.

[Albert Zeyer @albertzeyer]

@ID_AA_Carmack You could also hope that torch.compile will automatically handle that for you. I think JAX goes a similar route.

[Albert Zeyer @albertzeyer]

Nix wants to cleans all procs of the build user, via setuid(uid) + kill(-1, SIGKILL)(https://t.co/3qU1W4sus2). When nix is run via apptainer/singularity with --fakeroot, it will kill all running procs of your current user even outside of apptainer. --pid is a good idea here...

[Albert Zeyer @albertzeyer]

@ID_AA_Carmack Note that this can increase memory consumption, because those objects are only deleted when they go out of scope. So by reassigning e.g. a tensor, that often would free the previous assigned tensor.

[Albert Zeyer @albertzeyer]

@giffmana The same is also true for infinite context windows (e.g. RNNs/LSTMs), where the effective context lengths was sth like 10-1000 frames (depending on task, also probably model size). But that's maybe more expected.

[Albert Zeyer @albertzeyer]

@utkarsh_2105 One big speedup you get from your Triton kernel is due to the fusing. When using torch.compile, you can get a similar effect. Did you try that?

[Albert Zeyer @albertzeyer]

@KunhaoZ I always wondered, why does it say "Publisher arXiv", but there is no ArXiv link, and I also don't find it on ArXiv?

[Albert Zeyer @albertzeyer]

@giffmana sun -> son

[Albert Zeyer @albertzeyer]

@giffmana I remember that my sun was quite afraid when he was hearing the roar sounds coming out of the computer.

The live coding is a bit challenging with the patience of a 4 year old who has a lot of ideas what to implement.

[Albert Zeyer @albertzeyer]

@jxmnop Europe:
- Self-attention
- Torch
- Successful recipes for RNN LMs
- LSTM

Canada:
- LayerNorm

Asia:
- Residual networks
- CNNs
- Relu