Barni S

[Matteo Pagliardini @MatPagliardini]

A tweak in the architecture of #Transformers can significantly boost accuracy!

With direct access to all previous blocks’ outputs, a 48-block #DenseFormer outperforms a 72-block Transformer, with faster inference!

A work with @akmohtashami_a,@francoisfleuret, Martin Jaggi.
1/🧵

[Stas Bekman @StasBekman]

Getting multi-node SLURM launchers to work can be a time consuming business.

Here are 4 launchers with each component of the script clearly defined so that you can just plug your program in and it'd just work:

https://t.co/dGDs3RFkyt

[Göktuğ @GuvercinGoktug]

Transformers vs Convolution:
--------------------------------

Neural network architectures designed for dense prediction tasks like segmentation and depth estimation generally consist of an encoder to extract the feature representations of given input and a decoder to aggregate all those features into final prediction. In accordance with this, the studies in neural architectural research mostly focus on the design of the decoder to find out correct feature aggregation strategy. The main problem is that it is not possible to retrieve or recover vanished information in decoder part, so encoder has also significant role here.

In convolutional encoders, downsampling operation is commonly applied by either pooling operations or strided convolution, and this is performed in progressive manner. Its first contribution, of course, increases total scope where the features can see in terms of entire image, which is called as receptive field. Furthermore, as the feature maps are downscaled more and more, their low level representations are brought together for the generation of more generalizable and high level features. Since total resolution of those feature maps is constantly decreasing in this process, the network become capable of generating more number of features without exceeding memory and computational constraints.

Transformer backbones do not do such downsampling; they forego that stage, and maintain feature representations in constant dimensionality after initial embeddings are computed. This actually provides huge advantage for transformer-based encoders because convolution loses out granularity in feature space due to downsampling, and how much it can be recovered in decoder stage is still controversial. This kind of problem has the potential to affect the preservation of contours, geometrical consistency and continuity inside histopathological images, curvilinear systems like vessels and anatomical structures observed in endoscopic procedures (colon, lumen etc). Eventually, their segmentation and depth estimation are getting more difficult.

Another interesting point that we need to care about while comparing transformers with convolutional encoders is actually their global and local scope hierarchy: Convolution operation is actually carried out by small learnable filters, so convolution layers can perceive and learn only the local region which their filter fits into. This shows that they posses local learning strategy, and thereby reaching limited receptive field per convolution layer. That is why the design of deep architectures by stacking so many layers to acquire sufficiently broad context and receptive field becomes a must in convolutional models.

To mitigate these problems in convolutional encoder, the following strategies can be used:

- ASPP
- Usage of skip connections
- DualScaleNet approach
- Deformable convolution
- Training with higher input resolution

Recommended Papers:

- Vision Transformers for Dense Prediction
- UNETR: Transformers for 3D Medical Image Segmentation
- ConvNets Match Vision Transformers at Scale
- CvT: Introducing Convolutions to Vision Transformers
- Are Convolutional Neural Networks or Transformers more like human vision?

[Elad Hazan @HazanPrinceton]

most exciting paper *ever* from our @GoogleAI lab at @Princeton: @naman33k @danielsuo @XinyiChen2
https://t.co/aSkBZJ6S9t
*** Convolutional filters predetermined by the theory, no learning needed! ***

[Jascha Sohl-Dickstein @jaschasd]

Have you ever done a dense grid search over neural network hyperparameters? Like a *really dense* grid search? It looks like this (!!). Blueish colors correspond to hyperparameters for which training converges, redish colors to hyperparameters for which training diverges.

[Mustafa Hajij @HajijMustafa]

Topological deep learning (TDL) is a new frontier in deep learning with immense possibilities. Check out our new position paper on TDL https://t.co/FOBhXbI4QQ
@theopapamarkou @ghadazamzmi @ninamiolane @tolga_birdal @mmbronstein @PetarV_93

#TopologicalDeepLearning

[Valeriy M., PhD, MBA, CQF @predict_addict]

A new architecture outperforms all transformers using partial differential equations.

'PDETime: Rethinking Long-Term Multivariate Time Series Forecasting from the perspective of partial differential equations'.

When it comes to hashtag#timeseries hashtag#forecasting transformers is what you do not need -> https://t.co/MpkENNkjlF

#timeseries #forecasting

[Sasha Rush @srush_nlp]

New Griffin paper is really interesting and contains a lot of implementation details
https://t.co/ilmfnDtV58 . Implementation is in Pallas which is a Jax like frontend to Triton/TPU lowering. They show that Associative Scan is inherently worse than Linear Scan in this context. (not sure if this is TPU specific.)

[Tanishq Mathew Abraham, Ph.D. @iScienceLuvr]

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

abs: https://t.co/zUaktEeyBE

New paper from @GoogleDeepMind that proposes two architectures, Hawk and Griffin, that utilize a novel gated linear recurrent layer to replace multi-query attention. These architectures are scaled up to 14B params:

"Hawk-3B exceeds the reported performance of Mamba-3B (Gu and Dao, 2023) on downstream tasks, despite being trained on half as many tokens. Griffin-7B and Griffin-14B match the performance of Llama-2 (Touvron et al., 2023) despite being trained on roughly 7 times fewer tokens."

[Barni S @bharanis]

@foxish_ Congratulations 🎉🎈🍾

[Barni S @bharanis]

@sanandaraman Wordle 203 2/6

⬛⬛🟩🟨⬛
🟩🟩🟩🟩🟩

[Aleksa Gordić (水平问题) @gordic_aleksa]

[🧠 collective intelligence 🧠] I've been intrigued by the cellular automata (CA) concept for a long time, and by the potential, mutually beneficial interaction between CAs and deep learning, so I decided to dig a bit deeper. Here are some interesting resources I found:

1/🧵

[Barni S @bharanis]

@PrasannaPendse The world needs more mangalpani !!

[Barni S @bharanis]

@jeremyjkun Is there a mathematical term for represented by. Isomorphic , subset , ..

[Tomasz Malisiewicz @quantombone]

The Fourier transform is a neural network. Cool blog post for Fourier nerds.

@vincesitzmann check this out, it is relevant to our discussion about SIREN and Hilbert Spaces!

https://t.co/WlLDggQewS

[Barni S @bharanis]

@ian__manchester @ccanonne_ @linguanumerate 😂

[Will Kurt @willkurt]

I mapped out the mathematical connections between all the major probability distributions! The Gamma function is at the root of everything

[Barni S @bharanis]

Infants tailor their attention to maximize learning | Science Advances https://t.co/eoEqqVjj8M

[Andrej Bauer @andrejbauer]

Can someone explain in plain math what a basic subatomic particle is (modeled as)? It's some sort of an eigenvector, isn't it? Please don't hold back on math.