Insu Han

Assistant Professor @KAIST

[Insu Han @insu_han]

Most infinitely wide NTK and NNGP kernels are based on the ReLU activation. In https://t.co/k9JMnT1eoe, we propose a method of computing neural kernels with *general* activations. For homogeneous activations, we approximate the kernel matrices by linear-time sketching algorithms.

[Insu Han @insu_han]

We open-source NNGP and NTK for new activations within the Neural Tangents dev https://t.co/qKD7oi3ta1 and sketching algorithm at https://t.co/hDniQjO74s

Joint work with Amir Zandieh @ARomanNovak @hoonkp @Locchiu @aminkarbasi

[Insu Han @insu_han]

Most infinitely wide NTK and NNGP kernels are based on the ReLU activation. In https://t.co/k9JMnT1eoe, we propose a method of computing neural kernels with *general* activations. For homogeneous activations, we approximate the kernel matrices by linear-time sketching algorithms.

[Insu Han @insu_han]

Still, 
computing full NTK matrices is a big pain, e.g., 5-layer Convolution NTK requires 151 GPU hours.
We accelerate the NTK approximation by sketching techniques and provide a tight point-wise error bound. Our approximation takes only 1.5 GPU hours (x106 speedup) 🫢

[Insu Han @insu_han]

We study another simple approach for the dual kernel by Gauss-Hermite quadrature:

[Insu Han @insu_han]

In addition, we propose how to automatically compute the dual kernel of the derivative without the activation, which is useful to characterize the NTK with an unknown activation (e.g., normalized Gaussian) or whose dual kernel of the derivative is unavailable (e.g., GeLU, ELU).

[Insu Han @insu_han]

Our derivations are based on (1) an explicit expression of dual kernel by Hermite polynomials and (2) the fact that Hermite polynomials can play a role of random features of monomial kernels. They allow inputs from the entire *R^d* space.

[Insu Han @insu_han]

We first characterize a kernel function of a single-layer neural network (a.k.a. duel kernel) of various activations. This is a key block for the NNGP and NTK of deeper architectures.

[Elvis Dohmatob @dohmatobelvis]

Good news: Our paper on Scalable learning and MAP inference in nonsymmetric Determinantal Point Processes https://t.co/ZsJPo0WlDb has been accepted (oral) at #ICLR2021. Joint work with @mikegartrell, @insu_han, Victor-Emanuel Brunel, and Jennifer Gillenwater.

[Elvis Dohmatob @dohmatobelvis]

Happy to share our recent preprint with
@mikegartrell, Insu Han, V.-E. Brunel, J. Gillenwater,
on Scalable learning and inference in nonsymmetric det point point processes https://t.co/CRPE8jVWgi.

This is follow-up on our previous work https://t.co/1hoXFsGCKV

Please RT.