Chu-Cheng Lin

#NLProc and #ML @jhuclsp. Pronouns: i/ta.

[Chu-Cheng Lin @kitsing_l]

Our work "Limitations of Autoregressive Models and Their Alternatives" to appear at #NAACL2021 ! w/ @AaronJaech , Xin Li, Matt Gormley, @adveisner .
https://t.co/A3WX3goPTp
Session 14D (5PM PDT Wed Jun 9 2021)

[Jason Eisner @adveisner]

Everyone's using big autoregressive language models. But ... they predict the next word with a polysized circuit (computation graph). So they can't accurately model settings where that prediction is NP-hard. 😢

[Chu-Cheng Lin @kitsing_l]

@nouhadziri This is interesting work! Wondering if you have thought of connecting this to our previous work on limitations of LMs in general https://t.co/A3WX3g7MRp , to justify assumptions of your theoretical results (Prop. D.1 for example?)

[Chu-Cheng Lin @kitsing_l]

Paper link: https://t.co/XxNeIcUxsK Joint work with @aryamccarthy

[Chu-Cheng Lin @kitsing_l]

I'm presenting our spotlight paper "On the Uncomputability of Partition Functions in Energy-Based Sequence Models" at ICLR poster session 5 until 3:30pm EDT! #ICLR2022

[Chu-Cheng Lin @kitsing_l]

Even though many parameter estimation methods have been proposed, they stop working for model families that are expressive enough to parametrize our pathological EBM.

[JHU CLSP @jhuclsp]

Capabilities of autoregressive AI models will always be limited by their inability to reason like humans, says @JHUCompSci PhD candidate and CLSP member Chu-Cheng Lin (@kitsing_l). Read more about Lin's research in the latest from the @HubJHU! https://t.co/oXo1CuxMkI

[Austin Blodgett @austinblodgett5]

Take a look at our ACL camera-ready presenting our structurally-comprehensive AMR-to-text alignments. @complingy
https://t.co/n9H9v6PCbL

[Chu-Cheng Lin @kitsing_l]

@jacobmbuckman @adveisner Sure! Let me know if you have any other questions / comments 😀

[Chu-Cheng Lin @kitsing_l]

@jacobmbuckman @adveisner A bit more on ‘tell whether a prefix is valid’: assuming we could model this language using an AR, this AR would assign nonzero prob to a prefix iff the prefix is a TM that can halt. Does this make sense?

[Chu-Cheng Lin @kitsing_l]

@jacobmbuckman @adveisner ...which is already undecidable, let alone the polytime requirement. On the other hand *verifying* a TM + an exec trace is doable in polytime.

[Chu-Cheng Lin @kitsing_l]

@jacobmbuckman @adveisner (I was referring to our proof of thm 5 in the paper: https://t.co/VRbwRPA2QS )

[Chu-Cheng Lin @kitsing_l]

@jacobmbuckman @adveisner In our construction in the paper, we simply don’t have non-halting-machine prefixes in our language. Having a no halt special token would probably make it impossible for an EBM to score the string in polytime.

[Chu-Cheng Lin @kitsing_l]

@jacobmbuckman @adveisner Oh, sorry 😞 I had a typo above. Meant to write ‘ time polynomial in string *length*’

[Chu-Cheng Lin @kitsing_l]

@jacobmbuckman @adveisner By weights I meant unnormalized probabilities (in the context of EBMs)

[Chu-Cheng Lin @kitsing_l]

@jacobmbuckman @adveisner Sorry I wasn’t clear: we can weight strings in time polynomial in string weight, regardless of their validity. This can be done in polytime for some Turing machines (you just need to simulate their execution trace over time).

[Chu-Cheng Lin @kitsing_l]

@jacobmbuckman @adveisner (I wrote "assign positive probability to p(x'#)" a few messages ago but it was not accurate. I meant that x'# is a good prefix with positive-weight continuations. I should have written Z(x'#) )

[Chu-Cheng Lin @kitsing_l]

@jacobmbuckman @adveisner Note that in both cases, an energy-based model can assign positive weights to (and only to) valid strings, in polynomial time.