Jose Cordova Garcia

.

[Tiffany Vlaar @TiffanyVlaar]

📣 Hiring! Reach out to discuss PhD opportunities on maths of deep learning and climate change AI in my group at @UofGlasgow. Drop me an email (with motivation & CV) and find more info on my websites: https://t.co/RtbPzKnPaf and https://t.co/eoieK9ciZd #PhD #ML #AI #Mathematics

[Anshul Gandhi @PACELab_SBU]

(please RT) The Call for Papers for @ACMSigmetrics 2025 is now online! The first deadline is July 31/Aug 7. As an added and welcome incentive, I hear that the acceptance rate will be higher this time :-) https://t.co/CMvpbG2Obz

[Bartolomeo Stellato @b_stellato]

Missed the chance to register for the @Princeton Workshop on Optimization, Learning, and Control? Don't worry, you can still join us! 🎉

We'll be live streaming the event on June 27-28, 2024. 📅

🔗 Tune in here: https://t.co/oHhqKabNBC

#OLC24 #Optimization #Learning #Control

[Robert M. Gower @gowerrobert]

New optimization paper alert! "Directional Smoothness and Gradient Methods: Convergence and Adaptivity". We develop new sub-optimality bounds for gradient descent that depend on the local smoothness (curvature) along the optimization path.

[Jose Cordova Garcia @jecordov]

This 🧵 will make anyone in optimization and learning smile, nicely summarized, adding to reading list

[Mateo Díaz @mateodd25]

This paper made me smile a lot while working on it, so I want to share a bit about it https://t.co/egypZTIky9.

We draw a parallel story to the Eckart-Young Theorem (from numerical analysis) in stochastic optimization/learning problems.

(with Josh Cutler and Dima Drusvyatskiy)

[justin theroux’s hair plugs @mcstardently]

Spotify has gotten kinda unbearable to use now but I’m like 10 years deep so I’m going down with the sinking ship I guess

[Mark Tenenholtz @marktenenholtz]

Useful training tip: when using Adam, don't double your learning rate when you double your batch size.

Instead, scale it by sqrt(2).

I wasn't aware of this trick, but empirically I have noticed over the years that you need sublinear scaling, otherwise losses go crazy.

[Jose Cordova Garcia @jecordov]

Tuning binary clf thresholds made easy in scikit-learn!

[Guillaume Lemaitre @glemaitre58]

We merged my most wanted scikit-learn feature: a pair of estimators to modify the decision threshold of a binary classifier. Nice use case: https://t.co/YPShaNyIl2; and avoid some traps: https://t.co/pvXEwCWCCg. Thanks to all reviewers and contributors.

[Jose Cordova Garcia @jecordov]

We had a very interesting discussion about using AI for Sustainability at the Graduate School of Business! Thanks @Ramavill88 for the invitation to @espae's Faculty Colloquium

[ESPAE, Escuela de Negocios de la ESPOL @espae]

📣El Dr. José Córdova brindó una charla sobre "Aplicaciones de Analítica e Inteligencia Artificial para Sostenibilidad" en el Seminario para Profesores de ESPAE.

🤖Se exploró cómo la IA y la analítica están transformando la predicción de consumo de energía.

[Luciana Benotti @LucianaBenotti]

Atención #nlproc Latinoamérica! #naacl está pensando cambiar su nombre para representar a todo el continente (como dice actualmente su constitución). Nos encantaría escuchar todas sus voces en esta encuesta!

Me ayudan con un retweet?

[Jose Cordova Garcia @jecordov]

Insightful advice on studying and doing research in optimization

[Konstantin Mishchenko @konstmish]

Is optimization still a good topic to work on for a PhD? Here is my detailed response to a student who reached out asking this question.

[Alejandro Macias @doctormacias]

Generalmente culpamos al frío por las infecciones respiratorias. Sin embargo, es la falta de ventilación la que nos enferma, pues en tiempo de frío nos enclaustramos, cerramos puertas y ventanas, y empezamos a respirar todos el mismo aire cargado de virus en aerosoles.

[Sabrina Shankman @shankman]

We hear a lot about waste from the clean energy industry. It's definitely something to be aware of and account for ... but I found this graphic in @DanGearino's latest @insideclimate newsletter to be pretty eye-popping. https://t.co/UWb8KUhUcu

[Damek @damekdavis]

This 2019 paper empirically shows that the performance of SGD saturates after a "critical batch size."

1. How are ppl finding critical batch size?
2. What's the best theoretical explanation?
3. Is there any conflicting empirical evidence?

[Jose Cordova Garcia @jecordov]

Insights on third order magic in first order methods 🙌

[Jeremy Cohen @deepcohen]

I think the reason why second-order methods keep underperforming relative to first-order methods in deep learning is that first-order methods are more powerful than the theory gives them credit for. First-order methods + large step sizes can implicitly access specific **third

[Jose Cordova Garcia @jecordov]

True commitment

[“paula” @paularambles]

thinking about the time the PM on my team was annoyed that none of us engineers were reading her docs so she created a github repo and opened a PR with her latest doc and asked us to review it

[Yann LeCun @ylecun]

There is a basic misunderstanding.
High-dimensional *input* makes modeling hard.
High-dimensional *parameter space* makes model estimation easy.
People have known this for a very long time.
For example, ADMM & EM procedures artificially increase the dimension of the parameter space with auxiliary variables so as to make optimization easier.
Kernel methods are based on the idea that if you have one parameter per training sample, you can learn whatever you want (you can get good generalization only if your kernel function is appropriate).
The intuition that making neural net bigger makes them easier to train (and makes local minima go away) is very old, even if theories that explain why are more recent.
The idea that had a hard time becoming mainstream is that a widely over-parameterized neural net can *generalize* well.
That was contrary to every statistics textbook, and every machine learning theory results.

[Jose Cordova Garcia @jecordov]

This course is available to all majors at @espol and we had students from 6 different ones! Surely all of them showed they will contribute in their own field to make things smarter! All projects used Teachable Machines by @GoogleAI 2/2

[Jose Cordova Garcia @jecordov]

This year we set our "Smart Environments" course at @espol_fiec to be 1/2 IoT/1/2 AI. Yesterday it was our last day and we saw very cool uses of pose estimation and object detection along with sensors and actuators for smart applications in buildings, healthcare, agriculture 1/2

[Jose Cordova Garcia @jecordov]

Gorilla's strategy to make API calls 🦍📞

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

There’s been a lot of recent discussion about the Gorilla language model. This unique model learns the ability to coordinate, access, and use a variety of different deep learning model APIs as tools for problem solving. Here’s how the LLM is trained to do this...

Tool usage in LLMs. Recently, research on language models has heavily explored the idea of integrating LLMs with external tools, either via a prompting (e.g., HuggingGPT) or fine-tuning approach (e.g., Toolformer). LLMs can learn to use a variety of tools (e.g., search engines, calculators, etc.) by just learning to generate a call to an API associated with the tool.

Deep Learning Models as Tools. One of the most interesting types of tools that an LLM can learn to use is other deep learning models. Countless models exist for solving specific, useful tasks (e.g., object detectors, classifiers, image generators, etc.) and can oftentimes be queried for their output using online APIs.

Gorilla. The Gorilla LLM uses a fine-tuning approach to learn how to leverage a massive set of external deep learning model APIs when solving problems. Using the self-instruct framework, authors generate a dataset of valid API calls to over 1,600 different deep learning models (from HuggingFace, PyTorch Hub, and TensorFlow Hub) that are used to fine-tune the LLM.

Models as tools. The deep learning models that Gorilla learns how to use span a variety of domains, including CV, audio, RL, and more. The relevant information and documentation for each deep learning model API is stored as a json object. GPT-4 generates ten user question prompts and associated responses to go with each API, which are then filtered based on correctness.

Retrieval-aware fine-tuning. The fine-tuning process used by Gorilla is a bit different than typical SFT. The model uses retrieval-aware fine-tuning to learn how to make valid API calls. This might sound complex, but really it just means that, in the fine-tuning dataset, we prepend every call to an API with up-to-date documentation for that API.

“Supporting a web scale collection of potentially millions of changing APIs requires rethinking our approach to how we integrate tools.” - from Gorilla paper

Why is Gorilla unique? Because of its retrieval-aware fine-tuning approach, the Gorilla model learns to leverage an API's current documentation to craft valid API calls. As such, the model can adopt a retrieval approach at test time that retrieves up-to-date documentation, allowing the model to properly leverage thousands of tools with (potentially) changing documentation when attempt to solve a problem.