Pedram

PhD Fellow (Copenhagen) | prev. Danske Bank SWE 🏦 roulette/poker croupier 🎰 | all bets are my own

[Pedram @pedrambakh]

When I'm introduced to a new board game, each friend takes a turn explaining their most essential strategy, so I end up with virtually every single game detail. Sometimes, we even use the first 20 minutes reading said details from the rulebook... As soon as we start playing, however, nearly all these details end up being irrelevant for actually playing the game. We really only need to look at the rulebook, so I can convince Jonas that it is indeed possible to achieve world domination in a single turn, although I seemingly did nothing for 3 hours and swept in at the very last minute to take it all (Board game: Risk, 6-players).

Anyways, for that reason, I skip almost all details when introducing someone to a new game, and try to mention only the 2 or 3 compressed scenarios I think are the most important to understand (filling in details as needed). When it works, it's certainly much more effective, yet my friends' approach, in contrast, always works. But I don't think it's a lack of detail, more a question of how the scenarios are structured (also in the rulebooks). And even when it fails, just repeating the information, the same (word-for-word) information, once or twice, usually gets everyone on the same page.

To the same tune, many approaches towards making neural networks more efficient identify seemingly redundant or repeated details and remove them. Here, as with many other things in life, we might've slightly overlooked the significance of the details, especially those that are repeated to us. If we can't see the forest for the trees, we tend to burn a few of them, increasing our FOV. Rather, I'd argue, that many times, the forest is easier to see, were we to plant trees among those already there. In that sense, repeatability and similarity between details can also help reveal the bigger picture, not just obfuscate it.

Besides this (more personal) lens, we take up the perspective of algorithmic complexity in our recent work, Algorithmic Simplification of Neural Networks with Mosaic-of-Motifs, asking why neural networks, much like our board game rules for world domination, are suited for compression. We demonstrate that parameters of trained models have more structure and, hence, exhibit lower algorithmic complexity compared to the weights at (random) initialization. In turn, we present a constrained model parameterization (MoMos) that induces repeatability and structure in neural networks, yielding models with lower algorithmic complexity, including a theoretical justification for how the parameterization settings control this complexity.

Paper: https://t.co/T9vS7CBbv2
ABC: https://t.co/t4aRZIZyXV,asking why neural networks, much like our board game rules for world domination, are suited for compression. We demonstrate that parameters of trained models have more structure and, hence, exhibit lower algorithmic complexity compared to the weights at (random) initialization. In turn, we present a constrained model parameterization (MoMos) that induces repeatability and structure in neural networks, yielding models with lower algorithmic complexity, including a theoretical justification for how the parameterization settings control this complexity.

A repeated, but not redundant, thank you to my co-authors Tong Chen, Jonathan Wenshøj, Erik B Dam, and Raghavendra Selvan, for making it easier to see the forest for the trees, and a special thanks to Eduardo Yuji Sakabe for planting some more along the way.

[Pedram @pedrambakh]

@0xSero Have you set the Duolingo bird on fire as revenge, or what is this?

[Pedram @pedrambakh]

Energy efficiency is often conflated with carbon efficiency, and carbon efficiency with environmental sustainability, which is then mistaken for the sustainability of AI. Even so, ''green'' is not the same as ''sustainable''.

Perhaps, contra to intuition, it's not clear why environmental sustainability (climate awareness) actually works against social and economic sustainability (resource awareness), but neither dimension can solve for sustainability alone.

In our ICML 2026 paper (together with Pınar Tözün, Christian Igel, and Raghavendra Selvan), we lay out the dynamics surrounding contemporary AI development for sustainability and address the (often overlooked) social and economic facets. Importantly, we argue that neglecting the sustainability of AI is fueling a global AI arms race and the use of unprincipled hyperbole in recent discourse on AI.

Paper: https://t.co/wa432byhd9

[Pedram @pedrambakh]

If you ever think it pays off hating.. keep waiting.

[Pedram @pedrambakh]

@olgazaghen I highly recommend Bukhansan for a hike!

[Pedram @pedrambakh]

After high school, I worked as a roulette and poker croupier, even though I had never set foot inside a casino before; it simply seemed like an interesting job. What stood out to me was how easily betting strategies could be misjudged. A single large bet followed by a large win, and someone would walk away convinced the strategy worked, even though they had been losing small amounts for hours and were net negative overall. Too close to see the forest for the trees. If the budget did not allow losses to be recouped with a single large bet, however, it became blatantly obvious that the strategy was, at best, ineffective and, at worst, simply bad.

It seems that ML behaves similarly. A large and infrequent jump in performance, utility, or features masks a steady increase in cost. More parameters, more compute, and longer training cycles; the headline result becomes the justification rather than the strategy itself, and the slow accumulation of inefficiencies is disregarded. When we instead constrain ourselves to work more efficiently, inefficiencies show up immediately, and progress does not depend on increasing the budget.

Surely it is easier to accept the direction of bigger and more, often mistaken for progress. Yet, if this consistently coincides with growing overhead, then the strategy requires closer inspection. My interest in algorithmic complexity theory, compression, and quantization stems from this very instinct, where the focus is on the small losses, not just the occasional large wins, and we question whether we are confusing scale with understanding, performance with progress, and features with utility, to justify our strategies.

From this perspective, I recently started my PhD in Sustainable ML at the University of Copenhagen (@uni_copenhagen), supervised by Raghavendra Selvan (@raghavian) and Erik B. Dam, as part of the Sustainable Artificial INTelligence for Sciences (SAINTS) Lab.  And while solid progress has been made since this observation, I’ve only become more excited to work on ideas for ML strategies that are less dependent on the budget and more worthwhile.

[Raghav{endra}| ರಾಘವೇಂದ್ರ @raghavian]

So, I have been working hard on a book for > 1 year now. Something that I am extremely excited about, as it is trying to present a holistic view of my research domain at the intersection of #ArtificialIntelligence and #Sustainability. It is simply called "Sustainable AI".

[Raghav{endra}| ರಾಘವೇಂದ್ರ @raghavian]

100k pip downloads of #Carbontracker 🥳🥳🥳
Kudos to the dev team.
https://t.co/Uinzp0AyHx

[Raghav{endra}| ರಾಘವೇಂದ್ರ @raghavian]

Your regular reminder that #Carbontracker (first published in 2020) can be used to measure the #energy consumption and #carbonfootprint of not only your #DeepLearning models but any compute intensive jobs.
Check out the website: https://t.co/Uinzp0AyHx

[Frederik Lizak Johansen @fupperlidt]

#KDD2024 🚀
I'm delighted to share that our latest paper, featuring #CHILI, a graph #ML dataset for inorganic materials chemistry, will be presented at @kdd_news!

Kudos to @UlrikFJ1 @SodeAndy @raghavian @kirsten_mj !🎉

[Videnskab.dk @videnskabdk]

AI vil sluge lige så meget energi som hele Sverige. Nu har københavnske forskere en løsning https://t.co/470LBwJ1vd #dkgreen #dkklima #dkvid

[Raghav{endra}| ರಾಘವೇಂದ್ರ @raghavian]

More coverage for our work (with @pedrambakh, @christian_igel) presented at @ieeeICASSP on energy consumption aware #DeepLearning in the Danish magazine @videnskabdk
@DIKU_Institut

[Raghav{endra}| ರಾಘವೇಂದ್ರ @raghavian]

.@pedrambakh is ready! Drop by at 3A.10 in the first poster session of @ieeeICASSP starting now to hear about Energy consumption aware neural architecture search. #ICASSP2024 #DeepLearning #EnergyEfficiency

[Raghav{endra}| ರಾಘವೇಂದ್ರ @raghavian]

Two of our papers will be presented at @ieeeICASSP. First time conference attendees @pedrambakh and @Sebeliassen will be presenting [1] and [2], respectively.

#ICASSP2024
[1] https://t.co/V6jQdBSXQ8
[2] https://t.co/p5BKSRjeC3

[Raghav{endra}| ರಾಘವೇಂದ್ರ @raghavian]

Our work was featured in an evening news broadcast on @tv2newsdk yesterday, and @pedrambakh was amazing!
Thanks @Maria_Hornbek for the work!

If you have subscription, you can watch the segment here:
https://t.co/EEpe6Uh4Sk

[SCIENCE - KU @science_ku]

A single ChatGPT prompt is estimated to consume, on average, as much energy as forty mobile phone charges. Computer scientists at SCIENCE have created a recipe book for designing AI models that use much less energy without compromising performance: https://t.co/YPgsD5ZtQB #ai

[Raghav{endra}| ರಾಘವೇಂದ್ರ @raghavian]

In our second @ieeeICASSP paper, we present EC-NAS: a tabular dataset for performing energy consumption aware neural arch. search.

Now also with a press release from @UCPH_Research
https://t.co/XNJKUlB23d

[Raghav{endra}| ರಾಘವೇಂದ್ರ @raghavian]

Excited about the first of our two @ieeeICASSP papers. This one with an oral pres.

We improve upon existing activation compression for GNNs. Down to 2-bit quantization with almost no performance loss, with some nice variance minimization properties.

https://t.co/p5BKSRjeC3

[Dustin Wright @dustin_wright37]

Come work w/ me and @IAugenstein on factual text generation in Copenhagen!