Coffee and crackers

common human

[Yohan @yohaniddawela]

One of my favourite papers from this year:

A new global dataset measuring local government corruption.

Here's the breakdown and how to access it:

[Hudson Golino @GolinoHudson]

Do you like AI/LLMs/Network Psychometrics? Have you heard about our new AI-GENIE framework? And have you heard about our new Exploratory Graph Model? Let's combine everything together? Read below...

AI-GENIE:

Our new method termed AI-GENIE (Automatic Item Generation and Validation via Network-Integrated Evaluation; Russell-Lasalandra, Christensen, & Golino, 2024), introduces a fully automated system that revolutionizes human assessment development by developing new items for tests, scales, questionnaires, and surveys using LLMs
and automatically validating them in silica using my network psychometric methods.

We've also developed the AIGENIE R Package to seamlessly integrate R, Python, and cloud-based LLM services.

Exploratory Graph Model:

The Exploratory Graph Model (EGM) introduces a mathematical framework for psychological measurements through networks rather than latent variables, treating psychological dimensions as emergent from direct variable interactions. EGM provides a formal alternative to traditional psychometric approaches (Factor Analysis), offering new possibilities for network-based research.

Using AI-GENIE to develop items of Pathological Narcissism

Narcissism is a self-regulatory system for maintaining positive self-views through managing one’s self-image, emotions, and social interactions, which in its healthy form drives achievement and ambition while preserving self-esteem, though potentially at the cost of some interpersonal friction (Pincus & Lukowitsky, 2010). Pathological narcissism, on the other hand, is a dysfunctional self-regulatory pattern characterized by oscillating between grandiose self-inflation and vulnerable self-depletion, leading to impaired coping with disappointments, emotional distress, and interpersonal difficulties, often accompanied by other psychiatric conditions (Pincus & Lukowitsky, 2010).

We used the recently developed AI-GENIE to develop items of pathological narcissism (grandiose and vulnerable narcissism) using large language models. The items were generated following the steps described by Russell-Lasalandra et al. (2024), and the embeddings of the items (high-dimensional continuous vectors ) were used to check if the structure of pathological narcissism fits better an EGM or a factor model.

EGM Structure of Pathological Narcissism:

Fit of the EGM Structure vs. fit of a Factor Model:

Items:

The generated items and their respective communities are presented below.

Community 1 (Interpersonal Insensitivity):

• I rarely consider others feelings when focusing on my own needs,

• I often find it difficult to genuinely care about the problems of others,

• I struggle to put myself in other peoples shoes,

• I often overlook the feelings of others while pursuing my own goals,

• I struggle to genuinely connect with others emotional experiences,

• I seldom take the time to understand how others feel,

• I find it challenging to comprehend the emotions of those around me.

Community 2 (Grandiose Exhibition):

• I enjoy being the center of attention no matter the occasion,

• I often make sure I’m noticed even if it means creating a scene,

• I actively seek out opportunities to be in the spotlight,

• I enjoy flaunting my accomplishments to gain admiration,

• I seek out chances to highlight my successes for all to see,

Community 3 (Self-Critical):

• I am my harshest critic and often believe I will never be good enough,

• I constantly compare myself unfavorably to others,

• I am relentless in finding faults in myself,

• I constantly second-guess myself and feel I never measure up,

• I am unforgiving towards myself for even minor failings,

• I selectively forget instances where I failed to meet expectations,

• I often berate myself for not being perfect,

• I criticize myself harshly for perceived failures.

Community 4 (Shame/Vulnerability):

• I am easily overwhelmed by a sense of shame when others criticize me,

• I find myself avoiding situations where I might feel embarrassed,

• I am easily humiliated by my perceived shortcomings,

• I am strongly affected by the thought of others noticing my vulnerabilities,

• I feel intense discomfort when my imperfections are brought to light,

• I experience deep shame when others discover my weaknesses,

• I feel exposed and ashamed when my faults are pointed out.

[Hudson Golino @GolinoHudson]

The Exploratory Graph Model is now LIVE!

TL;DR:

• The paper introduces the Exploratory Graph Model (EGM), a new mathematical framework for analyzing psychological measurements using networks instead of traditional latent variable models
• Key innovations:
- Treats psychological dimensions as emerging from direct interactions between observed variables
- Does not assume underlying latent variables
- Provides a formal mathematical basis for community structures in networks
- Allows for both exploratory and confirmatory network modeling
• The researchers conducted simulation studies comparing EGM to traditional factor analysis (EFA):
- Parameter estimates were most accurate when the analysis method matched the data generation method
- Traditional fit measures could distinguish between EGM and factor models better than chance
- Model detection accuracy improved when multiple fit measures agreed
- Loading size and correlation strength affected which model fit better
• Three empirical examples were analyzed (Misinformation Susceptibility Test, Beck Depression Inventory, and a Narcissism scale):
- EGM fit better for some applications
- Results demonstrate how EGM can be applied to real psychological data
• Main implications:
- Provides a new way to conceptualize psychological measurement without latent variables
- Opens up new possibilities for network-based psychological research
- Complements existing network approaches in psychology
- May help improve measurement accuracy in psychological assessment
• The paper represents a significant advance in psychometric modeling by formalizing network approaches as an alternative to traditional latent variable methods
The paper can be found here:
https://t.co/Std3G2AgW7
(the main new functions) can be found in our EGAnet package here:EGM https://t.co/Yrneuzmykg
https://t.co/GdNZqP5TVT

[Voice Of Baceprot (VOB) @baceprotvoices]

Petuah bahasa Sunda sengaja kami masukan untuk menghardik ingatan. Bahwa manusia sebenarnya pernah memuliakan alam. Pernah mesra merawat bumi.

Mighty Island telah mengudara!

[Lorwen Harris Nagle, PhD @LORWEN108]

I'm 62.

Stress haunted me for years.

Yes, I tried therapy, medication, and countless self-help books.

Everything failed until I discovered this ancient Japanese concept to reduce stress.

If you're feeling stress, open this...🧵

[Hudson Golino @GolinoHudson]

🚨Do you like AI/LLMs and Network Psychometrics? 💡Want to know how to generate AND validate items in silica automatically? Introducing our new paper: "Generative Psychometrics via AI-GENIE: Automatic Item Generation and Validation via Network-Integrated Evaluation." by @larafromUVA, Alex Christensen and Hudson Golino. More below...

Introduction:

In our paper, AI-GENIE (Automatic Item Generation and Validation via Network-Integrated Evaluation) is introduced as a novel methodology for fully automated scale development and validation.

It leverages large language models (LLMs) and advanced network psychometric techniques to streamline the item generation and validation process.

Traditional scale development is resource-intensive, time-consuming, and costly, often requiring extensive human expert intervention, and costly data collection for psychometric validation.

Recent advancements in AI and LLMs offer promising solutions to generate expert-quality text for scale items.

The challenge lies in efficiently selecting and validating non-redundant, high-quality items that accurately represent intended psychological constructs, and that present adequate dimensionality (structural validity) AND item/dimension stability.

AI-GENIE aims to automate the entire process, from item generation to validation, enhancing efficiency and scalability in psychological assessment creation.

Previous research has shown that AI-generated items can create adequate psychological assessments, but the item selection process remains resource-intensive (with rounds of in-human data collection).

AI-GENIE eliminates the need for extensive human expert involvement in generating, selecting, and validating items, potentially saving researchers significant time and money.

The methodology combines open-source LLMs, generative AI, and network psychometrics to facilitate scale generation, selection, and validation.

AI-GENIE is the first fully automated methodology to generate, assess, and validate the quality of AI-generated items for psychometric scales.

Methodology:

The study used five large language models (Gemma 2, Llama 3, Mixtral 8x7b, GPT 3.5, and GPT 4o) to generate items for the Big Five personality traits.

A Monte Carlo simulation was conducted, generating 1,500 samples of at least 300 items each, using different temperature settings (low, medium, high) for each model.

The importance of prompt engineering is emphasized, with the study using a few-shot prompting technique to improve output quality and consistency.

The prompt included examples from John and Srivastava's Big Five assessment and assigned the model the persona of an expert personality methodologist.

The main prompt instructed the model to generate ten new items (one reverse-keyed and one regular-keyed for each of the Big Five traits) in a specific format, encouraging creativity and comprehensiveness.

To reach the target of 300 items, the prompt was parsed 30 times (10 items each) per condition.

The study aimed for an equal distribution of item types (10 types total), but noted that LLMs can be inconsistent in following exact instructions.

Low temperature models tended to produce less equitable distributions of item types due to their repetitive nature, while high temperature models produced more balanced distributions.

We acknowledge that unequal representation of item types is not a significant problem for most practical applications, as adjustments can be made when generating a single item pool.

Our methodology demonstrates a novel approach to automated item generation for psychological scales using AI, with considerations for prompt design, model behavior, and output quality.

Results:

AI-GENIE demonstrated effectiveness in item pool development and structural validity with item and dimension stability across all tested conditions.

All 15 conditions (5 models x 3 temperature settings) showed improvement in average final Normalized Mutual Information (NMI) compared to the initial NMI.

The highest temperature Llama 3 model showed the most significant improvement, with an average increase of 15.4% in NMI.

GPT 4o's lowest temperature model also performed well, with an average NMI improvement of 12.3%.

Mixtral's lowest temperature model showed the smallest improvement, but still increased NMI by 2.3% on average.

Even the smallest improvement indicates that AI-GENIE is capable of handling items generated by any of the five tested models at any temperature setting.

These results suggest that AI-GENIE is a robust and versatile tool for improving the quality of item pools generated by AI in psychological scale development.

GPT-3.5 with a temperature of 0.5 or 1 achieved the highest accuracy, with a normalized mutual information of basically 1 after AI-GENIE was implemented.

Steps of AI-GENIE:

AI-GENIE's process begins with embedding the items using OpenAI's text-embedding-3-small model, translating the semantic meaning of each item into a high-dimensional numeric vector. This step enhances the representation of items relative to traditional Likert scale responses.

The second phase establishes a baseline performance using Exploratory Graph Analysis (EGA). It determines an optimal step size for the Walktrap algorithm by maximizing the Normalized Mutual Information (NMI) between the EGA-detected communities and the known Big Five personality traits. This step provides a benchmark for subsequent item reduction.

Next, AI-GENIE applies Unique Variable Analysis (UVA) to identify and reduce redundancies in the item pool. UVA uses weighted topological overlap to detect locally dependent items, with a modified cut-off value of 0.20 to capture highly similar items more effectively than the standard 0.25 threshold.

Following redundancy reduction, the process repeats the step size optimization for the Walktrap algorithm. This ensures that the optimal step size is determined for the reduced item pool, preparing for the final stability analysis.

The final step employs bootEGA to assess item stability across multiple bootstrapped samples. Items with stability below 0.75 are considered unstable and removed. This process iterates until all remaining items meet the stability threshold, typically requiring 2-3 rounds. The result is a refined, stable item pool ready for review.

Throughout these steps, AI-GENIE leverages advanced psychometric techniques and machine learning to automate and optimize the item selection process for psychological scale development, significantly reducing the need for manual expert intervention.

Examples of before and after AI-GENIE:

Item and Dimension Stability:

Structural Validity before and after AI-GENIE:

Pre-print in the comments!

[via (she/her)☃︎ @itsmeowpage]

a love letter to all my friends 💌. Feel free to share anywhere without credits.

[Mike Cheung @mikewlcheung]

I recently hosted a 3-hour online workshop on "Conducting Meta-Analytic Structural Equation Modeling (MASEM) with R." You may watch the video here: https://t.co/Wr4fjbv0Og and access the materials here: https://t.co/9F5DSo8LZM. #RStats #MetaAnalysis #StructuralEquationModeling

[Nelly; @nrqa__]

Canva is OLD.

Modern people now use this insane tool to effortlessly design stunning visuals with one click.

Here’s how:

[Mushtaq Bilal, PhD @MushtaqBilalPhD]

Collecting relevant papers for literature review takes time — a lot of time.

Try Iris, an AI-powered app that will help you collect relevant papers in minutes.

Here's how to use it:

[Kiara Minto she/her @Dr_KiaraM]

Looking for participants aged 18-35 who currently live in Australia, for paid (AUD$20 for 1 hr) interviews about attitudes and beliefs about sexual consent. If you're interested in participating, please get in touch for more details.

[Lauren Breen @drlaurenjbreen]

New paper identifying 3 profiles of health professionals’attitudes on voluntary assisted dying led by ⁦⁦@dr_morganalw⁩ https://t.co/j6f5lK7kR1

[Coffee and crackers @wibisonosusilo]

I'm taking part in the International Women's Day Fun Run, presented by National Storage this March, to support women with breast cancer. You too can support, by helping me reach my goal. Donate to my fundraising page here - https://t.co/LndaR7mJGB.

[Coffee and crackers @wibisonosusilo]

I'm taking part in the International Women's Day Fun Run, presented by National Storage this March, to support women with breast cancer. You too can support, by helping me reach my goal. Donate to my fundraising page here - https://t.co/LndaR7mJGB.

[Mushtaq Bilal, PhD @MushtaqBilalPhD]

Here's how to supercharge your literature review using an AI-powered app called Litmaps:

[Aleksandr Volodarsky @volodarik]

ChatGPT has crossed 1M+ users in just 5 days.

To compare, it took Netflix 41 months, FB - 10 months, and Instagram - 2.5 months.

But many haven’t yet realized its full potential.

Here are the 10 mindblowing things you can do using it right now:

[Samantha Stanley @SamanthaStanley]

I have spent the last few days immersed in responses from 2546 Australians explaining what makes them angry about climate change. I'll share the prevalence of each category of anger (& their correlates) at #NatureFeelz next week

[mat marques, ph.d. ||| @[email protected] @fabiovelo]

Susilo Wibisono presenting important qualitative data on religious motives for involvement in environmental action in Indonesia
#SASP2022

[Akshay 🚀 @akshay_pachaar]

An essential tool for a Data Analyst is Tableau. 📊

Master Tableau for FREE!! 🚀

A Thread 🧵👇