This Liella ED, 'Mirai wa Kaze no You ni,' is one of my favorite songs and ending themes. I'm sure many people share a special sense of nostalgia and affection for the original five members of Liella.
#lovelive#ラブライブ#Liella
Am I overreacting for being upset my girlfriend called me her ‘friend’?
We've been together for 8 months. Yesterday, she introduced me to her coworker as ‘my friend.’
I just stood there smiling like an idiot.
Later I told her it bothered me. Her response: It's just a word. Why are you making it such a big deal?
She wasn't just dismissing me. She was treating our relationship like a low frequency term in a massive corpus, technically present but not important enough to stand out.
To understand what actually matters in a document, we use TF-IDF. It measures how important a word is, not just by how often it appears but by how rare it is across all documents.
TF-IDF is a numerical statistic that reflects how important a word is to a document relative to a collection of documents.
Formula
TF-IDF = TF(t, d) × IDF(t)
TF(t, d) = c(t) / k(d)
IDF(t) = ln(N / (1 + df(t)))
Full Form
TF-IDF = Term Frequency-Inverse Document Frequency
Where:
- c(t) = count of term t in doc d
- k(d) = total terms in doc d
- N = Total no. of docs
- df(t) = No. of docs containing term t
Let's take an example:
A news platform wants to identify the most important keywords in an article about space exploration.
Corpus (4 articles)
1. NASA launches Mars rover mission
2. SpaceX rocket launches satellite
3. NASA announces Moon base plans
4. Weather forecast predicts rain
Step 1: Calculate TF for NASA in A1
- Count of NASA = 1
- Total terms = 5
- TF = c(t) / k(d)
- TF = 1 / 5
- TF = 0.20
Step 2: Calculate IDF for NASA
- N = 4 documents
- ‘NASA’ appears in Article 1, 3
- df(NASA) = 2
- IDF = ln(N / (1 + df(t)))
- IDF = ln(4 / (1 + 2))
- IDF = ln(4 / 3)
- IDF = 0.29
Step 3: Calculate TF-IDF for ‘NASA’
- TF-IDF = TF(t, d) × IDF(t)
- TF-IDF = 0.20 × 0.29
- TF-IDF = 0.06
Step 4: Calculate TF-IDF for ‘launches’
- TF in Article 1 = 1/5 = 0.20
- ‘launches’ appears in Article 1, 2
- df(launches) = 2
- IDF = ln(N / (1 + df(t)))
- IDF = ln(4 / (1 + 2))
- IDF = ln(4 / 3)
- IDF = 0.29
- TF-IDF = TF(t, d) × IDF(t)
- TF-IDF = 0.20 × 0.29
- TF-IDF = 0.06
Step 5: Calculate TF-IDF for ‘Mars’
- TF in Article 1 = 1/5 = 0.20
- ‘Mars’ appears in Article 1
- df(Mars) = 1
- IDF = ln(N / (1 + df(t)))
- IDF = ln(4 / (1 + 1))
- IDF = ln(4 / 2)
- IDF = 0.69
- TF-IDF = TF(t, d) × IDF(t)
- TF-IDF = 0.20 × 0.69
- TF-IDF= 0.14
Final Answer
“Mars” has the highest TF-IDF (~0.14) because it's unique to this article.
“NASA” and “launches” score lower (~0.06) because they appear in multiple documents.
The rarer the term, the more it defines that specific document.
Congratulations 🎉, you've just learned TF-IDF.
Bonus: Applications in AI/ML
1. Search Engines:
Ranking documents by relevance, pages where your search term is frequent AND rare elsewhere rank higher.
2. Text Classification:
Transforming raw text into weighted feature vectors for spam detection, sentiment analysis, and topic categorization.
3. Keyword Extraction:
Identifying the most distinctive terms in a document for automatic tagging and summarization.
4. Recommendation Systems:
Finding similar documents or content by comparing TF-IDF vectors using cosine similarity.
I'm entering the #ShadowverseWB Lobby Bananza: Rupies Galore Day 1 giveaway for a chance to win 10 Legendary Card Pack tickets! 🎁
👇 Enter for your chance to win now!
https://t.co/spmkgNMP8t
Losing my mind over this Death Stranding trailer man
Kaiju battles.. more actions sequences.. high speed vehicle chase scenes.. fighting giant horrific monsters all in between walking through the most beautiful enviroments you've ever seen.
I'm so fucking back.