Patrick Ogbonna

5 Common Data Cleaning Mistakes You Might Be Making Cleaning data is important, but small mistakes can mess up your results. Here are five mistakes you should avoid: 1. Ignoring Missing Data You see empty cells and think, "It’s just a few blanks—no problem." But missing data can change your results. Instead of deleting them, ask yourself why they are missing. Should you fill them in? Should you remove them? Think before you act. 2. Deleting Duplicates Without Checking You spot repeated entries and delete them right away. But wait—what if they are supposed to be there? A customer could have made two purchases, or an employee could appear twice for a reason. Always check before removing. 3. Not Fixing Inconsistent Formats One column has "Jan 1, 2024," another has "01-01-24." Some names are in lowercase, others in uppercase. If your formats don’t match, your data won’t work properly. Standardizing everything from the start makes your work easier. 4. Removing Outliers Without Thinking You see one number that looks too high or too low, and you delete it. But what if it’s real? A big sale, a rare event—outliers can tell an important story. Always check before removing them. 5. Skipping Error Checks You finish cleaning and move on, but did you check for mistakes? If someone’s birth year is 1800, or a price is negative, that’s a clear error. A quick check can save you from bad analysis.

204

170

14K

Patrick Ogbonna @pattdata

over 1 year ago

@ud_analyst @gbolahaann @KhennieNectar Love the simple icon and color combo

pattdata retweeted

Annie🦋

@DabereNnamani

over 1 year ago

ANOVA (Analysis of Variance) is a statistical test used to determine whether there are significant differences between the means of three or more groups. It helps answer: Are the group means statistically different from each other? 1. Null Hypothesis (H₀): All group means are equal. 2. Alternative Hypothesis (H₁): At least one group mean is different. 3. Use Case: When comparing more than two groups. If you only have two groups, a t-test is simpler. ANOVA compares two types of variation: 1. Between-group variation: Differences between the group means. 2. Within-group variation: Variability of data points within each group. If the between-group variation is much larger than the within-group variation, it suggests the means are significantly different. Types of ANOVA: 1. One-way ANOVA: Tests the impact of one factor (e.g., comparing test scores across three teaching methods). 2. Two-way ANOVA: Tests the impact of two factors and their interaction (e.g., comparing test scores by teaching methods and gender). Example: One-Way ANOVA Scenario: You test three diets (A, B, C) to see if they lead to different weight loss results. Data: Group A: [4, 5, 6] Group B: [7, 8, 9] Group C: [3, 4, 5] Steps: 1. Calculate the mean for each group. 2. Measure the variation between and within groups. 3. Compute the F-ratio (a statistic that compares the variations). 4. Check the F-value against a critical value or p-value: If p-value < 0.05, reject the null hypothesis (significant difference exists). ANOVA tells you if there's a difference but not which groups differ. For that, use a post-hoc test (e.g., Tukey's test). Data should meet these assumptions: 1. Groups are independent. 2. Data is normally distributed. 3. Variances are roughly equal (homogeneity of variance).

417

332

31K

pattdata retweeted

Annie🦋

@DabereNnamani

over 1 year ago

A subquery is a query inside another query. Think of it as a mini-question that helps answer the main question. Why Use Subqueries? Sometimes, you need to get some data first (the subquery) to use it in your main query. Example: Scenario: You want to find employees who earn more than the average salary in a company. Step 1: Start with the Subquery The subquery calculates the average salary: SELECT AVG(salary) FROM employees; Step 2: Use it in the Main Query Now, find employees earning more than that average: SELECT name, salary FROM employees WHERE salary > (SELECT AVG(salary) FROM employees); Here, the subquery (SELECT AVG(salary) FROM employees) runs first, calculates the average, and passes it to the main query. Types of Subqueries: 1. Single-row subquery: Returns one value (like an average or a max value). 2. Multi-row subquery: Returns multiple values (like a list of IDs or names). 3. Correlated subquery: Depends on the main query and runs for every row. Tips for Understanding: Subqueries are enclosed in parentheses (). They can be in the SELECT, WHERE, or FROM clause. Always think: What does the subquery do first?