ML from Scratch Episode 2: All about Data

Data is the single most important element in Machine Learning. It is the one ingredient you absolutely need to get right for the entire recipe to turn out well. So before we dive headfirst into any algorithm, it is crucial to understand data better—because it is the foundational bedrock of any ML objective. Understanding data means understanding the underlying patterns it carries. And often, this is not quite straightforward. Different forms of data hide different kinds of patterns, and it is our job to format, process, and transform them accordingly so that our algorithms can actually understand, interpret, and learn these patterns.

Let’s roll up our sleeves and get our hands dirty with the data pipeline.

Data Collection and Data Processing

Data Sampling

Sampling is the technique through which we select the dataset we want to train or test our model on from the available data in the wild. The sampling strategy directly dictates the class representation, diversity, and overall efficiency of the dataset during training. So, we have to get this right.

Probabilistic Sampling Techniques:

1. Simple Random Sampling: Data points are chosen entirely at random, with or without replacement. It is the gold standard of fairness, but it isn’t always practical.
2. Stratified Sampling: Here, we divide the population into subgroups (strata) based on similar traits and sample proportionally from each. This ensures uniformity and guarantees that minority groups are represented.
3. Cluster Sampling: The population is clustered into natural groupings, and data is randomly sampled from each set or selected sets based on our requirements. It saves costs but introduces slight bias.
4. Systematic Sampling: Sampling happens by choosing data in a systematic approach—like picking every k-th row in the dataset. It is simple but can accidentally introduce periodic biases if the data has hidden cycles.

Non-Probabilistic Sampling Techniques:

• Convenience sampling: Choosing data based purely on availability, legality, and ease of access. Quick, but often terribly biased.
• Judgemental sampling: The researcher cherry-picks data points based on pre-determined criteria or expert knowledge. Useful for specific case studies but highly subjective.
• Quota sampling: Data points are sampled until a certain quota or criteria is met. It ensures coverage but lacks the randomness of probabilistic methods.

Handling Missing Data

Now that we have our raw sample, the next inevitable roadblock is dealing with missing values and anomalies. Real-world data is messy—and we need to deal with it. Here a few ways to handle missing data:

Deletion Methods:

• Listwise deletion: Dropping the entire row if even a single value is missing. Simple, but wasteful.
• Pairwise deletion: Selectively ignoring only the missing value during specific calculations (like correlations). Saves data but leads to inconsistent sample sizes across analyses.
• Column deletion: Removing the entire feature if it has too many missing values. A last resort.

Imputation Methods: Imputation refers to replacing the missing value with some other plausible value.

• Mean/Median/Mode imputation: Replacing with the central tendency of that column. Quick, but it artificially reduces variance.
• Last Observation Carried Forward (LOCF): Common in time-series, where we assume the last known value persists until a new one arrives.
• Constant imputation: Filling with a fixed value like 0 or “Unknown”. Use with extreme caution.

Using Machine Learning to fill missing values: Leveraging ML techniques like K-Nearest Neighbors (KNN) or MICE (Multiple Imputation by Chained Equations) to predict the missing value based on other features. This is computationally expensive but often yields the most accurate imputations.

Outlier Detection and Treatment

We have handled missing data, but we still have one more gremlin—outliers. These are abnormal data points that can skew patterns and mislead our models. Detecting and managing them is a good practice.

Outlier Detection:

Statistical Methods:

• Z-score: Measures the distance of a point from the mean in terms of standard deviations. Typically, any point beyond +/- 3 standard deviations is flagged as an outlier.
• Inter Quartile Range (IQR): Defines outliers as data points that fall below Q1 - 1.5 × IQR or above Q3 + 1.5 × IQR. This is highly robust against skewed data, which is why I personally lean towards it more often.

Machine Learning Methods:

• DBSCAN: A density-based clustering algorithm that identifies outliers as points in low-density regions. Highly effective for spatial data.
• One-class SVM: This tries to separate normal data points from outliers by drawing a hyperplane around the dense regions of the data.

Outlier Management:

• Trimming: Simply eliminating the outlier values entirely. Effective if outliers are few and clearly due to data entry errors.
• Imputation: Replacing outliers with a more suitable value (like the median). It retains the data point while neutralizing its extreme effect.
• Transformation: Applying a mathematical function (like log or square root) to the entire column. This squashes extreme values and reduces skewness, often making outliers less disruptive.

Feature Scaling

Our data looks clean now, but there is still one glaring issue: the features are on completely different scales. We can’t feed this directly to an algorithm that relies on distance or gradient calculations. Enter Feature Scaling—the art of bringing all features down to a comparable range. It ensures that features with larger magnitudes do not dominate model training or distance calculations.

Normalization (Min-Max Scaling): Normalization shifts and rescales data so that all values fall strictly within a specified range, usually 0 to 1 (or -1 to 1 for data with negative values).

                           X_norm = (X - X_min)/(X_max - X_min)

I use this when I know my data doesn’t have heavy tails, and I need bounded outputs—like in image processing where pixel values are already in a fixed range. The massive downside? It is incredibly sensitive to outliers. One colossal outlier will squash all normal data points into a tiny, indistinguishable fraction.

Standardization (Z-Score Normalization): Standardization transforms data so that it has a mean (μ) of 0 and a standard deviation (σ) of 1.

                                X_std = (X - μ)/σ

Notice that it doesn’t bound values to a specific range. Values can be negative or larger than 1. I prefer standardization for algorithms that assume a Gaussian distribution, like Linear Regression, Logistic Regression, or PCA. It is much less affected by outliers than Min-Max scaling, though outliers will still impact the calculation of the mean and standard deviation.

Categorical Encoding Techniques

Algorithms are mathematical beasts—they eat numbers, not text. So, we must convert categorical variables into numerical representations.

• One-Hot Encoding: Creates binary columns for each category. Perfect for nominal data where there is no order (e.g., colors: Red, Blue, Green). The downside? If you have a feature with 100 unique values, you just exploded your dimensions—enter the curse of dimensionality.
• Ordinal Encoding: Assigns integers to categories based on a natural order (e.g., “Low” = 0, “Medium” = 1, “High” = 2). Great for ordinal variables, disastrous if applied to nominal ones (the algorithm will assume a false numerical relationship).
• Target Encoding: Replaces a category with the mean of the target variable for that category. This is incredibly powerful, but it is also a dangerous game—it is an open invitation for data leakage (we’ll talk about that shortly). Always do target encoding inside cross-validation folds to prevent peeking.
• Count Encoding: Replaces a category with its frequency in the dataset. Simple and surprisingly effective for high-cardinality features.

Feature Engineering and Feature Selection

Feature Engineering: This, to me, is where the human magic happens. Raw features are rarely enough. Sometimes you need to create interaction terms (multiplying two features), split datetime into day/month/year, or aggregate transaction histories into a single “recency” score. It is a mix of domain knowledge, intuition, and sheer experimentation. I often spend more time engineering features than I do tuning hyperparameters—and it pays off.

Feature Selection: Not all features are created equal. Some add value, others add noise. Feature selection is the process of ruthlessly cutting down the unhelpful ones. It fights overfitting and reduces training time.

• Filter Methods: Statistical tests like correlation or chi-squared to rank features. Fast, but independent of the actual model.
• Wrapper Methods: Forward selection (adding features one by one) or backward elimination (removing them). Computationally expensive, as you are effectively training the model multiple times.
• Embedded Methods: Algorithms that have built-in feature selection, like Lasso (L1 regularization), which shrinks useless coefficients all the way down to zero.

Curse of dimensionality: as feature count grows, the data becomes sparse in high-dimensional space, and distance-based methods especially degrade — more features isn’t always better. We will explore techniques like PCA and SVD in future blog posts on how to handle this dimensionality issue.

Handling Imbalanced Data

Here is a classic pitfall: if your dataset has 99% cats and 1% dogs, a model that blindly predicts “cat” will have 99% accuracy but is practically useless for our purposes. Imbalanced data forces the model to be lazy.

• Resampling Techniques:
  • Undersampling: Randomly remove samples from the majority class. Fast, but you throw away valuable data.
  • Oversampling: Duplicate samples from the minority class. Prone to overfitting.
  • SMOTE (Synthetic Minority Oversampling Technique): This clever algorithm creates synthetic samples by interpolating between existing minority points. I’ve found it works wonders in practice.
• Algorithmic Adjustments: Many algorithms allow you to assign class weights. This penalizes the model more heavily for misclassifying the minority class, forcing it to pay attention.
• Metric Choice: If you are in an imbalanced domain, do not use accuracy. Use Precision, Recall, or the F1-Score instead.

Train-Test Splits, Validation, and Data Leak Management

This is the cardinal sin of ML: Data Leakage. It happens when information from outside the training dataset is used to create the model, allowing it to “cheat” during training.

• The Right Order: Always, always split your data into training and testing sets before you do any scaling, imputation, or feature selection. If you scale using the global mean before splitting, you have just leaked information from the future into your training set.
• Validation Sets: We need a middle ground to tune hyperparameters. I rely on K-Fold Cross-Validation, where I split the training data into K folds, iteratively train on K-1 folds and validate on the remaining 1. It gives a robust estimate of model performance without wasting too much data on a single validation set.
• The golden rule: The test set is a sealed vault. You open it only once—at the very end—to get the final, honest performance score.

Domain-Specific Data Processing

Data comes in different flavors, and we have to adapt our preprocessing to match the medium:

• Text (NLP): Requires tokenization (splitting words), removing stopwords, and converting to embeddings (like Word2Vec or BERT embeddings). Context is everything.
• Images (Computer Vision): We usually handle resizing, cropping, and normalization (scaling pixel values to [0,1] or [-1,1]). Data augmentation (rotating, flipping) is also a form of preprocessing to artificially expand the dataset.
• Time-Series: Cannot be shuffled randomly! Temporal order is sacred. We create lag features, rolling statistics (like 7-day moving averages), and ensure our train/test splits respect the chronology to avoid look-ahead bias.

Data Quality and Bias Considerations

We often obsess over model architecture and entirely neglect the social context of the data. Garbage in, garbage out—but biased data in also means biased decisions out.

• Representation Bias: If your training data lacks diversity (e.g., facial recognition datasets trained predominantly on lighter skin tones), your model will fail for underrepresented groups.
• Measurement Bias: This occurs when the way we collect data varies across different groups.
• The Human Check: I’ve started asking myself a crucial question for every dataset: “Does this data truly represent the real-world environment I intend to deploy this model in?”

Building robust ML systems is not just about mathematical rigor; it is also about understanding the limitations of our inputs. Be critical, be curious, and always interrogate your data.

We’ve finally navigated the entire data pipeline. It might seem exhaustive, but trust me, getting this phase right saves you days of debugging weird model behavior later. In the next episode, we will get started with our first blog on ML algotithms and we will discuss on the topic - Rregressions! Until then Stay tuned!