What Happen When We Don't Use Train-Test Split in Machine Learning?

WHAT HAPPEN WHEN WE DON’T USE TRAIN-TEST SPLIT IN MACHINE LEARNING?

Machine learning model are design to learn pattern from the data and give output based on the pattern to an unseen dataset. When I started my learning journey in machine learning I used to train and test model on same dataset, at first I used to be happy by seeing the accuracy but I was just in my delusional state until I went deep in the topic Train-Test Split and found out I was doing a wrong thing and it can lead to danger and create problem called overfitting when working on large or important datasets.

LET US UNDERSTAND THE CORE PROBLEM BY TAKING AN REAL LIFE EXAMPLE

Imagine a student who is preparing for final exams and teacher have gave the practice question and student memorized the practice question and student can answer that question perfectly and luckly the same practice question appears in the exam and student score 100% fully accurate.

Does this mean the student understand the subject well? It is not necessary the student may have just memorized the subject and appear in the exam.

What if the question was slightly change does the student can be equally accurate? If the student only memorized the question instead of learning the concept it can lead to failure if the question is changed.

The machine learning model also work in similar way if train and test on same data instead of learning the pattern and generalize to new unseen data it just memorized.

WHAT IS TRAIN-TEST SPLIT?

Train-Test Split is a technique used in machine learning which Is used to divide the dataset in two half one for training and another for testing.

Training data-It is used for learning the model learn and generate pattern from this data.

Testing data-It is used for evaluation and check whether the model can work well on unseen data or not.

If explaining train_test by above example then simply it is:

X_train=Question used for learningy_train=Correct answer for those questionX_test=New unseen questionY_test=Real answer for evaluation

Now let us see an example by taking a dataset and perform on the dataset on without train-test split and one with train-test split

WITHOUT TRAIN-TEST SPLIT

CODE:

importpandasaspd
# Load dataset
df=pd.read_csv("breast_cancer.csv")
# Remove unnecessary columns
df=df.drop(['id', 'Unnamed: 32'], axis=1)
# Convert diagnosis column
# M = 1 (Malignant)
# B = 0 (Benign)
df['diagnosis'] =df['diagnosis'].map({'M': 1, 'B': 0})
# Features and target
X=df.drop('diagnosis', axis=1)
y=df['diagnosis']
# Handle missing values if any
X=X.fillna(X.mean())
# Feature scaling
fromsklearn.preprocessingimportStandardScaler
scaler=StandardScaler()
X=scaler.fit_transform(X)
# Logistic Regression
fromsklearn.linear_modelimportLogisticRegression
clf=LogisticRegression(max_iter=10000)
# Train on FULL dataset
clf.fit(X, y)
# Predict on SAME dataset
predictions=clf.predict(X)
# Accuracy
fromsklearn.metricsimportaccuracy_score
accuracy=accuracy_score(y, predictions)
# Output
print("Predicted Values:\\n")
print(predictions)
print("\\nActual Values:\\n")
print(y.values)
print("\\nAccuracy:")
print(accuracy)

I have taken a breast cancer Wisconsin(Diagnostic) dataset and I have used as my input feature all the other columns except diagnosis like (radius_mean,area_meanetc)and saved it in X variable and diagnosis is my output feature which is stored in Y. I have import LogisticRegression because it is a classification problem and the logistic regression predicts probability between 0 and 1.

Now clf=LogisticRegression() this is simply just creating a model till this the model is empty and has nothing in it then here comes the most important line clf.fit(X,y), clf usually stand for classifier and .fit method is used to learn pattern from the data.

I have also used Feature Scaling because he dataset contain some feature with large values and some with small values so to make all the input feature into similar numerical range I have used it in my code. Logistic Regression problem mainly used feature scaling

Here, the model is tested and trained in same data and seen all the example then it can be extremely accurate because it has nothing that is unseen.

OUTPUT:

LETS US UNDERSTAND OVERFITTING

Overfitting is an important concept which is necessary while learning train-test split.

Overfitting happens when the model learns the training dataset too perfectly so instead of learning the general pattern it memorized the dataset so that’s why accuracy will be high in training data and poor in testing data.

So this is the reason evaluation on unseen data is widely important.

WITH TRAIN-TEST SPLIT

CODE:

importpandasaspd
# Load dataset
df=pd.read_csv("breast_cancer.csv")
# Remove unnecessary columns
df=df.drop(['id', 'Unnamed: 32'], axis=1)
# Convert diagnosis column
df['diagnosis'] =df['diagnosis'].map({'M': 1, 'B': 0})
# Features and target
X=df.drop('diagnosis', axis=1)
y=df['diagnosis']
# Handle missing values
X=X.fillna(X.mean())
# Feature scaling
fromsklearn.preprocessingimportStandardScaler
scaler=StandardScaler()
X=scaler.fit_transform(X)
# Train-Test Split
fromsklearn.model_selectionimporttrain_test_split
X_train, X_test, y_train, y_test=train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)
# Logistic Regression
fromsklearn.linear_modelimportLogisticRegression
clf=LogisticRegression(max_iter=10000)
# Train model
clf.fit(X_train, y_train)
# Predictions
predictions=clf.predict(X_test)
# Accuracy
fromsklearn.metricsimportaccuracy_score
accuracy=accuracy_score(y_test, predictions)
# Output
print("Predicted Values:\\n")
print(predictions)
print("\\nActual Values:\\n")
print(y_test.values)
print("\\nAccuracy:")
print(accuracy)

Now here we have split the dataset in two half by the help of the Train-Test split technique while train-test split we have to give X(our input feature),Y(our target column or output feature) and test size should be declare we can see I have taken 0.2 which means the 20% data will be used for testing and rest 80% is used for training.This 20% is the unseen data which is used for evaluating the model.I have also used random_state=42 this means it control the randomness of data splitting,it ensures that the data is split in the same way eveytime the code is run and if you are wondering why 42 then it doesnot have any mathematical meaning just commonly used number among programmers.

Output

Here you can see the output of both the approaches and without train test the accuracy is only 98.7% and with train test the accuracy is 97.3%.

CONCLUSION:

In conclusion, train-test split is the important step in machine learning because it helps us to evaluate whether model has learned the general pattern or just memorized the training data. A model can be highly accurate when not using train-test split but result can be often misleading,but when testing on unseen data the model is more reliable.

But does it hold always the same scenario? Does the accuracy is always high in the model which have not used train-test technique? Does train-test technique have any relation with the accuracy of model or just it is used as an evaluation method?

All the query will be answer in the next blog till then stay tuned.