How to deal with unbalanced data: SMOTE vs. Equalizer ?

I have a dataset which is very unbalanced (Binary class: only 1.5% are positive. All the rest are negative samples).

In order to build a model i constructed the following two flows.

I have a got feeling that Flow 2 is wrongly designed... but i am not sure why.

So i cannot explain why the results are totally different.

I need to help please to get better understanding what did i do wrong here.

The results I got are as follows (in average between iterations)

Flow 1: (AttributeSelectedClassifier + SMOTE):

Loop 20 -> Split %80 -> SMOTE on training data -> Train using AttributeSelcetedClassifier -> End Loop -> print AVG of collected scores

TP: 1.4

FP: 19

TN: 3087

FN: 32.6

Recall: 0.04

Precision: 0.069

Accuracy: 0.98

Cohen' Kappa: 0.044

AUC: 0.775

Flow 2: (AttributeSelectedClassifier + Equalizer):

Loop 20 -> Equalize Sample -> Split %80 -> Train using AttributeSelcetedClassifier -> End Loop -> print AVG of collected scores

TP: 27.9

FP: 9.85

TN: 24.15

FN: 7.1

Recall: 0.79

Precision: 0.74

Accuracy: 0.79

Cohen' Kappa: 0.508

AUC: 0.826

Notes:

  • the configuration of the AttributeSelectedClassifier in both flows is the same:

    ranker: ExhaustiveSearch

    evaluator: CfsSubsetEval

    classifier: RandomForest

  • in Flow 1, I run SMOTE on the minimal class (the positive) so that it will be same size as negatives

  • I tried to change the AttributeSelectedClassifier to RandomForest and got similar results

  • The equalizer in Flow 2 is basically doing under-sampling for the big class (the negatives) by randomly selecting similar number of instances like the positives.