Concept Drift Detection with DDM

[ngabrek]

Hi there,

I'm currently playing around with the package and stumbled over some things that confused me a bit.
Is it intended that the drift detectors are only capable of detecting just one concept drift? After removing the 'break' statement in the conditional statement of 'detector.drift' from the different examples you give in the documentation, the detector constantly detects a drift until the stream is fully iterated. So, do I understand it correctly, that I kind of have to reset the detector before trying to detect further drifts in a stream?

Thank you and I hope this is the right place to ask my question.

[jaime-cespedes-sisniega]

Hi @ngabrek,

Concept drift detectors can detect multiple "drifts/changes" (removing that break line that you mention), but if they occur consecutively, they likely belong to the same concept drift.

Is it possible to continue attempting to detect "other concept drifts" after the initial detection using DDM? Yes, but this will likely be challenging. Due to the inherent nature of DDM [1], it is not intended to be employed for ongoing detection of "other concept drifts" following the initial detection.

[1] https://link.springer.com/chapter/10.1007/978-3-540-28645-5_29

This is reflected in the following example that I have implemented. We examine the outcomes of resetting the detector versus not doing so.

We simulate a scenario where we are monitoring the errors of a binary model (DDM is designed for binary problems only). A value of 0 signifies correct classification of our sample, while 1 indicates the opposite. 3 different streams:

• 100 samples with 10% error rate.
• 100 samples with 30% error rate.
• 400 samples with 10% error rate.

For the initial 100 samples, our error rate is around 10%. Suddenly, the error rate significantly increases, with approximately 30% of our predictions being misclassified in the subsequent 100 samples. DDM detects the drift at step 115 (excluding the DDM warning zone preceding drift detection for simplicity). From now on, the behavior diverges based on whether the detector is reset or not:

• DDM (No reset): To observe DDM (No reset) leaving the drift zone, I had to "artificially" amplify the size of the last data stream by four times (400 samples) compared to the other two streams (100 samples each). DDM leaves the drift zone before step 500. As stated previously, DDM is not designed to operate continuously after the initial detection.

• DDM (Reset): No subsequent drift is detected because the error rate of subsequent samples reverts to the initial 10% on average.

I leave you the code of the example I mentioned. I hope it has been useful.

from frouros.detectors.concept_drift import (
    DDM,
    DDMConfig,
)
from frouros.metrics import PrequentialError
import numpy as np
import matplotlib.pyplot as plt

np.random.seed(seed=31)

error_samples_a = np.random.binomial(n=1, p=0.1, size=100)   # 10% error (0 means no error, 1 means error)
error_samples_b = np.random.binomial(n=1, p=0.3, size=100)   # 30% error (0 means no error, 1 means error)
error_samples_c = np.random.binomial(n=1, p=0.1, size=400)   # 10% error (0 means no error, 1 means error)
stream = np.concatenate((error_samples_a, error_samples_b, error_samples_c))

min_num_instances = 10  # Minimum number of instances before starting to check for drift

# Detectors
detector_no_reset = DDM(config=DDMConfig(min_num_instances=min_num_instances))
detector_reset = DDM(config=DDMConfig(min_num_instances=min_num_instances))

# Metrics
metric_no_reset = PrequentialError(alpha=0.99)
metric_reset = PrequentialError(alpha=0.99)

error_values_no_reset, error_values_reset = [], []
drift_no_reset_idxs, drift_reset_idxs = [], []
start_look_for_drift_reset_idx = [min_num_instances]
for i, value in enumerate(stream):

    # DDM (No reset) update
    error_values_no_reset.append(metric_no_reset(error_value=value))
    _ = detector_no_reset.update(value=value)
    if detector_no_reset.drift:
        print(f"DDM (No reset) - Change detected at index {i}")
        drift_no_reset_idxs.append(i)

    # DDM (Reset) update
    error_values_reset.append(metric_reset(error_value=value))
    _ = detector_reset.update(value=value)
    if detector_reset.drift:
        print(f"DDM (Reset) - Change detected at index {i}")
        drift_reset_idxs.append(i)
        detector_reset.reset()  # reset detector, have to wait to collect min_num_instances again
        metric_reset.reset()
        start_look_for_drift_reset_idx.append(i+min_num_instances)


# Plot error values and see where drift was detected
fig, axs = plt.subplots(4, 1, figsize=(16, 8))
fig.suptitle("DDM Drift Detection")
axs[0].set_title("DDM (No reset)")
axs[0].set_xlabel("Steps/Instances")
axs[0].set_ylabel("Error Rate") # 1 - Accuracy
axs[0].plot(error_values_no_reset)
axs[0].axvline(x=min_num_instances, color='lime', linestyle='--', label="Start looking for drift 1")
if drift_no_reset_idxs:
    axs[0].axvline(x=drift_no_reset_idxs[0], color='r', linestyle='--', label="Fist drift detected")
axs[0].legend()
axs[1].hlines(1,1,len(stream))
axs[1].eventplot(drift_no_reset_idxs, orientation='horizontal', colors='r')
axs[1].axis("off")
axs[2].set_title("DDM (Reset)")
axs[2].set_xlabel("Steps/Instances")
axs[2].set_ylabel("Error Rate") # 1 - Accuracy
axs[2].plot(error_values_reset)
for i, (idx, color) in enumerate(zip(start_look_for_drift_reset_idx, ["lime", "darkgreen"])):
    axs[2].axvline(x=idx, color=color, linestyle='--', label=f"Start looking for drift {i+1}")
if drift_reset_idxs:
    axs[2].axvline(x=drift_reset_idxs[0], color='r', linestyle='--', label="First drift detected")
axs[2].legend()
axs[3].hlines(1,1,len(stream))
axs[3].eventplot(drift_reset_idxs, orientation='horizontal', colors='r')
axs[3].axis("off")
plt.savefig('ddm_reset.png')
plt.show()

If you have any other questions, please let me know.

[ngabrek]

@jaime-cespedes-sisniega, thank you for the in-depth answer! Now I fully get the idea.