Added a config for custom_dimensions_max_length issue-16150

[jorgeuos]

Added a config for custom_dimensions_max_length and refactor plugins/CustomDimensions/Tracker/CustomDimensionsRequestProcessor.php:prepareValue() to use the config instead of being hardcoded.

Description:

I will change the max length for custom dimensions in:
https://github.com/matomo-org/matomo/blob/4.11.0-rc1/plugins/CustomDimensions/Tracker/CustomDimensionsRequestProcessor.php#L175

To see how it behaves before I apply my fix.

I'm tracking custom dimensions with a default installation of Matomo. No extra configuration or plugins added.

The default custom_dimension_1 type is varchar(255) in the database.

The tracking code is:

    var _paq = window._paq = window._paq || [];
    var customDimensionId = 1;
    var customDimensionValue = 'LongValue1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
    789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901ThisShouldBeCutOff';
    _paq.push(['setCustomDimension', customDimensionId, customDimensionValue]);

I'm modifying the php function like this:

private static function prepareValue($value)
{
    $inputMaxLength = 267;
    $return = mb_substr(trim($value), 0, $inputMaxLength);
    return $return;
}

Story 1:

• As a user, I want to track a custom dimension with a value that exceeds the limit, so that I can observe how Matomo handles it when the value length is capped at 250 characters.

$inputMaxLength = 250; // 204 OK

Stored value in MySQL:

LongValue1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901

Which equals 250 characters exactly.
custom_dimension_1 type is varchar(255) in the database.

Story 2:

• As a user, I want to track a custom dimension that reaches the 255-character limit, so that I can verify how Matomo handles and stores the full length.

$inputMaxLength = 255; // 204 OK

Stored value in MySQL:

LongValue1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901ThisS

Which equals 255 characters exactly.
custom_dimension_1 type is varchar(255) in the database.

Story 3:

• As a user, I want to track a custom dimension that exceeds the 255-character limit, so that I can observe Matomo’s behavior when the input length causes data storage to fail without warning.

$inputMaxLength = 256; // 400 Bad Request

Stored value in MySQL:

'' // empty string

This means that MySQL is not storing the value because it is too long. No error is shown in the UI and there is no Tracking failures in the log.
custom_dimension_1 type is varchar(255) in the database.

Story 4:

• As a user, I want to track a custom dimension with a value longer than the default maximum length by adjusting the database schema to support larger values, so that I can ensure the system behaves correctly with longer data.

1. I need to change the default column type for custom dimensions in the database from varchar(255) to text.
   According to this issue in Github: Allow custom dimensions of any size (custom_dimension_X fields of unlimited length column (TEXT type) instead of VARCHAR 255) #16150 we can alter the column type in the database like this:

ALTER TABLE matomo_log_visit
MODIFY COLUMN custom_dimension_1 TEXT,
MODIFY COLUMN custom_dimension_2 TEXT,
MODIFY COLUMN custom_dimension_3 TEXT,
MODIFY COLUMN custom_dimension_4 TEXT,
MODIFY COLUMN custom_dimension_5 TEXT;

Which would also help with another unrelated issue: https://forum.matomo.org/t/error-while-updating-to-4-0-0-b1-have-to-change-some-columns-to-text-or-blobs/39482

After a successful change in the database, I will track a custom dimension with a value that is too long so that I can see how Matomo behaves.
custom_dimension_1 type is now TEXT in the database.

2. Move on to the test case:

$inputMaxLength = 256; // 204 OK

Stored value in MySQL:

LongValue1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901ThisSh

Which equals 256 characters exactly.
custom_dimension_1 type is TEXT

Story 5:

• As a user, I want to track a custom dimension with a value that approaches the TEXT field limit (65,535 characters), so that I can verify that Matomo handles maximum-length values properly.

$inputMaxLength = 65535; // Max length for TEXT

Stored value in MySQL:

LongValue1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901ThisShouldBeCutOff

Which equals 250 characters exactly.
custom_dimension_1 type is TEXT in the database.

Conclusion

The default maximum length for custom dimensions is 250 characters. If a value exceeds this limit, it will be truncated and stored, unless the length exceeds 255 characters (the database limit for varchar(255)), in which case it will not be stored at all. Additionally, no error is displayed in the user interface, and no tracking failures are logged.

To fix this issue, a user needs to change the column type for custom dimensions in the database from varchar(255) to TEXT.

Potential edge cases thought about:

• The TEXT type is not as efficient as varchar(255) for shorter values. TEXT fields are generally less efficient than VARCHAR fields for smaller values. This can have performance implications for large datasets with frequent updates or queries on these fields.
• The TEXT type can store up to 65535 characters, which is a lot more than the 250 characters needed for custom dimensions.
• Switching to TEXT fields could result in increased storage use, particularly if a large number of custom dimension values exceed the original 255-character limit, leading to larger data entries.
• If the custom dimension columns are indexed for performance reasons, switching to a TEXT field could limit the ability to index effectively, as MySQL has restrictions on how much of a TEXT field can be indexed. This could potentially affect query performance, especially if the dimensions are frequently used in queries.
• A user needs to alter the column type in the database manually.
• A user updates only the custom_dimension_1 column, but there are 5 custom dimensions in total. The user needs to update all of them.
• A user only updates the custom_dimension columns in the matomo_log_visit table, but there are other tables where custom dimensions are stored.
  The log_link_visit_action and log_conversion tables also have custom dimension columns that need to be updated.
• It’s essential to ensure that any schema changes are consistently applied across all relevant tables.
• Users might not fully understand why they would need to manually update the schema or why the field behaves differently without adjusting the database, leading to confusion. This should be well-documented in the UI or support materials to minimize misunderstandings.

Review

• Functional review done
• Potential edge cases thought about (behavior of the code with strange input, with strange internal state or possible interactions with other Matomo subsystems)
• Usability review done (is anything maybe unclear or think about anything that would cause people to reach out to support)
• Security review done
• Wording review done
• Code review done - Any help here would be appreciated.
• Tests were added if useful/possible - There is no existing tests for prepareValue() feature, and none where added.
• Reviewed for breaking changes - I don't think this will break backwards compatibility.
• Developer changelog updated if needed - Not sure about this one, help would be appreciated.
• Documentation added if needed - Added in the code
• Existing documentation updated if needed - Not updated any docs

[jorgeuos]

This is to fix the issue-16150

[github-actions]

If you don't want this PR to be closed automatically in 28 days then you need to assign the label 'Do not close'.

[github-actions]

This PR was last updated more than one month ago, maybe it's time to close it. Please check if there is anything we still can do or close this PR. ping @matomo-org/core-reviewers

[jorgeuos]

Ah, makes sense!
Thank you @michalkleiner