Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doris support deducing partitions #280

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

beyond-up
Copy link
Contributor

@beyond-up beyond-up commented Dec 26, 2022

#149

Signed-off-by:

Pre-Checklist

Note: Please complete ALL items in the following checklist.

  • I have read through the CONTRIBUTING.md documentation.
  • My code has the necessary comments and documentation (if needed).
  • I have added relevant tests.

Purpose

Some description about what this PR wants to do.

Approaches

Some description about how this PR achives the purpose.

Related Issues

#149

New Behavior (screenshots if needed)

N/A

@beyond-up
Copy link
Contributor Author

@BlockLiu please help me code review, thx!

.collect(Collectors.toList())
);
DorisPartitionTemplate dorisPartitionTemplate = writerConfiguration.getNecessaryOption(DorisWriterOptions.PARTITIONS, CommonErrorCode.CONFIG_ERROR);
List<DorisPartition> dorisPartitions = parseTemplateToDorisPartitions(dorisPartitionTemplate);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, but I have one question.

  • Old partition option support flexible partition range. For example, ["start_range": "2022-01-01", "end_range": "2022-02-01"].
  • In this PR, DorisSink always generate partitions of single day.
  • So could you add a switch option to decide if using this template?

Copy link
Contributor Author

@beyond-up beyond-up Dec 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BlockLiu Thanks for your suggestion, now I understand what you mean.
So if I add a parameter representing the day, week, month, quarter and year to the template to represent the step size of the partition range , is it okay? For example, if use "month" , DorisSink will generate partitions by one month.
I think there are very few scenes to partition by random number of days like 4 days or 11 days.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I mean we can keep both two method for setting doris partition(s).

  • The first way is deducing partition as your codes.
  • And the second way is the original method, _i.e., directly setting doris partition.

So we don't need a "step size", but an option to decide which way to use.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like this:

bool useDeducedPartition = xxx // get from job configuration
if (useDeducedPartition) {
   // use your codes
} else {
   // use the original codes
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK,ths!

@@ -0,0 +1,25 @@
package com.bytedance.bitsail.connector.doris.partition;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be a license header. You can copy from other source files.

@lichang-bd
Copy link
Member

It seems that it has not been updated for a while, and I look forward to your continuous contribution. @beyond-up

@beyond-up beyond-up closed this Feb 21, 2023
@beyond-up beyond-up deleted the dev_doris_partition branch February 21, 2023 07:09
@beyond-up beyond-up restored the dev_doris_partition branch February 21, 2023 07:13
@beyond-up beyond-up reopened this Feb 21, 2023
@beyond-up beyond-up closed this Feb 21, 2023
@beyond-up beyond-up deleted the dev_doris_partition branch February 21, 2023 07:40
@beyond-up beyond-up restored the dev_doris_partition branch February 21, 2023 07:40
# Conflicts:
#	bitsail-connectors/connector-doris/src/main/java/com/bytedance/bitsail/connector/doris/option/DorisWriterOptions.java
#	bitsail-connectors/connector-doris/src/main/java/com/bytedance/bitsail/connector/doris/sink/DorisSink.java
@beyond-up beyond-up reopened this Mar 22, 2023
@beyond-up beyond-up marked this pull request as draft March 22, 2023 10:07
@beyond-up beyond-up marked this pull request as ready for review April 12, 2023 02:35
@beyond-up beyond-up marked this pull request as draft April 12, 2023 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

3 participants