Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(mito): Skeleton for scanning a region #2230

Merged
merged 20 commits into from
Aug 28, 2023

Conversation

evenyag
Copy link
Contributor

@evenyag evenyag commented Aug 22, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This PR implements a skeleton for scanning a mito2 region. The mito engine creates a ScanRegion to handle a ScanRequest.

A ScanRegion collects SSTs and memtables to scan without actually reading them. It creates a Scanner to actually scan these targets in Scanner::scan(). It may use different scan modes for different scan requests so the Scanner is an enum. Now we only implement a sequential scan mode SeqScan.

classDiagram
class ScanRegion {
    -VersionRef version
    -ScanRequest request
    ~scanner() Scanner
    ~seq_scan() SeqScan
}
class Scanner {
    <<enumeration>>
    SeqScan
    +scan() SendableRecordBatchStream
}
class SeqScan {
    -ProjectionMapper mapper
    -Option~TimeRange~ time_range
    -Option~Predicate~ predicate
    -Vec~MemtableRef~ memtables
    -Vec~FileHandle~ files
    +build() SendableRecordBatchStream
}
class ProjectionMapper {
    ~output_schema() SchemaRef
    ~convert(Batch) RecordBatch
}
ScanRegion -- Scanner
ScanRegion o-- ScanRequest
Scanner o-- SeqScan
Scanner -- SendableRecordBatchStream
SeqScan o-- ProjectionMapper
SeqScan -- SendableRecordBatchStream

Loading

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

@evenyag evenyag force-pushed the feat/mito2-handle-query branch 2 times, most recently from 2559f0a to 2a8cada Compare August 24, 2023 03:45
@evenyag evenyag marked this pull request as ready for review August 24, 2023 08:14
@codecov
Copy link

codecov bot commented Aug 24, 2023

Codecov Report

Merging #2230 (8d8ccb9) into develop (8d446ed) will decrease coverage by 0.38%.
Report is 6 commits behind head on develop.
The diff coverage is 86.64%.

@@             Coverage Diff             @@
##           develop    #2230      +/-   ##
===========================================
- Coverage    85.22%   84.85%   -0.38%     
===========================================
  Files          705      709       +4     
  Lines       115706   116304     +598     
===========================================
+ Hits         98613    98687      +74     
- Misses       17093    17617     +524     

@evenyag evenyag self-assigned this Aug 24, 2023
@evenyag evenyag mentioned this pull request Aug 24, 2023
2 tasks
src/mito2/src/error.rs Outdated Show resolved Hide resolved
src/mito2/src/memtable/version.rs Outdated Show resolved Hide resolved
@evenyag evenyag requested a review from waynexia August 25, 2023 08:56
src/mito2/src/read/scan_region.rs Show resolved Hide resolved
src/mito2/src/read/stream.rs Outdated Show resolved Hide resolved
@evenyag evenyag mentioned this pull request Aug 28, 2023
2 tasks
Copy link
Contributor

@v0y4g3r v0y4g3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM

src/mito2/src/memtable/version.rs Show resolved Hide resolved
src/mito2/src/read/scan_region.rs Outdated Show resolved Hide resolved
@v0y4g3r v0y4g3r enabled auto-merge August 28, 2023 06:48
@v0y4g3r v0y4g3r added this pull request to the merge queue Aug 28, 2023
Merged via the queue into GreptimeTeam:develop with commit e2522df Aug 28, 2023
13 checks passed
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
* feat: define stream builder

* feat: scan region wip

* feat: create SeqScan in ScanRegion

* feat: scanner

* feat: engine handles scan request

* feat: map projection index to column id

* feat: Impl record batch stream

* refactor: change BatchConverter to ProjectionMapper

* feat: add column_ids to mapper

* feat: implement SeqScan::build()

* chore: fix typo

* docs: add mermaid for ScanRegion

* style: fix clippy

* test: fix record batch test

* fix: update sequence and entry id

* test: test query

* feat: address CR comment

* chore: address CR comments

* chore: Update src/mito2/src/read/scan_region.rs

Co-authored-by: Lei, HUANG <[email protected]>

---------

Co-authored-by: Lei, HUANG <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants