[dev:main merge] #308
Merged
[dev:main merge] #308
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
New SectionNumber and PartitionNumber properties added to MemoryRecord. Add example showing how to leverage partition numbers to load adjacent records. Add MemoryRecord extensions methods to decouple code from constants. Log warnings in handlers when files are missing, e.g. a handler has been removed from the default pipeline. Breaking changes: * changes to ABSTRACTIONS assembly * new tags added to records * new props added to pipeline status file
The terminology used is "section number", which can be applied to multiple data formats. Formats: * Powerpoint: page number => slide number * Excel: page number => worksheet number * PDF: section number is a reliable page number * Word: section number is an approximate page number. The value is not reliable because OOXML doesn't capture how a document is rendered, and word documents can be paginated differently across different editors. Page number are not stored yet, there's work required on the text chunker, to connect chunks to page numbers.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
… fixes (#309) ## Motivation and Context (Why the change? What's the scenario?) Some functional tests config used a different schema, making it harder to copy settings from service to examples to tests. Redis config needs an update given the new tag for partition number. Redis functional tests checking for error message were not aligned with latest code. ElasticSearch config settings were missing in Service appsettings.json ## High level description (Approach, Design) * Add "KernelMemory" config prefix where missing. * Update Redis config and Redis settings. * Add ES default settings.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context (Why the change? What's the scenario?)
The current memory storage doesn't capture the position of a partition, making it hard or impossible to sort memories in relation to their position from the original file. Position can be useful to retrieve "adjacent" memories by position in the doc.
Similarly, memory records do not capture the "section" from which they are extracted, e.g. the page number, worksheet number, slide number, segment number, scene number etc.
Synchronize config files schema across projects (service, examples, tests)
High level description (Approach, Design)