Identify URLs and output them in TEI #1099
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a continuation of #1097. In this PR the URLs are output in the XML-TEI under the tags
<ref type="url" target="cleaned_version">url appearing from the data extraction</ref>
.As with #1097 we exploit the PDF annotations for clickable URLs to correct the extracted information from the regular expressions (I tried to improve them but did not obtained anything more robust).
The extracted URLs are aggregated to the notes and processed in the same data flow, thus with different formatting rules, as they are appearing in the text. This change will help in future if we need to add the identification of more elements. The implementation could be improved with more specialised data structures instead of the
Triple
object, to carry out the information.Here some output examples:
Here an example with sentence segmentation:
with sentence segmentation
TODO:
- fix some corner cases, where the.
or the)
are wrongly retained, these are cases when the regex catches too much (usually one or a few characters)- sometimes the URL refs are attached to the following text, it seems that a space is missing somehow- validate the XML-TEI schema