CollateX: Normalize txt

Feature Requests

Description

Some manuscripts provide their text with diacritics while others don't. As a result, CollateX doesn't recognize words that are actually the same (except for the diacritics) as different and produces false positives. In order to avoid this, we have to strip all diacritics from the text before we save the TXT files.

Code blocks to consider:

• "Tashkil from ISO 8859-6", "Combining maddah and hamza", "Other combining marks", see https://unicode-table.com/en/blocks/arabic/ for the Arabic script
• "Syriac punctuation and signs", Syriac points (vowels)", "Syriac marks" see https://unicode-table.com/en/blocks/syrian/ for Syriac script

This list may be extended.

User Stories

As a scholar I need normalized text in order to get good results with CollateX. Using the texts as they are produces too many errors.

Classification

Is this feature an enhancement of existing code or a completely new feature?

• enhancement
• new feature

Related Tickets

#50 (closed)

/cc Mathias Göbel, Frank Schneider, Michelle Weidling

changed milestone to %Ahikar Version 0.11.0

added Ahikar label

marked this issue as related to #50 (closed)

added Doing label

changed milestone to %Ahikar Version 0.12.0

The vocalization of the Arabic texts should be kept according to our talk with the scholars on 6th October.

mentioned in merge request !45 (merged)

mentioned in commit a5ad021e

mentioned in commit 5e2cc0c2

added In Review label and removed Doing label

added waiting for code review label and removed In Review label

mentioned in commit fc3d1653

added In Review label and removed waiting for code review label

closed

removed In Review label