)i DSrSSKrSSKrSSKJr \R 4Sjrg)z Some useful functions for converting and disambiguating between common alternative orthographies (ways of writing) the same text. N) sanscriptcU[R:XGaUn[R"SSU5n[R"SSU5n[R"SSU5n[R"SSU5n[R"SSU5n[R"SS U5n[R"S SU5n[R"S S U5n[R"S SU5n[R"SSU5n[R"SSU5n[R"SSU5n[R"SSU5n[R"SSU5n[R"SSU5n[R"SSU5n[R"SSU5n[R"SS U5n[R"S!S"U5n[R"X![R S#9nU$[ R"S$RX55 [R"SSU5$)%aG Given some (devanAgarI?) sanskrit text, this function produces a "key" so that 1] The key should be the same for different observed orthographical forms of the same text. For example: :: - "dharmma" vs "dharma" - "rAmaM gacChati" vs "rAma~N gacChati" vs "rAma~N gacChati" - "kurvan eva" vs "kurvanneva" 2] The key should be different for different for different texts. - "stamba" vs "stambha" This function attempts to succeed at [1] and [2] almostall the time. Longer the text, probability of failing at [2] decreases, while probability of failing at [1] increases (albeit very slightly). Sources of orthographically divergent forms: - Phonetically sensible grammar rules - Neglect of sandhi while writing - Punctuation, spaces, avagraha-s. - Regional-language-influenced mistakes (La instead of la.) Some example applications of this function: - Create a database of quotes or words with minimal duplication. - Search a database of quotes or words while being robust to optional forms. Also see equivalent function in the scala indic-transliteration package. z\P{IsDevanagari}z\sz\p{P}u[०-९।॥॰ऽ]|[॑-॔]u[यरल]्ँuम्u[ङञणन]uमuँ|ंuॐu ओम्u[ळऴ]uलu([क-हक़-य़])्\1+z\1u[कग]्ख्uख्u[कग]्घ्uघ्u च्छ्uछ्u ज्झ्uझ्u त्थ्uथ्u द्ध्uध्u ड्ढ्uढ्u प्फ्uफ्u ब्भ्uभ्)_from_tozgot script {} for '{}') r DEVANAGARIregexsub transliterate OPTITRANSloggingwarningformat)textencoding_schemekeys c/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/indic_transliteration/deduplication.pyget_approx_deduplicating_keyr sJ)...ii+R5iir3'ii"c*ii92sCii+Xs;ii(%5ii 8S1ii{C0ii E3/ii3UC@ii+Xs;ii+Xs;ii#6ii#6ii#6ii#6ii#6ii#6ii#6%%ciFYFYZ 077NOyyD)))__doc__r r indic_transliterationrrrrrrs) +8A7K7KF*r