Dj/UdZddlmZddlZddlmZddlmZideddhd ed d hd ed hd edhdedhdeddhdeddhdedhdedhdedhdedhdedhdehdd ed!d"hd#ed$hd%edhd&ed'hed(hed)hed*hed+hed,hed-hed.d/hed0hed0hed1d2hed3hed3hd4 Z d5e d6<d7Z d8e d9<d:Z d8e d;<d<f dNd=Z e e Zd5e d><dOd?ZiZd5e d@<e j#D]=\ZZeexseZedAeDZej-eeezee<?e e dBCZd5e dD<dPdEZdPdFZdQdGZedHdIhZdJe dK<dRdLZ dSdMZy)TaAccuracy-evaluation tables and predicates. Tools for asking *"is this detection acceptable, given what was expected?"* Used by the test suite and by benchmarking/diagnostic scripts (e.g., ``scripts/compare_detectors.py``, ``scripts/diagnose_accuracy.py``). This module owns three kinds of tables: 1. **Directional supersets** (:data:`SUPERSETS`) -- detecting a superset encoding when the expected encoding is a subset is correct (e.g., detecting UTF-8 when expected is ASCII), but not the reverse. 2. **Bidirectional encoding groups** (:data:`BIDIRECTIONAL_GROUPS`) -- groups of encodings where any member is interchangeable with any other. 3. **Bidirectional language groups** (:data:`LANGUAGE_EQUIVALENCES`) -- ISO 639-1 codes for languages that are nearly indistinguishable by statistical detection (e.g., Slovak/Czech, Malay/Indonesian). The corresponding predicates are :func:`is_correct`, :func:`is_equivalent_detection`, and :func:`is_language_equivalent`. ) annotationsN)Callablelookup_encodingASCIIzutf-8cp1252zTIS-620z iso8859-11cp874z ISO-8859-11GB2312gb18030GBKBig5 big5hkscscp950 Shift_JIScp932shift_jis_2004zShift-JISX0213zEUC-JP euc_jis_2004z EUC-JISX0213zEUC-KRcp949CP037cp1140z ISO-2022-JP> iso2022_jp_2iso2022_jp_extiso2022_jp_2004z ISO2022-JP-1rrz ISO2022-JP-3rz ISO-8859-1z ISO-8859-2cp1250cp1251cp1256cp1253cp1255cp1254cp1257z utf-16-lez utf-16-bezutf-16z utf-32-lez utf-32-bezutf-32) z ISO-8859-5z ISO-8859-6z ISO-8859-7z ISO-8859-8z ISO-8859-9z ISO-8859-13zUTF-16z UTF-16-LEz UTF-16-BEzUTF-32z UTF-32-LEz UTF-32-BEdict[str, frozenset[str]] SUPERSETS))rrrtuple[tuple[str, ...], ...]BIDIRECTIONAL_GROUPS))skcs)ukrubgbe)msid)nodasvLANGUAGE_EQUIVALENCESc|SN)xs P/root/workspace/ytshorts/venv/lib/python3.12/site-packages/chardet/evaluation.pyr6wscfi}|D](}tfd|D}|D] }|||<*|S)zJBuild a lookup: key -> frozenset of all equivalent keys in the same group.c3.K|] }|ywr2r3).0n normalizes r5 z%_build_group_index..|s7A9Q<7s) frozenset)groupsr<resultgroupnormednames ` r5_build_group_indexrDusM )+F-777 -D&,F9T? # -- Mr7_LANGUAGE_EQUIVcL||k(rytj|}|duxr||vS)aCheck whether *detected* is an acceptable language for *expected*. Returns ``True`` when *expected* and *detected* are the same ISO 639-1 code, or belong to the same equivalence group in :data:`LANGUAGE_EQUIVALENCES`. :param expected: Expected ISO 639-1 language code. :param detected: Detected ISO 639-1 language code. :returns: ``True`` if the languages are equivalent. TN)rEget)expecteddetectedrAs r5is_language_equivalentrJs48    )E   2U!22r7_NORMALIZED_SUPERSETSc#:K|]}t|xs|ywr2r)r:ss r5r=r=sDA*/a/Dc t|xs|Sr2r)r;s r5r6r6soa.@.EAr7)r<_NORMALIZED_BIDIRc||duS|yt|xs|j}t|xs|j}||k(S)aCheck whether *detected* names the very same encoding as *expected*. Only alias and case differences are normalized away, so ``windows-1252`` matches ``cp1252``. Unlike :func:`is_correct`, this credits neither supersets nor byte-order variants: detecting UTF-8 for an ASCII file is ``False`` here. This is the stricter convention some other detectors score themselves against. Reporting it alongside :func:`is_correct` makes the two conventions directly comparable instead of leaving the difference implicit. :param expected: The expected encoding name, or ``None`` for binary files. :param detected: The detected encoding name, or ``None``. :returns: ``True`` if both names resolve to the same encoding. NF)rlowerrHrInorm_expnorm_dets r5is_exact_matchrVsU"4x(.sC+*?*?*B1Cs$$)r]r<join)textnfkds r5_strip_combiningrcs)   .D 77CdC CCr7)¤€)rerdzfrozenset[tuple[str, str]]_EQUIVALENT_SYMBOL_PAIRScR||k(ry||ftvryt|t|k(S)uReturn True if characters *a* and *b* are functionally equivalent. Equivalent means: - Same character, OR - Same base letter after stripping combining marks, OR - An explicitly listed symbol equivalence (e.g. ¤ ↔ €) T)rfrc)abs r5_chars_equivalentrjs4 Av 1v)) A "21"5 55r7c|||duS|yt|xs|j}t|xs|j}||k(ry |j|}|j|}||k(ryt |t |k7ryt dt||dDS#ttf$rYywxYw)uCheck whether *detected* produces functionally identical text to *expected*. Returns ``True`` when: 1. *detected* is not ``None`` and both encoding names normalize to the same codec, OR 2. Decoding *data* with both encodings yields identical strings, OR 3. Every differing character pair is functionally equivalent: same base letter after stripping combining marks, or an explicitly listed symbol equivalence (e.g. ¤ ↔ €). Returns ``False`` if *detected* is ``None``, either encoding is unknown, or either encoding cannot decode *data*. :param data: The raw byte data that was detected. :param expected: The expected encoding name, or ``None`` for binary files. :param detected: The detected encoding name, or ``None``. :returns: ``True`` if decoding with *detected* yields functionally identical text to decoding with *expected*. NFTc3:K|]\}}t||ywr2)rj)r:rhris r5r=z*is_equivalent_detection..4sX41a A&XrN)strict)rrRdecodeUnicodeDecodeError LookupErrorlenallzip)datarHrIrTrUtext_exptext_dets r5is_equivalent_detectionrws.4x(r"__annotations__r$r0rDrErJrKitems_subset _supersets_key_normedrGrPrVrXrcrfrjrwr3r7r5rsI.#$,+( Y* ++( y,01+(9gY'+( i $ +(  9i[ ! +(  I{G, - +(G%567+(i!1 23+( i()+(I~./+( i "+( Yz "+(.9RS/+(0I~/?@A1+(2I0123+(6)XJ'7+(8)XJ'9+(:XJ'XJ'XJ'XJ'XJ'hZ(k23H:&H:&k23H:&H:&U+( $+h5162&'2  ' #  .@@U-V*V3*4605$??,YGZ 7 # .wDDDDG"7";";D)+"NQX"X$Y 0B$E0,  4#LD8A84 6 .Y .Y%.Y1;.Y .Yr7