Djed UdZddlmZmZmZddlmZmZddlm Z ddl m Z ddl m Z mZddlmZehdZeeed <ehd Zeeed <ehd Zeeed <ehdZeeed<ehdZeeed<eeeedZeeeefed<ehdZeeed<ej;Dcic]\}}|e|c}}Zeeefed<eeZ eed<dedede!fdZ"dede#e de#e fdZ$dede#e de#e fdZ%dZ&d d d!d"Z'eeefed#<d$Z(d%Z)d&Z*d'Z+dedefd(Z,deded)d*de!fd+Z-dede#e de#e fd,Z.de#e d-ede#e fd.Z/dede#e de#e fd/Z0d0Z1d1Z2de#e de#e fd2Z3dede#e de#e fd3Z4dedede!fd4Z5dede#e d5e!de#e fd6Z6d7d8dede#e d5e!de#e fd9Z7y:cc}}w);u\Stage 13: post-processing rank corrections. After statistical scoring produces a ranked list of candidates, a chain of rank corrections fixes up the ranking when bigrams alone are insufficient — see :func:`postprocess_results` for the order. The steps: dead-heat priors (superset preference, era prevalence), rare-language arbitration (ADR-0005), confusion-group resolution (delegated to :mod:`chardet.pipeline.confusion`), niche Latin demotion, KOI8-T promotion, classic-Mac line-ending promotion, and last of all the decode-safety flip, which hands a winner whose only multi-byte evidence is an undecodable trailing sequence to the best rival that can decode the caller's complete input. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. )dangling_tail_with_ascii_prefixdecodes_completelydecodes_without_error)RARE_LANGUAGES get_enc_index) _COMPAT_NAMES)DetectionResult)confusion_pair_winnerresolve_confusion_groups)REGISTRY> iso8859-1 iso8859-15cp1252_COMMON_LATIN_ENCODINGS>._ISO_8859_10_DISTINGUISHING>rrrrrrrrrrrrr r!r"r$r%r&r'r(r)r*r+r3r<_ISO_8859_14_DISTINGUISHING>rBrCrDrE_WINDOWS_1254_DISTINGUISHING>r,r-r.r/r0r1r4rGrC_HP_ROMAN8_DISTINGUISHING)z iso8859-10z iso8859-14cp1254z hp-roman8_DEMOTION_CANDIDATES> rrrr"_KOI8_T_DISTINGUISHING_DEMOTION_DELETE_KOI8_T_DELETEencodingdatareturnctj|}|yt|jd|t|k(S)alReturn True if encoding is a demotion candidate with no distinguishing bytes. Checks whether any byte in *data* falls in the set of byte values that decode differently under the given encoding vs iso-8859-1. If none do, the data is equally valid under both encodings and there is no byte-level evidence for preferring the candidate encoding. NF)rdgetlen translate)rfrgdeletes Z/root/workspace/ytshorts/venv/lib/python3.12/site-packages/chardet/pipeline/postprocess.py_should_demoteros; ! !( +F ~ t~~dF+ ,D 99resultsct|dkDr|djt|dj|r|dj}|dj}|ddD]}|jtvst |j||j |j}|Dcgc]}|j|k7s||us|}}|Dcgc]}|j|k(s|}}|g||cS|Scc}wcc}w)aDemote niche Latin encodings when no distinguishing bytes are present. Some bigram models (e.g. iso-8859-10, iso-8859-14, windows-1254) can win on data that contains only bytes shared with common Western Latin encodings. When there is no byte-level evidence for the winning encoding, promote the first common Western Latin candidate to the top and push the demoted encoding to last. rN)rkrfro confidencerr language mime_type) rgrqdemoted_encodingtop_confrpromotedxothersdemoted_entriess rn_demote_niche_latinr~s  G q AJ   + 71:.. 5"1:..1:(( =Azz44*JJ!**akk '!**8H*HQVWZA/6"XGW9W1"X"X <6z!_promote_koi8t..sQDAq!**:PaQs!!N) rfnext enumeraterkrlrertr rurv) rgrq koi8t_idx koi8t_resultrxrzrryr|s rn_promote_koi8trs gaj))X5QIg$6QSWXI 4>>$ /0CI=y) 1:(("  ! !   ! !  " "   !*' 2E1a9n!EE"6"" NFs 0 C >C g-C6?cp932cp949) shift_jisshift_jis_2004euc_kr_DEAD_HEAT_SUPERSETSg{Gzt?i@chtj|}|yt|j}|| zS)zHReturn the lowest era bit for *encoding* (lower = more prevalent today).i@)r rjintera)rfinfors rn _era_rankrIs1 << !D | dhh-C #:rpruz str | Nonec:tj|}|sy|dt}t}|d}t dt |D]*}||}|dk\s|dk\r|j |dz|z|},|sy|D]\} } } || |k7r|D] } | | s yy)u0Return True if *encoding*'s winning model weights a high-byte bigram present in *data*. A candidate whose model assigns zero weight to every non-ASCII bigram in the data earned its statistical score purely from ASCII bigrams — noise that cannot distinguish encodings. Only the variant that actually won (*language*) counts: another language's variant having weight for those bytes says nothing about why *this* result is on top. Only called on dead heats, so the Python-level scan of the (capped) data is off the hot path. FNrrsr[T)rrj_EVIDENCE_SCAN_MAX_BYTESsetrangerkadd) rgrfruvariantswindowseenprevrblangtable_keyidxs rn_has_high_byte_evidencerRs""8,H  ++ ,FUD !9D 1c&k " 1I 4<19 HHdai1_ %  %eT  DH$4  CSz  rpc~|r|dnd}||jt|dkr|St||j|jr|Sd}t |j}t dt|D]U}||}|j|j |j z tkDrn!t |j}||ksR|}|}W|dk(r|S||}t|j|j |j|j} t|D cgc] \} }| |k7s |} } }| g| Scc}} w)aoBreak statistical dead heats in favour of the more prevalent era. When several encodings score within :data:`_DEAD_HEAT_EPSILON` of the top result and the top result's models carry no weight for any high-byte bigram in the data, the ranking is an artifact of ASCII-bigram noise. Promote the candidate from the most prevalent era (modern web > legacy ISO > Mac > regional > DOS > mainframe) so evidence-free dead heats resolve to the likeliest real-world answer. A top result whose models do weight observed high-byte bigrams won on real evidence and is kept, however small its margin. rNrs) rfrkrrurrrt_DEAD_HEAT_EPSILONr rvr) rgrqtopbest_idx best_rankrryrankchosenrzjrests rn_prefer_prevalent_on_dead_heatrss8 '!*TC {cll*c'lQ.>tS\\3<<@H#,,'I 1c'l #  AJ ::   >>ALL (+= = $ ) IH 1} X F&:J:JH$G, >$!QX A >D >  t  ?s  D9-D9rc||}t|j|dj|j|j}t |Dcgc] \}}||k7s |}}}|g|Scc}}w)zDMove ``results[i]`` to the top, carrying the current top confidence.r)r rfrtrurvr)rqrryrzrr{rs rn_promote_to_toprsk A GAJ))1::q{{H$G, 7$!QQA 7D 7  t  8s A'A'cr|r|dnd}||jt|dkr|Stj|j}||St dt|D]T}||}|j |j z t kDr|S|j|k(s;t||sHt||cS|S)zAPromote a Windows superset over its base encoding on a dead heat.rNrrs) rfrkrrjrrtrrr)rgrqrsupersetrrys rn_promote_superset_on_dead_heatrs  '!*TC {cll*c'lQ.>#'' 5H 1c'l #/ AJ >>ALL (+= =  N :: !&;D(&K"7A. . / Nrpg{Gz?g333333?c|r|dnd}|?|j3|jtvs!|jtk\st |dkr|St dt |D]c}||}|j|jz tkDr|S|j |jD|jtvsWt||cS|S)aDemote a rare-language winner that leads a prevalent rival by a coin flip. Fires only when the winner's language is in :data:`~chardet.models.RARE_LANGUAGES`, its absolute confidence is inside the evidence-free zone, and a prevalent-language candidate sits within :data:`_RARE_ARBITRATION_MARGIN`. Genuine rare-language text fails both gates: even short files score confidently, and their entire neighborhood is same-language variants. rNrrs) rfrurrt _RARE_ARBITRATION_MAX_CONFIDENCErkr_RARE_ARBITRATION_MARGINr)rqrrrys rn_arbitrate_rare_languagers '!*TC <<  <<~ - >>= = w>ALL (+C C  N :: !3  ::^ +"7A. ./ Nrpc|r|dnd}||jt|dkr|St|jtk(r|S|jdk(r|S|j ddk\s|j dtkr|Stdt|D]}||}|j|jz tkDr|S|j8t|jtk(sUtd|j|jfD}t||j|j||jk(r|St||cS|S) uNPromote a classic-Mac candidate when line endings are bare ``\r``. Classic Mac OS is the only platform that terminated lines with a lone carriage return, so data with several ``\r`` bytes and no ``\n`` is near-certainly Mac-era text. When a LEGACY_MAC candidate scores within :data:`_CR_MAC_BAND` of a non-Mac top result, promote it — unless the pair has a distinguishing-byte map and the byte-level evidence says the current top wins: a platform prior must not overturn direct evidence that confusion resolution may have just used to establish the top. rNrzxx  rsc3&K|] }|| yw)N)rrs rnrz2_promote_mac_on_cr_line_endings..st?Os)rfrkr_LEGACY_MAC_ERArufindcount_CR_MAC_MIN_LINESrrt _CR_MAC_BAND frozensetr r)rgrqrrrylangss rn_promote_mac_on_cr_line_endingsrsG '!*TC {cll*c'lQ.>/1  ||u yy1 5 14E E 1c'l #/ AJ >>ALL (< 7 " N! :: !i &;&N"%,, !;E&dCLL!**eL<<  N#7A. .'/( Nrpcjt||sytj|}|duxs t||S)apCheck that *data* decodes completely under *encoding* and its output name. The flip's promise is that the caller's ``data.decode(result)`` works, and the caller sees the *public* name: ``compat_names=True`` (the default) can remap to a strictly narrower codec (``euc_jis_2004`` is reported as ``EUC-JP``), so a rival must decode under both names to be promoted. ``prefer_superset=True`` can also narrow (cp125x leaves codepoints undefined that iso-8859-x maps), but that is an opt-in output transform applied to every detection, not only promoted ones, and is out of this step's hands. FN)rrrj)rgrfdisplays rn_decodes_under_public_namesr&s8 dH -)G d? ?0w??rpinput_truncatedcL|r|dnd}|sG|E|j9t|dks+td|ddDrt||js|St dt|D]7}||}|jt ||js+t ||cS|S)a Promote a strictly decoding rival over a winner with no real evidence. Byte-validity filtering runs incremental decoders with ``final=False``, tolerating an incomplete multi-byte sequence at the end because detection input is often a prefix of a larger whole. When chardet examined the caller's *entire* input, that tolerance can hand back an encoding the caller's very next ``data.decode()`` will reject --- a four-byte ``iso-8859-1`` word ending in ``0xE1`` detected as utf-8 (issue #380). Fires only when the input was not truncated by chardet itself (the orchestrator's ``max_bytes`` slice or ``UniversalDetector``'s buffer cap --- either way these bytes are not the whole story and the caller was told so by *input_truncated*), the tail can actually hold a dangling sequence (a high byte in the final four, multi-byte winner), and the winner's tolerant decode is **non-empty pure ASCII** --- its only multi-byte evidence is the dangling tail itself. An empty tolerant decode (the whole input is one clipped sequence) is zero evidence, not ASCII evidence, and disqualifies the flip. The best-ranked rival that decodes the input completely under both its internal and public names then takes the top slot, regardless of the confidence gap: an all-ASCII-evidence winner detected nothing the rival did not also detect, and any statistical lead it holds comes from ASCII bigrams the rival matched equally well. The scan sees the ranking as given, which under ``full_ranking=False`` is pruned; if no listed rival decodes, the winner stands (measured across a 648-case accent-final sweep, the pruned ranking always carried a decodable rival). The pure-ASCII condition is what makes the unconditional flip safe. A short mid-character CJK cut has a correct answer that cannot decode the input --- flipping it to whichever single-byte codec happens to decode the bytes trades a right answer for a wrong one, and a 5-40 byte sweep measured exactly that under a gap-based rule (34 correct CJK answers lost, Big5 becoming cp1125). Such a winner has decoded real multi-byte characters and keeps its ranking; with the pure-ASCII condition in place, the same sweep measures zero lost answers at any gap. rNrc3&K|] }|dk\ yw)r[Nr)rrs rnrz+_prefer_decodable_on_tie..ps0190srs)rfrkanyrrrr)rgrqrrrrys rn_prefer_decodable_on_tier8sV '!*TC ; <<  wrs$ 9.,&+4+3/8/1/Ys^1n/8 "/Ys^"T09(0in-6-9S>B.-** 3d3 #./*3L* #/C.H.H.J&*S(Cx&$sEz"455 :S : :$ :  / " /@  / " /J(d38n  !%3,SWB' ' / "' /'TT/2stO?T  / " /J  $(  / " /@/ / / "/ //d@e@s@t@$A A / "A A  / AP" T T / "T T  / TW &sG