Dj*|dZddlmZmZmZmZmZmZddlm Z ddl m Z m Z m Z ddlmZmZmZddlmZede zzZdZd ed eed fd ed eeeeedzffdZd eed fd ed eeeeedzeeefeeeeeedzeefffdZd eed fd ed eeeeedzffdZddd ed eed fded ee fdZ y)zStage 3: Statistical bigram scoring. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. ) BigramProfile_get_model_norms get_enc_index get_rowmaxscore_best_languagescore_with_profile)DetectionResult)_CONFUSION_BAND_CONFUSION_FLOOR_RATIO_STRICT_TIER_MAX_CONF)_COMMON_LATIN_ENCODINGS_DEMOTION_CANDIDATES_RARE_ARBITRATION_MARGIN) EncodingInfo@data candidates.profilereturnNcg}|D]A}t||j|\}}|dkDs$|j|j||fC|S)zFScore every candidate fully (no pruning). Returns (enc, score, lang).)r)rnameappend)rrrscoresencslangs Z/root/workspace/ytshorts/venv/lib/python3.12/site-packages/chardet/pipeline/statistical.py _score_allr +sS 35F/%dCHHgF4 s7 MM388Q- ./ Mc t}t}t}|j}|j}|j }g}g} |D]} |j | j} | !| jr8t| D])\} \} }}|j| j| ||| f+et| D]u\} \} }}||}d}|D]}|||||zz }|j |}| td}n|dkDr |||zz }nd}| j|| | j| ||fw| jdd|| fS)u~Flatten candidate model variants for pruned scoring. Returns ``(mb_entries, sb_entries)`` where multi-byte entries are ``(enc, lang, table, key, variant_index)`` and single-byte entries are ``(upper_bound, variant_index, enc, lang, table, key)`` sorted by descending bound. The upper bound multiplies each lead byte's total profile weight by the model's maximum weight for that lead byte — at most 256 terms versus one term per distinct bigram for a full score. ``variant_index`` is the variant's position in the encoding index, so exact score ties resolve to the same variant the full path keeps. rinfrc |dS)Nr)es rz!_split_variants..ls !A$r!Tkeyreverse) rrrrow_freq nonzero_rows input_normgetr is_multibyte enumeraterfloatsort)rrindexnormsrowmaxr+r,r- mb_entries sb_entriesrvariantsvirtabler)rmub_dotb1 model_normubs r_split_variantsr@9s$ OE  E \FH''L##J@BJGIJD99SXX&      *3H*= D&&T5#!!388T5#r"BC D &/&9 D "B"ucBF" 0"R&8B<// 03J!5\c!zJ67   r2sxxucB C DD0OOO5 z !!r!c 8t}t||\}}iiiddddtdtdtdzdtddf fd }|D]\}}}} } ||t ||| ||  |D]K\} } }}}} t z } tkrt| tz} | | krn||t ||| || Mt z } tkrt| tz} jDcgc] \}}|| k\s |}}}g}td |Dr|jtd |vr|jd |rg|D]b}|j|vrt!|j#|jgD])\} \}}} ||jt ||| || +d|Dcgc]J}j#|jddkDr)|j|j|jfLc}Scc}}wcc}w) uScore candidates, skipping single-byte variants that provably cannot matter. Multi-byte variants are always scored fully — the orchestrator may later boost their confidence based on structural coverage, so no raw-score bound can rule them out. Single-byte variants are scored in descending upper-bound order (see :func:`_split_variants`) and skipped once their bound falls more than ``_PRUNE_MARGIN`` below the running second-best encoding score: such variants can affect neither the winner, nor position 1, nor any candidate within the confusion band of the top score. Encodings that ``postprocess_results`` inspects regardless of rank (the common Western Latin trio for niche-Latin demotion, KOI8-T for the KOI8-R promotion) are force-scored when their trigger could fire. Because confusion resolution can promote position 1 or any candidate within the band into position 0 before those triggers are evaluated, the trigger check covers every encoding near the top, not just the statistical winner. Returns (enc, score, lang) tuples for encodings scoring above zero, in candidate order. renc_namerrNr9rcj|}|||ks ||k(r ||k\ry||<||<||<| k(r|y|kDr || y| kDr| yyN)r.) rCrrr9prev best_lang best_scorebest_vitop1top1_enctop2s rrecordz_score_pruned..records~~h'  Ta4iB'(BSz _score_pruned..s 71$ $ 7szkoi8-rzkoi8-t)rr@strr1intr _PRUNE_MARGINr minr itemsanyextendr rrr0r.)rrr3r6r7rMrCrr:r)r9r? threshold trigger_floorr&rnear_topforcedrrGrHrIrJrKrLs @@@@@@r _score_prunedr\ps`4 OE,ZAJ #%J')I GH D DcDjcd&+5L&$sBx+GUC@$KL/9L*B$s=( ' 'It.D'DEI > x+GUC@$KL8=(M ##M42H+HI (..0GdaA4FGHGF 7h 77 -.8 h  TCxxv%+4EIIchh4K*L T&&T5#sxx!3GUC!H$PRS T  T   >>#((C (3 . :chh'388)<= H s? H H?AHF) full_rankingr]c 4|r|sgSt|}|jdk(rgS|st|jtkrt |||}n t ||}|jdd|Dcgc]\}}}t|||c}}}Scc}}}w)aScore all candidates and return results sorted by confidence descending. :param data: The raw byte data to score. :param candidates: Encoding candidates to evaluate. :param full_ranking: When ``True``, score every candidate fully so the returned list is complete (needed by ``detect_all``). When ``False`` (the default), single-byte candidates that provably cannot affect the top of the ranking may be skipped; the winner, position 1, and all candidates within the confusion band of the top score are identical to the full ranking. :returns: A list of :class:`DetectionResult` sorted by confidence. rc |dS)Nr%)xs rr'z"score_candidates..s adr!Tr()encoding confidencelanguage) rr-lennonzero_MIN_NONZERO_FOR_PRESCREENr r\r2r )rrr]rrrrrs rscore_candidatesrhs$ z D!GS  s7??+.HHD*g6z73 KKNDK1$   D!T !dC  s5B)!__doc__chardet.modelsrrrrrrchardet.pipeliner chardet.pipeline.confusionr r r chardet.pipeline.postprocessr rrchardet.registryrrSrgbytestuplelistrQr1r rRr@r\boolrhr%r!rrss-  *)1+>>     lC'(   %UC$J& '( 4"lC'(4" 4" sC$JsC/ 01uc3d E36 78:4"nnlC'(n n %UC$J& '(nj " "lC'(" "  / "r!