DjNdZddlZddlmZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZdd lmZddlmZddlm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+e deddZ,dZ-dZ.de/de0e/de/de1e fdZ2dZ3dZ4dZ5d Z6d!Z7d"e8d#e9e*d$fd%e de9e*d$ffd&Z:d'd(d"e8d)e1e9e/e;fd#e9e*d$fd%e d*eefddd.d/d'd'd0d"e8d1ed2e?d3e0e/dzd4e0e/dzd5e/d6e/d*e run_pipeline() -> _run_pipeline_core() -> _make_fallback_or_none().  zL is excluded by include_encodings/exclude_encodings; returning encoding=None) stacklevelg?N)rrr)warningswarn UserWarningrr)rr!r"s [/root/workspace/ytshorts/venv/lib/python3.12/site-packages/chardet/pipeline/orchestrator.py_make_fallback_or_noner,8sQw l!H<(K L   ~ X$ N OOg?gffffff?datavalid_candidates.ctxc4g}|D]}|jrt|||}||j|j<|tkr@|j 0t |t |jdtz |_|j tkrt||||j }||j|j<|tkr|j tk\rt|||}|tkr|j!| t#|S)aEliminate CJK multi-byte candidates that lack genuine multi-byte structure. Four checks are applied in order to each multi-byte candidate: 1. **Structural pair ratio** (valid_pairs / lead_bytes) must be >= ``_CJK_MIN_MB_RATIO``. Catches files with many orphan lead bytes. 2. **Minimum non-ASCII byte count**: the data must contain at least ``_CJK_MIN_NON_ASCII`` bytes > 0x7F. Tiny files with 1-5 high bytes can accidentally form perfect pairs and score 1.0 structurally. 3. **Byte coverage** (non-ASCII bytes in valid multi-byte sequences / total non-ASCII bytes) must be >= ``_CJK_MIN_BYTE_COVERAGE``. Latin text has many high bytes that are NOT consumed by multi-byte pairs; genuine CJK text has nearly all high bytes accounted for. 4. **Lead byte diversity**: the number of distinct lead byte values in valid pairs must be >= ``_CJK_MIN_LEAD_DIVERSITY``. Genuine CJK text draws from a wide repertoire of lead bytes; European false positives cluster in a narrow band (e.g. 0xC0-0xDF for accented Latin). Returns the filtered candidate list. Structural scores are cached in ``ctx.mb_scores`` for reuse in Stage 2b. N)non_ascii_count) is_multibyter mb_scoresname_CJK_MIN_MB_RATIOr5len translater_CJK_MIN_NON_ASCIIr mb_coverage_CJK_MIN_BYTE_COVERAGE_CJK_DIVERSITY_MIN_NON_ASCIIr_CJK_MIN_LEAD_DIVERSITYappendtuple)r1r2r3gatedencmb_score byte_coveragelead_diversitys r+_gate_cjk_candidatesrHks:!#E   /c3?H&.CMM#(( #++""*&)$i#dnnT:6V2W&W#""%77;c30C0CM)6COOCHH %55""&BB!= 0.95) receive a confidence boost proportional to coverage. When nearly all non-ASCII bytes form valid multi-byte pairs, the structural evidence is strong and should increase the candidate's ranking relative to single-byte alternatives whose bigram models may score higher on small samples. Note: boosted confidence values may exceed 1.0 and are used only for relative ranking among candidates. ``run_pipeline`` clamps all confidence values to [0.0, 1.0] before returning to callers. c38K|]\}}|vs |ywN).0r8_sc enc_lookups r+ z/_score_structural_candidates..s&&T3tz?Q 4s  c3:K|]}|jr|ywrN)r6)rPes r+rSz/_score_structural_candidates..sJa1>>JsNrIgffffff?c|jSrN)rxs r+z._score_structural_candidates..s q||r-Tkeyreverse)r6r8rBlistr_STAT_SCORE_MAX_BYTESrr=getrArrrr sort) r1rKr2r3rJrUvalid_mb single_byteresultsboostedrcoveragerRs @r+_score_structural_candidatesris#4,+q~~ +J*;HJ#3JJK '' ( %h % %% G&(G  ;<::3??&&qzz373 t  NNJJ H =qzz1;;  NN1   LL+TL: N9+s D(D(resultc|j|S|jdnd}t|j|j|j|S)zXDefault ``mime_type`` to ``text/plain`` (text) or ``application/octet-stream`` (binary).z text/plainr)r rrrr)rjmimes r+_with_default_mimermsD # !??6<.s'GS'Gsrr)rurqrVc |dS)NrWrOrYs r+r[z$_run_pipeline_core..ps QqTr-Tr\rrI)rs)$r r:r frozensetr,r rrr rrr r _BINARY_RESULTrrrrr rrrHr6r7rar8rrArb _STRUCTURAL_CONFIDENCE_THRESHOLDrirr`r_rrB)r1rtrurorprqrrrJrsr3 candidatesr! bom_resultutf1632_result escape_result magic_result utf8_precheckascii_precheck markup_resultr2rKrDscore_ best_scorere stat_datas r+_run_pipeline_corerss  C%>TY)>O   D  .?ARSJ''GJ'GGG % '+A  D!J*"5"5"@| -T2N!n&=&=&H +40M!  " " .  " "g - %L~ %M"$'N   " di 0 *$/M ]%;%;w%F  %&&-*@*@@&&'1 !**!,,!**!++  0mWM !n&=&=&H ]%;%;w%F*$ ; %&7BUVV,D2BCH %&7BUVV24<   MM%%chh/E}0sC@s{!((#((E):; <>4@)!, : 9 92! ) G*'?++,IE*:$;,WG %&7BUVV tWo NNr-c t||||||||| } t|| } | D cgc]?} | jdk(r,t| j| j d| j n| A} } | D cgc] } t| } } | s d} t| | D cgc]S} | j dkDr@t| jt| j d| j| j n| Uc} Scc} wcc} wcc} w)aRun the full detection pipeline. :param data: The raw byte data to analyze. :param encoding_era: Filter candidates to a specific era of encodings. :param max_bytes: Maximum number of bytes to process. :param include_encodings: If not ``None``, only return these encodings. :param exclude_encodings: If not ``None``, never return these encodings. :param no_match_encoding: Encoding returned when no candidate survives. :param empty_input_encoding: Encoding returned for empty input. :param full_ranking: When ``True``, statistical scoring evaluates every candidate so the returned list is complete (``detect_all`` needs this). When ``False``, candidates that provably cannot affect the top of the ranking may be omitted from the tail. :param input_truncated: Pass ``True`` when *data* is already a truncated view of the caller's input (``UniversalDetector`` caps its buffer at ``max_bytes``, which this function cannot see from ``len(data)`` alone). Truncation by the ``max_bytes`` slice here is detected either way; the flag only ever widens it. :returns: A list of :class:`DetectionResult` sorted by confidence descending. )rsrorprqrrrJzxxNz/pipeline must always return at least one resultg?) rrrrrrr rm RuntimeErrormin) r1rtrurorprqrrrJrsrergmsgs r+ run_pipeliners@! '+++1! GT7+G   ::   ALL$ D  G /66!!$6G6 ?3    <<#   C c$:AJJ T    7 sAC1/C6AC;)B__doc__r(chardet._utilsr chardet.enumsrchardet.pipelinerrrrr chardet.pipeline.asciir chardet.pipeline.binaryr chardet.pipeline.bomr chardet.pipeline.escaper chardet.pipeline.languagerchardet.pipeline.magicrchardet.pipeline.markuprrchardet.pipeline.postprocessrchardet.pipeline.statisticalrchardet.pipeline.structuralrrrchardet.pipeline.utf8rchardet.pipeline.utf1632rchardet.pipeline.validityrchardet.registryrrrzr{r`strryr_r,r9r<r>r@r?bytesrBrHfloatboolrirmintrrrOr-r+rs,%0-+:4/R<9 .<89 ' ( $( PP s^PP / P2   "3 3L#-.3 3 <  3x 5 5E#u*-.5L#-.5  5  5 /5pVV?V'hO 04/3% '!hO hOhOhO !~, hO !~, hOhOhOhOhO /hO\'A 04/3% '!A AAA !~, A !~, AAAAA /Ar-