Dj+rdZddlmZmZmZmZmZmZddlm Z dZ de de de dzfd Z de d ee dee fd Zy) aThree-tier language detection for filling DetectionResult languages. Tier 1: hardcoded mapping for single-language encodings (e.g. Big5 -> Chinese). Tier 2: statistical bigram scoring against the encoding's language-model variants. Tier 3: decode to UTF-8 and score against the UTF-8 byte-level language models. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. )_THIN_RARE_MAX_BYTESRARE_LANGUAGES BigramProfilehas_model_variantsinfer_languagescore_best_language)DetectionResultidataencodingreturnNc|dk(r|S |j|djddS#tttf$rYywxYw)aPDecode data from encoding and re-encode as UTF-8 for language scoring. Returns None if the encoding is unknown. For UTF-8, returns data as-is. Uses ``errors="ignore"`` because the data already passed byte-validity filtering for the detected encoding; any residual invalid bytes are irrelevant for language scoring. utf-8ignore)errors surrogatepassN)decodeencode LookupError TypeError ValueError)r r s W/root/workspace/ytshorts/venv/lib/python3.12/site-packages/chardet/pipeline/language.py_to_utf8rs[7 {{8H{5<< O=   J /s#-AAresultsc Z|dt}dt|cxkxr tknc}g}d}d}d}|D]v}|xr2|jduxr"|jtvxr|j du}|j |j|s|j |f|j } |rdn t| } | ,|r*t| r| t|}t|| ||\} } |xr| duxr| tv} | | rM|rKtdr@t|| } | r2| |k7r t| }| }t| d||\} }| | |tvr|} |r| | tvr|j |1| |j |F|j t| |j| |jy|S)ubFill missing ``language`` fields on text results via the three-tier algorithm. Tier 1: single-language encodings via hardcoded map (instant). Tier 2: multi-language encodings via statistical bigram scoring (lazy). Tier 3: decode to UTF-8, score against UTF-8 language models (universal fallback). Binary results (``encoding is None``) are passed through unchanged, as are results that already have a non-``None`` language — except a :data:`~chardet.models.RARE_LANGUAGES` label on a thin input, which is re-derived through the same scoring so the thin-rare demotion band applies to statistically-attached labels too, not only to labels this function computes. A re-derivation can only *demote* to a prevalent language; it never swaps one rare label for another. :param data: The raw byte data the results were produced from. Truncated to the first 2 KB internally — bigram language models converge quickly. :param results: A list of :class:`DetectionResult` from the pipeline. :returns: A list of results with ``language`` filled in where possible. Nr)profiledemote_thin_rarer)_LANG_SCORE_MAX_BYTESlenrlanguagerr appendrrrrrr confidence mime_type)r rthinfilledr utf8_profileutf8_profile_srcresultrecheckr lang_escalate utf8_data utf8_langs rfill_languagesr..s, && 'D s4y // /D$&F$(G)-L%)3  ,t+ ,>1 ,t+  ?? "v'B7 MM& ! ??tN8$<  MM& ! \ MM& ! MM&*;*;T6CSCST c3h M)__doc__chardet.modelsrrrrrrchardet.pipeliner rbytesstrrlistr.r/rr7sj - 5CEDL$T T/T /Tr/