Dj,iUdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z mZddlmZmZejdejdfZe j&j)dZej,dj.Zej,d j.Zed ed DZd Zd ZiZe e!e!fe"d<ejFD]4Z$e%e$jLdk(se$jLdee$jN<6 d2dede(de)e!de(de e!eff dZ*dede+e e!efe e!e,fffdZ-ej\de+e e!efe e!e,fffdZ/de e!effdZ0de e!efde e!e)e+e!dzee!fffdZ1ej\de e!e)e+e!dzee!fffdZ2de!de!dzfdZ3de!de4fdZ5de e!e,ffdZ6ej\de e!effd Z7ej\defd!Z8Gd"d#Z9 d3d$e9d%d&d'e!de,fd(Z:e;hd)Zd-Z? d4d.d/dede!d$e9dzd0e4de+e,e!dzff d1Z@y)5zModel loading and bigram scoring utilities. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)_kernel) dot_packed pack_profile)REGISTRYlookup_encodingi)z.soz.pydz>Iz>dc#,K|] }|dvrdndyw)) rN).0bs U/root/workspace/ytshorts/venv/lib/python3.12/site-packages/chardet/models/__init__.py r9s" DE8 8Aa? ssCMD2sCRM1_SINGLE_LANG_MAPrdataoffsetnames chunk_sizereturnc(t|}|dz}tj}i}t}d} |} t|} d} t||krdt|z } |jr|j |j| }nF| | kr)|| | |z}| t|z } |j || }n| sd} |j }|snknj| t|z } ||z }t|dk(r+t|||t|<|jnt|dkDrnt||krt||k(rd}|jr|j |jd}|sM|jsA| | kr<|| | |z}| t|z } |j |d}|s|js| | kr<|s|js| s|j }| t|z } | |k7st||k7rd| d|}t||S) uDecompress the model tables from ``data[offset:]``, one per name. Each model is stored as its own bytes object rather than a memoryview slice of one big blob: mypyc compiles bytes indexing in the scoring hot loop to a native C array access, while memoryview indexing goes through a boxed generic call. Decompression is incremental, one 64 KB table at a time — materializing the whole multi-megabyte blob and slicing it would transiently double the allocation and strand the freed pages in process RSS. Trailing compressed bytes are ignored, as with whole-blob ``zlib.decompress``; the decompressed size is validated instead. :raises ValueError: If the decompressed size is not exactly ``len(names) * 65536``. rFTrz&corrupt models.bin: decompressed size z != expected ) lenzlib decompressobj bytearrayunconsumed_tail decompressflushbytescleareof ValueError)rrrr num_models expected_sizedecompmodelstableproducedposendflushedneedpiecechunkextramsgs r_decompress_tablesr:Is$"UJ&M    !F!F KEH C d)CG f+ "s5z!  ! !%%f&<& >| LC >> D   '   2v T ""r ctdS)zLoad all bigram models from the bundled models.bin file. Each model is a bytes object of length 65536 (256*256). Index: (b1 << 8) | b2 -> weight (0-255). :returns: A dict mapping model key strings to 65536-byte lookup tables. rr^rr r load_modelsras  q !!r r/ci}|jD]>\}}|jdd\}}|j|gj|||f@t |D]}t |}|||vs||||<|S)zBuild a grouped index from a models dict. :param models: Mapping of ``"lang/encoding"`` keys to 65536-byte tables. :returns: Mapping of encoding name to ``[(lang, model, model_key), ...]``. /r)itemssplit setdefaultrClistr)r/indexkeymodellangencenc_name canonicals r_build_enc_indexros=?Elln= UIIc1% c b!(($s);<= K/#H-  Ye%;$XE) / Lr c(ttS)zTReturn a pre-grouped index mapping encoding name -> [(lang, model, model_key), ...].)rorarr r get_enc_indexrqs KM **r encodingc,tj|S)zReturn the language for a single-language encoding, or None. :param encoding: The canonical encoding name. :returns: An ISO 639-1 language code, or ``None`` if the encoding is multi-language. )rgetrrs rinfer_languagervs    ))r c|tvS)zReturn True if the encoding has language variants in the model index. :param encoding: The canonical encoding name. :returns: ``True`` if bigram models exist for this encoding. )rqrus rhas_model_variantsrx s } &&r ctdS)zAReturn cached L2 norms for all models, keyed by model key string.rr`rr r_get_model_normsrzs  q !!r c t}tjjd} |j dj }t j|j dj j}d}|ddtk(rV|d||k(rNt||t|dzzk(r1t|Dcic]\}}||||dzz||d zdzzc}}S|rtjd t d |j#Dcic]%\}|t%fd t'dddD'c}}S#ttf$rd}d}YwxYwcc}}wcc}}w)uReturn per-model row-maximum tables for upper-bound prescreening. For each model, entry ``b1`` of its 256-byte table holds the maximum weight in the model's row for lead byte ``b1``. Because every bigram weight is bounded by its row maximum, a dot product against the row maxima (256 terms) upper-bounds the dot product against the full table (65536 terms) — statistical scoring uses this to rule out candidate models without scoring them fully. Loads the precomputed ``rowmax.bin`` (written by ``scripts/train.py`` in the same model order as ``models.bin``). The file starts with a ``CRM1`` magic and the SHA-256 of the ``models.bin`` it was derived from: a stale or mismatched file would silently under-estimate row maxima and break the upper bound that pruning depends on, so anything that does not match the *current* ``models.bin`` byte-for-byte is rejected and the tables are derived from the models directly (slower, but always correct). rOz rowmax.binrPr $Nr<rrzpchardet rowmax.bin is missing or does not match models.bin; deriving row maxima from the models (slower startup)rQrRc3@K|]}t||dzyw)rN)max)rstartr0s rrzget_rowmax..Gs!Uu3uUUS[12Usrr)rarTrUrVrWrXhashlibsha256digestFileNotFoundErrorOSError _ROWMAX_MAGICr! enumeraterYrZr[rdr(r@)r/rVr models_digest header_sizerrir0s `r get_rowmaxrs~&]F    % %&6 7E~~l+668 NN< ( 3 3 5 &(  K RaM! ; = 0 Is6{S'88 8 $F+ 3 kAG+kQUcM.IJ J    C   !,,.   C UUaPS@TU UU + w '   sAE E(!*E.E%$E%ctjjdjd}|j }t |dk7r.t jdt |dtdddzS|S) uReturn a 65536-byte IDF weight table for bigram profile construction. Loads a precomputed table from ``idf.bin`` (generated at training time). For each bigram index, the weight reflects how discriminative that bigram is across all models: - Bigrams in every model (common ASCII) → weight 1 (minimal signal) - Bigrams in one model → weight 255 (maximum signal) - Bigrams not in any model → weight 1 (unknown, treat as neutral) rOzidf.binrz chardet idf.bin has wrong size (z"), falling back to uniform weightsrQrR) rTrUrVrWrXr!rYrZr[r\s rget_idf_weightsrLsv    # #$4 5 > >y IC >> D 4yE .s4yk:. .    Kr c veZdZdZdZdeddfdZdeedeed eed eddf d Z e d e eefddfd Z y) BigramProfileuPre-computed bigram frequency distribution for a data sample. Computing this once and reusing it across all models reduces per-model scoring from O(n) to O(distinct_bigrams). Each bigram is weighted by its IDF (inverse document frequency) across all models — bigrams unique to few models get high weight, bigrams common to all models get weight 1. ``nonzero`` lists the indices carrying weight, in first-encounter order. The weights are reachable two ways, and which one a profile fills depends on how it was built: * ``idx_arr``/``val_arr`` — parallel ``array('i')`` buffers, what :func:`score_with_profile` reads for a streaming profile. Filled by the streaming constructor only. * ``values`` — a plain list parallel to ``nonzero``, filled by :meth:`from_weighted_freq` for the small focused profiles confusion resolution builds, which are scored inline instead. ``row_freq`` aggregates the weights by lead byte (256 entries) and ``nonzero_rows`` lists the lead bytes with non-zero total; together with per-model row maxima (:func:`get_rowmax`) they let statistical scoring compute a cheap upper bound on a model's score. **Input limit.** Weights are packed as int32, which holds any value an input of at most 16 MB can produce (``255`` per occurrence against a 2.1-billion ceiling). Detection truncates to ``max_bytes`` long before that; construct a profile directly from a larger buffer and packing raises :exc:`OverflowError`. ) freqidx_arr input_normnonzero nonzero_rowsrow_freqval_arrvalues weight_sumrrNct|dz }|dkrEg|_g|_g|_t\|_|_g|_g|_d|_ d|_ yt}dgdz}g}d}t|D]V}||}||dz} || k(r t|r|dz| z} || } || dk(r|j| || xx| z cc<|| z }X|j||g|y)a?Compute the bigram frequency distribution for *data*. Each bigram is weighted by its IDF (inverse document frequency) across all loaded models. Bigrams unique to few models get high weight; bigrams common to all models get weight 1. :param data: The raw byte data to profile. rrNrr=)r!rrr _EMPTY_PACKEDrrrrrrrr@_ASCII_WHITESPACE_TABLErC_finish) selfr total_bigramsidfrrw_sumrb1b2idxws r__init__zBigramProfile.__init__sD A A #%DI&(DL%'DK)6 &DL$,')DM+-D #$DO%(DO #+}% AaBa!eB Rx3B77b.CCACyA~s# INI QJE $ T7B.r rrrrc*||_||_||_d}dgdz}|r:tt |D]"}||}|||zz }|||dz xx|z cc<$n$|D]} || }|||zz }|| dz xx|z cc<!t j ||_||_tdD cgc] } || s |  c} |_ |stst\|_ |_ nt||\|_ |_ |strg|_y||_ycc} w)aQStore the frequency data and derive the norm and row aggregates. Single finalization path shared by both constructors so pruning fields cannot silently diverge between them. Exactly one of *freq* and *values* carries the weights, never both. The streaming constructor passes the dense *freq* table it had to build and leaves *values* empty; the sparse constructor fills *values* parallel to *nonzero* and passes no table, which is what lets it skip a 65536-entry allocation per call. Neither is retained. *freq* is read here to derive the norm, the row aggregates and the packed buffers, and is then dropped -- scoring reads ``idx_arr``/``val_arr`` or ``values``, never the dense table, so keeping it alive would pin 512 KB per profile for nothing. rrr=N)rrrr@r!mathsqrtrrr_KERNEL_COMPILEDrrrrr) rrrrrnorm_sqrrvrrs rrzBigramProfile._finishs(.  $ cCi 3w<( /1I1q5 q)Q.) /  (I1q5 "a'" ())G,  */*EB RE ))6 &DL$,)5gt)D &DL$,!%5B D Fs , D7D weighted_freqc|d}g}g}d}|jD]/\}}|s |j||j|||z }1|jg||||S)uCreate a BigramProfile from pre-computed weighted frequencies. Computes ``weight_sum`` and ``input_norm`` from *weighted_freq* to ensure consistency between the stored fields. Deliberately does not build the dense 65536-entry ``freq`` table the streaming constructor uses. Callers here pass a handful of bigrams — confusion resolution's focused profiles hold a median of eight — and allocating a 65536-element list per call cost about 24us, which measured as roughly 40% of the whole bigram-rescore stage. Scoring reads ``nonzero``/``values``, so the table is never needed. :param weighted_freq: Mapping of bigram index to weighted count. :returns: A new :class:`BigramProfile` instance. r r)rdrCr)clsrprofilerrrrcounts rfrom_weighted_freqz BigramProfile.from_weighted_freqsu$c('--/ JCs# e$   GVU3r ) __name__ __module__ __qualname____doc__ __slots__r(rrgintr classmethoddictrrr rrres> I,/U,/t,/\1A3i1Ac1AS 1A  1A  1AftCH~/r rrrjzbytes | bytearray | memoryview model_keyc\t|ts t|}|jdk(ryt}|r|j |nd}|7d}t dD]}||}|s |||zz }t j|}|dk(ry|j}|j} | r-d} t t|D]}| |||| |zz } nKtr"t|j|j|} n#d} |j} |D]} | || | | zz } | ||jzz S)anScore a pre-computed bigram profile against a single model using cosine similarity. ``bytearray``/``memoryview`` tables are accepted for compatibility but copied to ``bytes`` first: the narrow type lets mypyc compile the dot-product loop with native byte indexing (and keeps compiled and pure-Python installs accepting the same argument types). rNrr) isinstancer(rrzrtr@rrrrr!rrrrr) rrjrrG model_normsq_sumrrrrdotrrs rscore_with_profilersI eU #e S   E)29%Ju AaA!a% YYv& SooG ^^F s7|$ 1A 5$vay0 0C 1 '//5A|| *C 5:S ) )C * *w111 22r >brcygagdRARE_LANGUAGESzxxgQ?F)demote_thin_rarerc4|s|yt}|j|}|y| t|}d}d}d}d} |D]<\} } } t|| | } | |kDr| }| }|s | tvs)| t k7s3| |kDs9| }| } >|r||tvr| ||z t kr| }||fS)a Score data against all language variants of an encoding. Returns (best_score, best_language). Uses a pre-grouped index for O(L) lookup where L is the number of language variants for the encoding. If *profile* is provided, it is reused instead of recomputing the bigram frequency distribution from *data*. :param data: The raw byte data to score. :param encoding: The canonical encoding name to match against. :param profile: Optional pre-computed :class:`BigramProfile` to reuse. :param demote_thin_rare: Pass true only when the caller has judged the *original* input thin (under :data:`_THIN_RARE_MAX_BYTES` before any transcoding). A :data:`RARE_LANGUAGES` winner that leads the best prevalent-language variant by less than the measured noise band is then reported under the prevalent language instead. The length judgment deliberately lives with the caller: this function may receive transcoded bytes or a profile without its source data, so ``len(data)`` here is not a reliable proxy for input size. Language-fill callers pass this; encoding-ranking callers must not, so that candidate ordering stays byte-identical. :returns: A ``(score, language)`` tuple. The score is always the best cosine similarity across variants; the language matches it except when ``demote_thin_rare`` fires, in which case the label is the best prevalent-language variant's while the score remains the rare winner's. N)rNr)rqrtrrr ART_LANGUAGE_THIN_RARE_MARGIN)rrrrrrhvariants best_score best_langbest_prevalentbest_prevalent_langrkrjrss rscore_best_languager]sD GO OEyy"H%J IN&*"* 'eY wy 9 z>JI N* $N"N"&  '   !  '  +  '*; ;' y  r )i))N)Ararray functoolsrimportlib.resourcesrTrrErYr"chardetrchardet._kernelrrchardet.registryrrr__file__endswithrStruct unpack_fromr?rBr(r@rr>rrrstr__annotations__r_encr! languagesrJrrgr:tuplefloatrMcacher^rarorqrvboolrxrzrrrr frozensetrr_THIN_RARE_MAX_BYTESrrrr rrs   46S!;5;;s#34 ##,,_=t$00&--%11  INs    $&$sCx.% HOO 8D 4>>a&*nnQ&7#8 CIE EE%)#YE