Dj)2UdZddlZddlmZmZmZdZdZdZdZ dZ d Z d Z d Z d ezZeed <dededefdZdededzfdZdededzfdZdededzfdZdedefdZddededefdZy)aStage 1a+: UTF-16/UTF-32 detection for data without BOM. This stage runs after BOM detection but before binary detection. UTF-16 and UTF-32 encoded text contains characteristic null-byte patterns that would otherwise cause binary detection to reject the data. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)ASCII_TEXT_BYTESDETERMINISTIC_CONFIDENCEDetectionResulti gQ??g?gffffff?g333333?_NULL_SEPARATOR_ALLOWEDdata null_fracreturncD|tk\ry|jdt S)uReturn True if the data looks like ASCII with null byte separators. :param data: The raw byte sample to examine. :param null_frac: The positional null fraction for this UTF-16 candidate (i.e. fraction of null bytes in even positions for BE, or odd positions for LE) — not the total null fraction across all bytes. Checks two conditions: 1. The positional null fraction is below ``_NULL_SEPARATOR_MAX_FRACTION`` 2. Every non-null byte is printable ASCII or common whitespace When both conditions are met, the nulls are likely field separators (e.g. ``find -print0``), not UTF-16 encoding artifacts. FN)_NULL_SEPARATOR_MAX_FRACTION translater )r r s V/root/workspace/ytshorts/venv/lib/python3.12/site-packages/chardet/pipeline/utf1632.py_is_null_separator_patternr=s%00~~d$;< <<cn|dt}t|tkryt|}||St |S)aDetect UTF-32 or UTF-16 encoding from null-byte patterns. UTF-32 is checked before UTF-16 since UTF-32 patterns are more specific. :param data: The raw byte data to examine. :returns: A :class:`DetectionResult` if a strong pattern is found, or ``None``. N) _SAMPLE_SIZElen_MIN_BYTES_UTF16 _check_utf32 _check_utf16)r sampleresults rdetect_utf1632_patternsrQsD-< F 6{%%& !F    rc ttdzz }|tkryd||dz}tfdtdtdD}tfdtdtdD}||k(r8||z dkDr0 j d}t |rt dtdS tfd td tdD}tfd td tdD}||k(r9||z dkDr1 j d }t |rt d tdS yy#t$rYwxYw#t$rYywxYw)aCheck for UTF-32 encoding based on 4-byte unit structure. For valid Unicode (U+0000 to U+10FFFF = 0x0010FFFF): - UTF-32-BE: the first byte of each 4-byte unit is always 0x00 - UTF-32-LE: the last byte of each 4-byte unit is always 0x00 For BMP characters (U+0000 to U+FFFF), additionally: - UTF-32-BE: the second byte is also 0x00 - UTF-32-LE: the third byte is also 0x00 Nc34K|]}|dk(s dywrN.0ir s r z_check_utf32..{JaT!W\J rc3:K|]}|dzdk(sdyw)r!rNr"r#s rr&z_check_utf32..}s Oqd1q5kQ>NOsrz utf-32-beencoding confidencelanguagec34K|]}|dk(s dywr r"r#s rr&z_check_utf32..sIQDGqLqIr(c34K|]}|dk(s dywr r"r#s rr&z_check_utf32..r'r(z utf-32-le) r_MIN_BYTES_UTF32sumrangedecode_looks_like_textrrUnicodeDecodeError)r trimmed_len num_units be_first_nullbe_second_nulltext le_last_null le_third_nulls` rrrgsxd)s4y1}-K%%   Dq IJ5CIq#9JJMOE!SY$:OON !ny&@3&F ;;{+D%&(7!&I%3t9a"8IILJ5CIq#9JJMy ]Y%>%D ;;{+D%&(7!& )"   ""    s$-E -E EE E'&E'cttt}||dzz}|tkry|dz}t fdt d|dD}t fdt d|dD}||z }||z }|t k\xrtd|| }|t k\xrtd|| }|s|syd||fd||fg} ||kDr| jd} d } d } d} d }| D]U\}}} d|j|}|rd }t|s,| dz } t|}|| | dkDrtnd zkDsP|} |} |} W| y| r| dk(xs | tk\}n |xr | tk\}|rt| t d Sy#t$rYwxYw)aCheck for UTF-16 via null-byte patterns in alternating positions. UTF-16 encodes each BMP character as two bytes. For characters whose code-point high byte is 0x00 (Latin, digits, basic punctuation, many control structures), one of the two bytes in each unit will be a null. Even for non-Latin scripts (Arabic, CJK, Cyrillic, etc.) a significant fraction of code units still contain at least one null byte. Non-UTF-16 single-byte encodings never contain null bytes, so even a small null-byte fraction in alternating positions is a strong signal. The null pattern only establishes that the data is UTF-16-like. Byte order is always chosen by decoding both ways and comparing text-quality scores, with the null signal breaking near-ties (see below). r1Nc34K|]}|dk(s dywr r"r#s rr&z_check_utf16..Kad1glKr(rc34K|]}|dk(s dywr r"r#s rr&z_check_utf16..rAr(r!z utf-16-lez utf-16-begFTgr*)minrrrr3r4_UTF16_MIN_NULL_FRACTIONrreverser5r7r6 _text_quality_QUALITY_TIE_MARGIN_MIN_TEXT_QUALITYrr)r sample_lenr9 be_null_count le_null_countbe_fracle_frac le_qualified be_qualifiedsides best_encoding best_qualitybest_qualifiedviablequalified_side_decodedr+_frac qualifiedr<qualityaccepteds` rrrs  SY -J*q. J$$aIK5J#:KKMK5J#:KKMi'Gi'G ++ G*4 +O.O"/  I"   s(E00 E<;E<r<c`|sy|dd}td|D}|t|z tkDS)z9Quick check: is decoded text mostly printable characters.FNc3JK|]}|js|dvsdyw) r!N) isprintable)r$cs rr&z#_looks_like_text.. sJ!AMAJs##)r3r_MIN_PRINTABLE_FRACTION)r<r printables rr6r6s8  $3ZFJvJJI s6{ "%< <10% control characters or >20% combining marks (category ``M*``). NrgLr!MZsr]Cg?g?r)r unicodedatacategoryord) r<rbrnlettersmarksspacescontrols ascii_lettersr_catscores rrFrFs(&5\F F AAvG E FHM  ""1% q6S= qLG1v|" Vs] QJE D[AM aKF Vs] MH !|c qy3 aKE ma3 &&E2v&1*   Lr)r[)__doc__rjchardet.pipelinerrrrr2rrDrHrGr`rr bytes__annotations__floatboolrrrrstrr6intrFr"rrr}s XX   $")+;!;;=U=u==( % Od,B ,5u54!75pdud4!7dN=3=4=99C9%9r