Update dependency chardet to v7 - #133
Open
renovate[bot] wants to merge 1 commit into
Open
Conversation
Contributor
Author
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
>=5,<7→>=7,<8Release Notes
chardet/chardet (chardet)
v7.4.3Compare Source
Patch release: fixes a crash when input contains null bytes inside a
<meta charset>declaration.Bug Fixes
ValueError: embedded null charactercrash when input contained a<meta charset>declaration with a null byte in the encoding name (e.g.b'<meta charset="\x00utf-8">').codecs.lookup()raisesValueErroron embedded nulls, andlookup_encoding()was only catchingLookupError. Also added defensiveValueErrorcatches in_validate_bytes()and_to_utf8()for completeness. (#369, thanks @DRMacIver for the report)Full Changelog: chardet/chardet@7.4.2...7.4.3
v7.4.2Compare Source
Patch release: fixes a crash on short inputs and closes a bunch of WHATWG/IANA alias gaps.
Bug Fixes
RuntimeError: pipeline must always return at least one resulton ~2% of all possible two-byte inputs (e.g.b"\xf9\x92"). Multi-byte encodings like CP932 and Johab could score above the structural confidence threshold on very short inputs, but then statistical scoring would return nothing, leaving an empty result list instead of falling through to the fallback. (#367, #368, thanks @jasonwbarnett)Improvements
<meta charset>labels likex-cp1252,x-sjis,dos-874,csUTF8, and thecswindows*family all resolve correctly through the markup detection stage. Every alias was driven by a failing spec-compliance test, not speculative. (#366)Full Changelog: chardet/chardet@7.4.1...7.4.2
v7.4.1Compare Source
Bug Fixes
utf-16/utf-32instead ofutf-16-le/utf-16-be/utf-32-le/utf-32-be. The endian-specific codecs don't strip the BOM on decode, so callers were getting a stray U+FEFF at the start of their text. BOM-less detection is unchanged. (#364, #365)Full Changelog: chardet/chardet@7.4.0...7.4.1
v7.4.0.post2Compare Source
v7.4.0.post1Compare Source
v7.3.0Compare Source
License
Features
mime_typefield to detection results — identifies file types for both binary (via magic number matching) and text content. Returned in alldetect(),detect_all(), andUniversalDetectorresults. (#350)pipeline/magic.pymodule detects 40+ binary file formats including images, audio/video, archives, documents, executables, and fonts. ZIP-based formats (XLSX, DOCX, JAR, APK, EPUB, wheel, OpenDocument) are distinguished by entry filenames. (#350)Bug Fixes
Performance
dataclasses.replace()with directDetectionResultconstruction on hot paths, eliminating ~354k function calls per full test suite runBuild
v7.2.0: chardet 7.2.0Compare Source
Features
include_encodingsandexclude_encodingsparameters todetect(),detect_all(), andUniversalDetector— restrict or exclude specific encodings from the candidate set, with corresponding-i/--include-encodingsand-x/--exclude-encodingsCLI flags (#343)no_match_encoding(default"cp1252") andempty_input_encoding(default"utf-8") parameters — control which encoding is returned when no candidate survives the pipeline or the input is empty, with corresponding CLI flags (#343)-l/--languageflag tochardetectCLI — shows the detected language (ISO 639-1 code and English name) alongside the encoding (#342)Fixes
Full changelog: https://chardet.readthedocs.io/en/latest/changelog.html
v7.1.0: chardet 7.1.0Compare Source
Features
# -*- coding: ... -*-and# coding=...declarations on lines 1–2 of Python source files are now recognized with confidence 0.95 (#249)chardet.universaldetectorbackward-compatibility stub so thatfrom chardet.universaldetector import UniversalDetectorworks with a deprecation warning (#341)Fixes
++or+wordpatterns (#332)detect()call — model norms are now computed during loading instead of lazily iterating 21M entries (#333)detect()now returns chardet 5.x-compatible names by default (#338)iter_unpackyielded fewer tuples instead of raising)Performance
load_models()struct.iter_unpackfor bulk entry extraction (eliminates ~305K individualunpackcalls)New API parameters
compat_namesparameter (defaultTrue) todetect(),detect_all(), andUniversalDetector— set toFalseto get raw Python codec names instead of chardet 5.x/6.x compatible display namesprefer_supersetparameter (defaultFalse) — remaps legacy ISO/subset encodings to their modern Windows/CP superset equivalents (e.g., ASCII → Windows-1252, ISO-8859-1 → Windows-1252). This will default toTruein the next major version (8.0).should_rename_legacyin favor ofprefer_superset— a deprecation warning is emitted when usedImprovements
"utf-8"instead of"UTF-8"), withcompat_namescontrolling the public output formatlookup_encoding()toregistryfor case-insensitive resolution of arbitrary encoding name input to canonical namesFull changelog: https://chardet.readthedocs.io/en/latest/changelog.html
v7.0.1Compare Source
Fixes
+4bafdea3...) were misdetected as UTF-7, breaking tools like tox_SINGLE_LANG_MAPmissing aliases for single-language encoding lookup (e.g.,big5→big5hkscs)TypeErrorin UTF-7 codec handlingImprovements
New Contributors
v7.0.0Compare Source
Ground-up, MIT-licensed rewrite of chardet. Same package name, same public API — drop-in replacement for chardet 5.x/6.x. Just way faster and more accurate!
Highlights:
detect()anddetect_all()with no measurable overhead; scales on free-threaded Python 3.13t+Breaking changes vs 6.0.0:
detect()anddetect_all()now default toencoding_era=EncodingEra.ALL(6.0.0 defaulted toMODERN_WEB)LanguageFilteris accepted but ignored (deprecation warning emitted)chunk_sizeis accepted but ignored (deprecation warning emitted)Configuration
📅 Schedule: (UTC)
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.