Python 3.15 goes global with UTF-8 support

Decades of cross-platform character corruption finally comes to an end for Windows coders

After a week’s delay, the new version of Python, v3.15, has been released. With it comes a proper profiler, an experimental just-in-time compiler, and a tightening of dictionary structures.  It also offers a time-saving Lazy Import option for programs to not load modules until needed, the implementation of which ironically halted this release until a few last-minute regressions were sorted.

But of all these new delicacies spread before the Pythonistas, the data wranglers of the bunch may most appreciate default UTF-8 support across all platforms. And when we say “all platforms,” we’re really just throwing shade at Windows (as usual). Python has defaulted to macOS and Linux Python environments for some time now. 

It’s a small but important update. About half of all Python developers use the language for processing and analyzing data, and about 58 percent of Python users build on Windows, according to various Python Software Foundation surveys.

REG AD

Party crashers

REG AD

Earlier versions of Python defaulted to the host system's locale encoding. This led to much consternation for Windows developers whose scripts opened UTF-8-encoded data sets, only to have non-ASCII characters corrupt or crash the program. Culprits could be as innocuous as a smart quote or a fancy apostrophe.

Managed by The Unicode Consortium, with the help of the IETF and ISO, UTF-8 is an 8-bit binary encoding of the Unicode catalog of all the world’s written symbols. ASCII offered only 128 characters to cover the world’s knowledge, but UTF-8 provides over 1.1 million slots, which is plenty of room for smart quotes, emojis, musical and math notations, all the Chinese dialects, and even a few extinct languages (for research purposes).

There’s also UTF-16 and UTF-32, but UTF-8’s combination of efficiency and inclusiveness made it the default for the Web and most modern operating systems.

By the time Python Enhancement Proposal 681 was proposed in 2022 to make UTF-8 the preferred encoding, the language was already behind the times. That year, Java settled into UTF-8 with JDK 18, and Ruby 3.0 defaulted to the encoding scheme in 2020.   

Today all the cool languages and runtimes pledge allegiance to UTF-8, including Rust, Go, and Node.js (JavaScript itself prefers UTF-16 for internal encoding). Most code editors, from Visual Studio Code to Windows Notepad, prefer UTF-8.  Data sharing and configuration formats such as JSON, TOML, and YAML all require 8-bit encodings.

Python had long understood UTF-8, but programmers had to put a flag in their code alerting the language to use it – if they knew they had to do so at all. Many didn’t and were left scratching their heads over why their data processing scripts worked just fine on MacOS or Linux but then would flame out on Windows (or some other obscure system that defaulted to a C/POSIX locale).

Programmers love puzzles

To acquaint developers with the new aspects of the language, longtime Python programmer Barry Warsaw (AKA “Uncle Barry”) created a terminal-based adventure game called “whatsnewt” that offers 18 puzzles, each one built on a new feature found in Python 3.15, including a puzzle for UTF-8 support, “The Encoding Tower of Babel.”

REG AD

Once you've downloaded Python 3.15, do a “pip install whatsnewt” and then run the game on the Python command line. It even works on Windows, we hear. ®