In which I revisit one of my favorite bugs, the invalid surrogate pair.

The modern answer
If you're doing string manipulation in JavaScript and you care about not corrupting characters, use Intl.Segmenter:

const seg = new Intl.Segmenter(undefined, { granularity: "grapheme" });
const segments = [...seg.segment("👩‍🚀A👍")].map((s) => s.segment);
// → ['👩‍🚀', 'A', '👍']
This splits by grapheme clusters rather than code units. No orphaned surrogates, no split emoji. It's what .slice() should have been doing all along, but of course UTF-16 predates emoji by decades.

Once you know about it, you start seeing it in the wild. Any code that does str.slice(0, 1) or str[0] to get "the first character" is potentially broken.

Where IO::Handle Description I did not find how to open file with :utf8 flag.

https://perldoc.perl.org/Encode#UTF-8-vs.-utf8-vs.-UTF8

https://www.effectiveperlprogramming.com/2011/08/know-the-difference-between-utf8-and-utf-8/

Tom's Unicode Scripts So Life is Easier

local $Data::Dumper::Useqq = 1;
print Data::Dumper::Dumper( $_ );

$VAR1 = "\30\31\32\e\34\35\36\37 !\"#\$%&'";
$VAR1 = "\30\31\32\e\34\35\36\37 !\"#\$%&\x{100}";

Ever wonder about that mysterious Content-Type tag? You know, the one you’re supposed to put in HTML and you never quite know what it should be? Did you ever get an email from your friends in…

I wonder why most modern solutions built using Perl don't enable UTF-8 by default.

Saying that “Perl should [somehow!] enable Unicode by default” doesn’t even start to begin to think about getting around to saying enough to be even marginally useful in some sort of rare and isolated case. Unicode is much much more than just a larger character repertoire; it’s also how those characters all interact in many, many ways.

uni queries the Unicode database from the commandline.

There are four commands: identify codepoints in a string, search for codepoints, print codepoints by class, block, or range, and emoji to find emojis.

There’s more to spaces than the key you instinctively hit between words with one of your thumbs. Let’s find out what other space characters there are, what their heritage is, and how they can be useful today.

Perl versions >5.8 automatically tag text as UTF-8 whenever the interpreter knows for certain that it is UTF-8 encoded. Examples of this include using chr() with a UTF-8 code-point (>255), a string containing the \x{code-point} syntax, data from files opened with a :utf8 encoding layer, or any data explicitly tagged as UTF-8 encoding using (for example) utf8::decode().

invalid