What's new in Perl 5.8.0

Jan 16, 2003 by Artur Bergman

It’s been nearly six months since the release of Perl 5.8.0, but many people still haven’t upgraded to it. We’ll take a look at some of the new features it provides and describe why you should investigate them yourself.

Unicode

Perl 5.8 - at last! - properly supports Unicode. Handling Unicode data in Perl should now be much more reliable than in 5.6.0 and 5.6.1. In fact, quoting the most excellent perluniintro, which is suggested reading, ‘5.8 is the first recommended release for serious Unicode work.’

The Unicode Character Database, which ships with Perl, has been upgraded to Unicode 3.2.0, 5.6.1 had 3.0.1. Most UCD files are included with some omissions for space considerations.

Perl’s Unicode model is straightforward: Strings can be eight-bit native bytes or strings of Unicode characters. The principle is that Perl tries to keep its data as eight-bit bytes for as long as possible. When Unicodeness cannot be avoided, the data is transparently upgraded to Unicode. Native eight-bit bytes are whatever the platform users (for example Latin-1), Unicode is typically stored as UTF-8.

Perl will do the right thing with regard to mixing Unicode and non-Unicode strings; all functions and operators will respect the UTF-8 flag. For example, it is now possible to use Unicode strings in hashes, and correctly use them in regular expressions and transliterations. This has fully changed from 5.6 where you controlled Unicode support with a lexically scoped utf8 pragma.

To fully use Unicode in Perl, we must now compile Perl with perlio – the new IO system written by Nick Ing-Simmons that we will cover later – together with the new Encode module written by Dan Kogai. Together, these allow individual filehandles to be set to bytes, Unicode or legacy encodings. Encode also comes with piconv, which is a Perl implementation of iconv and enc2xs, which allows you to create your own encodings to Encode, either from Unicode Character Mapping files or from Tcl Encoding Files. From Sadahiro Tomoyuki comes Unicode::Normalize and Unicode::Collate, surprisingly used for normalization and collating.

Perl Threads

Just like 5.8 is the first recommended release for Unicode work, it is also the first recommended release for threading work. Starting with 5.6, Perl had two modes of threading: one style called 5005threading, mainly because it was introduced with 5.005, and ithreads, which is short for interpreter threads. Gurusamy Sarathy introduced ithreads as a step forward from multiplicity to support the psuedofork implementation on Win32. However, in 5.6, there was no way to control these threads from Perl; this has now changed with the introduction of two new modules in 5.8.

The basic rule for this thread model is that all data is cloned when a new thread is created, so no data is shared between threads. If one wants to share data, then there is a threads::shared module and the new : shared; variable attribute. Controlling the new threads is done by using the threads module. More reading can be found in the respective modules and the Perl Thread Tutorial page

.

New IO

Perl can now rely on its own bugs instead of the bugs of your underlying IO implementation! In Perl 5.8, we are now using the PerlIO library, which replaces both stdio and sfio. The new IO system allows filters to be pushed/popped from a filehandle for doing all kinds of nifty things. For example, the Encode module, as mentioned earlier in the Unicode discussion, uses PerlIO to do the magic character set conversions at the IO level.

Interested parties that want to create their own layers should look at the library API, the IO layer API, the PerlIO module, and the PerlIO::via manpage.

Safe Signals

No more random segfaults caused by signals! We now have a signal handler that just raises a flag and then dispatches the signal between opcodes so you are free to do anything you feel like in a signal handler (Since it isn’t run at async time, it isn’t really a signal handler). This has potential for conflicts if you are embedding Perl and relying on signals to do some particular behavior, but I suppose if you really like having the chance of a random segfault on receiving a signal, then you can always compile perl with PERL_OLD_SIGNALS. This will, however, not be threadsafe.

New and Improved Modules

Perl 5.8 comes with 54 new modules, many of them are included of CPAN for various reasons. One goal has been to make it easy for CPAN.pm to be selfhosting; this has meant including libnet and a couple of other modules.

We have been working on testing a lot so the Test::More family of modules were natural to include. Then there was a push to make Perl more i18n friendly, so 5.8.0 includes several i18n and l10n modules as well as the previously covered Unicode modules. There many modules that provide access to internal functions like the PerlIO modules, threads module and sort, the new module that provides a interface to the sort implementation you are using. Finally, we also thought it was time to include Storable in the core.

We also have a bunch of updated modules included: Cwd is now implemented in XS, giving us a nice speed boost. B::Deparse has been improved to the point that it is actually useful. Maintenance work on ExtUtils::MakeMaker has made it more stable. Storable supports Unicode hash keys and restricted hashes. Math::BigInt and Math::BigFloat have been upgraded and bugfixed quite a lot, and they have been complemented by a Math::BigRat module, and the bigrat, bigint and bignum pragmata for lexical control of transparent bignumber support.

Speed Improvements

Even if this release includes a lot of new features, there are some optimizations in there as well! We have changed sort to use mergesort, which for me is rather surprising since I have been told since I was a toddler to use quicksort. However, the old behavior can be controlled using the sort module; we even have a mystery stable quicksort!

Once again, we have changed the hashing algorithm to something called One-At-A-Time, so all of you who depend on the order of hashes, this is a good reason to fix your programs now!

Finally, map has been made faster, as has unshift.

Testing

We hope this should be the most stable release of Perl to date, as an extensive QA effort has been spearheaded by Michael Schwern that has led to several benefits. We now have six times the amount of test cases, testing a cleaner codebase with more documentation. The Perl Bug database has been switched to Request Tracker; we should thank Richard Foley for his work on perlbugtron, which has now been retired. After several discussions on what a memory leak is, several memory leaks and naughty accesses have been fixed. Tools used have been third degree, purify, and the most excellent open-source alternative, valgrind.

More Numbers

Nicholas Clark, Hugo van der Sanden and Tels have done some magic keeping integers as integers as long as possible, and when finding bugs in vendors number-to-string and string-to-number they have coded around these to increased precision. We should all be happy that 42 is now 42 and not 42.000000000000001 - imagine what the aliens would do if they found out!

Documentation

I have mentioned several documentations pages earlier, they are part of the 13 new POD files included in Perl; in addition to this, all README.os files have been translated into pod. Interestingly, there are several new tutorials, including a regular expressions tutorial, a tutorial on pack and unpack, a debugging tutorial, a module creation tutorial, and ‘a gentle introduction to perl’. There is also a new POD format specification written by Sean M. Burke.

Deprecations

Several deprecations have occurred in this release of Perl. In future versions of Perl, 5005threads will be gone and replaced by ithreads. Pseudo-hashes will be killed but the fields pragma will work using restricted hashes; suidperl, which, despite everything, isn’t safe and the bare package; directive, which had unclear semantics.

A few things have been removed and forbidden: blessing a refence into another ref is one; self-tying of arrays and hashes led to some weird bugs and have been disabled, as they touched some rarely tested codepaths. The [[.c.]] and [[=c=]] character classes are also forbidden because they might be used for future extensions. Several scripts that were outdated have been removed and the upper case comparison operators have also got the ax.

The War of the Platforms

Perl 5.8 works on several new platforms and the EBDIC platforms were regained. However, sadly we lost Amiga; so any volunteers that want to make the Amiga port work again are very welcome.

Odd and Ends

There is a long list of new small changes in Perl 5.8, the biggest of these small changes are restricted hashes, which can be used from the new Hash::Util module and allows you to lock down the keys in a specific hash; this will possibly be used as a replacement for pseudohashes for the fields pragma.

For the full and gory details, check out the whole Perl delta documentation.