eiffelroom - Unicode - Comments

Relevance

colin-adams — Sun, 01 Apr 2007 10:29:33 -0700

The main benefit for homogeneous clusters is simpler heuristics - there is no possibility of confusing Latin-1 with UTF-8.

If you can't eliminate the possibility of one or another, I know of know way to disambiguate them, in general.

Of course, there are lots of cases where it is easier to see which of the two is meant. But in other cases, not.

So, starting from the case where the file is pure ASCII. Did the author intend it to be treated as Latin-1 or UTF-8?

In this case, it doesn't matter (the only possibility is the type of manifest string constants, but these are defined to be of type STRING).

But all we have to do is to mutate one character in a string literal, and immediately (if we choose the mutation carefully), the case becomes undecidable.

Colin Adams

Is this relevant?

manus_eiffel — Fri, 30 Mar 2007 10:12:00 -0700

The source code will be using plain text file (i.e. sequence of character codes that are between 0 and 255), UTF-8 or any other Unicode encoding. Once you have the encoding then the semantics is properly defined.

Of course if one library author is using Unicode characters beyond 255, the user of that library will be forced to use a Unicode encoding for his source code, but is this relevant to the project specification? I don't think so.

Heuristics

colin-adams — Fri, 30 Mar 2007 10:07:50 -0700

See also Heuristics for detecting class text encoding. Colin Adams

I believe it does not matter

manus_eiffel — Mon, 19 Mar 2007 14:17:00 -0700

I believe it does not matter whether or not those routines are in STRING_GENERAL. It might be better to have them outside, possibly that you may want to serialize the data in something else than a string and to reduce code duplication it makes more sense outside.

For {STRING_32}.out, I don't think it is a major issue. The default implementation of `out' is compiler defined at the moment to be STRING. In the future we may want to change this to be STRING_32, but for the time being, being a truncated version of the STRING_32 representation is fine to me since `out' has different semantics depending on the Eiffel class. In my opinion using `out' for encoding would be really wrong.

Serializing

colin-adams — Mon, 19 Mar 2007 07:42:04 -0700

I agree that the encoding form doesn't matter, and that the compiler should be free to choose whichever it likes (and UTF-32 would be my choice too).

But for serializing, STRING_GENERAL should have the following routines (bodies omitted):

to_utf8: !STRING_8 is

      -- Serialization of `Current' as bytes of UTF-8 representation.

   do

   ensure

      not_shorter: Result.count >= count

   end

to_utf_16_be:  !STRING_8 is

      -- Serialization of `Current' as bytes of UTF-16BE representation.

   do

   ensure

      not_shorter: Result.count >= 2 * count

   end

to_utf_16_le:  !STRING_8 is

      -- Serialization of `Current' as bytes of UTF-16LE representation.

   do

   ensure

      not_shorter: Result.count >= 2 * count

   end

to_utf_32_be:  !STRING_8 is

      -- Serialization of `Current' as bytes of UTF-32BE representation.

   do

   ensure

      four_times_longer: Result.count = 4 * count

   end

to_utf_32_le:  !STRING_8 is

      -- Serialization of `Current' as bytes of UTF-32LE representation.

   do

   ensure

      four_times_longer: Result.count = 4 * count

   end

The question remains what {STRING_32}.out should produce. Perhaps it should be platform-specific (and may be finer grained than just Windows v. POSIX - different Windows configurations may have different natural defaults - I'm not sure about this).

Colin Adams

The form doesn't matter

manus_eiffel — Sun, 18 Mar 2007 16:56:00 -0700

The form doesn't matter since it is most likely hidden from the user point of view (the user manipulate a sequence of characters and nothing more). Nevertheless, it might be better if it is kept 32-bit since it is faster.

Regarding the encoding scheme, it cannot be set on the application level since many libraries might choose a different encoding, or you might have a need to read different encoding. So it has to be configurable and this should be outside the STRING class.

Unicode encoding

colin-adams — Sun, 18 Mar 2007 01:30:00 -0700

There are two Unicode encodings to be considered - the Unicode Encoding Form, and the Unicode Encoding Scheme.

The former is one of UTF-8, UTF-16, and UTF-32. This is what is used internally within the program. It is a classic time/space trade-off. ISE use UTF-32, which waste memory to speed computing time. Either UTF-16 or UTF-8 would be slower, but UTF-16 is rarely significantly slower. I don't see any linguistic cultural differences affecting the issue.

The Unicode Encoding Schemes are byte serializations of the encoding forms. The full list is UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, and UTF-32LE. The natural default tends to map to your computer hardware + O/S, except there is a disk-space consideration here: The UTF-32* use more disk space. Which is the most economical DOES depend upon the linguistic culture - in Europe UTF-8 is cheapest, in East Asia, UTF-16* are cheapest. I don't know what {STRING_32}.out produces with ISE 5.7.

So there are two possible sets of set_unicode_encoding features.

Colin Adams

STRING_8/STRING_32

thomas.beale — Sat, 17 Mar 2007 19:15:33 -0700

I would also really like to know why we should have STRING_8 etc. It makes no sense to me at all. Currently the compiler is doing tricks to convert all STRINGs in our code to STRING_8, but certain things like checking generated_type.is_equal("STRING") break; anywhere where dynamic_type() from INTERNAL is used with STRING objects might or might not work. And I don't see why issues of UTF encoding (not the same as unicode per se) should be exposed at a developer-visible level.

What we need is to be able to say at the beginning of an application set_unicode_encoding_utf8 or set_unicode_encoding_utf16 and everything just works. The default should be whichever makes sense in your linguistic culture (UTF-8 in all european languages).

- thomas

Real issue

manus_eiffel — Sat, 17 Mar 2007 09:27:28 -0700

What I meant is that it is not an issue of encoding or character set, but an issue with memory representation of Eiffel strings. Indeed the existing legacy C code wrapping are using this directly. So changing STRING to support unicode will break those programs. This is one of the reason why EiffelBase introduced C_STRING so that it works regardless of the memory representation of Eiffel strings.

My solution

colin-adams — Sat, 17 Mar 2007 05:25:18 -0700

Well, two queries actually.

maximum_code: INTEGER is

      -- Maximum value of `code' permitted by `character_set'

   do

      -- 255 for ISO-8859-x, 1114111 for Unicode

      -- compiler can optimize this as a builtin query

      -- in the case that only 1 character set is used in

      -- a system. Otherwise it would be an attribute

   ensure

      positive_code: Result > 0

   end

and

character_set: STRING is

      -- Name of character set used in `Current'

   do

       -- same considerations as for `maximum_code'

   ensure

     result_not_empty: Result /= Void and then not Result.is_empty

     result_is_ascii: -- whatever 

   end

With these two queries, all incompatibilities can be coped with (including your c-string stuff - the compiler can include transcoding when necessary).

Note that the latter query was also needed for ETL2, for supporting multiple or alternate encodings, such as ISO-8859-2.

Solution?

manus_eiffel — Sat, 17 Mar 2007 00:14:56 -0700

The issue is that you had legacy code wrapping C interfaces required 8-bit strings and having Unicode strings would have broken those API. Therefore the separation was and is still needed.

What do you mean by a query?

Multiple string types

colin-adams — Fri, 16 Mar 2007 23:24:55 -0700

I don't think there was a backwards-compatibility problem - at least, not one that needed the STRING_*/STRING_32 separation as a solution. A much simpler fix of adding a query would have done the trick.

Flip of a coin

manus_eiffel — Fri, 16 Mar 2007 22:19:00 -0700

That would be soon possible when we have converted all our legacy code that only handle STRING_8 will be adapted to work with STRING_32 as well.

Limitations - Yes, UTF-8

peter_gummer — Fri, 16 Mar 2007 15:41:35 -0700

Yes, the assumption here is that the strings are all UTF-8. (I tried to make that clear, especially in the title). This assumption is sound for our purposes.

For this reason, the official EiffelStudio version of SYSTEM_STRING_FACTORY probably should not adopt my "fix". Other good reasons for EiffelStudio to come up with a better fix than this are that my implementation is inefficient, and it would create a dependency of the base library on the gobo library. I don't mind my own project having a dependency on Gobo - the project already uses Gobo - but this is not ok in general.

I agree with your idea of generating STRING_32 when reading the data. I was thinking along those lines when I attempted (unsuccessfully) to map STRING to STRING_32. I modified my project's config to use base as a cluster rather than a library; then I copied all of the mappings from base.ecf to my config, editing STRING to map it to STRING_32 rather than STRING_8. But I quickly abandoned that route, because it wouldn't even compile. Some line in a library (something like true_string: STRING is "True") couldn't convert a STRING_8 to a STRING_32. I could have left the mapping alone, I suppose, and done a global search and replace of STRING with STRING_32 in our code; but that is very invasive, and I'd be surprised if it worked given that the libraries we call would still be producing STRING_8 objects.

I really don't like this STRING_8 / STRING_32 idea. I programmed in C# for two years, developing an internationalised application, and I was barely conscious of the fact that my strings and characters were Unicode. It just worked. I acknowledge that Eiffel is contending with a backward-compatibility problem here, but I would be much happier if I could just flip a switch in the config file so that all of my STRING objects instantly became Unicode.

Limitations

manus_eiffel — Fri, 16 Mar 2007 08:37:37 -0700

As far as I can tell, this implies that all your Eiffel strings are UTF-8, as otherwise it might not work for characters that are above 128. But if you get your data from UTF-8, wouldn't it be better to generate STRING_32 instead when reading the data. Once done, the STRING_32 would convert nicely with .NET System.String.