Re: Unicode-based FreeBSD

Tim Kientzle wrote:
>> Going to UTF-8 might fix some of the character issues
>> but we would be in the same shoes when it comes to characters
>> which are in -16 and -32 but not in -8.
>
> You need to read the Unicode/ISO10646 standards again;
> you do not understand them.
You are right, I do not understand them. As I mentioned, I am not a 
Unicode expert and I have never claimed to be one.
>
> There are no characters in UTF-32 that are not in UTF-8.
>
> UTF-32, UTF-16, and UTF-8 all use exactly the same characters.
>
> UTF-8 encodes Unicode characters from U+000000 to U+10FFFF, using 1 to 
> 4 bytes per character.
>
> UTF-16 encodes Unicode characters from U+000000 to U+10FFFF, using 2 
> to 4 bytes per character.
>
> UTF-32 encodes Unicode characters from U+000000 to U+10FFFF, using 4 
> bytes per character.
>
> Practically speaking, UTF-8 is a bit more convenient for file
> storage and transmission (including terminal support), UTF-16
> or UTF-32 can be slightly more convenient for internal
> string manipulation.  But all three encodings use exactly
> the same characters.
>
> Tim Kientzle
I cannot confirm you are 100% right because I am not an expert in 
Unicode. However, after some reading, I can see there is no "character 
loss" by using one form of Unicode than the other. Therefore, I stand 
corrected on that issue. I still think there should be support for 
UTF-16 and UTF-32 in FreeBSD in general but it is outside the scope of 
the topic (Unicode in syscons).

Tz-Huan Huang wrote:
> How do you define ``support''?
>
> If you mean software-level support, vim supports UTF-16, firefox
> supports UTF-16/UTF-32, perl supports UTF-16/UTF-32, etc.
>
> If you mean system-level support, there are two cases:
>
> 1. The system internal text representation is still in UTF-8, just add
> UTF-16/32
> support for terminal, stdin/stdout/stderr, etc. I think it's not so
> hard (I might be
> wrong because I don't know terminal at all) but I don't see any reason to set
> locale to UTF-16 or UTF-32.
>
> 2. The system internal text representation is changed to UTF-16 or UTF-32.
> This is another story and I have no comment on it.
>
>   
By support I meant full handling of Unicode characters which meant both 
1 and 2. Although, in connection to my discovery above, I think it is 
better if the internal handling is (continued to be) done in UTF-8.


Með kveðju / With regards,
Svavar Kjarrval (svavar_at_kjarrval.is)
s. 863-9900