Trond, dette høres ut som noe du har peiling på. Jeg fant meldingene på news, gnu.announce.
De bør kanskje få hjelp til å forbedre støtten for de norske og samiske språkene?
Subject: Language Info Needed for GNU Aspell Newsgroups: gnu.announce To: info-gnu@gnu.org Date: Tue, 23 Mar 2004 15:44:54 -0500 (EST) Reply-To: kevina@gnu.org
[Please distribute this document as widely as possible.]
GNU Aspell 0.60 should be able to support most of the Word Languages. This includes languages languages written in Arabic and other scripts not well supported by an existing 8-bit character set. Eventually Aspell should be able to support any current language not based on the Chinese writing system.
GNU Aspell is a spell checker designed to eventually replace Ispell. Its main feature is that it does a much better job of coming up with possible suggestions than just about any other spell checker out there for the English language, including Ispell and Microsoft Word. However, starting with Aspell 0.60 is should also be the only Free (as in Freedom) that can support most languages not written in the Latin or Cyrillic scripts.
However I, the author of Aspell, know very little about foreign languages (ie non-English) and what it takes to correctly spell check them. Thus, I need other people to educate me.
If you speak a foreign language I would appreciate if you would take the time too look over the following material and email me with any additional information you may have.
The first part gives a thorough analysis of the languages which Aspell can and cannot support. If you find any of this information is incorrect please inform me at kevina@gnu.org.
When Aspell 0.60 is released I would like to have dictionaries available for as many languages as possible.
Therefore, if you know of a Free word list available for a language that is not currently listed as having a dictionary available I would appreciate hearing form you. I am especially interested in working with someone to add support for languages written in the Arabic script. The encoding of the Arabic is quite complicated and I want to be sure that Aspell can correctly handle it.
I would also appreciate some help converting Ispell dictionaries to Aspell. So, if you would like to help convert some of the dictionaries listed as being available for Ispell please contact me.
The second part lists languages related issues involved in correctly spell checking a document. If you can offer any additional insight on any of the issues discussed, or know of any additional complications when spell checking a given language, I would appreciate hearing from you.
The last part discusses why Aspell uses 8-bit characters internally for your reading pleasure.
All of this material is also included in the Aspell 0.60 manual which you can find at http://aspell.net/devel-doc/man.
Languages Which Aspell can Support **********************************
Even though Aspell will remain 8-bit internally it should still be be able to support any written languages not based on a logographic script. The only logographic writing system in current use are those based on hànzi which includes Chinese, Japanese, and sometimes Korean.
Supported =========
Aspell 0.60 should be able to support the following languages as, to the best of my knowledge, they all contain 220 or fewer symbols and can thus, fit within an 8-bit character set. If an existing character set does not exists than a new one can be invented. This is true even if the script is not yet supported by Unicode as the private use area can be used.
Code Language Name Script Dictionary Gettext Available Translation
aa Afar Latin - - ab Abkhazian Cyrillic - - ae Avestan Avestan - - af Afrikaans Latin Yes - ak Akan Latin - - an Aragonese Latin - - ar Arabic Arabic - - as Assamese Bengali - - av Avar Cyrillic - - ay Aymara Latin - - az Azerbaijani Cyrillic - - az Latin - -
ba Bashkir Cyrillic - - be Belarusian Cyrillic Planned Yes bg Bulgarian Cyrillic Yes - bh Bihari Devanagari - - bi Bislama Latin - - bm Bambara Latin - - bn Bengali Bengali Planned - bo Tibetan Tibetan - - br Breton Latin Yes - bs Bosnian Latin - -
ca Catalan/Valencian Latin Yes - ce Chechen Cyrillic - - ch Chamorro Latin - - co Corsican Latin - - cr Cree Latin - - cs Czech Latin Yes Yes cu Old Slavonic Cyrillic - - cv Chuvash Cyrillic - - cy Welsh Latin Yes -
da Danish Latin Yes - de German Latin Yes Yes dv Divehi Thaana - - dz Dzongkha Tibetan - -
ee Ewe Latin - - el Greek Greek Yes - en English Latin Yes Yes eo Esperanto Latin Yes - es Spanish Latin Yes Incomplete et Estonian Latin Planned - eu Basque Latin - -
fa Persian Arabic - - ff Fulah Latin - - fi Finnish Latin Planned - fj Fijian Latin - - fo Faroese Latin Yes - fr French Latin Yes Yes fy Frisian Latin - -
ga Irish Latin Yes Yes gd Scottish Gaelic Latin Planned - gl Gallegan Latin Yes - gn Guarani Latin - - gu Gujarati Gujarati - - gv Manx Latin Planned -
ha Hausa Latin - - he Hebrew Hebrew Planned - hi Hindi Devanagari - - ho Hiri Motu Latin - - hr Croatian Latin Yes - ht Haitian Creole Latin - - hu Hungarian Latin Planned - hy Armenian Armenian - - hz Herero Latin - -
ia Interlingua (IALA) Latin Yes - id Indonesian Latin Yes - ie Interlingue Latin - - ig Igbo Latin - - ik Inupiaq Latin - - io Ido Latin - - is Icelandic Latin Yes - it Italian Latin Yes - iu Inuktitut Latin - -
jv Javanese Javanese - - jv Latin - -
ka Georgian Georgian - - kg Kongo Latin - - ki Kikuyu/Gikuyu Latin - - kj Kwanyama Latin - - kk Kazakh Cyrillic - - kl Kalaallisut/Greenlandic Latin - - kn Kannada Kannada - - ko Korean Hangeul - - kr Kanuri Latin - - ks Kashmiri Arabic - - ks Devanagari - - ku Kurdish Arabic - - ku Cyrillic - - ku Latin - - kv Komi Cyrillic - - kw Cornish Latin - - ky Kirghiz Cyrillic - - ky Latin - -
la Latin Latin - - lb Luxembourgish Latin Planned - lg Ganda Latin - - li Limburgan Latin - - ln Lingala Latin - - lo Lao Lao - - lt Lithuanian Latin Planned - lu Luba-Katanga Latin - - lv Latvian Latin - -
mg Malagasy Latin - - mh Marshallese Latin - - mi Maori Latin Yes - mk Makasar Lontara/Makasar - - ml Malayalam Latin - - ml Malayalam - - mn Mongolian Cyrillic - - mn Mongolian - - mo Moldavian Cyrillic - - mr Marathi Devanagari - - ms Malay Latin Yes - mt Maltese Latin Planned - my Burmese Myanmar - -
na Nauruan Latin - - nb Norwegian Bokmal Latin Yes - nd North Ndebele Latin - - ne Nepali Devanagari - - ng Ndonga Latin - - nl Dutch Latin Yes Yes nn Norwegian Nynorsk Latin Yes - nr South Ndebele Latin - - nv Navajo Latin - - ny Nyanja Latin - -
oc Occitan/Provencal Latin - - or Oriya Oriya - - os Ossetic Cyrillic - -
pa Punjabi Gurmukhi - - pi Pali Devanagari - - pi Sinhala - - pl Polish Latin Yes - ps Pushto Arabic - - pt Portuguese Latin Yes Yes
qu Quechua Latin - -
rm Raeto-Romance Latin - - rn Rundi Latin - - ro Romanian Latin Yes Yes ru Russian Cyrillic Yes Yes rw Kinyarwanda Latin - -
sa Sanskrit Devanagari - - sc Sardinian Latin - - sd Sindhi Arabic - - se Northern Sami Latin - - sg Sango Latin - - si Sinhalese Sinhala - - sk Slovak Latin Yes - sl Slovenian Latin Yes - sm Samoan Latin - - sn Shona Latin - - so Somali Latin - - sq Albanian Latin Planned - sr Serbian Cyrillic - Yes sr Latin - - ss Swati Latin - - st Southern Sotho Latin - - su Sundanese Latin - - sv Swedish Latin Yes - sw Swahili Latin Planned -
ta Tamil Tamil Planned - te Telugu Telugu - - tg Tajik Latin - - tk Turkmen Latin - - tl Tagalog Latin - - tl Tagalog - - tn Tswana Latin - - to Tonga Latin - - tr Turkish Latin - - ts Tsonga Latin - - tt Tatar Cyrillic - - tw Twi Latin - - ty Tahitian Latin - -
ug Uighur Arabic - - ug Cyrillic - - ug Latin - - uk Ukrainian Cyrillic Yes - ur Urdu Arabic - - uz Uzbek Cyrillic - - uz Latin - -
ve Venda Latin - - vi Vietnamese Latin - - vo Volapuk Latin - -
wa Walloon Latin Planned Incomplete wo Wolof Latin - -
xh Xhosa Latin - -
yi Yiddish Hebrew - - yo Yoruba Latin - -
za Zhuang Latin - - zu Zulu Latin Planned -
Notes on Latin Languages ------------------------
Any word that can be written using on of the Latin ISO-8859 character sets (ISO-8859-1,2,3,4,9,10,13,14,15,16) can be written, in decomposed form, using the ASCII characters, the 23 additional letters:
U+00C6 LATIN CAPITAL LETTER AE U+00D0 LATIN CAPITAL LETTER ETH U+00D8 LATIN CAPITAL LETTER O WITH STROKE U+00DE LATIN CAPITAL LETTER THORN U+00DE LATIN SMALL LETTER THORN U+00DF LATIN SMALL LETTER SHARP S U+00E6 LATIN SMALL LETTER AE U+00F0 LATIN SMALL LETTER ETH U+00F8 LATIN SMALL LETTER O WITH STROKE U+0110 LATIN CAPITAL LETTER D WITH STROKE U+0111 LATIN SMALL LETTER D WITH STROKE U+0126 LATIN CAPITAL LETTER H WITH STROKE U+0127 LATIN SMALL LETTER H WITH STROKE U+0131 LATIN SMALL LETTER DOTLESS I U+0138 LATIN SMALL LETTER KRA U+0141 LATIN CAPITAL LETTER L WITH STROKE U+0142 LATIN SMALL LETTER L WITH STROKE U+014A LATIN CAPITAL LETTER ENG U+014B LATIN SMALL LETTER ENG U+0152 LATIN CAPITAL LIGATURE OE U+0153 LATIN SMALL LIGATURE OE U+0166 LATIN CAPITAL LETTER T WITH STROKE U+0167 LATIN SMALL LETTER T WITH STROKE
and the 14 modifiers:
U+0300 COMBINING GRAVE ACCENT U+0301 COMBINING ACUTE ACCENT U+0302 COMBINING CIRCUMFLEX ACCENT U+0303 COMBINING TILDE U+0304 COMBINING MACRON U+0306 COMBINING BREVE U+0307 COMBINING DOT ABOVE U+0308 COMBINING DIAERESIS U+030A COMBINING RING ABOVE U+030B COMBINING DOUBLE ACUTE ACCENT U+030C COMBINING CARON U+0326 COMBINING COMMA BELOW U+0327 COMBINING CEDILLA U+0328 COMBINING OGONEK
Which is a total of 37 additional Unicode code points.
All ISO-8859 character leaves the characters 0x00 - 0x19 and 0x80 - 0x99 unmapped as they are generally used as control characters. Of those, 0x02 - 0x19 and 0x80 - 0x99 may be mapped to anything in Aspell. This is a total of 62 characters which can be remapped in any ISO-8859 character set. Thus, by remapping 37 of the 62 characters to the previously specifed Unicode code-points, any modified ISO-8859 character set can be used for any Latin languages covered by ISO-8859. Of course decomposing every single accented character wastes a lot of space, so only characters that can be not be represented in the precomposed form should be broken up. By using this trick it is possible to store foreign words in the correctly accented form in the dictionary even if the precomposed character is not in the current character set.
Any letter in the Unicode range U+0000 - U+0249, U+1E00..U+1EFF (Basic Latin, Latin-1 Supplement, Latin Extended-A, Latin Extended-B, and Latin Extended Additional) can be represented using around 175 basic letters, and 25 modifiers which is less than 220 and can thus fit in an Aspell 8-bit character set. Since this unicode range covers any possible Latin language this special character set can be used to reperesnt any word written using the Latin script if so desired.
Hangeul -------
Koren in generally written in hangeul or a mixture of hanja and hangeul. Aspell should be able to spell check the hangeul part of the writing. In Hangeul letters individual letters, known as jamo, are grouped together in syllable blocks. Unicode provided code points for both jamo and the combined syllable block. The syllable blocks will need to be decomposed into jamo in order for Aspell to spell check it.
Syllabic ========
Syllabic languages use a separate symbol for each syllable of the language. Since most of them have more than 240 distinct characters Aspell can not support them as is. However, all hope is not lost as Aspell will most likely be able to support them in the future.
Code Language Name Script am Amharic Ethiopic cr Cree Canadian Syllabics ii Sichuan Yi Yi iu Inuktitut Canadian Syllabics oj Ojibwa Ojibwe om Oromo Ethiopic ti Tigrinya Ethiopic
The Ethiopic Syllabary ----------------------
Even though the Ethiopic script has more than 220 distinct characters with a little work Aspell can still handle it. The idea is to split each character into two parts based on the matrix representation. The first 3 bits will be the first part and could be mapped to `10000???'. The next 6 bits will be the second part and could be mapped to `11??????'. The combined character will then be mapped with the upper bits coming first. Thus each Ethiopic syllabary will have the form `11?????? 10000???'. By mapping the first and second parts to separate 8-bit characters it is easy to tell which part represents the consonant and which part represents the vowel of the syllabary. This encoding of the syllabary is far more useful to Aspell than if they were stored in UTF-8 or UTF-16. In fact, the exiting suggestion strategy of Aspell will work well with this encoding with out any additional modifications. However, additional improvements may be possible by taking advantage of the consonant-vowel structure of this encoding.
In fact, the split consonant-vowel representation may prove to be so useful that it may be beneficial to encode other syllabary in this fashion, even if they are less than 220 of them.
The code to break up a syllabary into the consonant-vowel parts does not exists as of Aspell 0.60. However, it will be fairly easy to add it as part of the Unicode normalization process once that is written.
The Yi Syllabary ----------------
A very large syllabary with 819 distince symbols. However, like Ethiopic, it should be possible to support this script by breaking it up.
The Unified Canadian Aboriginal Syllabics -----------------------------------------
Another very large syllabary.
The Ojibwe Syllabary --------------------
With only 120 distinct symbols, Aspell can actually support this one as is. However, as previously mentioned, it may be beneficial to break it up into the consonant-vowel representation anyway.
Unsupported ===========
These languages, when written in the given script, are currently unsupported by Aspell for one reason or another.
Code Language Name Script ja Japanese Japanese km Khmer Khmer ko Korean Hanja + Hangeul pi Pali Thai th Thai Thai zh Chinese Hanja
The Thai and Khmer Scripts --------------------------
The Thai and Khmer scripts presents a different problem for Aspell. The problem is not that there are more than 220 unique symbols, but that there are no spaces between words. This means that there is no easy way to split a sentence into individual words. However, it is still possible to spell check these scripts, it is just a lot more difficult. I will be happy to work within someone who is interested in adding Thai or Khmer support to Aspell, but it is not likely something I will do in the foreseeable future.
Languages which use Hànzi Characters ------------------------------------
Hànzi Characters are used to write Chinese, Japanese, Korean, and were once used to write Vietnamese. Each hànzi character represents a syllable of a spoken word and also has a meaning. Since there are around 3,000 of them in common usage it is unlikely that Aspell will ever be able to support spell checking languages written using hànzi. However, I am not even sure if these languages need spell checking since hànzi characters are generally not entered in directly. Furthermore even if Aspell could spell check hànzi the exiting suggestion strategy will not work well at all, and thus a completely new strategy will need to be developed.
Japanese --------
Modern Japanese is written in a mixture of "hiragana", "katakana", "kanji", and sometimes "romaji". "Hiragana" and "katakana" are both syllabaries unique to Japan, "kanji" is a modified form of hànzi, and "romaji" uses the Latin alphabet. With some work, Aspell should be able to check the non-kanji part of Japanese text. However, based on my limited understanding of Japanese hiragana is often used at the end of kanji. Thus if Aspell was to simply separate out the hiragana from kanji it would end up with a lot of word endings which are not proper words and will thus be flagged as misspellings. However, this can be fairly easily rectified as text is tokenized into words before it is converted into Aspell's internal encoding. In fact, some Japanese text is written in entirely in one script. For example books for children and foreigners are sometimes written entirely in hiragana. Thus, Aspell could prove at least somewhat useful for spell checking Japanese.
Languages Written in Multiple Scripts =====================================
Aspell should be able to check text written in the same language, but in multiple scripts, with some work. If the number of unique symbols in both scripts is less than 220 than a special character set can be used to allow both scripts to be encoding in the same dictionary. However this may not be the most efficient solution. An alternate solution is to store each script in its own dictionary and allow Aspell to chose the correct dictionary based on which script the given word is written in. Aspell currently does not support this mode of spell checking however it is something that I hope to eventually support.
Notes on Planned Dictionaries =============================
be Belarusian Ispell Dictionary available bn Bengali Unoffical Aspell Dictionary available `http://www.bengalinux.org/downloads/' et Estonian Ispell Dictionary available fi Finnish Ispell Dictionary available gd Scottish Ispell Dictionary available. Gaelic `http://packages.debian.org/unstable/text/igaelic' gv Manx Ispell Dictionary available. `http://packages.debian.org/unstable/text/imanx' he Hebrew Ispell Dictionary available hu Hungarian MySpell dictionary expanded to over 500 MB. Will add once affix support is worked into the dictionary package system. lb Luxembourgish MySpell dictionary planned. lt Lithuanian MySpell dictionary expanded to over 500 MB. Will add once affix support is worked into the dictionary package system. mt Maltese Unofficial Aspell Dictionary available, but broken link to source. `http://linux.org.mt/article/spellcheck' sw Albanian Ispell Dictionary available sw Swahili Available at `http://sourceforge.net/projects/translate'. Offical version comming soon. ta Tamil Word list available at `http://www.developer.thamizha.com/spellchecker/index.html'. Working with them to create an Aspell dictionary. wa Walloon Ispell Dictionary available zu Zulu Available at `http://sourceforge.net/projects/translate'. Offical version comming soon.
References ==========
The information in this chapter was gathered from numerous sources, including:
* ISO 639-2 Registration Authority, `http://www.loc.gov/standards/iso639-2/'
* Languages and Scripts (Offical Unicode Site), `http://www.unicode.org/onlinedat/languages-scripts.html'
* Omniglot - a guide to written language, `http://www.omniglot.com/'
* Winkipedia - The Free Encyclopedia, `http://wikipedia.org/'
* Ethnologue - Languages of the World, `http://www.ethnologue.com/'
* World Languages - The Ultimate Language Store, `http://www.worldlanguage.com/'
* South African Languages Web, `http://www.languages.web.za/'
* The Languages and Writing Systems of Africa (Global Advisor Newsletter), `http://www.intersolinc.com/newsletters/africa.htm'
Special thanks goes to Era Eriksson for helping me the information in this chapter.
Language Related Issues ***********************
Here are some language related issues that a good spell checker needs to handle. If you have any more information about any of these issues, or of a new issue not discussed here, please email me at kevina@gnu.org.
German Sharp S ==============
The German Sharp S or Eszett does not have an uppercase equivalent. Instead when `ß' is converted to `SS'. The conversion of `ß' to `SS' requires a special rule, and increases the length of a word, thus disallowing inplace case conversion. Furthermore, my general rule of converting all words to lowercase before looking them up in the dictionary won't work because the conversion of `SS' to lowercase is ambiguous; it can be `ss' or `ß'. I do plan on dealing with this eventually, however.
Compound Words ==============
In some languages, such as German, it is acceptable to string two words together, thus forming a compound word. However, there are rules to when this can be done. Furthermore, it is not always sufficient to simply concatenate the two words. For example, sometimes a letter is inserted between the two words. I tried implementing support for compound words in Aspell but it was too limiting and no one used it. Before I try implementing it again I want to know all the issues involved.
Context Sensitive Spelling ==========================
In some language, such as Luxembourgish, the spelling of a word depends on which words surround it. For example the the letter `n' at the end of a word will disappear if it is followed by another word starting with a certain letter such as an `s'. However, it can probably get more complicated than that. I would like to know how complicated before I attempt to implement support for context sensitive spelling.
Unicode Normalization =====================
Because Unicode contains a large number of precomposed characters there are multiple ways a character can be represented. For example letter a* can either be represented as
U+00E5 LATIN SMALL LETTER A WITH RING ABOVE or U+0061 LATIN SMALL LETTER A + U+030A COMBINING RING ABOVE
By performing normalization first Aspell will only see one of these representations. The exact form of normalization depends on the language. Give the choice of
1. Precomposed character
2. Base letter + combining character(s)
3. Base letter only
if the precomposed charter is in the target character set then (1), if both the base and combing character is present than (2), otherwise (3).
Words With Spaces or other Symbols in Them ==========================================
Many languages, including English, have words with non-letter symbols in them. For example the apostrophe. These symbols generally appear in the middle of a word, but they can also appear at the end, such as in an abbreviation. If a symbol can _only_ appear as part of a word than Aspell can treat it as if it were a letter.
However, the problem is most of these symbols have other uses. For example, the apostrophe is often used as a single quote and the abbreviations marker is also used as a period. Thus, Aspell can not blindly treat them as if they were letters.
Aspell currently handles the case where the symbol can only appear in the middle of the word fairly well. It simply assumes that if there is a letter both before and after the symbol than it is part of the word. This works most of the time but it is not fool proof. For example, suppose the user forgot to leave a space after the period:
... and the dog went up the tree.Then the cat ...
Aspell would think "tree.Then" is one word. A better solution might be to then try to check "tree" and "Then" separately. But what if one of them is not in the dictionary? Should Aspell assume "tree.Then" is one word?
The case where the symbol can appear at the beginning or end of the word is more difficult to deal with. The symbol may or may not actually be part of the word. Aspell currently handles this case by first trying to spell check the word with the symbol and if that fails, try it without. The problem is, if the word is misspelled, should Aspell assume the symbol belongs with the word or not? Currently Aspell assumes it does, which is not always the correct thing to do.
Numbers in words present a different challenge to Aspell. If Aspell treats numbers as letters than every possible number a user might write in a document must be specified in the dictionary. This could be easily be solved by having special code to assume all numbers are correctly spelled. But what about something like "4th". Since the "th" suffix can appear after any number we are left with the same problem. The solution would be to have a special symbol for "any number".
Words with spaces in them, such as foreign phrases, are even more trouble to deal with. The basic problem is that when tokonizing a string there is no good way to keep phrases together. One solution is to use trial and error. If a word is not in the dictionary try grouping it with the previous or next word and see if the combined word is the dictionary. But what if the combined word is not, should the misspelled word be grouped when looking for suggestions? One solution is to also store each part of the phrase in the dictionary, but tag it as part of a phrase and not an independent word.
To further complicate things, most applications that use spell checkers are accustom to parsing the document themselves and sending it to the spell checker a word at a time. In order to support word with spaces in them a more complicated interface will be required.
Notes on 8-bit Characters *************************
There is a very good reason I use 8-bit characters in Aspell. Speed and simplicity. While many parts of my code can fairly be easily be converted to some sort of wide character as my code is clean. Other parts can not be.
One of the reasons because is many, many places I use a direct lookup to find out various information about characters. With 8-bit characters this is very feasible because there is only 256 of them. With 16-bit wide characters this will waste a LOT of space. With 32-bit characters this is just plain impossible. Converting the lookup tables to some other form, while certainly possible, will degrade performance significantly.
Furthermore, some of my algorithms relay on words consisting only on a small number of distinct characters (often around 30 when case and accents are not considered). When the possible character can consist of any Unicode character this number because several thousand, if that. In order for these algorithms to still be used some sort of limit will need to be placed on the possible characters the word can contain. If I impose that limit, I might as well use some sort of 8-bit characters set which will automatically place the limit on what the characters can be.
There is also the issue of how I should store the word lists in memory? As a string of 32 bit wide characters. Now that is using up 4 times more memory than charters would and for languages that can fit within an 8-bit character that is, in my view, a gross waste of memory. So maybe I should store them is some variable width format such as UTF-8. Unfortunately, way, way to many of may algorithms will simply not work with variable width characters without significant modification which will very likely degrade performance. So the solution is to work with the characters as 32-bit wide characters and than convert it to a shorter representation when storing them in the lookup tables. Now than can lead to an inefficiency. I could also use 16 bit wide characters however that may not be good enough to hold all of future versions of Unicode and it has the same problems.
As a response to the space waste used by storing word lists in some sort of wide format some one asked:
Since hard drive are cheaper and cheaper, you could store dictionary in a usable (uncompressed) form and use it directly with memory mapping. Then the efficiency would directly depend on the disk caching method, and only the used part of the dictionaries would relay be loaded into memory. You would no more have to load plain dictionaries into main memory, you'll just want to compute some indexes (or something like that) after mapping.
However, the fact of the matter is that most of the dictionary will be read into memory anyway if it is available. If it is not available than there would be a good deal of disk swaps. Making characters 32-bit wide will increase the change that there are more disk swap. So the bottom line is that it will be cheaper to convert the characters from something like UTF-8 into some sort of wide character. I could also use some sort of disk space lookup table such as the Berkeley Database. However this will *definitely* degrade performance.
The bottom line is that keeping Aspell 8-bit internally is a very well though out decision that is not likely to change any time soon. Fell free to challenge me on it, but, don't expect me to change my mind unless you can bring up some point that I have not thought of before and quite possible a patch to solve cleanly convert Aspell to Unicode internally with out a serious performance lost OR serious memory usage increase.