PermaLinkNow In The Code Bin: All Unicode Characters DB
12:26:09 AM

I started my career in the IT business 27 years ago as a international products specialist at Wang Laboratories. I became something of an expert on dealing with different character sets supported on different localized versions of Wang hardware and software, and one of the indispensible tools that I used was the All Characters Document. It was a Wang Word Processing document that contained one of every possible character that could be represented, conveniently laid out in a matrix with 16 characters per row, making it easy to read the hexadecimal value of each character's actual byte value. It was not a big document, as in those days all character sets were just 7 or 8 bits, and Wang systems did not support any type of ISO 2022 style escape mechanism to extend character sets.


Even after I moved into working on email products at Wang, I did follow the evolving standards, so I was aware of what was going on with ISO 10646 and Unicode, but over the years, my knowledge of internationalization and localization has lagged a bit. On modern systems, most of the time there's nothing much to worry about, because everything just works. In the Lotus Notes and Domino world everything is LMBCS or a properly tagged MIME character set, conversions are done automatically when you need them in LotusScript and Java, and if you're working in the C API there are functions available for conversion whenever necessary. The few occasions where I've had to dig into character set issues have typically involved uncertainty about the system or JVM default character sets rather than any confusion about what's going on in the Lotus software.


A few months ago, however, I started looking into a MIME conversion issue involving a subset of the Japanese characters known as "Hankaku-Kana". I won't go into the details of the issue here, other than to say that it took me much longer than it really should have to analyze, understand, and solve the problem -- and that even when I had the solution I was thrown off track by one particular document because it turned out that it contained damaged LMBCS. About half-way through the process I realized that what I really needed was the Lotus Notes and Domino equivalent of the All Characters Document, so I created it.


Well... what I created is more than just a document. I created a database with one document per character, and a simple view that displays them. It's technically an All Unicode Characters Database, since the starting point for it was a file on the Unicode Consortium web site called UnicodeData.txt (there's a description of the format of the file here). I'm not 100% certain that everyting in LMBCS is also in Unicode, and I'm not sure that everything in Unicode is actually in the file, so there may be some gaps, but for many if not most intents and purposes, I think my database is a complete enough reference. Here is a screen shot of the All Characters view:


unicodeData1.jpg



In addition to creating the individual documents per charcter, I also created a couple of true All Characters Documents. I created one document that has 16 characters per line of rich text, and another with each character rendered on its own line of rich text. I did not go the trouble or programming a nice tabular matrix, so it's not quite as easy to work with these documents as it was with the one that I used at Wang, but in both cases I did put the code point values in the document along with the characters so these documents can serve as nice self-documenting test case data for any process that needs to work on rich text and be sure that it is handling all character conversions correctly. Here's a screen shot of the one-character-per-line document:


unicodeData2.jpg



One caveat to bear in mind when working with this database is that its ability to actually show all Unicode characters is potentially limited by a couple of things.There are large blocks of Unicode code points that are not rendered as characters on the screen on my computer, and for any given case I don't really know what the limiting factor is. I can think of two. I've already mentioned that I'm not sure that everything in LMBCS is in Unicode, but I'm also not sure that everything in Unicode is also in LMBCS. In fact, it wouldn't surprise me at all if there are things that have been added to Unicode since after LMBCS was finalized. Any Unicode characters that are not available in LMBCS will not display correctly in the database. Also, fonts come into play. I set up the form and rich text to use the font @Arial Unicode MS, but I'm not 100% certain that this font really renders all of Unicode. Naturally, and character that this font doesn't render correctly will not display correctly in the database. As i said above, however, for most intents and purposes this database is complete enough.


At 21 MB, the database is a bit larger than I want to have downloaded directly from my blog, and unlike many Notes databases it doesn't compress much at all when zipped, so I've uploaded it to the Code Bin on the OpenNTF site. You can download the database UnicodeData.nsf from there.


UPDATE: A new version has been posted. Read about it here.

This page has been accessed 537 times. .
Comments :v

1. Dennis van Remortel02/15/2010 05:04:10 AM
Homepage: http://www.d2k.nl/domino


Thanks!




2. Ben Langhinrichs02/15/2010 10:41:56 AM
Homepage: http://www.GeniiSoft.com/showcase.nsf/GeniiBlog


Terrific resource, Rich! Thanks for making this available.




3. David Schaffer02/15/2010 01:46:49 PM
Homepage: http://bloginprogress.us


I can't open the downloaded database. Says it has local encryption.




4. Richard Schwartz02/15/2010 02:06:34 PM
Homepage: http://www.poweroftheschwartz.com


No wonder zip didn't shrink it previously! That really should have been a clue for me to look at the encryption.

I'm in the process of posting a non-encrypted version, zipped.




5. Richard Schwartz02/15/2010 02:25:10 PM
Homepage: http://www.poweroftheschwartz.com


Non-encrypted version is now posted -- in zip file this time, and the link in the article has been updated to point to it.




6. vesoftware11/05/2013 10:17:33 PM
Homepage: http://vesoftware.blogspot.com/2013/10/itupokercom-agen-poker-online-indonesia.html


Agen Bola Promo 100% SBOBET IBCBET Casino Poker Tangkas Online
http://vesoftware.blogspot.com/2013/10/agen-bola-promo-100-sbobet-ibcbet.html
ITUPOKER.COM AGEN POKER ONLINE INDONESIA TERPERCAYA
http://vesoftware.blogspot.com/2013/10/itupokercom-agen-poker-online-indonesia.html
alfaonline.com : Toko belanja online murah, Promo heboh jual barang hanya Rp 1,-
http://vesoftware.blogspot.com/2013/10/alfaonlinecom-toko-belanja-online-murah.html
BELAJAR SEO
http://balapseo.blogspot.com/




7. ninest12304/23/2016 04:47:08 AM


http://www.oakleyvault.us.com/
http://www.oakley--sunglasses.com.co/
http://www.oakleysunglassesoutlet.us.org/
http://www.oakleysunglasses.us.com/
http://www.oakleysunglasses.com.co/
http://www.rayban-sunglass.us.com/
http://www.ray-ban-sunglass.us.com/
http://www.rayban--sunglasses.org/
http://www.polo-ralph-lauren.us.com/
http://www.polo-ralph-lauren-outlet.us.com/
http://www.burberry-outlet-store.us.com/
http://www.burberryoutlet-store.us.com/
http://www.michaelkors-outlet-online.us.org/
http://www.outletonline-michaelkors.com/
http://www.michaelkors-outlet-online.us.com/
http://www.michael-korsoutlet.in.net/
http://www.michaelkors-outlet-store.us.com/
http://www.michaelkors-handbags.in.net/
http://www.uggs-outletboots.in.net/
http://www.uggsoutlet.us.com/
http://www.uggs-outlet.in.net/
http://www.cheapuggboots.in.net/
http://www.cheap-uggs.in.net/
http://www.louisvuitonoutlet.us.com/
http://www.louisvuitton-outlets.org/
http://www.louisvuitton.ar.com/
http://www.louisvuitton-outlet-store.com/
http://www.louis-vuittonhandbags.org/
http://www.redbottoms.in.net/
http://www.christianlouboutin-outlet.in.net/
http://www.christian-louboutin-shoes.in.net/
http://www.christian-louboutin.org.uk/
http://www.rolexreplicawatches.us.com/
http://www.replica--watches.com/
http://www.air-jordanshoes.net/
http://www.nike--outlet.com/
http://www.toryburch-outletonline.in.net/
http://www.guccioutlet-online.us.com/
http://www.prada-handbags.us.com/
http://www.prada-outlets.in.net/
http://www.longchamp-outlet.in.net/
http://www.longchamphandbags.in.net/
http://www.longchamphandbagsoutlet.us.com/
http://www.tiffany-and-co.us.com/
http://www.tiffanyjewelry.us.com/
http://www.nikeairmaxinc.net/
http://www.nikeair-max.net/
http://www.nike-freerun.com/
http://www.chanel-handbags-outlet.in.net/
http://www.katespadeoutlets.cc/
http://www.katespade-handbags.in.net/
http://www.coachoutlet-online.in.net/
http://www.coachfactory-outletonline.in.net/
http://www.coachoutletstoreonline.jp.net/
http://www.coachpurse.in.net/
http://www.true-religionoutlet.in.net/
http://www.truereligionjeansoutlet.us.com/
http://www.truereligionoutlets.in.net/
http://www.cheaptruereligion.us.com/
http://www.nike-airmax-pas-cher.fr/
http://www.nikeair-max.fr/
http://www.airjordanpas-cher.fr/
http://www.nikefreerun.fr/
http://www.nike-roshe-run.fr/
http://www.sac-longchamp-pas-cher.fr/
http://www.sac--longchamps.fr/
http://www.poloralphlauren-pascher.fr/
http://www.louboutin-pas-chere.fr/
http://www.sac-michaelkors-pascher.fr/
http://www.sac-burberry-pascher.fr/
http://www.lunette-oakley-pascher.fr/
http://www.lunette-rayban-pas-cher.fr/
http://www.hollister-abercrombiefitch.fr/
http://www.sachermes.fr/
http://www.vanspas-cher.fr/
http://www.timberland-paschere.fr/
http://www.newbalancesoldes.fr/
http://www.converse-pas-cher.fr/
http://www.the-north-face.fr/
http://www.niketnrequin-pascher.fr/
http://www.polo-lacoste-pascher.fr/
http://www.nikeblazerpaschere.fr/
http://www.nikeair-force.fr/
http://www.sac-guesspascher.fr/
http://www.sac-vanessa-bruno.fr/
http://www.scarpehogan--outlet.it/
http://www.rayban-sunglasses.me.uk/
http://www.michaelkorshandbags-uk.org.uk/
http://www.michaelkors-outlet-canada.ca/
http://www.michael--korsoutlet.ca/
http://www.lululemon-outlet-canada.ca/
http://www.hollistersale.org.uk/
http://www.thenorth--face.co.uk/
http://www.mulberryhandbagsoutlet.co.uk/
http://www.ralphlauren-outlet.co.uk/
http://www.nike--airmax.co.uk/
http://www.nikeairmax.org.uk/
http://www.nike-free-run.org.uk/
http://www.abercrombie-and-fitch.org.uk/
http://www.nike-roshe-run.org.uk/
http://www.nike-trainers.org.uk/
http://www.nike-air-huarache.co.uk/
http://www.longchampbags.org.uk/
http://www.hollister-abercrombie.es/
http://www.vans-shoes-outlet.com/
http://www.nikerosherun.net/
http://www.maccosmeticsoutlet.us/
http://www.abercrombie-andfitch.us.com/
http://www.hermesbirkinbags.com.co/
http://www.the-northfaceoutlet.us.com/
http://www.the-north-face.in.net/
http://www.newbalance-shoes.in.net/
http://www.reebok-outlet.in.net/
http://www.lululemonoutlets.in.net/
http://www.cheap-nfljersey.in.net/
http://www.cheap-wedding-dresses.org/
http://www.beatsbydrdreheadphones.in.net/
http://www.montblanc-pens.us.com/
http://www.ferragamo-shoes.net/
http://www.giuseppe-zanotti.in.net/
http://www.instylerionicstyler.us.com/
http://www.babyliss.in.net/
http://www.mcm-handbags.net/
http://www.chiflatiron.cn.com/
http://www.insanityworkout.in.net/
http://www.ghdhair-straighteners.org.uk/
http://www.bottegaveneta-handbagsoutlet.com/
http://www.valentino-shoes.us.com/
http://www.herveleger.us/
http://www.asicso.com/
http://www.soccer--shoes.net/
http://www.soccerjerseys.com.co/
http://www.celine-handbags.org/
http://www.p90x3workout.us.com/
http://www.jimmychoo-shoes.us.com/
http://www.hollister-clothingstore.us.com/




8. gabbar bhaijaan12/15/2016 06:58:21 AM


http://merrychristmas-happynewyear2017.org/best-merry-christmas-and-happy-new-year-wishes-messages-2017.html

http://merrychristmas-happynewyear2017.org/happy-new-year-quotes-2017-inspirational-resolution-quotes.html

http://merrychristmas-happynewyear2017.org/merry-christmas-greetings-wishes-messages-2016.html

http://merrychristmas-happynewyear2017.org/happy-new-year-messages-sms.html




9. kiss day01/14/2017 12:42:05 AM
Homepage: http://https://kissdayimages2017.com


[URL=https://kissdayimages2017.com]kiss day images[/URL]
[URL=https://kissdayimages2017.com]kiss day images 2017[/URL]
[URL=https://kissdayimages2017.com]kiss day quotes 2017[/URL]
[URL=https://kissdayimages2017.com]kiss day sms 2017[/URL]
[URL=https://kissdayimages2017.com]kiss day pictures 2017[/URL]




10. kiss day01/14/2017 12:45:49 AM
Homepage: http://https://kissdayimages2017.com



<a href="https://kissdayimages2017.com">kissdayimages2017.com</a>
<br>
<a href="https://kissdayimages2017.com">kiss day images 2017</a>
<br>
<a href="https://kissdayimages2017.com">kiss day images</a>
<br>
<a href="https://kissdayimages2017.com">kiss day 2017</a>
<br>
<a href="https://kissdayimages2017.com">kiss day quotes 2017</a>
<br>
<a href="https://kissdayimages2017.com">kiss day sms 2017</a>
<br>
<a href="https://kissdayimages2017.com">kiss day pictures 2017</a>




11. kiss day01/14/2017 12:47:14 AM
Homepage: http://https://kissdayimages2017.com


https://kissdayimages2017.com
https://kissdayimages2017.com
https://kissdayimages2017.com
https://kissdayimages2017.com




12. Ranbirkapoor01/25/2017 01:48:10 AM
Homepage: http://www.inspirationallovequotesimages.com/2016/11/happy-promise-day.html


Pretty nice information. it has a better understanding. thanks for spending time on it.
http://www.inspirationallovequotesimages.com/2016/11/happy-rose-day.html
http://www.inspirationallovequotesimages.com/2016/11/happy-hug-day.html
http://www.inspirationallovequotesimages.com/2016/11/valentines-day-wishes.html
http://www.inspirationallovequotesimages.com/2016/12/easter-greetings.html
http://www.fathersday-2014.net/2016/06/fathers-day-poems-from-child.html
http://www.inspirationallovequotesimages.com/2015/03/inspirational-love-quotes-Him-Her.html




13. nike air max03/14/2017 05:16:29 AM
Homepage: http://www.nikeshoes.us.org


kate spade handbags http://www.katespadesoutlet.us
coach factory outlet http://www.coachoutletonline.cc
Coach Outlet Store Online http://www.coachoutletstore.in.net
Michael Kors Outlet http://www.michaelkorsoutletstore.eu.com
polo outlet http://www.outletralphlauren.in.net
Polo Ralph Lauren Outlet Store http://www.polosoutlet.us.com
yeezy boost 350 http://www.yeezyadidasboost.us.com
Red Bottom Shoes http://www.redbottomsshoe.us.com
adidas nmd http://www.nmdadidasshoes.us.com
Christian Louboutin Outlet http://www.christianlouboutinsshoes.us.com
true religion outlet http://www.outlettruereligionjeans.us.com
true religion jeans outlet http://www.true-religionjeans.com
Pandora Jewelry http://www.pandorajewelrysale.us
Ferragamo Shoes http://www.ferragamoshoes.eu.com
North Face Jackets http://www.thenorthfaces.us.com
Polo Ralph Lauren Outlet http://www.poloralphlaurenus.us.com
kate spade outlet http://www.outletkatespade.us.com
Ray Ban Sunglasses http://www.raybanoutlet.cc
Coach Outlet Store Online http://www.coachsoutlet.us.com/
Coach Factory Outlet Online http://www.coachoutletonlines.us.org/
Coach Outlet Online http://www.coachfactoryoutletstore.eu.com/
Coach Outlet http://www.coachsoutletonline.us.com/
Michael Kor Factory Outlet http://www.michaelkorsmk.us.com/
Michael Kors Outlet Online http://www.michaelkorsoutleto.us.com/
Michael Kors Outlet http://www.michaelkorsofficials.us.com/
Michael Kors Factory Outlet Online http://www.michaelkorsoutletstoreofficial.eu.com/
michael kors outlet http://www.michaelkorsofficial.in.net
Coach Outlet Clearance http://www.coachoutletstoreonline.net
Michael Kors Online http://www.michaelkorsonline.eu.com
Coach Outlet online http://www.coachoutletcoach.us.com
Michael Kors Outlet Online http://www.outlet-michaelkors.us
nike shoes http://www.nikeairmax90.us.com
nike outlet http://www.nikeshoes.us.org
cheap Jordan http://www.jordancheap.us
retro jordan shoes http://www.jordanshoe.us.com
tory burch outlet http://www.outlet-toryburch.in.net
tory burch sale http://www.tory-burch-outlet.us.com
oakley outlet online http://www.oakleyoutlet.ca
cheap oakley sungalsses http://www.oakleysunglasses-outlet.in.net
ray ban outlet online http://www.raybanoutlet.com.co
ray ban sunglasses outlet http://www.rayban-sunglassesoutlet.in.net
coach outlet http://www.coachoutletstoreus.us.com
the north face outlet http://www.thenorthfaces.name
Michael Kors Bags http://www.michaelkorsoutlets.us.org
Coach Factory Outlet http://www.coachfactoryoutletofficial.name
north face jackets http://www.northfacejacketsoutlet.com
Coach Outlet http://www.coachfactoryoutletstore.us




Enter Comments^


Email addresses provided are not made available on this site.





You can use UUB Code in your posts.

[b]bold[/b]  [i]italic[/i]  [u]underline[/u]  [s]strikethrough[/s]

URL's will be automatically converted to Links


:-x :cry: :laugh: :-( :cool: :huh: :-) :angry: :-D ;-) :-p :grin: :rolleyes: :-\ :emb: :lips: :-o
bold italic underline Strikethrough





Remember me    

Monthly Archive
Responses Elsewhere



About The Schwartz

rss.jpg


All opinions expressed here are my own, and do not represent positions of my employer.