Non-Breaking Hyphen

October 2, 2020

We begin by creating a string with an embedded unicode character:

(define s (string-append "1000136650" (string #\x7db) "1"))
> s
"1000136650ߛ1"

For some reason, the end of the string is reversed when copied from my terminal window to WordPress. Then we write a function asciistr similar to the Oracle function of the same name:

(define (asciistr s)
  (map char->integer (string->list s)))
> (asciistr s)
(49 48 48 48 49 51 54 54 53 48 2011 49)

Now you can see the string is in the correct sequence. You can run the program at https://ideone.com/O6Dumh, where you will see
that Guile substitutes a different character for the non-breaking hyphen.

Pages: 1 2

5 Responses to “Non-Breaking Hyphen”

  1. Dale said

    A non-breaking hyphen is 0x2011

  2. Tim said

    I’m not sure what the “correct” behaviour is here. Obviously crashing the system wasn’t it, but should this field just be left untouched? Or should the non-breaking hyphen have been normalised to a regular hyphen? And if it’s the latter, is this an easy thing to do in general? (How many other characters look like hyphens?) Honestly, raising an error in the case that the invoice number contains non-numeric characters seems like the least evil thing. Otherwise we’ll wind up having to resort to OCR somewhere down the line.

    Anyway, thanks for the amazing supply of interesting problems!

  3. programmingpraxis said

    @Tim: The correct behavior is to decide whether to accept unicode characters either everywhere or nowhere. If you decide to accept unicode characters in the user interface, the database backend should accept them. If you decide the database backend will not accept unicode characters, the user interface should reject them (strip them, or beep at the user).

    What has happened historically is that the vendor had a custom user interface and database backend; both used plain ascii. Parts of the system date to the 1980s. The vendor recently (in the last two years) updated the user interface so it could be used from any web browser, but kept the database backend (their advertising now features the word “mobile” prominently). Apparently, the user interface now allows unicode, but the database backend doesn’t. I filed a bug report, but it will be ignored (there are some bugs in the bug-tracking database that are over twenty years old).

    Frankly, I am perfectly happy with the current state of affairs. If the vendor wrote good programs, I would be out of a job.

  4. Daniel said

    Here’s a solution in Python.

    def is_ascii(string: str) -> bool:
        try:
            string.encode('ascii')
            return True
        except UnicodeEncodeError:
            return False
    
    print(is_ascii('programming-praxis'))
    print(is_ascii('programming\u2011praxis'))
    

    Output:

    True
    False
    
  5. Alex B said

    @Praxis – Working in Japanese text, where Unicode is not the norm, and sometimes with ‘standard’ encodings not conforming to the standards, I feel your pain here.

    Just one comment though, non-breaking hyphen is U+2011 (0x2011, 8209[sub]10[/sub]); U+07db (0x7db, 2011[sub]10[/sub]) is Nko Letter Sa. Interestingly, this has Right-To-Left Bidirectional Class, which may be why the order is reversed when copying from terminal to WordPress.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: