Terminal shows non-ascii? characters in UTF-16 hex codes

Todoroki
todoroki@todoroki-VJZ13B ~>printf "ä\n"
                           echo "ä"
                           ä
ä
ä
\udcc3\udca4: \u30b3\u30de\u30f3\u30c9\u304c\u898b\u3064\u304b\u308a\u307e\u305b\u3093

according to a UTF-16 decode tool, \u30b3\u30de\u30f3\u30c9\u304c\u898b\u3064\u304b\u308a\u307e\u305b\u3093 is コマンドが見つかりません (= "command not found"), which is the correct Japanese output I expect.

From the printf and echo result, UTF-8 seems working correctly.

This happens in all shell outputs, such as ls (Japanese characters in filenames shows up in UTF-16 hex format)
EDIT: ls output was not utf-16, but something called "Octal Escape Sequence" (where becomes \346\234\210)

ls in a directory which contains 3 folders named C, あいう, and outputs:

todoroki@todoroki-VJZ13B ~/test> ls -l
total 12
drwxr-xr-x 3 todoroki todoroki 4096 10月  4 15:02  C/
drwxr-xr-x 2 todoroki todoroki 4096 10月 11 09:04 ''$'\343\201\202\343\201\204\343\201\206'/
drwxr-xr-x 2 todoroki todoroki 4096 10月 11 09:05 ''$'\346\234\210'/

(and this is weird because of the file creation dates are shown correctly, while the directory name one isn't)

less vi nano behaves more strange; a file (a.txt, created with gedit) like below

あ
い
う
ä

will show as

in less (it complains "a.txt" may be a binary file. See it anyway?):

<E3><81><82>
<E3><81><84>
<E3><81><86>
<C3><A4>

in vi:

�~A~B
�~A~D
�~A~F
ä

and in nano:

 ^a^b
 ^a^d
 ^a^f

I don't remember what I had done, but it was correctly showing Japanese letters at least two days ago (and for more than 6 months).

What could be the problem and the way to recover from this?

Todoroki

I had accidentally updated my fish config file to read ~/.profile,
which included a line saying locale=C.

I changed this to locale=C_UTF8 and everything recovered.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Find non-ascii characters from a UTF-8 string

From Dev

URL encode ASCII/UTF16 characters

From Dev

Convert string with hex ASCII codes to characters

From Dev

Print non ASCII characters as their hex in PHP

From Dev

UTF16 hex to text

From Dev

Accept non ASCII characters

From Dev

fwrite with non ASCII characters

From Dev

Decoding strings of UTF-16 encoded hex characters

From Dev

Losing characters when converting between hex and ascii

From Dev

Convert extended ASCII character codes to utf-8 byte codes

From Dev

Running Ascii regex over non-ASCII characters with UTF-8

From Dev

Non-ASCII characters in UTF-8 mode regular expression

From Dev

Convert non-ASCII/UTF-8 characters into LaTeX codes

From Dev

Outputting non ASCII codes to a file

From Dev

Print non ASCII characters as their hex in PHP

From Dev

Escaping characters in JavaScript using their ASCII codes

From Dev

UTF16 hex to text

From Dev

fwrite with non ASCII characters

From Dev

Decoding strings of UTF-16 encoded hex characters

From Dev

Terminal urxvt shows squares instead of ASCII bold characters

From Dev

Printing an inputted list of ASCII codes as a list of characters

From Dev

Encoding unicode with 'utf-8' shows byte-strings only for non-ascii

From Dev

`cat -v` for non-printing non-ascii UTF characters

From Dev

Ubuntu Terminal shows ASCII codes character

From Dev

List directory contents on Windows in hex or other form that shows the non-ASCII chars in detail

From Dev

Removing non-ascii characters on utf-16 (Python)

From Dev

Sum of UTF-16 character codes

From Dev

Remove ASCII Hex Codes for Non-Alphanumeric Printable Characters from URL PHP

From Dev

How to convert string in UTF-8 to ASCII ignoring errors and removing non ASCII characters

Related Related

  1. 1

    Find non-ascii characters from a UTF-8 string

  2. 2

    URL encode ASCII/UTF16 characters

  3. 3

    Convert string with hex ASCII codes to characters

  4. 4

    Print non ASCII characters as their hex in PHP

  5. 5

    UTF16 hex to text

  6. 6

    Accept non ASCII characters

  7. 7

    fwrite with non ASCII characters

  8. 8

    Decoding strings of UTF-16 encoded hex characters

  9. 9

    Losing characters when converting between hex and ascii

  10. 10

    Convert extended ASCII character codes to utf-8 byte codes

  11. 11

    Running Ascii regex over non-ASCII characters with UTF-8

  12. 12

    Non-ASCII characters in UTF-8 mode regular expression

  13. 13

    Convert non-ASCII/UTF-8 characters into LaTeX codes

  14. 14

    Outputting non ASCII codes to a file

  15. 15

    Print non ASCII characters as their hex in PHP

  16. 16

    Escaping characters in JavaScript using their ASCII codes

  17. 17

    UTF16 hex to text

  18. 18

    fwrite with non ASCII characters

  19. 19

    Decoding strings of UTF-16 encoded hex characters

  20. 20

    Terminal urxvt shows squares instead of ASCII bold characters

  21. 21

    Printing an inputted list of ASCII codes as a list of characters

  22. 22

    Encoding unicode with 'utf-8' shows byte-strings only for non-ascii

  23. 23

    `cat -v` for non-printing non-ascii UTF characters

  24. 24

    Ubuntu Terminal shows ASCII codes character

  25. 25

    List directory contents on Windows in hex or other form that shows the non-ASCII chars in detail

  26. 26

    Removing non-ascii characters on utf-16 (Python)

  27. 27

    Sum of UTF-16 character codes

  28. 28

    Remove ASCII Hex Codes for Non-Alphanumeric Printable Characters from URL PHP

  29. 29

    How to convert string in UTF-8 to ASCII ignoring errors and removing non ASCII characters

HotTag

Archive