Japanese script not displaying properly

[geishaboy]

I have noticed of the past couple of days that Japanese script it not displaying like it should. I have been on these baords for a while and have never had any trouble with it.

And then just a little while after the Japanese Forum opened (ironic) it stopped working like it should

It cannot display Hiragana, wich is the native Japanese alphabet. It simply displays as question marks.

Katakana is ok カタカナ
Kanji Seems to work 漢字
But Hiragana is not working at all �?�ら�?��?�

Without the ability to post in Japanese it kind of makes the Japanese forum a bit redundant.

It would be a great help if someone could take a look into this and try and solve the problem.

[Theo]

We changed the character set from Latin (western) to UTF8 generic. Now is should be able to support any language that we can throw at it. The only thing that I can think of is that PHPBB tries to do this automatically and somehow mangles the data.

I'm not sure what the workaround is at this time. Maybe Ken knows something more about the inner workings of the forum and how to solve this. For prior posts, I'm not sure what the fix is.

[webmaster]

I haven't been able to figure this out either.

[Raoul Duke]

I have absolutely no idea about anything...lol, but if something used to work, and then you change it, and it suddenly stops working, can you not deduce that the problem has something to do with the fact that you changed it?

If it aint broke don't fix it!

Or it could be to do with the other problems we are experiencing - just chucking out suggestions.

I googled the fonts and came up with nothing.

[Theo]

That's the problem. With the way it way before it shouldn't have worked... We changed it to where it should and to fix other problems.

[webmaster]

Still haven't been able to find anything of any help, so lets go over what we know.

The charset of the pages has been specified to be UTF-8 for a long while now, and before any of the recent changes all the weird characters displayed fine.

On October 30 phpBB released an update, which I installed the next day, and on November 2nd the charset for the database was converted from latin1_swedish_ci to utf8_general_ci. I don't see any mention of wrongly displayed characters until November 3rd, which would indicate the database conversion has something to do with it, but someone did mention that they updated their board as well and afterwards had a similar problem. So I went through the entire change log of the update, but didn't see any changes that looked like it would have anything to do with it.

From what I can tell, characters only seem to get screwed up after they have been through the database. If a message is previewed before it is posted, then the characters in the preview all look fine, since the preview text is passed through the form's POST method and not stored in the database. And both the preview and final post go through the same formatting functions. The SQL statement that actually inserts the text into the database doesn't seem to provide any clues either:

$sql = "INSERT INTO " . POSTS_TEXT_TABLE . " (post_id, post_subject, bbcode_uid, post_text) VALUES ($post_id, '$post_subject', '$bbcode_uid', '$post_message')";

One thing I don't know is whether the characters are screwed up in the database or if they get screwed up when being pulled out of the database.

This is what UTF-8 encoding is supposed to look like when viewed as ASCII characters:

Code: Select all

HI: ã²
RA: ら
GA: ãŒ
NA: ãª

This is what we're seeing in the post:

Code: Select all

HI: �?�
RA: ら
GA: �?�
NA: �?�

[Dretlin]

Mhhh...as Theo already explained to me, I guess this is why my perivous characters screwed up and I couldnt login...

Also, I just added text to my sig...and it only shows up on my new posts and not the 1616 I have done previous to it...

[Mai]

Maybe the whole multilingual thing just hates you guys, Ken and Theo.

Wait..Er..Uh..nevermind.

[webmaster]

I added the following line right after the database connection:

mysql_query("set names 'utf8'",$this->db_connect_id);

Let's see if it makes any difference....

UTF-8:
Hiragana: ひらがな
Katakana: カタカナ
Kanji: 漢字

[Theo]

I'm not sure about Hiragana, Katakana, or Kanji. They all look Japanese to me.

[Mai]

It all just looks like gibberish to me.

I'm like..language-illiterate on some level.

Jeez, sometimes I feel stupid...

[geishaboy]

Yes, thank you. It all seems to be working fine now. But all Japanese posts from before are showing up as gibberish, wich means I have to go back through just about every post in the Japanese Language forum and re-wite it it can't be helped, I'm just happy its working again.

Also, as you can see, my username was in Japanese so that to got messed up, I re wrote it when these problems first appeard, but it displayed the same, and now that the language settings have been fixed my username has turned to gibberish.

I tried to change it again, but it wont let me. So it would be a great help if someone could change my username back to the original one 芸者男の子 some time soon. That or let me change it myself.

どうもありがとうがいました
(Thank you very much)

[Theo]

We are working to fix the old ones. We have a copy of all of the data from before the upgrade. We are trying to see what we can do to upgrade it. If we can we will just replace those articles that were affected. This is what we are hoping to do.

I'm not sure how to change your username to special characters, just back to normal. I'll defer to Ken on that.

[Mai]

Is it jsut me or does this all always get even more confusing?? ...

[Theo]

It's just you.

[Mai]

Well, that totally ruined my day. Jerk. [Total joke]