This is a mobile optimized page that loads fast, if you want to load the real page, click this text.

Forum update

thomas

Unswerving cyclist
Admin
Joined
14 Mar 2002
Messages
21,050
Reaction score
18,904
JREF Forum Update

Dear valued Members,

we are working hard on converting the forum to a new software.

THE FORUM

Skip this part if you are not interested in forum history.

Our forum has been online in its current format since 2002, relying on vBulletin software from the very start. vBulletin, founded by a few devoted British PHP wizards, was until recently considered to be the vanguard of forum software. Along came the vultures of venture capital in 2008, took over and wrecked one of the most popular forum scripts in less than four years. Not only that: they drove away the vB core team, who, not idle in the least, recreated the most amazing and promising piece of forum software, in the best tradition of vB's heydays.

Xenforo, innovative, fresh and written from scratch, seemed to have posed a substantial threat to Internet Brands, the vulture: they sued the enthusiastic team of XF in both, US and UK courts on various grounds, one more ludicrous that the next, thereby frivolously jeopardizing development as well as developers' and webmasters' livelihoods for two years.

Long story short, the case was dismissed "with prejudice" (the actual settlement remains confidential), and development was resumed four weeks ago. Late last summer the vultures released their concept of vBulletin 5 which is nothing short of a development disaster, prematurely released, half-baked, sluggish and devoid of any features. I have therefore decided to move the JREF forum to Xenforo as soon as possible.

Last Sunday night we converted our sister site, tokyocycle.com, to XF. The overall user feedback is very positive, given the fact that users in general hate change. We received tons of feedback and suggestions, which we implemented right away. See for yourself.

The software is very intuitive, user-friendly and easy to grasp for anyone who has used vB. It increased (inter)activity of and among users, who love the speed and the features, in particular the Like and the trophy system!

On top of that, the gallery and the classifieds will remain integrated into the new forum software.

However, there's

THE PROBLEM

Our database is eleven years old. The forum is of course bilingual, English and Japanese. Back then when we launched it in 2002, UTF-8, the lingua franca of encryptions, was not yet available. The collation of the database tables is Latin-1, the language MIME is x-sjis, and over the years many tables, but not all, were converted to UTF-8. The database is therefore a weird potpourri of standards which has effectively prevented us from migrating.

Many of the problems relating to the display of non-standard characters, mojibake, and many editor issues reported over the years arise from those database discrepancies. I have already invested countless hours in that mongrel of a database, to no avail. The Japanese content is not being converted and/or imported.

However, I haven't given up yet. I will keep on experimenting and, just a few days ago, made contact with a Tokyo-based database expert with connections to other MySQL sages. While spirits are high, we do have to prepare contingency plans.

One if them would be to turn the current Japanese language section here at the forum into a read-only repository and allow users to post related question to the new language subfora on the updated forum. So basically an archive of Japanese language users can browse, but not post to. Japanese content in other sections will have to be manually edited.

PLEA FOR FEEDBACK

I would like to invite our regular users and veteran members to share their opinion. Would a "nihongo repository," still accessible to the general public and Google, be an acceptable and viable solution to you? Are there any other suggestions on the issue? Please post them right here.

Furthermore, if there are any other wishes, ideas, complaints, suggestions of whatever nature, or features you would like to see in the future, please do not hesitate to voice them. You can also contact me privately by PM.

PERSONAL

Davey, our tireless co-admin, will - for the time being - lay down his official functions here at the forum due to a flurry of work-related and private issues. He will remain an active member of the forum. We wish him all the best on his endeavours.
 
Last edited:
All the previous threads will be archived and new ones will be in a new subfora, correct?
I feel we need to have the stickies made highly visible for new Nihongo forum participants to see. Is that possible?
If yes, I have no objections. I hope our revered veterans of the forum do, too.

@Davey, thank you for your hard work and good luck in your new endeavors! Keep in touch here and on FB!
 
That's correct, Epigene-san.

The forum will remain as it is, but the entire Nihongo Forum with its current content will be moved to jref.com/repository. It should be noted that this is the worst case scenario. I will continue to find another solution with the help of other developers to avoid splitting up the DB.

Thank you for your feedback!
 
I get the feeling.............

.......from Dave's posts on Facebook , that his job and life are busy and complicated at times. Hopefully you can find a good replacement to help you Thomas.

Uncle Frank
 
My hair's turning grey over this issue, but here's a little update: this week I worked hard to find a way to convert the Japanese threads in our forum database to utf8, no luck so far. I still hope there are ways to keep the entire forum intact.

I thank Epigene-san for her reply, and I'm hoping for more feedback from other regular contributors. How do you think about splitting up the DB and keeping the former Nihongo section in a read-only repository? It is likely that Japanese threads in other fora will be lost too (unless we move them into the repository as well). As stated above, this would be the ultima ration in case all conversion efforts fail.

Please share your thoughts.
 
Convert what you can. Archive what you can't. Quit giving yourself an ulcer over it.
 
I'm close to it, Mike.

So in the worst case a Nihongo archive it is going to be.
 
I'm close to it, Mike.

So in the worst case a Nihongo archive it is going to be.

Look in the bright side...they'll never be necroposted or threadjacked!
 
I will post here, and express what's on my mind, yet wish to make clear that I do do so with some reluctance. The reason for that is that I basically never have gotten involved in the Japanese sections, and, have not been so regular for several years, therefore do not reason, nor feel, that I have much right to say anything about it. If pushed, I'd say take the route of least resistance; as Mike has said above (and others, basically).

I want to take this opportunity to give my heartfelt, hats-off bow to Davey, for all that he has done for JREF. There have been times when I have worked to support him, and times when I have increased his work. I make a toast to your service, time, knowledge, energy, kindness, and smooth-operating on the forum, Davey !! Cheers !! (I just hope that all is on the positive, busy side, and all that... like, you know, moving on up, growing with the kids, and all that !! )

Thomas, don't do yourself in. As one, among many others, I know, there is great appreciation in most every fiber of that which makes the totality of myself, for all you have done for us who have been here for some while, who are active now, and who will, regardless of frequency, always be somewhere around these parts ! After a certain extent of playing around, if it still looks like it's not going to work, make the choice... and stay healthier for it.

MM
 
Thanks MM, I do appreciate the support. Given the latest developments, there is a good chance to preserve the DB.
 
MySQL (5.0.96). The problem are the collations and the conversion of Japanese multi-byte characters.
 
In there a reason why you are not using SQLServer?

How big is the Database now?
What is stopping you doing the conversions?
 
Hi Dave, if you refer to Microsoft SQL Server, I believe it only runs on Windows servers. We use Linux (CentOS), the DB is over 1GB in size. As outlined above, the collation of the tables is messed up, and with a few 100k lines of data, that's a lot of cleaning up. I asked a programmer take a look at it. He was quite positive today that the DB could be converted, but it would be quite a task.
 
Sorry, did not realise you were on Linux.

The DB is by no means a huge one, but not exactly small. How many tables and fields do you have? Even with 100k lines of data, cleaning it up should be possible with the use of some PHP script etc, so it should only take as long as designing some code to sort it out. I don't see how this could be a huge job??? I am not questioning you at all, I just hope you are not being taken for a ride. Of corse I have not seen the database, so it makes it really hard to comment.

Cheers Dave
 
Sorry, I meant 100s of k of lines; there are 265 tables and 2087 columns. The problem is not the size, but the collations. I have experimented for months, but frankly it's beyond me. I had one developer who usually works with Japanese DBs who couldn't help. This week will decide whether the entire DB can be converted or not.
 
I don't see how the collations are proving to be such a problem.

Can't you just use this for the Database,

ALTER DATABASE <database_name> CHARACTER SET utf8_general_ci;

And this for the Tables,

ALTER TABLE <table_name> CONVERT TO CHARACTER SET utf8_general_ci;

This should restore any mojibake for Chinese characters.

And if you have the commandline shell, this should all that be needed??? Just change the DB name ( "dbname" ) to your DB name.


If you want to convert it to a more general type you could use 'utf8_unicode_ci '.
 
We have tried that, Dave.

ALTER DATABASE databasename CHARACTER SET utf8 COLLATE utf8_unicode_ci;
ALTER TABLE tablename CHARACTER SET utf8 COLLATE utf8_unicode_ci;

It's changed the collation, but the data itself was not converted, resulting in more mojibake.

We also tried this

=> Converting your MySQL database to UTF8 - MoodleDocs
 
In that case, as you posted in your link, use 'ALTER TABLE', but as you already know you just can't do this and get the results you need. You will also need to use 'MODIFY' and a few other procedures to get the data type you need.

A better explaination is here of what could help you.

And as you already probably are doing, test on a backup Database.

Sorry if I am re-hashing anything.
 
Thanks Dave. Of course, we wouldn't touch the original DB.

Now I wonder whether we should upgrade MysQL...
 
It could not hurt to move to MySQL 5.6.10 if you are doing a complete revamp.
Here are some of the fixes, MySQL 5.6.10 .

I am interested as to how you get on, please post updates.

Dave
 
I really appreciate it, Dave! I'll talk to our programmer today or tomorrow (depending onhis progress) and will post updates as they come in.
 
@Davey, thank you for your hard work and good luck in your new endeavors! Keep in touch here and on FB!

I'm not gone yet, just taking a break Thank you for your kind words!

.......from Dave's posts on Facebook , that his job and life are busy and complicated at times. Hopefully you can find a good replacement to help you Thomas.

Uncle Frank
I have been pretty busy and still am.. life is life (grin) . Let's hope that thomas indeed can find someone that is able to help him more than I have been able to.



your words are too kind, but thank you for them. I'm looking forward to seeing you again either this or next year.
 

BTW Thomas, I did not realize you were that big into cycling. Here is my ride,

 
Cookies are required to use this site. You must accept them to continue using the site. Learn more…