Mantis Bugtracker          
testlink.org

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000212Testlink 1.6.xUser Interface Generalpublic2005-11-09 07:202005-12-07 03:33
Reporterkevin 
Assigned Tokevin 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version1.6 RC2 
Fixed in Version 
Summary0000212: "Â" symbol is appearing everywhere after upgrading from 1.5 to 1.6RC2
DescriptionAfter performing an upgrade installation (1.5 -> 1.6RC2)I am seeing an odd character appearing in all my test cases.

 is appearing everywhere.

I'm not sure why I'm seeing this and it doesn't appear to map to any specific character I had before. It seems to be appearing in addition to all my text so maybe some html markup from 1.5 is being misinterpreted in 1.6.
Additional InformationNo longer seeing the issue now that I have turned UTF-8 off in config.inc.php. I will continue to investigate but this should no longer be considered an open bug.
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0000293)
schlundus (reporter)
2005-11-09 20:17

Can you us provide a dump of your data?
(0000294)
schlundus (reporter)
2005-11-09 20:18

Please tell us also, your db-version and if your tables are in UTF-8?
Please try also to turn off UTF-8 support in your config.inc.php
(0000300)
kevin (reporter)
2005-11-09 23:36

Setting the following property in config.inc.php solves the problem and I no longer see the symbol (Thanks Schlundus!):

/** Set this to TRUE if your MySQL DB supports UTF8 (MySQL version >= 4.1) */
define('DB_SUPPORTS_UTF8', FALSE);

As far as my MySQL version I'm using I believe I'm on 4.1.11 :

"Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3746 to server version: 4.1.11"

I will determine if my tables are in UTF-8.
(0000301)
kevin (reporter)
2005-11-09 23:53

If in fact I do determine that my character set is Latin or something else besides UTF-8 - I'm going to document the process I took to upgrade my character set and provide this documentation for our installation manual.

I'm going to use this link as a starting point:
http://dev.mysql.com/doc/refman/4.1/en/charset-conversion.html [^]

I have Mysql 4.1, but I think we upgrade our DB from MySQL 3.X, and this is probably why I don't have UTF-8 characters even though I'm on MySQL 4.1.
(0000323)
kevin (reporter)
2005-11-11 22:52

We have confirmed DB was using Latin encoding. WE are investigating how to best proceed with converting to UTF-8. Re-opening this issue so I can provide detailed feedback.
(0000419)
kevin (reporter)
2005-12-02 01:37

I have determined that my 1.5 TestLink database contained non-ASCII, UTF-8 extended symbols even before I did the upgrade from 1.5 to 1.6. TestLink 1.5 did not bother to try and render extended character set symbols, so the problem was not seen while I was using TestLink 1.5. Since 1.6 does support all UTF-8 symbols, these additional characters that were somehow inserted into my 1.5 database began appearing - hence my initial thought that this was an upgrade problem. How these symbols got into my database in the first place I do not know - my database has been running since TestLink 1.0.4 so it could have happened sometime ago. Maybe these problem only effects me, but if it does effect other people here are the steps to fix it.

I have written a perl script which can scrape all extended symbols out of a mySQL dump file - removing these characters. Additionally I have figured out how to setup a database so that it believes it's database character set is UTF-8 and not latin. These steps are extensive - so I will write up a doc tonight/tomorrow which explains the steps I went through and the perl script I ran. This document should be included in release documentation - I will email this to Martin as soon as possible and before Sunday's deadline.

For now though - this bug can be closed.
(0000420)
kevin (reporter)
2005-12-02 01:38

I will provide documentation on:

1. how to remove non-ASCII extended symbols from a dump of a mySQL database

2. how to setup a new database so that it has the property:
Db characterset: utf8

When I send the documentation out and the team approves on this documenation I will close this bug.
(0000451)
kevin (reporter)
2005-12-07 03:33

Documentation is as follows :
TestLink 1.6 allows for UTF-8 encoded character rendering, therefore any extended character data that may have snuck
into your database and didn't show up in 1.5 may start appearing in 1.6 UI. You can turn UTF-8 support off
in testlink by modifying a value in the config.inc.php file, but then you will be missing out
on the ability to use characters beyond ASCII.

If you have the same problem I did and see lots of extended characters appearing
in your data after upgrading to 1.6 and having UTF-8 support turned on,
you should read through the following instructions.
Be sure to practice this excercise on a test machine before performing on your deployment
system.

The instructions will help you clear out any non-ASCII characters from your database
and setup your database to support UTF-8.

1. First make a backup of your current database using the mysqldump utility.
/usr/bin/mysqldump -u root testlink15 -p > testlink15.backup

2. Now edit testlink15.backup so schema definitions for EACH table has utf8 encoding specified.
Change the CHARSET for each table from latin1 to utf8.
For example the following line in the definition of 1 table which reads as follows :
ENGINE=MyISAM DEFAULT CHARSET=latin1 COMMENT='This table holds the bugs filed for each result';
should be changed to
ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='This table holds the bugs filed for each result';

3. Then ran testlink15.backup thru my the perl script below as follows:
./replaceScript.pl < testlink15.backup > testlink15.cleaned
replaceScript.pl is as follows :
#!/usr/bin/perl
while (<>) {
    chomp;
        tr/\000-\177/\040/cs;
        print $_, "\n";
}

4. Created an empty testlink16 db with utf8 charset as follows:
CREATE DATABASE testlink16 CHARACTER SET utf8;

5. install the tables into the new database
mysql testlink16 –u root –p < testlink15.cleaned

6. You can verify your database's "Db characterset" is now set to utf8 by using the following command:
login to mysql
use testlink16
mysql> \s
--------------
mysql Ver 14.7 Distrib 4.1.11, for redhat-linux-gnu (i386)
Connection id: 26
Current database: testlink15
Current user: bugz@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 4.1.11
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: latin1
Db characterset: utf8
Client characterset: latin1
Conn. characterset: latin1
UNIX socket: /var/lib/mysql/mysql.sock
Uptime: 36 min 55 sec


7. Run the upgrade installation provided by Testlink 1.6.

---------------------------------------------------------
Other resources:
what the heck is UTF-8 ?
http://www.joelonsoftware.com/articles/Unicode.html [^]

octal table (you can see octal values 000 - 177 are "normal ascii" characters).
The perl script that is provided searches based on octal values.
http://web.cs.mun.ca/~michael/c/ascii-table.html [^]

description of tr perl operation
http://www.unix.org.ua/orelly/perl/learn/ch15_05.htm [^]

- Issue History
Date Modified Username Field Change
2005-11-09 07:20 kevin New Issue
2005-11-09 20:17 schlundus Note Added: 0000293
2005-11-09 20:18 schlundus Note Added: 0000294
2005-11-09 23:36 kevin Note Added: 0000300
2005-11-09 23:53 kevin Note Added: 0000301
2005-11-10 04:32 kevin Assigned To => kevin
2005-11-10 04:32 kevin Priority urgent => normal
2005-11-10 04:32 kevin Status new => resolved
2005-11-10 04:32 kevin Resolution open => no change required
2005-11-10 04:32 kevin Additional Information Updated
2005-11-11 22:52 kevin Status resolved => feedback
2005-11-11 22:52 kevin Resolution no change required => reopened
2005-11-11 22:52 kevin Note Added: 0000323
2005-12-02 01:37 kevin Note Added: 0000419
2005-12-02 01:38 kevin Note Added: 0000420
2005-12-02 01:38 kevin Status feedback => resolved
2005-12-02 01:38 kevin Resolution reopened => fixed
2005-12-07 03:33 kevin Status resolved => closed
2005-12-07 03:33 kevin Note Added: 0000451
2005-12-07 03:33 kevin Fixed in Version => CVS



Copyright © 2000 - 2019 MantisBT Team
Powered by Mantis Bugtracker