Публикацията е защитена с парола. За да я видите, моля въведете паролата:

Парола:

Some implementation notes:

in gentoo you must enable the storage engines in /etc/make.conf

USE="unicode big-tables extraengine

In debian

This article is so valuable to me so I will copy/paste the last part of it here…

Data Archiving Techniques

Data archiving, a subset of Information Lifecycle Management (ILM), typically involves horizontally partitioning data so that a set of current information resides in one or more objects, while the historical/archived data is placed in separate objects. Oftentimes, historical data can be transferred to lower-cost storage devices (like EMC Centera) that allow easy disk access, but provide significant financial savings. Of course, the idea behind data archiving is to lessen response times by reducing the sheer amount of data needed for current access. However, for analytic purposes, there are times when both the current and archive data needs to be accessed together.

For data archiving environments, Archive tables can easily be accessed in standalone fashion or be set up in union views, which is the primary vehicle used to reference both current and archived data in one object. By combining MySQL’s Archive storage engine with the new 5.0 Federated table support, a DBA can smartly transfer historical data to cheapter storage devices and then reference both current and archived data when needed.

For example, let’s say you have a set of recent information regarding customer transactions in a transactional InnoDB table called CLIENT_TRANSACTION, and a set of historical transactions stored in an Archive table called CLIENT_TRANSACTION_HIST. To lessen load on the active server, you can

1. Transfer the history table to another MySQL server that utilizes cheaper storage devices.
2. Set it up to use the Archive storage engine.
3. Create a federated table link to the historical data from the current database.
4. Build a union view to reference both together when needed.

On the historical MySQL Server:

mysql> alter table client_transaction_hist engine=archive;
Query OK, 112050 rows affected, 0 warning (1.98 sec)
Records: 112050  Duplicates: 0  Warnings: 0

On the MySQL Server that contains active data:

mysql> select count(*) from client_transaction;
+----------+
| count(*) |
+----------+
|    18675 |
+----------+
1 row in set (0.00 sec)

mysql> CREATE TABLE client_transaction_hist (
    ->   client_transaction_id int(11) NOT NULL,
    ->   client_id int(11) NOT NULL,
    ->   investment_id int(11) NOT NULL,
    ->   action varchar(10) NOT NULL,
    ->   price decimal(12,2) NOT NULL,
    ->   number_of_units int(11) NOT NULL,
    ->   transaction_status varchar(10) NOT NULL,
    ->   transaction_sub_timestamp datetime NOT NULL,
    ->   transaction_comp_timestamp datetime NOT NULL,
    ->   description varchar(200) default NULL,
    ->   broker_id bigint(10) default NULL,
    ->   broker_commission decimal(10,2) default NULL
    -> ) ENGINE=FEDERATED
    -> COMMENT='mysql://mysql:password@serv1:3306/gim/client_transaction_hist';
Query OK, 0 rows affected (0.14 sec)

mysql> CREATE VIEW client_transaction_all as
    -> select * from client_transaction
    -> union all
    -> select * from client_transaction_hist;
Query OK, 0 rows affected (0.08 sec)

mysql> select count(*) from client_transaction_all;
+----------+
| count(*) |
+----------+
|    130725|
+----------+
1 row in set (1.20 sec)

Note: rather than use a SELECT * in the view definition above, the actual columns should be listed out; the above was only done to save space for this article.

The targets for Archive tables in data warehousing or data archiving environments are fact and summary tables as they usually contain the bulk of the information. Dimension tables are rarely large enough to warrant a switch to Archive.

Miscellaneous Archive Engine Notes

We talked earlier about the Archive engine being a good choice for storing data auditing results as only INSERT and SELECT are supported. Let’s check to be sure:

mysql> delete from test_archive where client_id = 50;
ERROR 1031 (HY000): Table storage engine for 'test_archive' doesn't have this option
mysql> update test_archive set price=120 where client_id = 50;
ERROR 1031 (HY000): Table storage engine for 'test_archive' doesn't have this option

We also talked about how Archive doesn’t support indexes at this time:

mysql> create index my_index on test_archive (client_transaction_id);
ERROR 1069 (42000): Too many keys specified; max 0 keys allowed

One special thing to note about Archive tables and indexes is that, if you want to alter another storage engine table to be an Archive table, you must first drop any indexes that have been defined on the table.

Archive tables are supported by the new sql_mode data integrity enhancements in MySQL 5.0, so you can ensure only proper data finds its way into your Archive tables:

mysql> create table test (c1 int) engine=archive;
Query OK, 0 rows affected (0.11 sec)

mysql> insert into test values ('hi');
Query OK, 1 row affected, 1 warning (0.05 sec)

mysql> select * from test;
+------+
| c1   |
+------+
|    0 |
+------+
1 row in set (0.00 sec)

mysql> set sql_mode='strict_all_tables';
Query OK, 0 rows affected (0.00 sec)

mysql> insert into test values ('hi');
ERROR 1264 (22003): Out of range value adjusted for column 'c1' at row 1

Conclusion

DBAs facing the problem of corporate data explosion have an excellent new tool to help them in the MySQL 5.0 Archive storage engine. Whether it’s a data warehousing, data archiving, or data auditing situation, MySQL Archive tables can be just what the doctor ordered when it comes to maintaining large amounts of standard or sensitive information, while keeping storage costs at a bare-bones minimum.

Updated: 2005-09-16

I was optimistic that I could make http://www.cenite.com, a price monitoring website to use the fulltext search of mysql. Unfortuntly I have found so many drawbacks that I have to leave this idea. The main source for information for me was:

http://dev.mysql.com/doc/refman/5.0/en/fulltext-search.html

http://devzone.zend.com/node/view/id/1304#Heading14

At first the speed was wonderfull. I was searching in 300k> records apx. 350mb. But then I have to surrender. I cant configure mysql to work as I want. I know that If I spend two days to become expert on C/C++ with Unicode I will success but this is not the case. I want working solution.

The resons that make me not to use mysql for searching:

There is no way to change the default operator by default it is OR. You must parse the user query and rewrite it.

I want automaticaly truncation on all my terms.

There is no way to tell MySQL what are characters, and what are not…..sorry, there are two ways:

1. Touching the sources,

2. Configuring in xmls

No documentation on both. Maybe there is …somewhere.

If you use the default configuration then you will wonder how you get or not get the required results.

Here is a summary of the comands that I use to tweak my mysql server:

SHOW VARIABLES LIKE ‘ft%’
SET @global.ft_min_word_len=2;
SET @local.ft_min_word_len=2;

But it is better to have those options in my.conf
[mysqld]
ft_min_word_len=3
ft_stopword_file=”C:\MySQL\stop.txt”

[myisamchk]
ft_min_word_len=3
ft_stopword_file=”C:\MySQL\stop.txt”

To check what characterset is using your mysql:

SHOW VARIABLES LIKE ‘character_sets_dir’;

/usr/share/mysql/charsets | E:MySQL Server 5.1sharecharsets

SHOW VARIABLES LIKE ‘characte%’
After that you need to rebuild your index with one of those commands
slow: REPAIR TABLE products QUICK;
slow: myisamchk –recover –ft_min_word_len=3 tbl_name.MYI
fastest: DROP INDEX …; CREATE INDEX….;

Alternatives

http://www.sphinxsearch.com/

http://endeca.com/

http://lucene.apache.org/solr/

• Statistics
  • ajaxMyTop, , phpMyTop
  • show full processlist – to check which command is currently running on the mysql
• InoDB monitor, Mysqlreport
• phpmyadmin OR http://phpminiadmin.sourceforge.net/
• CPU load info script
• http://www.maatkit.org/
• SQLite manager – http://iqk.sourceforge.net/

Status

SHOW status where Variable_name like ‘Th%’ or Variable_name like ‘%Connec%’ ;
SHOW [GLOBAL | SESSION] STATUS [LIKE 'pattern' | WHERE expr]

Check/Repair tables

mysqlcheck -u root -p***** –auto-repair –check –optimize –all-databases

Profiling

watch -n 0.5 ‘mysql -u root -ppass -e “SHOW FULL PROCESSLIST” | grep Query’

http://opendba.blogspot.com/2008/03/mysql-finally-ability-to-traceprofile.html

mysql> set profiling=1;
mysql> select count(*) from mysql.user;
mysql> show profile;

Dump

pg_dump -U test arachnid_archiv_test –inserts -h chaos.spider.bg –encoding=utf8 -f pgsql.sql

mysqldump -c -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASS -r $tfile --add-drop-table $DB

mysqldump -c -h localhost -u system3 system3_production -psomepassword -r system3_production.sql  --add-drop-table $DB

Dump for full backup with flushing of the log files

mysqldump -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASS
–single-transaction –all-databases –delete-master-logs –flush-logs –master-data=2
> backup_sunday_1_PM.sql

Encoding problems

http://www.hostbulgaria.com/tutorials/mysql-charset-encoding.aspx

SHOW VARIABLES LIKE ‘character_set_%’;
curl -i http://system3.spider.bg

Creating a database

create database re_production DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

Creating a user

GRANT ALL PRIVILEGES ON arachnid_production.* TO ‘payak’@'%’ IDENTIFIED BY ‘payakpassword’ WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON system3_production.* TO ‘payak’@'%’ IDENTIFIED BY ‘payakpassword’ WITH GRANT OPTION;

mysqladmin -u [user] -h localhost -p password ‘[new_password]‘

SQL for a table

SHOW CREATE TABLE tblname;

mysql tunel to another machine

ssh -N -f -l root -L 0.0.0.0:3307:91.196.240.132:3306 s1
open port 3307 on the local machine to 91.196.240.132:3306 and login into s1 with root

Replace text

UPDATE script_histories SET cod_script = replace(cod_script,”observer.ArchiveObserver(siteId)”,”observer.ArchiveObserver(siteId, script_id, owned_source_id)”);

Copy from one table to another

DELETE FROM system3_production.articles;
INSERT INTO system3_production.articles SELECT * FROM arachnid_from_screen.articles;

Sessions for Rails

select count(*) from sessions where updated_at < DATE_SUB(now(), INTERVAL 3 DAY);

Binnary loging

http://dev.mysql.com/doc/refman/5.0/en/recovery-from-backups.html

Check this attachment here: mysql-presentation on replication etc.

• See the status of the log files
  SHOW BINARY LOGS;
  SHOW MASTER STATUS;
• Clean the binary logs instantly
  RESET MASTER;
• Clean binary logs to date/name
  PURGE BINARY LOGS TO ”mysqld-bin.00XXXX’;
• Configurations in my.cnf
  log-bin
  server-id = 1
  expire_logs_days = 1
  max_binlog_size = 100M

Configuration

max_allowed_packet = 50M
wait_timeout=720
max_connections=1000
connect_timeout=20

query_cache_limit=8M #~~~ removed, 1M def. max pozwl. razmer za cache-hirane na edna zajawka
query_cache_size=128M #~~~ 32M, 0 def.
query_cache_type=1

Restoring the maintian Debian User

GRANT ALL PRIVILEGES ON *.* TO 'debian-sys-maint'@'localhost' IDENTIFIED BY '<password>' WITH GRANT OPTION;
Replace <password> with your debian-sys-maint password.

Copy the content to cp1251toUTF8.sh or use the commands manually

The script is making echo instead of running the commands because this will give you a chance to fix an error if occurs.

#!/bin/bash
echo "Set the params in the script and you will get the commands that you must run in order to get your database converted"

SOURCE_DATABASE=source_database_name
TARGET_DATABASE=new_database_name

USER=router
HOST=mysql.spider.bg

# ----------------------------------- No need to touch bellow
echo -e  "\n\n# Lets export the source database"
echo "mysqldump -u $USER -p -h $HOST --default-character-set=cp1251 --max_allowed_packet=64M $SOURCE_DATABASE > db1.sql"

# recode latin1..utf8 or what ever...
echo -e "\n\n# Lets convert it"
echo "recode -v -f windows-1251..UTF-8 < db1.sql > db2.sql"

echo -e "\n\n# Lets replace some sql creation statements. maybe you will want to do it manually"
echo perl -pi -e "s/cp1251/utf8/g" db2.sql
echo perl -pi -e "s/utf8_bulgarian_ci/utf8_general_ci/g" db2.sql

echo -e "\n\n# Lets create the target database"
echo mysql -u $USER -p -h $HOST mysql -e \"drop database $TARGET_DATABASE\; CREATE DATABASE $TARGET_DATABASE CHARACTER SET utf8 COLLATE utf8_general_ci\";

echo -e "\n\n# Lets do the initial import"
echo "mysql -u $USER -p -h $HOST $TARGET_DATABASE --max_allowed_packet=64M --default-character-set=utf8 < db2.sql"

As a bonus here is a command with which you can convert an all your html pages to utf8 also

find . -name '*.html' -exec recode -v -f windows-1251..UTF-8 \{\} +

This would recursively find all htmls in the current directory.

How to detect character sets

http://linux.die.net/man/1/enca

Migrating Latin1 -> UTF8

We have the mistake to enter all the data in the database (utf8) without setting the right connection encoding (set names utf8). In this case our content is stored as latin1 characters in the utf8 database.

Here is the magic that fixes the encoding found by my bright colleague bl8cki

alter table articles convert to character set latin1;
alter table articles change content content blob;
alter table articles change title title blob;
alter table articles change author author blob;
alter table articles change content content text character set utf8;
alter table articles change title title text character set utf8;
alter table articles change author author text character set utf8;

here is an example

mysql> CREATE TABLE `articles` (`id` int(11) NOT NULL auto_increment,   `title` mediumtext, PRIMARY KEY  (`id`) ) ENGINE=MyISAM AUTO_INCREMENT=10313630 DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.00 sec)

mysql> set names latin1;
Query OK, 0 rows affected (0.00 sec)

mysql>  insert into articles (title) values('закъсал');
Query OK, 1 row affected (0.00 sec)

mysql> alter table articles convert to character set latin1;
Query OK, 1 row affected (0.01 sec)
Records: 1  Duplicates: 0  Warnings: 0

mysql> alter table articles change title title blob;
Query OK, 1 row affected (0.01 sec)
Records: 1  Duplicates: 0  Warnings: 0

mysql> alter table articles change title title text character set utf8;
Query OK, 1 row affected (0.00 sec)
Records: 1  Duplicates: 0  Warnings: 0

mysql> set names utf8;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from articles;
+----------+----------------+
| id       | title          |
+----------+----------------+
| 10313630 | закъсал        |
+----------+----------------+
1 row in set (0.00 sec)

As the example shows it works with cyrilic (pasted in utf8) !