User Tools

Site Tools


tcms_install

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

tcms_install [2016/10/18 20:16] (current)
admin created
Line 1: Line 1:
 +====== Thumb CMS ======
  
 +Thumb CMS - is the next step in development of SmartCJ.
 +
 +It's been almost 10 years since the first version of SmartCJ was released and it's time to move ahead and adopt new technologies. New version uses php 5.3+ and mysql 5+, please, be aware of it.  ​
 +
 +===== What's new =====
 +
 +New version has a lot of improvements ​
 +
 +  * DB scheme has been changed to support large Dbs with categories having 100-200-300k galleries in each category. It doesn'​t matter how many galleries you have in your categories - it will be really fast now.
 +  * you can run cron jobs as often as you want optimizing server load, of course you have to understand that all actions will be executed as often as you run cronjobs. So if you do that every 5 minutes - all stats we be updated only once every 5 minutes.
 +  * All Db queries are optimized ​
 +
 +Now the script is absolutely invisible from outside :
 +
 +  * you can install the script into any folder ​
 +  * rewrites are simple and you can have any links 
 +  * you can cookieless mode and script won't send any cookies to users and save cookies into internal cache. That adds a bit of load and the script can not be tracked. ​
 +  * You can remove [[TCMS Rotation Parameter]] and still  the script can count hits even if galleries have more that 1 rotation thumb per gallery + [[Cell K]]
 +
 +Also there are some brand new features
 +
 +  * [[Thumb Split]] - that's a way to make your thumb unique by splitting an image into pieces ​
 +  * [[FaceDetect]] - make sure that if a face has a face it won't be cut out 
 +
 +
 +
 +===== Installation =====
 +
 +Now it really easy to install the script. You don't even have to download an install file, just run the following command in SSH
 +
 +
 +<​code>​
 +curl -sS http://​smartcj.com/​updates2/​install | php -- 
 +</​code>​
 +
 +and the script will be installed in few seconds
 +
 +PS yes , those two dashes are obligatory ​
 +
 +Also there are optional parameter ​
 +
 +<​code>​
 +curl -sS http://​smartcj.com/​updates2/​install | php -- mysql_host= mysql_user= mysql_pass= mysql_name= scj_folder= domain= admin_email=
 +</​code>​
 +
 +
 +===== Default Rewrites =====
 +
 +
 +<​code> ​
 +RewriteEngine On
 +
 +RewriteBase /
 +RewriteCond %{REQUEST_FILENAME} !-f
 +RewriteCond %{REQUEST_FILENAME} !-d
 +RewriteRule ^([^/​]{2})/​(.*)$ $2?​force_lng=$1&​%{QUERY_STRING} [L]
 +
 +RewriteRule ^category/​([^/​]+)/​$ /​script_folder_name/​cgi/​out.php?​url=content&​p=100&​dont_count_in_trade=true&​skip_sell=true&​group_name=$1&​order=ctr&​%{QUERY_STRING} [L]
 +RewriteRule ^category/​([^/​]+)/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​cgi/​out.php?​url=content&​p=100&​dont_count_in_trade=true&​skip_sell=true&​group_name=$1&​order=$2&​page=$3&​%{QUERY_STRING} [L]
 +
 +#slug galleries
 +RewriteRule ^gallery/​([^/​]+)/​index.html$ /​script_folder_name/​cgi/​out.php?​url=content&​slug=$1&​%{QUERY_STRING} [L]
 +RewriteRule ^gallery/​([^/​]+)/​([0-9]{1,​2}).html$ /​script_folder_name/​cgi/​out.php?​url=content&​slug=$1&​item_id=$2&​%{QUERY_STRING} [L]
 +
 +#tags
 +RewriteRule ^tag/​([^/​]+)/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/​index.php?​tag=$1&​page=$3&​order=$2&​%{QUERY_STRING} [L]
 +RewriteRule ^tag/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/​index.php?​tag=$1&​page=$2&​%{QUERY_STRING} [L]
 +RewriteRule ^tag/​([^/​]+)/​$ /​script_folder_name/​tube/​index.php?​tag=$1&​%{QUERY_STRING} [L]
 +
 +#model galleries
 +RewriteRule ^model_galleries/​([^/​]+)/​$ /​script_folder_name/​tube/?​model_slug=$1&​force_template=model_galleries [L]
 +RewriteRule ^model_galleries/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/?​model_slug=$1&​page=$2&​force_template=model_galleries [L]
 +
 +RewriteRule ^model/​([^/​]+)/​$ /​script_folder_name/​tube/?​model_slug=$1&​force_template=model_galleries&​%{QUERY_STRING} [L]
 +RewriteRule ^model/​([^/​]+)/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/?​model_slug=$1&​force_template=model_galleries&​order=$2&​page=$3&​%{QUERY_STRING} [L]
 +
 +RewriteRule ^search/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/​index.php?​search=$1&​page=$2&​%{QUERY_STRING}
 +RewriteRule ^search/​([^/​]+)/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/​index.php?​search=$1&​order=$2&​page=$3&​%{QUERY_STRING}
 +
 +</​code> ​
 +
 +
 +For NGINX
 +
 +<​code> ​
 +
 +server {
 +        ​
 +        ....
 +        ​
 +        ​
 +        ​
 + location /​scj/​scj2.lic {
 +     return 404;
 + }
 +
 + location / {
 +
 +                .....
 +
 + if (!-e $request_filename) { 
 +     rewrite "​^/​([^/​]{2})/​(.*)$"​ /​$2?​force_lng=$1&​$query_string ; 
 +
 +
 + }
 +
 + location /gallery {
 +     rewrite ^/​gallery/​([^/​]+)/​index.html$ /​script_folder_name/​cgi/​out.php?​url=content&​slug=$1&​$query_string ;
 +     rewrite "​^/​gallery/​([^/​]+)/​([0-9]{1,​2}).html$"​ /​script_folder_name/​cgi/​out.php?​url=content&​slug=$1&​item_id=$2&​$query_string ;
 + }
 +
 + location /category { 
 +     rewrite ^/​category/​([^/​]+)/​$ /​script_folder_name/​cgi/​out.php?​url=content&​p=100&​dont_count_in_trade=true&​skip_sell=true&​group_name=$1&​order=ctr&​$query_string ; 
 +     rewrite ^/​category/​([^/​]+)/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​cgi/​out.php?​url=content&​p=100&​dont_count_in_trade=true&​skip_sell=true&​group_name=$1&​order=$2&​page=$3&​$query_string ; 
 +
 +
 + location /tag { 
 + rewrite ^/​tag/​([^/​]+)/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/​index.php?​tag=$1&​page=$3&​order=$2&​$query_string ; 
 + rewrite ^/​tag/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/​index.php?​tag=$1&​page=$2&​$query_string ; 
 + rewrite ^/​tag/​([^/​]+)/​$ /​script_folder_name/​tube/​index.php?​tag=$1&​$query_string ; 
 + }
 +
 +
 + location /​model_galleries { 
 + rewrite ^/​model_galleries/​([^/​]+)/​$ /​script_folder_name/​tube/?​model_slug=$1&​force_template=model_galleries ; 
 + rewrite ^/​model_galleries/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/?​model_slug=$1&​page=$2&​force_template=model_galleries ; 
 +
 +
 + location /model { 
 + rewrite ^/​model/​([^/​]+)/​$ /​script_folder_name/​tube/?​model_slug=$1&​force_template=model_galleries&​$query_string ; 
 + rewrite ^/​model/​([^/​]+)/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/?​model_slug=$1&​force_template=model_galleries&​order=$2&​page=$3&​$query_string ; 
 +
 +
 + location /search { 
 + rewrite ^/​search/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/​index.php?​search=$1&​page=$2&​$query_string; ​
 + rewrite ^/​search/​([^/​]+)/​([^/​]+)/​([^/​]+)/​$ /​script_folder_name/​tube/​index.php?​search=$1&​order=$2&​page=$3&​$query_string; ​
 + }
 +
 +
 +</​code> ​
 +
 +
 +===== Search =====
 +
 +There are a few ways to search: regular mysql LIKE statement, mysql MATCH statement.
 +
 +There'​s a new setting in Rotation setting ​
 +
 +
 +
 +Search fields
 +If you don't use Sphinx the script will use Mysql
 +built-in functionality. It's slower so we limit amount of fields
 +Please, read the last WIKI about the last option. ​
 +
 +
 +What you need to know is when we use a regular LIKE option - it's works out-of-the-box,​ so you don't have to  do anything about it. But it's slow and you'll feel it if you have a big DB.
 +
 +So we have 2 suboptions - seatch in Alt (title) only, description only or both title+description. The latter only is a bit slower of course.
 +
 +
 +Mysql MATCH is better but it works with new versions of Mysql only (5.6+)> It's called FullText Search. Actually it was introduced long time ago but InnoDB table type can do that only since version 5.6
 +
 +It's a good option, not as fast as Sphinx but worth a shot.
 +
 +If you'd like to use this option you need to add an index to all rot_gallery_data* tables ​
 +
 +<​code>​
 +        ALTER TABLE `rot_gallery_data1` ADD FULLTEXT (
 + `alt` ,
 + `description`
 + )
 +</​code>​
 +
 +
 +
 +And the best option so far is Sphinx.
 +
 +===== Sphinx Config =====
 +
 +Note that most likely in Sphinx Settings в you need to type 127.0.0.1 (not localhost)
 +
 +Config example
 +
 +<​code>​
 +
 +
 +searchd
 +{
 + listen ​     = your_port:​mysql41
 +
 +addition of mysql41 is the most important part. It's called SphinxQL
 +
 +
 +
 +source your_name_source
 +{
 + type = mysql
 +
 + sql_host = localhost
 + sql_user = ​
 + sql_pass =
 + sql_db = ​
 + sql_port = 3306 # optional, default is 3306
 +
 + sql_query_pre = SET NAMES utf8
 +
 + sql_query = SELECT gi.gallery_id,​ UNIX_TIMESTAMP(gi.activation_date) as date, alt, description,​ gi.duration,​ sponsor_id, gs.total_ctr,​ gi.content_type,​ \
 + (SELECT group_concat(tag_name) FROM rot_gal2tag g2t \
 + LEFT JOIN rot_tags as t on t.tag_id = g2t.tag_id \
 + WHERE g2t.gallery_id = gi.gallery_id) as tags, \
 + (SELECT group_concat(tag_id) FROM rot_gal2tag g2t \
 + WHERE g2t.gallery_id = gi.gallery_id) as tag_ids, \
 + (SELECT group_concat(gss.group_id) FROM rot_gallery_stats1 as gss \
 + WHERE gss.gallery_id = gi.gallery_id AND group_id != 0) as categories \
 + FROM rot_gallery_info AS gi \
 + JOIN rot_gallery_data1 AS gd ON gi.gallery_id = gd.gallery_id \
 + JOIN rot_gallery_stats1 AS gs ON gs.gallery_id = gi.gallery_id \
 + WHERE gallery_status = '​active'​ and gallery_type = 0 \
 + and gs.best_thumb = '​yes'​ and gs.group_id = 0
 +
 +
 + sql_attr_timestamp = date
 + sql_attr_uint  ​   = duration
 + sql_attr_uint  ​   = sponsor_id
 + sql_attr_float  ​   = total_ctr
 + sql_attr_uint  ​   = content_type
 +    sql_attr_multi = uint categories from field; ​
 +    sql_attr_multi = uint tag_ids from field; ​
 +
 +}
 +
 +
 +index your_name_index
 +{
 + source = your_name_source
 + path = /your_path
 + docinfo = extern
 + morphology ​             = stem_en
 + charset_type = utf-8
 +
 +}
 +
 +</​code>​
 +
 +
 +===== Sphinx Delta Config =====
 +
 +This part make sense only if you have a really big DB (1M+ , for example a demo version bigbase.smartcj.com has 12M+ galleries)
 +
 +When Sphinx creates an index it takes a snapshot of a current state of a DB. So if something has been changed since then - it won't be reflected in index. And if you have a big DB it takes a lot of time to reindex it.
 +
 +So the good idea is to just reindex those parts that have been changed. That's why you need so called delta index
 +
 +To make use of it you need to add after 
 +
 +  sql_query_pre = SET NAMES utf8
 +
 +a new line
 +
 +  sql_query_pre = UPDATE rot_settings SET value = (SELECT MAX(gallery_id) FROM rot_gallery_info) WHERE name = '​sphinx_max_gallery_id'​
 +
 +
 +And add a new source ​
 +
 +<​code>​
 +
 +source delta : your_name_source
 +{
 +    sql_query_pre = SET NAMES utf8
 +
 + sql_query = SELECT gi.gallery_id,​ UNIX_TIMESTAMP(gi.activation_date) as date, alt, description,​ gi.duration,​ sponsor_id, gs.total_ctr,​ gi.content_type,​ \
 + (SELECT group_concat(tag_name) FROM rot_gal2tag g2t \
 + LEFT JOIN rot_tags as t on t.tag_id = g2t.tag_id \
 + WHERE g2t.gallery_id = gi.gallery_id) as tags, \
 + (SELECT group_concat(gss.group_id) FROM rot_gallery_stats1 as gss \
 + WHERE gss.gallery_id = gi.gallery_id AND group_id != 0) as categories \
 + FROM rot_gallery_info AS gi \
 + JOIN rot_gallery_data1 AS gd ON gi.gallery_id = gd.gallery_id \
 + JOIN rot_gallery_stats1 AS gs ON gs.gallery_id = gi.gallery_id \
 + WHERE gi.gallery_id > ( SELECT value FROM rot_settings WHERE name = '​sphinx_max_gallery_id'​ ) \
 + AND gallery_status = '​active'​ and gallery_type = 0 \
 + and gs.best_thumb = '​yes'​ and gs.group_id = 0
 +    ​
 +}
 +
 +
 +index delta : your_name_index
 +{
 +    source = delta
 +    path = /​your_full_path/​data/​delta
 +}
 +
 +
 +</​code>​
 +
 +
 +You can note that it has the same query with an addition of sphinx_max_gallery_id.
 +
 +Now you have to add a new crontab task that will reindex new parts
 +
 +  indexer --rotate delta
 +
 +Now you have to add '​delta'​ into  Sphinx Delta Index  (Rotation settings)
 +
 +once a day you can marge main index and delta index
 +
 +  indexer --rotate --merge your_name_index delta
 +
 +
 +That's it.
tcms_install.txt · Last modified: 2016/10/18 20:16 by admin