No data in Statistics pages

This applies since MOR X7.

Run following commands to fix it.
For MOR:

svn update /usr/src/mor

For M2:

svn update /usr/src/m2

Then:

service elasticsearch restart

Wait about one minute for Elastiscearh to start and then run command

elasticsearch resync full

ES (ElasticSearch) is a tool that we use to show statistics in MOR.
ES Sync Status shows how accurate is ES data compared to MySQL data.
ES periodically fetches data from MySQL.
Since this action is not in real time, some miss match is possible when you have active calls.
This miss match should be only noticeable when checking newest calls.
Eventually all calls should be synchronized between ES and MySQL. Small percentage miss match is normal.

If not all calls are synchronized

Check ElasticSearch count:

elasticsearch count

[root@CentOS-68-64-minimal ~]# elasticsearch count
{
 "count" : 956528,
 "_shards" : {
   "total" : 5,
   "successful" : 5,
   "failed" : 0
 }

Then check how many calls are in database. ElasticSearch count and value from database should be almost the same:

mysql> select count(*) from calls;
+----------+
| count(*) |
+----------+
|  1056672 |
+----------+

If for some reason there is a mismatch, run:

curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "threadpool.bulk.queue_size" : 100 } }'

elasticsearch resync

Possible reason of mismatch: https://stackoverflow.com/questions/20683440/elasticsearch-gives-error-about-queue-size

Check synchronized calls for specific period

Run script with the needed date:

/usr/src/mor/sh_scripts/elasticsearch/es_sync_check.sh 2019-05-01 2019-05-31

You will get similar output:

Checking period 2019-05-01 2019-05-31

2019-05-01 6066 6066 OK
2019-05-02 349448 349448 OK
2019-05-03 399457 399457 OK
2019-05-04 17114 17114 OK
2019-05-05 125 125 OK
2019-05-06 341779 405967 MISMATCH!
2019-05-07 325504 355798 MISMATCH!
2019-05-08 340145 369193 MISMATCH!
2019-05-09 346800 375679 MISMATCH!
2019-05-10 346138 382082 MISMATCH!
2019-05-11 12850 12850 OK
2019-05-12 1021 1021 OK
2019-05-13 323383 343793 MISMATCH!
2019-05-14 399852 463003 MISMATCH!
2019-05-15 386584 440355 MISMATCH!                                                                                                                                
2019-05-16 394110 450554 MISMATCH!                                                                                                                                
2019-05-17 364207 441448 MISMATCH!                                                                                                                                
2019-05-18 29805 29815 MISMATCH!                                                                                                                                  
2019-05-19 1091 1091 OK                                                                                                                                           
2019-05-20 350018 524564 MISMATCH!                                                                                                                                
2019-05-21 310804 486752 MISMATCH!                                                                                                                                
2019-05-22 295253 423898 MISMATCH!                                                                                                                                
2019-05-23 362126 458598 MISMATCH!                                                                                                                                
2019-05-24 380011 423197 MISMATCH!                                                                                                                                
2019-05-25 100113 101103 MISMATCH!                                                                                                                                
2019-05-26 3343 3343 OK                                                                                                                                           
2019-05-27 381988 465975 MISMATCH!                                                                                                                                
2019-05-28 402943 472638 MISMATCH!                                                                                                                                
2019-05-29 399595 470119 MISMATCH!                                                                                                                                
2019-05-30 380600 440080 MISMATCH!                                                                                                                                
2019-05-31 324217 352097 MISMATCH!                                                                                                                                
                                                                                                                                                                 
ES sync check completed

OK - means that calls are synchronized correctly in this date.

MISMATCH! - means that you need to re-synchronize this date.

Re-synchronize calls for specific period

If there are many calls in database, for example, 50 or 100 millions, re-synchronization might take few hours to become completed. If you are generating invoices, checking statistics for example, one month, 2 weeks etc then you can try to re-synchronize only this specific period. To do so, firstly, restart elasticsearch.

service elasticsearch restart

Make sure that all shards are loaded successfully.

elasticsearch count
{
 "count" : 956528,
 "_shards" : {
   "total" : 5,
   "successful" : 5,
   "failed" : 0
 }

Total must be the same as Successful. If it shows that:

elasticsearch count
{
 "count" : 956528,
 "_shards" : {
   "total" : 5,
   "successful" : 2,
   "failed" : 0
 }

Wait 10-15 minutes and re-check again. If it still shows that Successful shard quantity is lower than total, full re-sync must be done, as in the beginning of manual.
If all shards are loaded successfully, then write

elasticsearch resync interval "2018-12-01 00:00:00" "2018-12-31 23:59:59"

It will re-synchronize calls only for December.

Invoice cannot be generated due to disposition integrity errors

Try to delete all ES data and resync all calls.

rm -rf /var/lib/elasticsearch/cluster_*
service elasticsearch restart

Wait for 1-2 minutes. Then resync all calls.

elasticsearch resync full