HHVM fastcgi + Nginx performance fluctuations-open source projects facebook/hhvm
Currently we started using HHVM in a production envirionment and so far almost all results are pretty inpressive. Our overall transaction rate is greatly improved compared to PHP-FPM with APC. Almost all requests are under 500ms however every couple of requests (5 to 10 or so) results in a request time of 2 to even 5 seconds.
The page requested does not seem to make any difference and also requesting the same page over and over again will trigger this behavior within a couple of requests.
We are running HHVM in server mode with the following command line options:
/usr/bin/hhvm --mode server -vServer.Type=fastcgi -vServer.FileSocket=/usr/local/php55/sockets/admin.sock -vPidFile=/var/run/hhvm/admin.pid -vEval.Jit=true -vServer.ThreadCount=24 -vServer.APC.EnableApc=true
We are running nginx for the webserver with these relevant configurations (I am sorry if i forgot something importand here).
fastcgi_buffer_size 128k;
fastcgi_buffers 256 16k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_read_timeout 240;
fastcgi_intercept_errors on;
The server has 128GB of memory and 24 cores (hyperthreading so actually 12).
We did a fair bit of searching on https://github.com/facebook/hhvm/wiki/Runtime-options however most options aren’t explained very well so I have no idea what they do and testing them in a production environment is a bit scary.
Had anybody here had a similar problem or could maybe point me in a direction with some of the HHVM options I would be very grateful.
The HHVM version used is from http://www.hop5.in/yum/el6/
HipHop VM 3.0.1 (rel)
Compiler:
Repo schema: e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
And the the /etc/hhvm/config.hdf
Log {
Level = Warning
AlwaysLogUnhandledExceptions = true
RuntimeErrorReportingLevel = 8191
}
MySQL {
TypedResults = false
}
We are using supervisord to start HHVM so here the supervisor config as wel:
[program:hhvm]
stopasgroup=true
killasgroup=true
command=/usr/bin/hhvm --mode server -vServer.Type=fastcgi -vServer.FileSocket=/usr/local/php55/sockets/admin.sock -vPidFile=/var/run/hhvm/admin.pid -vEval.Jit=true -vServer.ThreadCount=24 -vServer.APC.EnableApc=true
user=admin
stdout_logfile=/var/log/hhvm/admin.log
stderr_logfile=/var/log/hhvm/admin.error.log
directory=/home/admin
umask=000
There is a php.ini in /etc/hhvm/php.ini but the content is empty. Also the pages tried all do some database connectivity but this is usually very minimal. For a complete picture also the my.cnf. The mysql version used is percona
[mysql]
# CLIENT #
port = 3306
socket = /var/lib/mysql/mysql.sock
[mysqld]
# GENERAL #
user = mysql
default-storage-engine = InnoDB
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysql.pid
[mysqld]
# MyISAM #
key-buffer-size = 32M
myisam-recover = FORCE,BACKUP
# SAFETY #
max-allowed-packet = 128M
max-connect-errors = 1000000
# DATA STORAGE #
datadir = /var/lib/mysql/
# BINARY LOGGING #
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 14
sync-binlog = 1
# CACHES AND LIMITS #
tmp-table-size = 128M
max-heap-table-size = 256M
query-cache-size = 10G
max-connections = 1000
thread-cache-size = 100
open-files-limit = 65535
table-definition-cache = 4096
table-open-cache = 4000
join-buffer-size = 1M
# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 512M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 73G
# LOGGING #
log-error = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes = 1
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log
Mysql version:
innodb_version 5.6.17-65.0
protocol_version 10
slave_type_conversions
version 5.6.17-65.0-56-log
version_comment Percona Server (GPL), Release 65.0, Revision 587
version_compile_machine x86_64
version_compile_os Linux