Sudden PHP ERROR & Slow Application

Hi,

I have been experiencing an issue lately with my servers/application, the development team is unable to find the root cause. No significant changes were made to the application or servers. The issue appeared suddenly.

Infrastructure

• 1 web server; Fedora 21, PHP, 2GB RAM

• 1 database server; Fedora 21, MySQL, 2GB RAM

Web Server

• Fedora 21

• Server version: Apache/2.4.10 (Fedora)

• PHP 5.6.7 (cli) (built: Mar 20 2015 06:12:07) Zend Engine v2.6.0, Copyright © 1998-2015 Zend Technologies with Xdebug v2.3.1, Copyright (c) 2002-2015, by Derick Rethans

Background

1) The servers are used for a native iphone application

2) An SSL was installed on Feb 17th

3) The servers were working effectively, last app testing was Feb 19th

4) No significant changes were made to the application or server after Feb 19th

Problem

1) After the Linode Xen Upgrade on Mar 7th & 8th, the server response slowed significantly

2) I am not saying that the Linode Xen upgrade is the cause but it is the only item remaining after troubleshooting

3) The extreme slowness was first noticed on Mar 11th

Issue 1: Upload Images

The app allows a user to upload a maximum of 10 images per post. The user can upload images from their Camera Roll, Google Images, or Take a Photo.

Use Case 1:

If you upload images from your Camera Roll or Google Images, the image size is usually 1MB or lower. In this case all 10 images are uploaded with no issues. However, the upload can take approximately 20-30 seconds. Sometimes it can go to 1 minute.

Use Case 2:

1) If you upload images from Take Photo (iPhone), a single image size is on average 5MB. The app seems to work with uploading 3-4 images without issue, but it takes more than 1 minute for the upload to be successful.

2) If you upload 5 images the system seems to terminate the connection. This is where you can check the Error Log and find a PHP Error. However, there are times where the app just crashes with no error.

3) If you upload 6 or more images the app crashes and no errors are reported in the error log.

Issue 2: Slow Downloads

The app downloads other user posts to your Home Page, these are users you are following. The app also downloads your own posts to your Profile Page. Both the Home Page and Profile Page are slow. The pages can take anywhere from 30-60 seconds to load.

PHP ERROR:

This PHP Error occurs only once in a while when the application does not crash but seems to have the connection terminated.

When the app does crash there is no Errors in the Error Log

When the app does crash, you can just open the app, then try to post again (posting more than 5 images usually crashes the app)

[Tue Mar 31 20:29:43.744128 2015] [:error] [pid 23957] [client 24.114.67.149:4894] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 39868997 bytes) in Unknown on line 0

Actions

Based on the PHP Error above, I have tried to test different configurations within the php.ini file with no success.

The available memory on the server under load is 1.7GB (total of 2GB)

The available CPU on the server under load is 97-98%

The load testing was conducted with JMETER

PHP.INI CONFIG TESTS

memory limit = 128M change to 256M, 512M, 1024M, and -1 (-1 was used for a test only, the default of 128M was re-entered as the issue remained)

post max size = 100M change to 512M

upload max filesize = 2M change to 5M, then 10M

max file uploads = 20 change to 50

enable realpathcachesize = 16k (these are disabled by default, I enabled and changed to 64k)

enable realpathcachettl = 120

Please let me know if you have any suggestions. I have tried to work with Linode support as well. I have provided the Linode support team with outputs for different performance metrics/commands and they have concluded that the RAM and CPU are sufficient.

Also, does anyone know how to check if APC or OpCache installed with PHP? I would like to know if either is installed and then perhaps install OpCache to see if there would be any benefits.

Thanks,

Ray