Testing PHP: Ropes to Skip, Ropes to Know

Recently, I spent a rather long evening tracking down a PHP application bug that appeared, unexpectedly, in a production environment. While "spelunking" the logs and other data, I asked the DevOps lead whether he had run the "run-tests.php" file on the installed PHP engine. Surprisingly, the DevOps guy gave me a look seem most often from deer just before they are hit by an oncoming semi.

PHP Engine Testing

DevOps practitioners are flooded with data and monitoring logs from their cluster and container orchestration suites. But there is something to be said for keeping things simple and doing the most direct and obvious tests first. Many DevOps folks tend to assume their Docker/K8s container PHP engines are installed/configured correctly (php.ini) and working as advertised.

Many also assume their "php.ini" configuration files have already been 'pruned' and 'streamlined' to match their web application's specific need. But, few have actually tested their php.ini file to confirm their presumptions. Running a test of the PHP engine directly, using the "run-tests.php" file is, in my humble opinion, a better alternative to assuming the php.ini file is configured correctly; following the "trust, but verify." motto."

Steps to Using the PHP Tests File

The steps to getting the 'run-tests.php' file that matches your installed PHP engine is very straight-forward.

How to Get and Run the PHP Test File

Obtaining the PHP test files for your installed PHP engine can be done directly from the command line.

Find your installed PHP executable

Simply type: "which php" at your command prompt (Linux/OSX) and you'll get the complete path to the PHP executable, typically located in the "/usr/local/bin/php' folder on most machines. Some PHP installations done for LAMP package installs may place it elsewhere.

$ which php $ /usr/local/bin/php

Get php.ini file info by using the phpinfo() from the command line

Next, at the command prompt enter the following command to get the contents of the php.ini configuration file for the installed PHP executable:

$ php -i

While this puts the entire phpinfo() function call output directly into the terminal window as text, it does not require using a web browser or a installed web server. Again, the command output will give you the location of your php.ini file which you should note if you need to confirm you are running your tests against the correct PHP configuration.

Locate PHP Engine version and build date

Next, assuming you did not build your installed PHP engine from source tarballs, then you need to scroll back through the output from the "php -i" command, and locate the first two lines which gives you the installed PHP Engine's version and build date:

PHP Verion => 5.5.38

Build Date => Feb 11 2018 23:42:27

Download and Unzip the PHP Engine source code

Next, using the PHP Version info from the previous command, use wget or curl to download the PHP engine source code from the PHP.net website. The command below is based on the info from the command above. Replace the 'php-5.5.38' slug in the following command example with your installed PHP version info:

$wget http://www.php.net/distributions/php-5.5.38.tar.gz

Next, unzip the downloaded zipped tarball using the following command. This will create a folder with the same name as the PHP Engine version: php-5.5.38

$tar zxvf php-5.5.38.tar.gz  

Locate, setup, and run the 'run-tests.php' file

Next, navigate into the folder created by the 'tar ..' command, and there you'll find the "run-tests.php" file among several files and folders.

Before you can run the PHP test files, you must set an environment variable to point to the full path of your installed PHP engine:

$export TEST_PHP_EXECUTABLE="/usr/local/bin/php"

Test the environment variable you just created using the echo command:

$ echo $TEST_PHP_EXECUTABLE

The echo command output should match the full path you got from the "$which php" command you executed earlier.

Now, you can execute the "run-tests.php" file using your installed PHP engine using the following command:

$ php .run-tests.php

Depending on your processor and memory in your machine, the tests should take no more than 10 - 15 minutes to run.

Be Patient, Wait for the Results

PHP engine tests are very extensive and quite exhaustive: over 13,000 tests of the PHP 5 engine alone. As a rule, these tests are run when the PHP engine is built from source. But, the 'run-tests.php' file can be run anytime and should be run whenever the php.ini file is changed, or when a module is added to it. Even if you install your PHP engine using a binary or one of the LAMP packages, run the tests and check the results.

The tests are extensive and can take time. Options to cut down the tests are available and the output is a basic listing of "PASS" and "FAIL" results. But the first time, it's a good practice to run all the tests at least once to give you a baseline. The "run-tests.php' file, while simple in structure, can give you very thorough, diagnostic view of the installed PHP engine, and it even includes a set of "Expected To Fail" tests.

TL;DR - Road Not Taken

DevOps practitioners who have so many other more sophisticated, GUI testing tools, may feel using a command line tool is a step backwards. Others may not do it because they are not familiar with these tests, or know how to read and interpret the test results. Opting to 'run-tests.php' file against your PHP engine will not only make you familiar with your PHP engine's internals, but it will also give you specific warnings and errors in your PHP installation that could go unnoticed. Many of the tests will not seem specific to your web application needs.

Clearly, running a script containing runs over 13,000 tests and outputs "PASS" or "FAIL" could seem like overkill, for each test prints details and references to fixes or reported bugs. Volumes of text output scrolling rapidly across the terminal is a clear candidate for TL;DR treatment. This may explain why DevOps practitioners who know about 'run-tests.php' avoid using it. This TL;DR 'road not taken' attitude however, might not be the best decision and in my case, running the tests did help track down a PHP application bug more quickly with less overhead.

Cutting Down the Terminal Clutter

If you are annoyed by the tsumani of terminal output, a judicious use of command line tools like 'grep', 'awk', and 'sed' can cut the flood of test result text down to size. Also, with a bit more work, and review of the available PHP engine test documentation, DevOps folks can even create their own customized tests, specifically designed for their web application environments without much heavy lifting.

But even without piping the test output through the standard grep, awk, and sed CLI text wrangling utilities, as the following PHP test run listing shows, this tool provides a clear, concise, and understandable test results summary. The results are also annotated with bug report numbers, highlighted by "#" with hyperlinks to the bug reports, so one can get more detailed descriptions of what the test found and what it is attempting to diagnose.

=====================================================================
TEST RESULT SUMMARY
---------------------------------------------------------------------
Exts skipped    :   20
Exts tested     :   60
---------------------------------------------------------------------

Number of tests : 13459             11467
Tests skipped   : 1992 ( 14.8%) --------
Tests warned    :    4 (  0.0%) (  0.0%)
Tests failed    :   51 (  0.4%) (  0.4%)
Expected fail   :   45 (  0.3%) (  0.4%)
Tests passed    : 11367 ( 84.5%) ( 99.1%)
---------------------------------------------------------------------
Time taken      :  915 seconds
=====================================================================

=====================================================================
EXPECTED FAILED TEST SUMMARY
---------------------------------------------------------------------
Test open_basedir configuration [tests/security/open_basedir_linkinfo.phpt]  XFAIL REASON: BUG: open_basedir cannot delete symlink to prohibited file. See also
bugs 48111 and 52176.
Inconsistencies when accessing protected members [Zend/tests/access_modifiers_008.phpt]  XFAIL REASON: Discussion: http://marc.info/?l=php-internals&m=120221184420957&w=2
Inconsistencies when accessing protected members - 2 [Zend/tests/access_modifiers_009.phpt]  XFAIL REASON: Discussion: http://marc.info/?l=php-internals&m=120221184420957&w=2
Bug #48770 (call_user_func_array() fails to call parent from inheriting class) [Zend/tests/bug48770.phpt]  XFAIL REASON: See Bug #48770
Bug #48770 (call_user_func_array() fails to call parent from inheriting class) [Zend/tests/bug48770_2.phpt]  XFAIL REASON: See Bug #48770
Bug #48770 (call_user_func_array() fails to call parent from inheriting class) [Zend/tests/bug48770_3.phpt]  XFAIL REASON: See Bug #48770
Bug #64896 (Segfault with gc_collect_cycles using unserialize on certain objects) [Zend/tests/bug64896.phpt]  XFAIL REASON: We can not fix this bug without a significant (performace slow down) change to gc
Fixed Bug #65784 (Segfault with finally) [Zend/tests/bug65784.phpt]  XFAIL REASON: This bug is not fixed in 5.5 due to ABI BC
Initial value of static var in method depends on the include time of the class definition [Zend/tests/method_static_var.phpt]  XFAIL REASON: Maybe not a bu

PHP Engine Testing: Next Steps

After running the tests and looking through the test output, you may discover its hidden value and want to explore how to use this tool in more detail. Valuable guidance and information about the 'run-tests.php' file, what it contains, and how to write custom tests, can all be found on the "PHP Internals" web site.