These Perl modules must installed to be able to use the test harness:

• ExtUtils::MakeMaker
• Test
• Apache::Test

The Bundle::ApacheTest CPAN module can be installed to gather many dependencies of Apache::Test:

perl -MCPAN -e 'install Bundle::ApacheTest'

Note: Crypt::SSLeay requires OpenSSL to be installed (only required for t/TEST -ssl): https://www.openssl-library.org/ More accurate results may be obtained by using the same openssl command line and libraries as consumed by APR-util and mod_ssl, due to X.509 formatting behavior differences.

For an extensive documentation see https://perl.apache.org/docs/general/testing/testing.html

To run the tests for all Apache web server modules, some additional CPAN modules will be required. If the tests don't work, make sure that you have up to date versions of each of these perl modules:

cpan App::cpanminus
cpanm Bundle::ApacheTest
cpanm HTTP::DAV DateTime Time::HiRes \
  Test::Harness IO::Socket::SSL \
  IO::Socket::IP IO::Select LWP::Protocol::https AnyEvent \
  AnyEvent::WebSocket::Client LWP::Protocol::AnyEvent::http FCGI

If the Bundle::ApacheTest fails due to native compilation issues with Crypt::SSLeay, you can install the remaining packages:

cpanm --quiet --showdeps  Bundle::ApacheTest|grep -v Crypt::SSLeay|xargs cpanm

If you don't care how it works and just want to run the tests:

1. Clone this repository:

   git clone --recurse-submodule https://github.com/apache/httpd-tests

2. You need an installation of Apache httpd (2.4.x or trunk).
3. Any DSOs you wish to use should be configured in that Apache's httpd.conf (the test harness will pick this configuration up).
4. Setup:

   perl Makefile.PL -apxs /path/to/apache/bin/apxs

5. Run the tests:

   t/TEST

6. Evaluate test output.

The test harness will run every .t file under the t/ directory. To run only the tests for PHP:

t/TEST -httpd /path/to/apache/bin/httpd t/php

That will start the test server, run the .t tests under t/php and shut down the test server. You can also control each of these steps individually.

Start the test server:

t/TEST -httpd /path/to/apache/bin/httpd -start

Run the PHP tests in the test environment:

t/TEST t/php

Stop the test server:

t/TEST -stop

Run the server under gdb (using -X):

t/TEST -d gdb

Note: At this point, you have a working test environment. You can look in t/conf for the test server configuration files. These are generated by the test harness. Once you have a working test environment, you do not need to specify httpd on the t/TEST command line. For instance, to start the server up again:

t/TEST -start

For a full regression test, you should have all modules loaded. Build the server with:

configure --enable-modules=reallyall --enable-load-all-modules ...

among other things. Edit the generated httpd.conf and comment all mpm modules that you do not want. Run t/TEST -clean again.

You will see some:

skipped: cannot find module 'XXX'

as not all modules are in every Apache release (but the tests run for all).

All in all, some >4k tests will run and the result needs to be: PASS

If you have a "PASS" at the end of t/TEST, congratulations! If not, this section gives some advice in order to find the cause.

0. If your test startup hangs forever in "waiting for server to warm up", but the test server is reachable under port 8529, you might be the victim of IPv4/6 confusion. The default servername configured is localhost and some operating systems define 127.0.0.1 as well as ::1 in /etc/hosts. If the test server listens only on 0.0.0.0 it might not answer requests to ::1 and that causes the test startup to hang. Solution: comment the ::1 line in /etc/hosts and see if that improves matters.

1. Run t/TEST -clean every time you change something in your Apache configuration. The test suite picks up certain things from your installed httpd.conf (such as LoadModule statements) and will not see your changes unless you clean it.

2. Failures in proxy.t may originate from the fact that the test script cannot open the specified port. This happens on some machines if you abort a test run and the socket is not properly shut down. Check if the error goes away after a reboot. (proxy.t tests are slow, so chances you interrupt tests at that point are good.)

3. Failures in access.t may result from reverse lookups not working or giving other answers than expected. In the case 0 above, if the test client connects via 127.0.0.1, a "Grant for localhost" might resolve to ::1 and therefore will not match the access rules of the tests. Solution: check that your servername is localhost (which is the default) and that it always resolves to 127.0.0.1.

4. If some ssl test cases fail, especially when t/ssl/proxy.t fails, the reason can be mismatches between your installed SSL library and the one httpd uses. The openssl binary found in your $PATH is used to create binary setup files by t/TEST. If another version of openssl then tries to read these from your Apache server process, it might fail. Try:

   t/TEST -clean
   PATH=<bin dir of correct openssl>:$PATH t/TEST
   

   If a lot of ssl tests fail, check in the error log for the presence of a certificate validation error. If you find it, check the expiration date of the TLS/SSL certificates used by the tests; they might be expired. Running t/TEST -clean should delete the old ssl certificates, so they'll be regenerated during the next run.

5. If you see failures in the modules/h2.t test cases, please notify the dev mailing list with a description of your setup. These tests are quite young, currently only valid in 2.4.x and later and interact with quite some other modules as well as OpenSSL versions installed. Some tests require mod_ssl so make sure to load it in the httpd conf.

6. Segmentation faults and core dumps occurring while executing the test suite might indicate a real problem but always run again the tests after a clean make install to avoid inconsistencies from old objects.

7. If you see error messages like "Parse errors: Bad plan. You planned X tests but ran Y." it usually means that you are missing a perl module or the tested httpd module depends on another one not loaded in the httpd config.

8. If you see SSL certificate errors, remove t/conf/ssl/ca prior to t/TEST -clean.

9. perl 5.28 in MacOS homebrew seems to hang the test suite. Invoking /usr/bin/perl Makefile.PL -apxs ... will cause an older perl to be used.

Sometimes it's possible that the test is passing properly for the first time, but when you run it again, the test might fail. It's important to run repetition smoke testing. For example to repeat the tests 5 times:

t/SMOKE -times=5

It's also possible that a test will pass when it's run after a particular test, but if moved to run after a different state it may fail. For this reason by default the tests run in random order.

Since it's important to be able to reproduce the problem with random testing, whenever -order=random is used, the used seed is printed to STDERR. It can then be fed into future tests via the APACHE_TEST_SEED environment variable.

By adding the option -order=repeat, the tests will be run in alphabetical order.

Combining these two important smoke testing techniques, one can run tests with:

t/SMOKE -times=N -order=(repeat|random)

For example, to run the mod_rewrite tests 5 times:

t/SMOKE -times=5 -verbose t/modules/rewrite.t

The tests can be repeated N times, and run in the following modes:

• randomize all tests
• repeat the whole test suite N times

For configuration options and default settings run:

t/SMOKE -help

For more information refer to the Apache::TestSmoke manpage.

The test server is configured with conf files like any normal Apache server. The tricky part is those conf files are generated by the harness just prior to starting the server. t/conf/httpd.conf is generated by t/conf/httpd.conf.in. If that does not exist, the harness will generate a working configuration and will include LoadModule (and AddModule for Apache 1.3) directives from the httpd.conf associated with the httpd binary you are using for testing.

If t/conf/extra.conf.in exists, t/conf/extra.conf will be generated from that, and an Include directive for that file will be put in the generated t/conf/httpd.conf.

t/conf/apache_test_config.pm is generated from the test configuration. It contains all the information about the configuration of your test server. You can access this information in test scripts by:

my $env = Apache::TestConfig->thaw;

Apache::TestConfig accesses apache_test_config.pm and returns a hash reference with all the information. Look through apache_test_config.pm; it's a lot of stuff. Once these conf files are generated, you have a working test environment, and they must be 'cleaned' if you wish to make changes to them:

t/TEST -clean

(Now you will have to specify your httpd binary when starting back up again.)

For more information on using the test harness and writing tests, see the README in Apache-Test and the examples in Apache-Test/t.

The test harness was originally written by Doug MacEachern and is discussed on the httpd dev mailing list.

It is also included in modperl-2.0 source along with tests for modperl-2.0.