How I made the integration tests work for 2 frameworks.

Introduction

Since I'm working on the Apie library and make it work in 2 frameworks (Symfony and Laravel) one of the issues I ran into were the integration tests. The main reason is that both frameworks provide a different testing class that they advise you to use:

For example Symfony integration tests typically extend a specific TestCase class:

// tests/Service/NewsletterGeneratorTest.php
namespace App\Tests\Service;

use Symfony\Bundle\FrameworkBundle\Test\KernelTestCase;

class NewsletterGeneratorTest extends KernelTestCase
{
    public function testSomething(): void
    {
        self::bootKernel();

        // ...
    }
}

The same can be said for Laravel:

use Illuminate\Foundation\Testing\WithoutMiddleware;
use Illuminate\Foundation\Testing\DatabaseTransactions;
 
class ExampleTest extends TestCase
{
    /**
     * A basic functional test example.
     *
     * @return void
     */
    public function testBasicExample()
    {
        $this->visit('/')
             ->see('Laravel 5')
             ->dontSee('Rails');
    }
}

Even if PHP would introduce multiple inheritance you can not just combine both base classes. And to make it worse is that the naming is so completely different that it means you can not just easily copy, paste a test for the other framework. For every feature I would need to write every integration tests twice! And it would get even worse if I would decide to add even more frameworks on the list.

A solution would be to write a dummy application written in Symfony and a dummy application written in Laravel that use the current branch code in vendor and do http requests (I could use symfony/browser-kit for this). However starting the application and keeps it in a stable setup or running multiple tests at the same time is error prone. It becomes even less feasible the moment you have to keep multiple applications running with multiple configurations.

Ideally you would write one tests with a data provider and provide all available frameworks. But this does not exist. So let's write it!

Shared interface

The ideal test case would be the use of data providers with a class that can boot the application.

So the test for Laravel and Symfony is handled by the same test with a structure like this:

class ATest {
    public function it_registers_an_apie_service_provider(): Generator
    {
        yield [new SymfonyTestApplication(new ApplicationConfig(templatingDisabled: true))];
        yield [new SymfonyTestApplication(new ApplicationConfig(templatingDisabled: false))];
        yield [new LaravelTestApplication(new ApplicationConfig())];
    }

    /**
     * @dataProvider it_registers_an_apie_service_provider
     * @test
     */
    public function it_registers_an_apie_service(TestApplicationInterface $testApplication)
    {
        $testApplication->bootApplication();
        $apieService = $testApplication->getServiceContainer()->get('apie');
        $this->assertInstanceOf(ApieFacade::class, $apieService);
        $testApplication->cleanApplication();
    }
}

So in the above situation I write an integration test that checks if my library created a service with service id 'apie' that is of class ApieFacade.

Now the only problem is how to make a SymfonyTestApplication and LaravelTestApplication class.

The ApplicationInterface

I ended up with this interface:

interface TestApplicationInterface
{
    public function bootApplication(): void;

    public function getServiceContainer(): \Psr\Container\ContainerInterface;

    public function cleanApplication(): void;

    public function getApplicationConfig(): ApplicationConfig;

    public function httpRequestGet(string $uri): \Psr\Http\Message\ResponseInterface;

    public function httpRequest(TestRequestInterface $testRequest): \Psr\Http\Message\ResponseInterface;
}

bootApplication and cleanApplication are called in the test, because we do not want to initialize the application in the data provider for obvious reasons.

getServiceContainer() and getApplicationConfig() can be used to make some assumptions which configuration is used.

httpRequestGet() and httpRequest() are used to test requests. The reason I use a custom TestRequestInterface is so I can write generic tests, for example a Rest API test can follow a very strict pattern.

Now that I made the interface it was time to make the actual implementations.

Symfony test application

I did make integration tests without the one offered by Symfony as it is relatively easy to make it work.  In general Symfony bootstraps itself like this (nowadays there is symfony/runtime package to make it more maintainable at the cost of less clear how it works):

$kernel = new AppKernel();
$response = $kernel->handle(Request::createFromGlobals());
$response->send();

We do not need to send the response, so we can skip this part and just test the response object the handle() method returned. 

If we run this kernel with only our bundle however it will fail, as it will complain that a 'http_kernel' service is missing. A fast google search shows that the Symfony framework bundle is required so we include this.

Another thing is related to apie/cms and the configurable dashboard page. Most Symfony applications use twig as template engine, but it is also possible to install Symfony without twig by not including TwigBundle.. We want to test the dashboard in both situations  so we create an ApplicationConfig object that tells the SymfonyKernel whether it should include the TwigBundle or not.

We also added hooks for the symfony security component, but this is also not included in the framework bundle, so we add the security bundle as well.

Kernel::getCacheDir()

If you create your test kernel and just extend Kernel you notice that it will create the cache files inside the same folder which makes it a messy folder structure. The reason for that is that we did not modify Kernel::getCacheDir(). We can follow different strategies and it effects how integration tests will function:

class TestKernel extends Kernel
{
    private string $cacheDir;
    public function __construct(private array $apieConfiguration)
    {
    	$this->cacheDir = sys_get_temp_dir() . DIRECTORY_SEPARATOR . uniqid(__CLASS__);
        parent::__construct('test', true);
    }
    public function getCacheDir(): string
    {
        return $this->cacheDir;
    }
    
    public function __destruct()
    {
        system('rm -rf ' . escapeshell_arg($this->cacheDir));
    }
    
    // ... other methods
}

The code above will let every integration test use its own cache directory, but this also means a significant performance hit as not a single test has anything in it's caching. Also if I want to check the contents of the generated cache folder I need to disable the __destruct method since the cache path is removed when the test is completed.

So I changed the getCacheDir() to follow a different strategy. We can json encode the entire config as array and use md5 around it so it is impossible to have exploits here.

class TestKernel extends Kernel
{
    public function __construct(private array $apieConfiguration)
    {
        parent::__construct('test', true);
    }
    public function getCacheDir(): string
    {
        return sys_get_temp_dir() . DIRECTORY_SEPARATOR . md5(json_encode([PHP_VERSION_ID, $this->apieConfiguration]));
    }
    
    // ... other methods
}

Because the kernel cache folder is shared now if you reuse the same configuration I can no longer keep the __destruct() method as it is because the PHP garbage collector can run it at any time including the next test case. I also use the current PHP version in the checksum so it is possible to even run the integration tests in multiple PHP versions at the same time.

A downside is that the measured code coverage could be different depending if the cache exists as some methods are only called by Symfony if the cache is missing or stale.

Laravel test application

I had less experience with testing Laravel applications. I worked with orchestra/testbench before. The setup is atrocious. It does its testing with a dummy application that is being installed as a composer package. Because this is always the same path you can not run multiple instances with different configurations.

Still I tried to do it without orchestra/testbench and failed miserably. While Symfony requires only a cache folder, Laravel assumes files on specific places. Also laravel facades need to be instantiated as many laravel packages, but also the framework itself uses lots of global 'helper' methods and facades to run.

So in the end I did use orchestra/testbench. I made a class that extends the TestCase class made in orchestra/testbench, but instead I implemented the methods found in the ApplicationInterface and just made it call the methods that exist in the TestCase class. Nasty, but it gets the job done.

The boot/register issue with service providers

The only issue I ran into is with the configuration overwriting part of orchestra/testbench and that we register our services as singletons. It took me a lot of debugging to figure out why adding event subscribers to the events service made the routing use the default apie configuration instead of the one I was overwriting.

As it turned out: when you write a service provider for laravel to register services you have to register them in register(). However it goes terribly wrong it you already instantiate services that I'm still registering as orchestra/testbench has not overwritten the configuration yet. And because we register the services as singletons to be consistent with Symfony, overwriting the configuration later will not unregister the already registered singletons. Apparently registering the events in register() resulted in my services to register too early.

As a fix I register all events in the boot() of the service provider instead of registering them when I register the service.

Conclusion

I still have to see how it works out, but it looks promising. Writing your own integration test classes does give me headaches and took much longer than I expected and fellow programmers will not like it yet as it is not documented yet how the custom integration tests work. I do hope that in the end I can write a library that works flawlessly in more than one framework without having to worry like: 'this works in Symfony, but not in Laravel. It looks like the main developer prefers Symfony'.

Maybe I can even extract it as it's own package.

In a newer blog item I will continue writing about the data provider library I wrote to make the data providers for me.