Using the never type in typescript

on
0

How a developer playground increase your library development speed

on

 


When you work on any common PHP library the workflow to test if your library works in an actual application is slow. You first have to make a PR, push your library code, update packagist, update the library in your application and test manually if the changes worked. As you can imagine this is slow.

How about integration tests to handle this?

We already have a setup to run our code in multiple application setups with even different frameworks. But the problem is that too many application setups will lead to a very long testsuite. Right now we already test our integration tests with 7 different application setups. We only have about 40 integration tests like this, so 7 * 40 = 280 test cases. If we would test all our 7 application setups with different databases (mysql, sqlite, postgres, mongodb) we would end up with 28 application setups per actual integration test.

Not only would the tests require a lot of installation requirements for installing all those database servers, it would also just waste time in testing something actual useful. We could of course write only one test with a database setup, but again, then we would miss things that will break. A better solution would be to be able to run all the tests with a configurable database connection, so a developer could test whatever he feels like testing.

Still it would not scale enough for now to test all these combinations. There's also the case that the integration tests do not test reality 1 on 1 as I noticed small issues in the integration tests compared to an actual application. For example in the test suite I did notice small bugs where an actual application responds differently. Funnily most of them were Laravel related:
  • some console commands were not registered in the testsuite, but in an actual Laravel application they do appear and work as intended.
  • Laravel integration tests keeps state between multiple HTTP requests that you are still logged in, while in reality you should not.
  • While I do test apie/cms for rendering a form, I do not test if the form actually works as this requires a browser test instead.

How about monorepo's?

Monorepo's can be found in many large PHP projects. For example Doctrine, Symfony and Laravel are all written as a monorepo. A monorepo is however not made to test your library code in an actual application, but only speeds up making changes to multiple couped libraries together.

For example if we have a library it's very common to separate it in 2 packages: one with the framework angnostic code and one with the bindings to a framework. You make changes to the library code and then you update your framework binding package. This is very time consuming as you could sometimes end up having to push the library code multiple times for insights you missed. You could write everything in one library, but then if you make multiple library/framework connections you end up downloading the entire internet or hiding actual dependencies and ask the programmer to install them themselves. In npm you could use peer dependencies for this, but that's not what peer dependencies were used for.

With a monorepo you get the best from separating packages and not keeping everything in one package at the same time:


This works for library to library, but it does not work well when you want to use this for an application as well. For example we would like to be able to see if our latest changes would work in any application setup if needed, for example: does our code work in an application set up with Mysql/Sqlite/PreSQL, MongoDB. etc.? We could only do this with an actual application (or very slow integration tests).

Playgrounds

I ran into the concept of a playground when I tried out how to make a Nuxt plugin. A Nuxt plugin only works inside a nuxt application, so they have the same problem that during the development of your code it would be beneficial to test your code right away in an actual application.
Basically if you decide to make a nuxt plugin, it will create a playground folder that contains a working Nuxt application. It will link to the nuxt plugin you are still developing and has a watch function.

When I saw this I knew it would be beneficial for development that the library comes with a playground.

Docker or no docker, that's the question?

We could run something like run a php server to mimic a server or we could run it with actual docker containers. I decdided to work with Docker as it will act as an actual HTTP server and it will be able to connect with an actual mysql database etc. It's slower and sometimes the setup could be a headache, but it means that it can run the same on anyone's machine.

Our docker-compose file.

We start with a simple docker-compose file:
version: "3"
volumes:
  volume_database_project:

services:
  mysql:
    image: mysql:8.0
    container_name: apie_playground_mysql
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: 'true'
      MYSQL_ROOT_PASSWORD:
      MYSQL_DATABASE: project
      MYSQL_USER: project
      MYSQL_PASSWORD: project
    volumes:
      - volume_database_project:/var/lib/mysql
    ports:
      - "3306:3306"

  phpmyadmin:
    image: phpmyadmin/phpmyadmin
    container_name: apie_playground_phpmyadmin
    links:
      - mysql
    environment:
      PMA_HOST: mysql
      PMA_PORT: 3306
      PMA_ARBITRARY: 1
    ports:
      - "81:80"

  web:
    build:
      context: ../
      dockerfile: ./playground/.docker/Dockerfile
    container_name: apie_playground_web
    volumes:
      - ./app/ApiePlayground:/var/www/html/src/ApiePlayground
      - ../packages:/packages
      - ./.docker/virtualhost.conf:/etc/apache2/sites-available/000-default.conf
      - ./.docker/supervisord.conf:/etc/supervisor/conf.d/supervisord.conf
    ports:
      - "80:80"
    environment:
      - APIE_STARTER_SETUP=maximum
      - APIE_STARTER_ENABLE_CMS=1
      - APIE_STARTER_FRAMEWORK=Symfony
      - APIE_STARTER_ENABLE_USER=0
      - DATABASE_URL=mysql://project:project@mysql:3306/project
    depends_on:
      - mysql

  mailhog:
    image: mailhog/mailhog
    container_name: apie_playground_mailhog
    ports:
      - "1025:1025" # smtp server
      - "8025:8025" # web ui
As you can see it was really setup to be run locally with the hardcoded usernames/passwords. We basically make a local Dockerfile that installs an apie application running the apie project starter. We create a custom repositories section in the composer.json as documented here to link to our local packages which are mount as a volume to the web container. This way any local change will be automatically reflected in our playground.
We also mount a /var/www/html/src/ApiePlayground to add our own entity resources to test. By default we link it to mysql. To be able to edit our database we can edit our database with phpmyadmin on port 81.

Our dockerfile

Most of our magic is handled in the Dockerfile. One thing with the Dockerfile is that we run into some bootstrapping issues as we need to copy the packages folder and configuration during the container build. Only when the container is built you will have the volumes available.
FROM php:8.2-apache

# Set the working directory inside the container
WORKDIR /var/www/html

# Install system dependencies
RUN apt-get update \
    && apt-get install -y \
        libicu-dev \
        libpq-dev \
        libzip-dev \
        unzip \
        jq \
        git \
        cron \
        nano \
        supervisor

# Install PHP extensions
RUN docker-php-ext-install \
    intl \
    pdo \
    pdo_pgsql \
    zip \
    pdo \
    pdo_mysql \
    opcache

# Install APCU
RUN pecl install apcu
RUN docker-php-ext-enable apcu

# Install symfony CLI
RUN curl -sS https://get.symfony.com/cli/installer | bash
RUN mv /root/.symfony5/bin/symfony /usr/local/bin/symfony

# Install Composer CLI
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

# You can add your own aliases here
RUN echo 'alias console="php /var/www/html/bin/console"' >> ~/.bashrc
RUN echo 'alias phpstan="/var/www/html/vendor/phpstan/phpstan/phpstan"' >> ~/.bashrc
RUN echo 'alias dsu="/var/www/html/bin/console d:s:u --force --complete"' >> ~/.bashrc

RUN echo "umask 0000" >> /root/.bashrc

RUN mkdir /packages
COPY ../../packages /packages

WORKDIR /var/www
ENV COMPOSER_ALLOW_SUPERUSER=1
COPY ./playground/app/.env /var/www/.env
RUN composer create-project apie/apie-project-starter html -s dev
COPY ./playground/edit-composer.json tmp.json
RUN jq -s 'add' tmp.json ./html/composer.json > tmp-merged.json
RUN mv tmp-merged.json /var/www/html/composer.json
WORKDIR /var/www/html
COPY ./playground/app/apie.yaml /var/www/html/config/packages/apie.yaml
RUN rm -rf vendor
RUN composer update

# Setup crontab
COPY ./playground/.docker/crontab /var/www/html/.docker/crontab
RUN crontab /var/www/html/.docker/crontab

# Start Supervisor, which will run cron -f by default.
CMD ["/usr/bin/supervisord"]
As you can see we run our playground in PHP8.2, we install composer and a few php extensions. We use the apie project starter to fill in our application and then with jq we replace some sections in the composer.json to use our local apie packages.
Once this is installed we can show our application with localhost. We can just modify the code in packages directly as if we do not have a composer package and see our changes on page refresh.
The first thing I noticed is that some big things basically were quite broken with Mysql 8, because I tested in Mysql 5.7 before.  The playground uses Mysql 8.0. Mysql 8.0 will complain about faulty GROUP BY and DISTINCT and this just happens to be the case with the text search. I also noticed that the filter search was defined as a json string and not as a one to many which can not be indexed in Mysql 8 as Doctrine maps it as a JSON type and not as a text field type. I could now easily fix these problems as if I work locally!

Future improvements.

Right now you can only test in a Mysql application, but I can imagine either running multiple versions in multiple containers linked to a specific hostname or adding a change button, so I can change whenever I feel to change. This playground could also be used in the future to write browser tests to test apie/cms much better. I think I can develop much faster now as I do not need to push my library code and have to update it in an actual application before I see my changes.

Comments

Post a Comment