Initial Investigation

In our initial investigation, we found that neither the web server nor the database showed significant load - the server was not struggling at all, yet simply could not serve the files as quickly as the older server which had much lower specs.

We upscaled resources in the Azure Portal, but performance remained unchanged (not a big surprise based on the server utilisation), meaning we were dealing with something that was a lot more complicated to diagnose/solve.

Docker Experiment

One of the key differences in the new hosting was the separation of the database and webserver from one machine into dedicated ones, we could see in the metrics that the queries were taking a bit longer which wasn't surprising due to a bit of added latency.

As a test we wrapped the application and MySQL database into a custom docker image and deployed that the same Azure PaaS server - the performance drastically improved, however the client had now lost the ability to connect to the MySQL from PowerBI (based Azure PaaS port restrictions). So while this was a great step forward it wasn't going to be a long term solution.

Code Optimization

While the issue was clearly server related, we dived into the code base to see if it was doing anything particularly unusual that could explain the issues.

There was a MySQL query that was taking a fairly dramatic amount of time (on both old and new sites) pulling back over 8000 lines from a table only to do further processing in memory which could be offloaded to the database server itself

There was also an extremely long dropdown box was causing the browser to lag out while rendering, as it turned out the user didn't actually use this functionality so we removed the dropdown and the associated db query.

These changes reduced server rendering time by 4+ seconds and client rendering time by 8 seconds.

Despite these improvements, the page still took over 2 seconds to render, compared to 0.5s on the old server. It was time to dive back into the docker solution and see why that was so impactful.

Root Cause: Azure Storage Overhead

On a hunch we updated the docker hosted solution to point at the hosted MySQL instance, curiously the site was still much faster. This was two versions on the site, hosted on the same exact hardware but performing very differently. Given the docker version had an added level of abstraction we would have expected it to perform slightly worse, not incredibly better.

Looking at what other experiements we could do we realised we could mount the other version of the site within the docker image itself simply by remoting in and creating some softlinks to switch between the directory on docker and the Azure PaaS persisted storage. As soon as we pointed docker at the Azure storage we noticed the exact same drop in performance, now we were getting somewhere.

Diving deeper into how Azure works it turns out on Linux based systems it mounts the home directory into the webserver as a network file system, which adds overhead to every IO request. For a PHP based website where the files are extremely modular this overhead quickly added up and had a massive impact on the overall performance (which also explains why we didn't see the same impact on our dotnet based solutions). Now that we were armed with this knowledge we quickly started to find articles about this specific issue.

Solutions

Microsoft had already implemented at least fixes for this problem several years earlier but neither were the default and neither were particularly well advertised.

The first article Speed up your application in Azure App Service was to enable a setting in the app service which copies the files locally on startup, eliminating the network traffic to the storage and increasing the performance just like the docker based solution. The downside was that this storage was readonly so any site that allows user uploaded content could not work.

The second solution was very similar and documented on the App Service GitHub page for Kudulite , again this was a number of years old but not a very well known setting. This setting also made the site blazingly fast, but also had the same issue where user uploaded was not persisted between reboots.

Azure PaaS has another feature that lets you mount a writable Azure Files container on top of the app cache storage allowing user uploads to be persisted between reboots.

In the end we ended up with a site that was significantly faster than the old one due to the extra enhancements we had made along the way, but more secure with lower maintenance overheads.