A Frustrating Battle with Nginx and FastCGI Timeouts

Introduction

As someone who works in software and systems engineering, you sometimes run into those deeply frustrating issues that feel like they’re consuming more time and energy than they should. Recently, I faced one of these moments with my Moodle and Streamlit applications running on Nginx, which both suddenly stopped working properly.

The issues started small — slow load times, occasional failure to load a page, and eventually, full-on timeouts. Nginx kept throwing alerts in the error logs about open sockets, and my pages would refuse to load. This post covers the painstaking process I went through to troubleshoot the issue, and finally, the solution that made both applications work seamlessly again.

The Setup

I run a couple of different applications, including Moodle for online learning and Streamlit for data-driven web apps, on my Nginx web server. Here’s the basic setup:

OS: Ubuntu 22.04.3 LTS
Nginx: To handle incoming web requests
PHP 8.1-FPM: For handling PHP-based Moodle
Streamlit: A data-focused Python app running on a different port

Initially, everything was working fine until both applications started facing timeout issues.

The Issue: “Hmmm… can’t reach this page”

The problem first appeared when I attempted to access the Moodle app through its URL. Instead of loading the page, the browser displayed: Hmmm… can’t reach this page. At first, I thought it might be a network glitch, but after a couple of refreshes, it was evident that something deeper was wrong.

Checking the Nginx error logs (/var/log/nginx/error.log), I noticed alarming messages like:

[alert] open socket left in connection
aborting

And this is where my frustrating journey began.

Troubleshooting the Root Cause

Checking PHP-FPM Service:
My first stop was the PHP-FPM service since it handles the PHP backend for Moodle. I made sure the service was running correctly by checking its status:
```
$ sudo systemctl status php8.1-fpm
```
Everything seemed fine there, but the problem persisted.

Nginx Configuration Review:
I reviewed my Nginx configuration for any obvious errors in handling the PHP requests or proxying them. The configuration seemed fine at first glance:

location ^~ /learn {
    limit_req zone=mylimit burst=20 nodelay;
    root /var/www/html;
    index index.php index.html index.htm;
    try_files $uri $uri/ /learn/index.php?$query_string;

    location ~ [^/]\.php(/|$) {
        fastcgi_split_path_info ^(.+?\.php)(/.*)$;
        fastcgi_pass unix:/var/run/php/php8.1-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
    }
}

Nginx Error Logs:
The error logs pointed to a serious problem with open sockets, which I suspected was related to timeouts during FastCGI communication.
```
[alert] open socket left in connection
```
Trying PHP Error Logs:
I also checked the PHP error logs for any fatal issues, but strangely, the logs didn’t exist yet. So, I created them manually and set the appropriate permissions.
```
sudo touch /var/log/php_errors.log
sudo chown www-data:www-data /var/log/php_errors.log
sudo chmod 644 /var/log/php_errors.log
```
Despite these efforts, there were still no PHP errors.

The Resolution: FastCGI Timeouts

After scouring through online documentation and forums, I stumbled upon a potential fix — adjusting the FastCGI timeout settings. It turns out that Nginx has strict timeout values that may need adjusting, especially for long-running PHP scripts or dynamic content served by FastCGI.

Here’s the adjustment I made:

location ~ [^/]\.php(/|$) {
   fastcgi_split_path_info ^(.+?\.php)(/.*)$;
   fastcgi_pass unix:/var/run/php/php8.1-fpm.sock;
   fastcgi_index index.php;
   include fastcgi_params;
   fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
   fastcgi_param PATH_INFO $fastcgi_path_info;

   # Add these FastCGI timeout settings
   fastcgi_read_timeout 300;
   fastcgi_send_timeout 300;
   fastcgi_connect_timeout 300;
}

Why This Worked:

FastCGI manages the communication between Nginx and PHP-FPM (in the case of PHP-based apps like Moodle). When the application takes longer to process a request than the configured timeout, Nginx will cut the connection. Extending the timeouts to a more reasonable value (in this case, 300 seconds) allows the server to handle longer-running requests without abruptly closing the connection.

After applying these changes and restarting Nginx, both Moodle and Streamlit apps began working perfectly again!

Conclusion

This experience was both frustrating and educational. I learned (yet again) the importance of reading error logs and digging into timeout configurations, especially when using a reverse proxy like Nginx. The issue with sockets and timeouts boiled down to simple misconfigurations, but the process to identify that took considerable effort.

If you’re dealing with Nginx and FastCGI-related timeout issues, I hope this post saves you time and effort. It’s always the little things in configuration that end up making a world of difference.