Creating a Stable Python Tools Site: The Journey from Passenger to CGI
500 errors, timeouts, and the occasional complete refusal to load.
Today, I finally sat down to overhaul the infrastructure. The goal was simple: create a robust, secure, and easily extensible platform for my Python tools. The journey to get there, however, involved a deep dive into the quirks of shared hosting.
The Problem: "Resource Temporarily Unavailable"
The site was originally built to run on Phusion Passenger, a popular app server. However, DreamHost (my hosting provider) has deprecated Passenger support on shared plans. This left the site in a zombie state—trying to launch, failing, and generating cryptic error logs.
I initially tried to modernize it using FastCGI (fcgid) with the flup library. It worked perfectly in my local staging environment, but the moment I pushed it to production, the logs lit up with:
RuntimeError: can't start new thread
[error] End of script output before headers: app.fcgi
It turns out that shared hosting environments are strictly resource-constrained. FastCGI tries to keep a persistent process running and uses threads to handle concurrent requests. The server's security policies were aggressively killing these threads, viewing them as resource hogs.
The Solution: Embracing Simplicity (CGI)
Sometimes, the "old" way is the best way.
Instead of fighting the server's limits with complex threading models, I switched the entire deployment to CGI (Common Gateway Interface).
Unlike FastCGI, which keeps a process alive, CGI spawns a fresh, clean Python process for every single request. * Is it slower? Marginally, on the first load. * Is it stable? Absolutely.
Because every request gets a fresh slate, there are no memory leaks, no "stuck" threads, and no "resource unavailable" errors. It just works.
New Architecture
The site now runs on a lightweight Flask application with a few key features:
- Secure Login: A simple, environment-variable-backed login system (
APP_PASSWORD) protects the tools. - Scalable Tools List: Adding a new tool is now as simple as adding a dictionary entry in
app.py. - Automated Deployment: I fixed the Git hooks. Now, a simple
git pushupdates the code, installs dependencies, and ensures the permissions are correct.
I've also added two key documentation files to the repo to ensure I never get stuck again:
* troubleshooting_server_issues.md: A cheat sheet for server quirks.
* new_tool_instructions.md: A step-by-step guide for expanding the site.
The site is now stable, secure, and ready for new tools. Sometimes you have to take a step back in complexity to take a step forward in reliability.