Dev->Prod: Interlude #1
by joe

Posted on 2018-11-20



Taking a break here from "real" development to tackle a brain-bender problem with the BLOX project. Given the project organization, it is almost impossible to tell when we are "developing" code and when we are "producing" a post. The UI looks exactly the same in either case.

Throw in the "generation" process that produces the static pages for the website and the transfer (via drag-and-drop FTP) of those pages to the public website, and we have four operational modes for the blox application. Add to all of that the fact that we constantly switch between operational modes - writing a bit of code and checking to see whether it works in development, adding material to drafts of blog posts, generating static pages for completed (or corrected) blog posts, and transferring posts to the website.

It's easy - and becoming all too common - to lose the thread of just where we are at any one moment. Time to fix that.


Not a regular Flask app

Let's take a tour of the blox application. (If you have not pulled the source code, open up the source view on bitbucket.) As it now stands, the project is essentially two applications. The easiest way to distinguish and describe the applications is by looking at the view functions.

The spiel_views.py and stanza_views.py modules are the guts of an online authoring and database maintenance application. The routes.py module most closely resembles a public-facing Flask website, drawing on the database contents to present fully-constructed web pages.

The generate.py module (under the .../blox project directory) transforms the natural Flask operation of the routes.py application into static web pages using the Frozen-Flask plugin.

The db.py, db_map.py and dbx_seed.py modules supply the database access and management functions, while the huddle.py module functions as the bridge between the database data elements and the display objects.

At this stage of the development, the templates are a bit disorganized, reflecting the cumulative entropy accrued as we tried different approaches to the development.

As mentioned in the Misfeature_1 and Misfeature_2 posts, for the last several weeks we have been doing development while operating against a live production database, making backups of the database, of course, while being very careful with the coding changes.*

No self-respecting programmer can live this way for long, though. We cannot safely do development against our production database, so we need to have a clean separation between development and production environments.

*(NOTE: We are grizzled professionals with nothing to lose. You should NOT try this at home.)

Overlapping, Simultaneous Dev and Prod Usage

It is the nature of our work on this application - and the blog we are writing with it - the the uses overlap. As we make changes to the application we also write blog entries to discuss and describe those changes (or previous changes, or planned changes).

The problem we face with this bifurcated application is that, having separated these functions, we cannot tell the difference between a development instance and a production instance without experiencing severe eye-strain. Both are running on localhost. Production and Development use the same REST addresses and the same template files. The only distinction between development and production is the port number in the browser address bar.

There's only one answer: we have to separate the production environment from the development environment - and we need to do it in such a way that deployments, which we expect to be frequent, are as simple as possible.

So today we dial back on the application coding to insert a developmental interlude and dedicate a few hours to our own comfort by implementing a responsible facility that will serve us well in the longer term.

Interlude #1: Commit a687306: Automate Dev & Prod Context Signals

The elements of this commit fall into two categories: 1) determine whether we are in production mode or development mode, and 2) adapt the running application instance to the proper mode.



Determine the Proper Mode
There are a jillion ways to organize your application to detect whether you are running in development or production mode. (We did the math. Came out to exactly a jillion. Go figure.)

The way we have done it is to set up a distinctive organization of the directories in our production environment. As with most of the things we do the first time, this is not intended to be a solution for the ages, but for now it greatly simplifies updating the production environment without any fiddling with environment variables or deployment scripts or hand-tweaking in either environment.

We set up a home directory for production at ~/Applications/BLOX. Under that directory, we setup a "code" directory named "blox" and a "data" directory named "blox_data". Under the blox_data directory, we keep our production database, which is named blox.sqlite.

In the development environment we keep the database in the Flask /instance directory, which is designated not to be included in the git repository.

So, to determine whether we are operating in production mode, we can check for the existence of the file ~/Applications/BLOX/blox_data/blox.sqlite in the environment where the blox_run.py module is running.

That's what is going on in the new dev_or_prod_mode() function in the __init__.py module. We start with the assumption that we are running in a development environment - safer in this case than assuming we are running production. (If we're wrong, we screw up development data - not the good stuff.) Then we use the Flask primitive blx.root_path to derive the absolute path of the production database. If we find the the production database file exists, we know we are in production mode and reset blx.config['RUN_MODE'] accordingly.

Adapt to the Execution Mode
Our chosen mechanism to distinguish development mode from production mode is to change the background color of the NavBar. In the Modern Business Bootstrap template, the NavBar is black. We picked a mustard yellow for development mode. (We changed our mind pretty quickly. A few minutes operating with the mustard yellow NavBar drove us to a more soothing medium green.)

We talked about several possibilities for implementing this change. The underlying question is how to communicate the Dev-Prod information to the templating system from the Python code. One way is to add a parameter to every render_template() call. That just seems ugly and unsophisticated. Other possibilities were equivalents of setting up a dev_mode template and a prod_mode template, then copying the correct one to the app/templates/include/jp_navbar.html (for the blog pages) - and doing something similar for the database maintenance pages with dbx_navbar.css.

Then we hit on the solution included with this commit. The Modern Business Bootstrap template provides a second CSS file (.../blox/app/static/css/modern_business.css)that we have used to tweak elements of the presentation, e.g., the structure and spacing of the stanza boxes in the sidebar. We made a third CSS file whose sole purpose is to change the NavBar background color for both subsystems. We leave the new CSS file ("dev.css") in the development project environment so that it changes the color of the NavBar, but if we detect it while running in production mode, we just delete it, leaving the production NavBar as default black.

That is what's going on with most of the rest of this commit. The bottom block of the dev_or_prod_mode() function is the IF statement that deletes this file when the blx.config['RUN_MODE'] is set for production.

The production pages based on the "blog-std" stencil inherit the change to app/templates/blog/includes/jp_css.html file through the app/templates/blog/blox_base.html template. But we have some template files for the static blog site that are one-offs (app/templates/index.html and app/templates/jp_about.html), so when we run in development mode we want to use this new dev.css file. They are not generated like the blog pages so we have to include the CSS file directly.

NOTE: Check out the sidebar dev.css Revisited for a bit of foreshadowing.

Deployment is a Snap

With the development --> production setup described above, we have a trivial deployment process: replace the production .../BLOX/blox directory with the development project .../blox directory. We've done it a few times already, and things work just fine. We delete the current production .../BLOX/blox directory, copy the current development .../blox directory, the paste the copy into the .../BLOX production directory. (Or you can drag and drop on the Mac and select "Replace" when Finder prompts you about the conflicting file names.)

The dev_or_prod_mode() function in the __init__.py module takes care of the only chore that remains after the drag-to-deploy step: removing the dev.css file that is responsible for making the NavBar green in the development operational mode. The first time that the blox program runs, it detects the file and deletes it, ensuring that the NavBar stays Modern Business black from that point on.

Two Tabs - Reading & Writing
We run the blox app in two browser tabs. In the first, we are running a Flask database maintenance and authoring sub-application to create the content for blog posts. In the second, we are running the routes.py sub-application to view the active pages that we'll freeze into static pages for the public website.

We think of them as the "writing" and "reading" tabs, respectively.

db.py Module is Unchanged
In the process of working out the dev-prod context automation, we made changes to the db.py module several times. For example, in one iteration, we were testing a blx.config[] value, and connecting to different dev and prod databases using other, different blx.config[] values.

As we wrote this post, we started to put in a description of changes to the db.py module, only to find that it was unchanged. Somehow, we managed to edit and re-edit db.py multiple times and wind up right back where we began.
"Deployment" Defined
Deployment, as commonly used, may connote a much more expansive and robust process than is required in the context of the blox application.

We are doing development and running production on the same machine - simultaneously authoring an application and using the authoring application.

Still, the separation of development and production is necessary, which leads to the need for deployment. Further, we are in a position as developer-users to indulge a high time preference and deploy the application to production whenever we do something cool.** This puts a premium on simplifying the process.

**Not just daily, but often multiple times per day. We're that good.
Commit a687306 Detritus
In reviewing commit a687306 for this post, we detected a few minor blemishes we overlooked. Little things, but they needed to be cleaned up. They make up commit a687306.

1. We removed a superfluous include of jp_navbar.html from dbx/base.html.

2. We're defining the value for blx.config['DEV_DB_FILE'] in the dev_or_prod_mode() function, but we can just as easily put them into .../blox/config.py, which seems a bit tidier. (We started to move the DEV_DB_DIR and DATABASE values to the config file as well. But calculating them at runtime is better.)

3.

dev.css Revisited
One effect of deleting the dev.css file in production is a constant, OCD-pounding repetition of a 404 error for every web page that attempts to retrieve the dev.css file - which is every blessed one of them.

In a later update, we just empty out the dev.css file instead of deleting it. Elimination of all those 404 messages let us work on this project with much less irritation.

Comments

It will be some time yet before we get a comments section working here. In the meantime feel free to send comments via email. On this site our name is Joe Python. The email address is our first name at joepython.com.

Edited: 2019-01-30 20:55:12(utc) Generated: 2019-06-10 17:29:58(utc)