Misfeature - Part 3
by joe

Posted on 2018-11-26



Misfeature - not a bug. Rather the misfeature is characterized as, "We did it that way on purpose, but it turned out to be a bad idea."
Joe, ca. 1988


Recognizing the Goal: Generate and Publish

Up to this point almost all of our efforts with BLOX have been directed to the process of creating blog posts, specifically what we call standard blog posts (like this one) that are based on the blog-std stencil.

That's now working pretty well - actually surpassing our original intent when we started the project. But our use of Frozen-Flask to generate static pages is tenuously functional; and our drag-and-drop-via-SFTP mechanism for "publishing" pages to this website leaves a lot to be desired, particularly in light of our desire to create posts on disparate topics that need to be separated in order to make later discovery and access coherent.

Spiel States

We've been dodging an inadequacy involving the various stages and states applicable to a spiel. There's a date-based hack to distinguish those spiels we want to appear in the ToC of active posts and those we don't (in main.html, the test '{% if s.post_year < "2018" %}' at line 8). So we keep drafts of posts from being published by giving them pre-2018 dates. More than a little clunky.

We actually want to publish some drafts, designated as drafts, publicly in the blog - either because the incomplete information is particularly timely or because we'd like to get feedback. We also want to publish non-public drafts for 'internal' review to an Internet-accessible site that makes the post editable by multiple contributors. This capability implies a means of exporting and importing posts between multiple BLOX databases.



Marginal Misfeature

To some extent, the need to revisit the spiel table schema is less a misfeature than a simple enhancement. It feels a bit like a misfeature because we dialed back the stencil-spiel elements of the database from our earliest efforts so radically - and now we need to expand them again.

Nevertheless, we now know that we want to expand our options to include multiple stencils, and we want to designate multiple alternative publication targets that we did not anticipate in our original conception for BLOX. We also want to partition blog posts into multiple 'threads' - think the difference between these posts following BLOX development and 'How To' posts like Python package installations and programming tips.

With these changes to the spiel table in place, we can proceed to upgrade the whole static page generation process to use the new information. And as part of that enhancement we will need to re-examine the mechanisms for selecting spiels to be generated - including an option to regen the entire site (or identifiable sub-sections of the site) on occasion.



Database Schema Changes

Our changes to the blox database schema are limited for this update. All we're going to do is add two fields to the spiel table: 1) a target field to identify the target location for this spiel in the current generation-publication run; and 2) a thread field to identify which category this spiel belongs to.

We'll do this by running the following Sqlite ALTER TABLE commands to add the fields and assign default values to them in the current database.


ALTER TABLE spiel ADD COLUMN thread text DEFAULT 'BLOX';
ALTER TABLE spiel ADD COLUMN target text DEFAULT 'local';

Here's what the current BLOX database schema looks like:


CREATE TABLE spiel (
             sid integer primary key,
         stencil text not null DEFAULT 'blog-std',
        filename text not null,
           title text not null,
          author text not null,
       post_date text not null,
             tab text not null,
       edit_date text DEFAULT '1950-09-17 04:56:03.141593',
    publish_date text DEFAULT '1950-09-17 04:56:03.141593'

);
CREATE TABLE stanza (
           spiel_sid integer,
               brand text null DEFAULT 'stanza',
                 seq integer,
           edit_date text DEFAULT '1950-09-17 04:56:03.141593',
              part_1 text,
              part_2 text,
    primary key (spiel_sid, brand, seq),
    foreign key (spiel_sid) references spiel (sid)
);
CREATE TABLE to_press (
             pub_type text,
             text_val text DEFAULT '',
              int_val integer DEFAULT 0
);

And here's what the BLOX database schema looks like (according to Sqlite) after the ALTER TABLE operations. (We've kept a more nicely formatted version in the repository.)


CREATE TABLE spiel (
             sid integer primary key,
         stencil text not null DEFAULT 'blog-std',
        filename text not null,
           title text not null,
          author text not null,
       post_date text not null,
             tab text not null,
       edit_date text DEFAULT '1950-09-17 04:56:03.141593',
    publish_date text DEFAULT '1950-09-17 04:56:03.141593'

, thread text DEFAULT 'BLOX', target text DEFAULT 'local');
CREATE TABLE stanza (
           spiel_sid integer,
               brand text null DEFAULT 'stanza',
                 seq integer,
           edit_date text DEFAULT '1950-09-17 04:56:03.141593',
              part_1 text,
              part_2 text,
    primary key (spiel_sid, brand, seq),
    foreign key (spiel_sid) references spiel (sid)
);
CREATE TABLE to_press (
             pub_type text,
             text_val text DEFAULT '',
              int_val integer DEFAULT 0
);

A Couple of Observations on the Changes
First, the Sqlite ALTER TABLE commands will have to be executed directly against (a copy of) the production database when we deploy.

Second, we have a minor hangup concerning the tidyness of database schemas. It just seems tacky to have table fields in any old order. For instance, some of us like to put fixed-length fields first to make SELECT * results easier to read when working from the command line interface. And even when the fields are not fixed length (such as the new fields we are adding), it's nicer to have shorter fields earlier in the display to aid with readability - again from the command line interface.

We'll revisit some of this much later when we explore making a stand-alone database migration utility for Sqlite.

Prep Spiel Table for Generate and Publish

Commit c92e8e1 turned out to be pretty easy overall, though we still managed to mangle it just a bit. See the sidebar "A Messy Commit" and the follow-on commit 5a0c14b.

As always, it's most helpful to have the commit opened up in a browser for review while reading the following discussion.

Two of the changes to add the target and thread fields consist of just a couple of lines of code each to include the new fields along with the existing field attributes in the Spiel class: the db_map.py module and the huddle.py module.

The non-logger-related changes to the dbx_seed.py module at lines 28-32 and line 57, are not much more complex.

Our addition of the target_options and thread_options in the util.py module are used by the spiel_views.py module to enable the dropdown selectors in the confirm_spiel_delete.html, the edit_spiel.html, and the new_spiel.html templates. The HTML <select> tags in the templates are pretty straightforward, building a list of 'options' and including a 'selected' attribute for the existing value when it is available.

Going to Seed

We now have a functional authoring system for producing the joepython.com blog.

The downstream processes of generating static pages and distributing them to selected locations remain marginally functional and somewhat manual. Nevertheless, we have accumulated enough capability to declare an open source correlative of "minimum viable product" with the existing application.

We have plans for many more extensions and enhancements to this BLOX application. For now, though, we will tie up loose ends and declare that we have reached the seed stage of development. This commit will carry the 'seed' tag and serve as the jumping off point for some future tutorials and lessons.

The seed release is the accumulation of all of the commits from the misfeature_3 development branch. Find it at commit b1e6cd3 in the Blox repository.


Surprise Ending
When we started writing this post a few days ago, we did not know that the impending commit would take us to the long-planned 'seed' tag. A pleasant surprise. At some point soon we'll have a recap post discussing some of the lessons learned on the path from 'germ' to 'seed'.

From here, we start making enhancements to the generation and publication processes based on the options available to us with the addition of the target and thread fields to the spiel table.

In a month or two that should take us from 'seed' to 'shoot'.


BLOX Repository-based Example
This post is based on the BLOX repository code.

This post will be a lot easier to follow if you open up a separate screen to review the commits in the BLOX repository.
Related Posts
Misfeature - Part 1: consolidates the rhyme table and the stanza table from the original BLOX database into a single stanza table with an additional brand field to distinguish between Rhymes and Stanzas.

Misfeature - Part 2: consolidates the rhyme_views.py and stanza_views.py modules - including changes to the related HTML template files.

Groundwork for Larger Tasks
Before we move on to the generate and publishing operations, we did a little prep by normalizing maintenance on the current spiel table. We fleshed out the addition (new_spiel) and update (update_spiel) transactions. And we added a deletion operation with confirm_delete_spiel and delete_spiel routes in the spiel_views.py module.

We enhanced selectors for post_date and stencil fields in both the new_spiel.html and edit_spiel.html templates with selector_options from the util.py module; and we made matching changes to the spiel_views.py module in parallel. Finally, we added the huddle.delete_spiel() and dbx_seed.db_delete_spiel() functions to actually make the database update happen.

These changes show up in BLOX commit 0a03614.

And, BTW, our drag-and-drop deployment of the new code worked just fine.
Safari? WTF?
The Safari browser does not handle <input type="data" ... > HTML tags like every other browser does.

For the last week we've been running "production" writing blog posts using the Safari browser. So, as we prepared for this post, we thought we needed to put a date selector into the new_spiel.html and the edit_spiel.html templates. But when we checked the code, the selectors were already there. we tested with other browsers to make sure.

Now we're using the Opera browser to run "production" writing this stanza - and all blog posts for the foreseeable future.

Does anyone know why Apple does this?
A Messy Commit
We made a small pedagogical mistake by mixing in some housekeeping with the substantive changes in commit c92e8e1.

We finally deleted the dbx_if.py module, an earlier version of the the dbx_seed.py module. And we stripped out all of the logging usage prepatory to making a cleaner logging setup.

All of those changes are mixed in with the substantive changes discussed in the main post. Sorry about that.

We made a couple of other changes in this commit that need some explanation. We added favicon code to the routes.py module to suppress the 404 messages that the Flask development server issues when it cannot find the favicon.ico file. Again this is one of those non-substantive things that hits us like fingernails on a chalk board.

Then there's the wince-inducing correction in the stanza_views.py module at lines 119-123. We deleted unneeded (and unwanted) parameters in the url_for() call. We spotted this mistake after we took out all of the logging, and the Flask development server logging resumed. The additional parameters showed up in the Flask log as query parameters, which stuck out like a sore thumb.

We preserved the output, and we may be able to do a little Flask diving to figure out the mechanism in the not-too-distant future.

And finally, just before we put this all to bed for a merge into the master branch, we discovered an oversight in the edit_spiel.html template that killed our ability to edit a subset of Rhymes in a Spiel. It's fixed in commit ae93483.

Comments

It will be some time yet before we get a comments section working here. In the meantime feel free to send comments via email. On this site our name is Joe Python. The email address is our first name at joepython.com.

Edited: 2019-01-30 20:55:12(utc) Generated: 2019-06-10 17:29:58(utc)