Misfeature - Part 1
by joe

Posted on 2018-11-15

Misfeature - not a bug. Rather the misfeature is characterized as, "We did it that way on purpose, but it turned out to be a bad idea."
Joe, ca. 1988

The original BLOX database had separate tables for Rhymes (left side of standard blog post) and Stanzas (right side of standard blog post). See the explanation of a standard blog post layout.

It was apparent, even in the early coding stages, that there was a great deal of commonality between the two tables. For example, the functions for keeping the entries ordered within their respective areas of the page (the order_edits() functions in rhyme_views.py and stanza_views.py modules) were almost identical. Other code for database accesses and updates and for view functions also matched closely.

This is a common occurrence in the development of database applications - two or more tables that have essentially the same data fields but are used for distinct purposes. In this case the rhyme table and the stanza table were intended to serve different purposes. But after using the BLOX application for a couple of months, the commonality became more apparent. Not only are the structures of the tables very close, but the actual use of the data in those tables is very similar - both in the presentation of the forms for entering the data and in the creation of the ultimate HTML output.

This relatively small structural change to the database schema imposes some necessary code changes and opens up opportunities for quite a few more. The same is true for the addition of a 'stencil' field in the spiel table.

The two separate tables now appear to be a misfeature. So we are going to change the database and code to correct that mistake. The effect should be to simplify the code, making the whole application easier to work with.

Two Tables --> a Flag

We replace the two table with a single table (stanza) and add a flag field (brand) to distinguish the "rhyme" entries from the "stanza" entries in the stanza table.

The before and after version of the stanza table definitions are shown below.

Before the change, we had two tables, rhyme and stanza:

            spiel_sid integer,
                mold text,
               verse text,
                 seq integer,
           edit_date text not null,
    primary key (spiel_sid, seq),
    foreign key (spiel_sid) references spiel (sid)
            spiel_sid integer,
              header text,
                body text,
                 seq integer,
           edit_date text not null,
    primary key (spiel_sid, seq),
    foreign key (spiel_sid) references spiel (sid)

After the change, the surviving stanza table has a 'brand' field to
distinguish rhymes from stanzas:

           spiel_sid integer,
               brand text null DEFAULT 'stanza',
                 seq integer,
           edit_date text DEFAULT '1950-09-17 04:56:03.141593',
              part_1 text,
              part_2 text,
    primary key (spiel_sid, brand, seq),
    foreign key (spiel_sid) references spiel (sid)

The parts of a Rhyme are used differently than the parts of a Stanza. In the Rhyme, part_1 identifies the HTML semantics that determine how the contents that make up part_2 are displayed - simple paragraph, blue box, heading, etc. But in a Stanza, both part_1 and part_2 are contents, each of which is styled in a specific way in order to display the stanza as a two-box structure on the page.

Where the original Rhyme and Stanza were made into separate types with distinct variable attributes, in the new versions, the separate types are distinguished by different implementations of their respective present() methods.

In this post we're going to focus on adapting the database maintenance operations to accommodate the consolidated stanza table. That involves almost all of the modules in the blox/app directory - and quite a few of the templates as well. We're going to change the Stanza and Rhyme objects, both attributes and functions, and some other supporting structures.

We'll start by comparing the Stanza object and the stanza table definition in the blox.sqlite database.

Refactoring in Stages

Take a look at BLOX commit 54130d7. Before consolidating all of the Rhyme-Stanza code into a single block that handles ordering and database operations for both, we have this intermediate step of adjusting the name references in the original code.

This is the first of several refactoring steps that we will implement in order to convert the code so that all references to the rhyme table in the database are eliminated in favor of using the single stanza table with the brand field to distinguish the rhymes from the stanzas. Each step is limited by the restriction that the resulting code must still be operational - a classical maintenance process.

The order of the succeeding steps is as follows:

1. Introduce the stanza.brand attribute into the database processing of spiel.stanzas[] in the huddle.py module and the dbx_seed.py module. (BLOX commit d6f8b0a.)

2. Replace huddle.update_rhymes() with calls to huddle.update_stanzas('rhyme') in the rhyme_views.py module. (BLOX commit 8516fd5 .)

3. Remove the vestigial huddle.update_rhymes() code from the huddle.py module. (BLOX commit dddb687 .)

4. Remove the vestigial rhyme table code from dbx_seed.py module. (BLOX commit 60ae894 .)

5. Move code from module util.py to the Stanza and Rhyme classes to implement the present() method for those classes. (BLOX commit ad12e94.)

Commit d6f8b0a

The first step in consolidating database update and retrieval code is to parameterize the stanza-handling code to accommodate both Rhymes and Stanzas. We use the current stanza-handling code and add the brand value to distinguish between Rhymes and Stanzas.

The tup_stanza named tuple in the huddle.py module is responsible for carrying update information to the dbx_seed.py module. So we need to add the 'brand' element to that tuple. We use the brand to parameterize the update_stanzas() method, passing the 'brand' parameter through to check_stanza_changes(), where we construct the db_actions tuple values that are used by db_update_stanzas() to make the actual database updates in the dbx_seed.py module.

The huddle.update_stanzas() method is called from the stanza_views.py module. Those calls have to be parameterized as well. Then we change the stanza-related code around the database operations correspondingly.

Commit 8516fd5

Commit 8516fd5 shows how nicely some refactoring can work out. With the previous changes made to parameterize the stanza-handling code in stanza_views.py huddle.py and dbx_seed.py, the changes required in rhyme_views.py module are almost trivial. All we did was replace three calls to huddle.update_rhymes() with calls to huddle.update_stanzas('rhyme').

Commit dddb687

Because we have the former huddle.rhyme_updates() changed over to calling parameterized huddle.spanza_updates(), we no longer need the methods update_rhymes(), check_rhyme_changes(), and re_sequence_rhymes() in the huddle.py() module - almost 100 lines of code.

Commit 60ae894

With the elimination of the rhymes-related update calls in the huddle.py module, there are no longer any calls to any database functions related to the former rhyme table. Therefore we can eliminate all of that code from the dbx_seed.py module. Another 100 lines of code gone.

Commit ad12e94

This commit is just a bit of tidying up and anticipation of future changes. The main change is to move code from the util.py module to the Rhyme and Stanza classes in order to implement the present() method for each of them. In each case, presentation consists of using the contents from database records to substitute text into the HTML patterns - a simple mechanism for inserting content elements into the standard_post.html template.

NOTE: This commit shows deletion of the same code from the huddle.py module as shown in the dddb687 commit. I don't know why.

Right Place

Having worked through the consolidation of the Rhyme and Stanza handling code, it looks like a lot of this code should be methods in the Spiel class. After all, the Rhymes and the Stanzas are attributes of the Spiel. But look again and you will see that some of this code deals with both the huddle.d_spiel and the huddle.e_spiel together. Those methods have to be implemented in the Huddle.

Seeing Some Payoff

We've been able to make all of the changes in this post while running the code live. We think that is because the scope of the BLOX application is so narrowly defined - very few database tables combined with a small selection of simple operations on the data in those tables.

Also, the form of the code is written to be easy to understand - nothing terribly subtle in any of the organization or expression of the operations.

This is gratifying because, one of the purposes of writing code in this style is to make it easier to maintain and adapt to changes in the application requirements. Expect many more changes in the BLOX requirements, planned and unplanned, in the near future.

BLOX Repository-based Example
This post is based on the BLOX repository code.

This post will be a lot easier to follow if you open up a separate screen to review the commits in the BLOX repository.
Related Posts
Misfeature - Part 2: consolidates the rhyme_views.py and stanza_views.py modules - including changes to the related HTML template files.

Misfeature - Part 3: separate update of the spiel table to support changes to the generate and publication operations.
What we found along the way:
During the refactoring we found (and introduced) several bugs in the application and fixed them en passant.

First, in the edit_stanzas.html template we were setting the list of stanzas to be edited to all of the stanzas in huddle.e_spiel instead of the selected_stanzas passed to the template. As a result, we were presented with the full set of stanzas to edit even when we selected a subset. You'll see that change in commit d6f8b0a in the BLOX repository.

Second, we introduced a bug when we put in the "Upd" buttons in the edit_rhymes.html template. We tried to get around it with use of a global flag in rhyme_views.py. Bad idea. A better solution was to add a parameter identifying the spiel to the "/update_all_rhymes/" path. That's resolved with changes to spiel.html and rhyme_views.py. (See BLOX commit dddb687).

Third, we found that the Modern-Business defined CSS class "lead" for lead-text squelched some internal styling - specifically the use of the <strong> tag. So we changed the definition of the 'lead-text' entry in the rhyme_elements dictionary in the util.py module to a simple increase in the font-size. (That code wound up as a class attribute in the Rhyme() class in the huddle.py module.)
As originally published (this may change) the last element of the main blog section in this post is a blue box. In reviewing the draft, we noticed the <hr> tag that follows the blue box looked ragged because the Comments section of the page, preceded by its own <hr> tag immediately follows.

We were really tempted to correct that effect by changing the definition of the "blue box" element in the "rhyme_elements" class dictionary in the Rhyme class by removing the <hr> tag at the end.

We tried taking it out, then put it back.

Mission creep is hard to control - particularly in this case because we don't have a detailed project specification. But making the change to redefine the "blue box" element would necessitate editing past posts to keep the formatting the way that we wanted from the first - or it would mean making at least one more "blue box" variant to handle the different semantics that arise from placing the box anywhere other than at the top of the post.

We're pretty sure that we'll revisit this later.


It will be some time yet before we get a comments section working here. In the meantime feel free to send comments via email. On this site our name is Joe Python. The email address is our first name at joepython.com.

Edited: 2019-01-30 20:55:12(utc) Generated: 2019-06-10 17:29:58(utc)