Eine Kleine Abstraction
by joe

Posted on 2018-12-25

A Time to Every Purpose
We have now tired of some of the circumnavigations required by the current BLOX implementation. The single most irritatingly awkward process involves inserting an entry into the middle of a post:
  • click the "Update Rhymes" button to take us back to the edit_spiel page,
  • then click the "+[placeholder]" button to add a Rhyme element to the end of the spiel.rhymes[] list,
  • click the "Edit All Rhymes" button to return to the edit_all_stanzas/../rhyme page,
  • then paging down to the end of the page to enter an interpolating entry number,
  • then pressing the "Upd" button to move the [placeholder] entry to the desired position on the page
  • then scrolling down to the position on the page to access the nascent entry
- and only then being able to edit the new entry in the proper context.

And, of course, it's even more irritating when we need to insert more than one entry.

Surely There Is an Easier Way

The process does not have to be that difficult. The truth is that we have always known there was a better way. We just had other priorities.

There's a bit of HTML code lurking in the edit_rhymes.html file. Here's what it looks like:

    <td style="font-size: 160%">
        <button type="submit" value="+↑" 
            name="add_up_{{ loop.index }}">+&uarr;</button>
        <button type="submit" value="+&darr;" 
            name="add_dn_{{ loop.index }}">+&darr;</button>
        <button type="submit" value="-" 
            name="del_{{ loop.index }}">-</button>

Let's ignore for the moment that the intent to comment out the row of small buttons was imperfectly executed and just concentrate on other matters.

We have been assured repeatedly that our comfort in dealing with the current, somewhat peculiar, user interface for BLOX is unique. No one else on the planet, we have been told, could possibly accept dealing with data as it is now presented. As highlighted in the introduction above, we are painfully aware of (at least some of) the existing user challenges in the current UI.

We highlight the code segment above as evidence that our own desires for a more accommodating UI are not so far off the beaten track. Those three buttons date from the earliest days of BLOX development, demonstrating our intent to accommodate more normal electronic intercourse from the beginning. We never actually implemented the operations indicated, and this more ambitious intent to reduce the effort to edit Rhymes and Stanzas was suppressed as part of the triage that reduced the database down to three (and then two) functioning tables.

Those buttons bear witness that our heart is pure.

This is what they look like, in context, when not commented out:

Where To Begin?

Reading from left to right, the function attached to each button is 1) insert above, 2) insert below, and 3) delete.

The current implementation of the two insert functions is described at the top of this post. The current implementation of the delete function is to enter a zero value in the sequence field and hit one of the update buttons ("Update Rhymes" or "Upd").

As a going in proposition, our plan is to implement the two insert buttons not only to do the insertions of a new, editable rhyme entry, but also to position the cursor to start editing in the textarea field of the new entry.

We suspect that we will use what we learn from the insert implementations to unify the Add New Rhyme and +[placeholder] operations as well. Right now the (highly obscure) delete operation is unconditional - click one of the update buttons with a zero value in a sequence box and your rhyme entry is gone. We will interpose a confirmation step in conjunction with the delete button as part of this overall upgrade.

'Insert' Is Not the Same as 'Append'

When we added the +[placeholder] button back in commit 73880b4, we stopped using the "Add New Rhyme" option on the edit_spiel page. Our first impulse when considering the re-introduction (and actual implementation) of these 'insert buttons' was to use the code that implements the +[placeholder] button. But that does not work because the +[placeholder] button implementation is a simple append operation. So we looked again at the code that implements the "Add New Rhyme" operation. We discovered that the add operation still works correctly as an insert command, but now has a bug when used for an append command.

This rather basic observation inspires a more broadly applicable implementation of the 'insert buttons'. As part of that implementation we'll clean up the newly discovered problem with our "Add New Rhyme" operation. But it turns out that our project suffers from other, more basic, failings.


We reviewed the 'insert' and 'append' code with a view to distinguishing and clarifying the code that implements those distinct operations. That review led us to look more closely at the existing code for maintaining the order of Rhymes and Stanzas within a Spiel as new entries are inserted or appended.

That code is a mess.

To be fair, it is a functioning mess that enables the creation of these blog posts - and that is no small accomplishment. We have seen worse offered for sale without the benefit of serviceable functionality. Nevertheless, we would not want to bequeath such a mess to a successor programmer tasked with maintaining the application, particularly since that programmer could well be our future self.

As part of the review, we questioned whether the 'proper' object for all of these operations was the Huddle(). What we found were multiple indicators that the modules spiel_views.py and stanza_views.py as well as templates for Rhymes, Stanzas and Spiels are relying on too much "inside information" concerning the Huddle object. The tipoff is direct use of the differences between the d_spiel and e_spiel attributes of the Huddle object. That realization spawned questions concerning the 'proper' object to interface with the database, and as always it seems, the 'proper' vector for communicating exceptions to the users and maintainers of the system.

Step 1: Eliminate references to huddle.e_spiel
Commit ffb1601 shows the changes implemented to do a better job of hiding Huddle() details. Specifically, we added a @property huddle.spiel to serve as the main object communicated from the database to the various view functions. This reorganization shows up in all of the "view" modules and most of their related templates. References to huddle.e_spiel were replaced in the modules and eliminated in the templates in favor of a function-scoped spiel variable. (See, e.g., changes in stanza_views.py and in and correlative changes to new_rhyme.html and edit_stanzas.html and edit_spiel.html.)

The resulting code is much cleaner - local references to a spiel object instead of the unwieldy huddle.e_spiel or huddle.d_spiel and simpler communication to templates with a spiel parameter rather than the entire huddle object.

This abstraction also sets us up better for a future "Undo" function that keeps a deeper stack of changes between updates.

Step 2: Normalize returns from database operations
Commits 1ed28a0 and ffb1601 implement a better orchestrated regime for database returns. Now almost all calls to the database via huddle.get...() operations return a tuple whose first element is a spiel or a list of spiels and whose second element is an error value. The error value is always None if the database operations were successful. Any other value (True, False, or message) indicates trouble of some sort. Other than the fixed meaning of 'success' for the None value, we have not settled on a structure for other return values.

Step 3: Implement the new buttons for edit_rhymes and edit_stanzas
In commit 1823f2b we implement the functionality for the new buttons. Here's how the button arrangement turned out after we worked through our issues:

The delete button [-] is moved to the right side of the text entry area - out of the way so that one cannot hit it by accident. That is lieu of putting in a confirmation screen as we did with the confirm_delete_spiel template. (We're trying to avoid managing more state information than necessary. But we won't be able to avoid this forever.)

At the same time, we eliminated the option to select a subset of Rhymes or Stanzas for editing from the edit_spiel template. We now have single buttons for selecting whether to edit "All Rhymes" or "All Stanzas".

Step 4: A couple of redos
We fudged a little bit in the previous section, making it sound as though we had smoothly updated all of the functionality associated with the new editing buttons. Actually, we were a bit precipitous with a couple of changes that we had to reverse or correct.

First, we thought we had eliminated the need for the +[placeholder] functionality with the introduction of the "insert above" and "insert below" buttons. It was only after we had 86-ed the +[placeholder] button that we discovered we were stymied upon creation of a new Spiel. It turned out that, with that button removed, we had no way of creating the first Rhyme or Stanza for the spiel. We contemplated adding a Rhyme automatically on creation of a new Spiel (and possibly a Stanza as well), but decided that really did not get us where we wanted to go. (Consider, e.g., what would we do if we then deleted the seed Rhyme or Stanza with no others in place?) So, amending that functional oversight, we replaced the +[placeholder] buttons with more specific "+Blank Rhyme" and "+Blank Stanza" buttons. (Commit 0a03f3b.) This is a good example, we think, of eschewing complicating automatic processes for simple manual steps that get the job done.

Second, we thought we had eliminated the need for the huddle.spiel @property with the regularization of the return values from database calls. But it then turned out we had used the huddle.spiel @property repeatedly within the Huddle() object itself. The better solution was to use it for all of the database operations and eliminate returning huddle.e_spiel anywhere. (Commit 1e7a626.)

Step 5: Worth the wait
The unifying concept behind the commits addressed in this post is identification of useful abstractions for modeling a real-world need in a piece of software.

Very early in the development process, we had the idea of keeping ourself on the edit_rhyme and edit_stanza pages as we added new entries for each, but we had other needs that took priority in the development queue.

One common trap for inexperienced developers is premature abstraction***. We invented the Huddle() type [the name is intentionally non-mnemonic] to preserve the current state of the information in the database for testing against the current edited state of the Spiel (including its Rhyme and Stanza elements). But that element has been plastic throughout the process so far, exchanging and sometimes duplicating attributes with the two Spiels that are its principle components. We have come to be a bit more confident about its continuing structure, so now seemed like a good time to tidy up.

BLOX is still a very small application. Nevertheless, it is starting to hit on all cylinders - with significant technical investment in the UI and the database, and more modest investment in the production-deployment process. It's about time to move beyond proof-of-concept level thinking to a this-is-real-software attitude.

***A malady not unknown to more experienced developers as well, of course. It takes a bit of discipline to resist throwing all the cool stuff one knows at even the simplest problems. Even more restraint is called for in the early stages of a larger project which is sure to benefit from advanced expertise.

Keeps on Ticking
It's worth emphasizing that we have overlaid a continuing implementation process that keeps us in production mode even though we are pounding away through this ongoing development process.

If the BLOX project trajectory seems chaotic, blame it on our flighty client with whom we negotiate the order of tasks. His priorities are his own, based as we understand it on his own measure of discomfort or desire.
Rule of Three
We all have our favorite rules of thumb to help guide us through problem solving of all sorts.

One expression concerning software development is the Rule of Three which cautions against automatic or immediate application of the principles like DRY or Once And Only Once. There's a nice discussion of premature abstraction at The Wrong Abstraction.
We Have Proof
As we have mentioned before (and repeatedly) we are our own client in the development of the BLOX project. Our client was impatient to 'get something going' right away. So we have stressed ready functionality over usability - more important to get data into and out of the database than to make a slick UI.

Add to that our comfort with HTML primitives, and you have a formula for a rough-looking web interface. But we began with more aspirational designs that were necessarily subordinated to the client's insistence on "productivity now"**

It's all true. We did indeed aspire to a normal, perfectly usable interface from the very beginning. How can you tell? Simple, just check out the edit_rhymes.html file (lines 27-33) in the BLOX initial commit.

**The client may not always be right - but he is always the client.


It will be some time yet before we get a comments section working here. In the meantime feel free to send comments via email. On this site our name is Joe Python. The email address is our first name at joepython.com.

Edited: 2019-01-30 20:55:12(utc) Generated: 2019-06-10 17:29:58(utc)