Making Re-Usable Code - Some Design Concepts

Matt Clarkson, 2019-10-30

Tutorial is hosted on gitlab, and displayed on readthedocs.

(in fact, if you are reading this on readthedocs, the page itself is generated from that jupyter notebook)

Pre-requisites

Ensure you have already:

These tutorials are important, as the same PythonTemplate and hence tox and virtualenv layout is used for this project.

The Problem

Problems with research software include:

  • Code that only one researcher can run.
  • At the end of a project, the code dies, is not re-used, and subsequent researchers feel compelled to re-implement it, in their own nuanced way, thereby wasting time, and also repeating the same loop. This may not be a huge concern in the era of deep-learning, as a new researcher will likely implement something newer and better. But if you want to use an algorithm in any other piece of code, the code must be designed for re-use.
  • Hard coded parameters, with unknown history. i.e. what params have been tested? When? With what version of code?

The Solution

A researcher should develop code that:

  1. is designed for re-use, i.e. a clear, simple interface, so the code can be directly embedded in other third party programs, such as GUI’s or other scripts, without cutting-and-pasting.
  2. has core functions, run and tested via unit tests.
  3. has command line entry points so an untrained user can just run it.
  4. can be used within jupyter notebooks, as this is good for development and supervision meetings.
  5. can be pip installed by others, and re-used as is, with almost zero effort.
  6. has no hard-coded parameters.

In this tutorial, we implement a simple classifier in TensorFlow, to demonstrate the above steps. The classification code itself is inspired by the standard Fashion MNIST tensor flow tutorials, such as this one and this one. The classifer per se, is not the point of the tutorial. The point is to demonstrate how to make code that can be widely re-used.

The Design

In this notebook, we will step through the design of our Fashion MNIST example, and explain the design choices.

Class interface

First, we can choose either a class-based interface, or a function-based interface. I chose class. A class is a way of grouping data-members and methods into a coherent concept, and providing encapsulation. So, just like a black box, the user doesn’t have to know the internals of how a class works, they can just use the interface. If you chose a function based approach, then related methods are not easily grouped together, and its not obvious how separate methods are related, or in which order the should or must be called. So in the long run, funtion-based code, with lots of functions gets more disjoint and messy. So I prefer a class.

So, in file sksurgerytf/models/fashion.py we basically have the following, written as pseudo-code:

class FashionMNIST:
    __init__(params)
    train()
    test(image_to_classify)

where

  • The constructor is responsible for initialising the network. See this article by Martin Fowler, and how the class is loading/preparing data. Also see this on RAII and books by Scott Meyer to get the idea that once the constructor is complete, you should have a fully usable object. i.e. you must never allow an unusable or unready object to exist.
  • train() method to train the network. In this example, the train() method is called from within the constructor. You could call it separately.
  • test() method to classify each new image. This would be something that a 3rd party user would call, without knowing what goes on inside the black box.

So, by using encapsulation, and a simple class API, we have addressed point 1 of our proposed solution.

Modules for Functions/Classes and Unit Tests

Under sksurgerytf/ we can put any other sub-modules, classes and functions as necessary.

For example:

sksurgerytf/maths/matrix_algebra.py

would have its corresponding unit test:

tests/maths/test_matrix_algebra.py

However, that said, its difficult to unit test large networks that take days/weeks to train. Unit tests must be fast, so in all likelihood, we are talking about testing small individual functions. Try to break out your functionality into bits you can test without running a full training cycle, i.e. test with dummy data.

So, by separating classes and functions, and having separate unit tests, we will address point 2 of our proposed solution.

Command Line Entry Point

For bash scripting, or for working with other people, it is useful to have a command line entry point. This repo provides a pattern that you can just copy for each command line entry point.

So, the top level python script:

sksurgeryfashion.py

contains:

from sksurgerytf.ui.sksurgery_fashion_command_line import main

which runs a command line parser in

sksurgerytf/ui/sksurgery_fashion_command_line.py

which calls through to the fashion.py module created above.

In this way, a non-trained user can just run the code, like this:

# To setup the same virtualenv as tox installed
source .tox/py36/bin/activate

# Run program, just printing command line args
python sksurgeryfashion.py --help

So, we now have the same code called by unit tests (point 2) AND a command line program (point 3), so, we have addressed point 3 of our proposed solution.

Running via Jupyter Notebook

Thanks to this blog post from Angelo Basile.

The reason we started with a standard python script is because once a network is developed, its more likely to be run in standard python scripts, on a cluster/GPU node, or embedded in a larger program using the Python import mechanism. So the design above supports this. However, Jupyter notebooks are useful for development, and for writing up weekly supervisions. Here is an example of how to run our FashionMNIST class inside a jupyter notebook.

NOTE:

This relies on 3 things:

  • In the top level tox.ini, see commands_pre=ipython kernel install --user --name=sksurgerytf which creates a python kernel inside the tox environment.
  • You must ensure you start jupyter within the tox environment.
# If not already done.
source .tox/py36/bin/activate

# Then launch jupyter
jupyter notebook
  • Then when you navigate to and run this notebook, select the sksurgerytf kernel from the kernel menu item, in the web browser.
[1]:

# Jupyter notebook sets the cwd to the folder containing the notebook.
# So, you want to add the root of the project to the sys path, so modules load correctly.
import sys
sys.path.append("../../")

# This will load the module, create the network, and run a simple training.
# If this completes without errors, we have a valid way of running notebooks.
from sksurgerytf.models import fashion as f
fmn = f.FashionMNIST()

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
flatten (Flatten)            (None, 784)               0
_________________________________________________________________
dense (Dense)                (None, 128)               100480
_________________________________________________________________
dense_1 (Dense)              (None, 10)                1290
=================================================================
Total params: 101,770
Trainable params: 101,770
Non-trainable params: 0
_________________________________________________________________
Train on 60000 samples, validate on 10000 samples
60000/60000 [==============================] - 4s 75us/sample - loss: 0.4985 - accuracy: 0.8249 - val_loss: 0.4560 - val_accuracy: 0.8366
10000/1 - 0s - loss: 0.3516 - accuracy: 0.8366

So, the ability to run the same class in a jupyter notebook, accomplishes point 4.

Make Code Pip-Installable

The top level setup.py is responsible for installing a python package/module in a users environment according to normal python conventions. Normally, to distribute code, you would do:

  • Install your code on PyPi.org. See this tutorial.
  • The 3rd party user then just does pip install to install it

Alternatively, the 3rd party user would:

  • git clone your repository
  • Does pip install . in the top-level folder.

Both of these methods need setup.py

So, you should edit setup.py, paying particular attention to:

  • Setting your name and email address
  • Ensuring install_requires matches requirements.txt for all 3rd party dependencies, including all the version numbers. Unfortunately, we don’t have a nice way of avoiding this duplication yet.

However, we also want our command line apps to be pip-installed to. The section:

'console_scripts': [
    'sksurgeryfashion=sksurgerytf.ui.sksurgery_fashion_command_line:main',
],

creates a new command line application called sksurgeryfashion that runs the same main function as above. This command line application does not have a .py extension, and does not need running via the python interpretter. To the end-user it looks like any other shell command (e.g. native unix/bash/windows commands).

So, for each of your command line apps, you should create an entry under console_scripts that will allow a 3rd party user to run your code, without knowing how any of it works.

This then accomplishes point 5 on the above list.

A Note On Object Re-Use

Consider the two code fragments, written as pseudo-code:

class FashionMNIST()
    __init__()
    set_param1(param1)
    set_param2(param2)
    set_param3(param3)
    train()
    test()

which only has constructor arguments, and no setters. Compare this with:

class FashionMNIST()
    __init__(param1, param2, param3)
    train()
    test()

which is best?

Arguably it depends what you are trying to achieve. However, it becomes interesting if you want to re-use an object. In the first example, its not obvious if calling the setters is mandatory, and in which order they should be called. Also, if you train() once, then call a setter and train() again, what should the expected behaviour be? The object is being re-used, so does the training re-start from scratch or where it left off? Does the new parameter value take effect for the second training, or does setting a parameter value cause the training to restart? All of these points can of course be clarified with effective documentation for each method.

However, the point about the second example is that its more obvious to the 3rd party that the object is designed not to have the parameter values set once the object is constructed. In other languages such as C++ this can be enforced via public, protected, private mechanisms. The fact that Python doesn’t protect variables shouldn’t dissuade you from the general idea that it is best to define the code to automatically indicate to the end user, the way in which a class is meant to, or designed to be used.

So, I would use these two types of class design to indicate what my intention was, in terms of re-use.

In general, I prefer to use constructor arguments, and no setters. The user should at least get the idea that to test new values, and new settings of hyperparameters for example, they are then required to make a new object, which makes it fairly obvious that the training is then starting from scratch. So, try to make the class design informative of how the class should/must be used.

Hard-Coded Parameters

General advice would be to avoid hard-coded parameters. However, a researcher may have spent a lot of time optimising a given parameter, and specifically does not want anyone else to change this value. Or, a parameter value may not have been extensively tested or validated, so what should a researcher do then? There is a trade off between providing 3rd party users with extra parameters to tweak and hence extra flexibility, and the downside of 3rd party users complaining “your code doesn’t work with parameter X set to value Y, why not?”

So the general advice might be better phrased as: “Provide tunable parameters, where you think there is some benefit to doing so, and you have reasonable expectation that it would be useful to 3rd parties”.

Then, following from the conversation above, you just have to decide if they should be constructor arguments, and no setter methods provided, or a class with default constructor, and setter methods.

Other Thoughts

  • What if your network is not a classifier? Same principles apply. Users still want a nice simple interface, even if the internal workings of your class is fairly complex.
  • What about global parameters? Don’t make global variables. Try class member variables, and initialise in constructor.

Logging

And finally, make sure you log as much stuff as reasonable.