Keeping it Clean - Use Python Modules to Organize Code

The problem

  I have said many times that it is easy to write code that simply “does something”. The difference between “someone who can code” and a seasoned developer comes down to a few different things, one of them being code organization/readability. I can’t count the number of projects that I expected to be short scripts, or even started as one-liners in bash, that grew into something way bigger. One of the things that can come back to bite in these projects is a reckless attitude of “It’s a small app. I don’t really need to worry about code organization.” or “I can read it, and who else is ever going to look at this anyway?”. I have learned my lesson the hard way about this kind of development and, while not perfect, have over the years become more apt to start from the beginning as if the code I am writing will be seen by the world and will need to be infinitely aded to. So the question becomes “How do I organize my code???”. While this is a fairly broad question this post will focus on one aspect of code organization in Python - modules and packages.

  Virtually anyone who has written any Python has used imports. Sometimes these come from the standard modules library and sometimes they are third-party. I know when I first started developing I didn’t really give much thought as to where these third-party imports originated. In my mind they were generated by the dark magicians of the Python world whose secrets I would never unlock. Then came the day when I had to face the truth about the application that I was working on - The 1800 line script that I was writing was totally unmanageable. Looking for a reference in the code was painful, debugging the code was a nightmare, and nobody (not even me) could possibly look at this monster and in any quickly tell what it did or how it did it. I was forced, as it were, to look for a way to solve this.

The solution

  In short, Python modules allow you to separate your code into separate files that can be imported and reused by other scripts. This is very obvious when installing a package from PyPi, but did you know that those modules are just Python scripts, like any other Python scripts, except organized and stored in a location where Python knows where to find them? Did you know that you can do the same thing, even without publishing the code to PyPi? Well you can! And it is insanely easy too!

How it works

  A Python module can be as simple as a file in the same directory as the script that imports it. To see this we can put the following into a file named foo.py

print("You imported foo!!!!!!!")

Then we can import it from a terminal

>>> import foo
You imported foo!!!!!!!
>>>

As you can see, when we imported foo the code in foo.py was executed. This is because, when looking for modules, Python looks first in the directory that the current script is running in. It then looks in your environment’s $PYTHONPATH, and then the system-defined locations. This little fact gives us a way to compartmentalize our code and even easily reuse it in another application.

In the example we put an statement outside of any function, variable, or class declaration. This caused the code to be executed immediately when the module was imported. This is generally used for initializing any resources that the code inside of the module needs but doesn’t really help our application. For a useful module we need to declare a callable or variable that is of some use to us. Lets replace the code in foo.py with this:

def imported_who():
    print("Imported foo!!!!!!")

Now lets import and call our function:

>>> from foo import imported_who
>>> imported_who()
Imported foo!!!!!!!
>>>

Because EVERYTHING in Python is an object we can treat the declared objects inside of foo.py as attributes of foo. This works with variables, functions, classes, etc. While just a silly example, you can easily see how this would allow us to divide our code into logical groups of functions/classes based on what they do.

Packages

Python modules can be organized into packages. A package is simply a directory that contains modules, and normally, a init.py file. The contents of init.py are executed and it’s variables, functions, and classes available for import when the module is imported. Modules (Python files inside of the package directory) are accessed as attributes of the package. For example lets create a package named my_package by creating a directory named my_package and putting the following into my_package/init.py:

print("Package imported")

def what_package():
    print("My Package!!!!!")

We can then import the package and it’s function like this:

>>> from my_package import what_package
Package imported
>>> what_package()
My Package!!!!!
>>>

As you can see, the code outside of any definition was executed upon import and we were able to import the function what_package() from the package. Now lets move our module file, foo.py, into the ./my_package directory. We can now access the function in module foo as a member of the my_package package:

>>> # Import the function directly from the foo module inside of my_package
>>> from my_package.foo import imported_who
Package imported
>>> imported_who()
Imported foo!!!!!!!
>>> # Now lets access it all the way back from my_package
>>> import my_package
>>> my_package.foo.imported_who()
Imported foo!!!!!!!
>>>

Wrapping up

Being able to modularizing code into modules, submodules, and packages is a skill that every Python developer should have. By modularizing our code into modules and packages we can

• Make our code more obvious in what it does since each module/package provides tools to accomplish a specific task
• Make our code reusable
• Save the next guy the frustration of having to look through hundreds or thousands of lines of code to find one little thing
• Reduce the chance of git conflicts arising from multiple developers committing to the same file at the same time since the code is spread out across more files
• Take our first steps towards contributing a Python module to the OpenSource community (These things don’t write themselves)

There is a lot more to the subject of modules and packages and I highly suggest looking at the official Python docs HERE

About the Author

Linux loving , Python slinging, OpenSource evangelizing Senior Solutions Architect at Quinovas