Adam's Digital Garden

Python Handbook

Reminders and code snippets for creating Python3 programs.

Development Tools

IDEs

The Language of Python

Vocabulary

Python Naming Conventions

The REPL

This is where the action starts. One liners and simple functions can be written and tested immediately. REPL is an acronym for Read-eval-print-loop. Meaning it will read a command, evaluate it, print it, and then loop to be ready for another command. In this handbook, when a command is written in the REPL it is indicated with »>.

Assuming you have correctly installed Python, to start the REPL use

$ python3

To navigate to different directories in the REPL use

import os # imports are explained under the Library Heading
os.chdir('full/path/to/dir')
os.getcwd() # To see which directory the REPL is looking in

Execute a file in your REPL

exec(open('FILE').read())

Helpful methods

dir()

Gives a directory of all the attributes a given object has. These attributes can be methods aka functions specific to an object. The attributes that start with __ are the special methods, Dunder methods or magic methods. Special methods are used by special operators such as + or %. For information look at SPECIALMETHODS under help().

help()

This could and maybe even should be the only source of knowledge for Python students. For information about any method use help(OBJECT.METHODNAME) for a brief text about what that method does. For example

help('TexT'.swapcase)

tells you what swapcase does. Instead of creating the instance like in the example I can write the type, str, int, float and so on or use the variable I labeled my instance as. Note that writing

help(30.real)

produces an error because with number Python assumed I was creating a float and just did so incorrectly.

help() also shows a methods signature. Between the parenthesis the suffix is required and the information between the square brackets is optional.

import pdb; pdb.set_trace()

Use this at the start of programs you want to debug. This runs the program through an interface similar to the REPL so you can debug it. Use h to see the list of available commands.

A Little About Objects

Truthy or falsey

Objects can be truthy or falsey. Empty strings are falesy, 0 is falsey, empty lists are falsey. See if something is truthy or falsey with bool(OBJECT)

Containers

Every object in python is given an unchanging id, you can see this with id class.

id(name)

Containers hold multiple objects

Lists are mutable object holders. You can add with .append or remove with .remove even insert with .insert. If you run a function on a list like sort(old_list) you will get a new list, but if your the method old_list.sort it will change your existing list.

create a new list with name_of_list = [].

When inserting to the list named_list.insert(0, 'hello') where 0 is the which index and ‘hello’ was the what. If inserting from the start you better use collections.deque as it uses a different data structure

list('Adam') # turns sequence of characters into separate elements in a list

tuples are immutable object holders. there are 3 ways to create tuples, but the Pythonic way is

p = ('first_element', 'look commas', 'party in the tuple')

Tuples are used as keys in dictionaries and use less memory than lists, because tuples are immutable. Generally used to return information from a function.

Sets are mutable unordered collection without duplicates. Look up operations are very cheap for sets as they use hash functions. Sets do not keep elements in order but the “in” operator works fast on sets than lists. sets are created with filled curly brackets {2,1} or directly from the set class integer_list = [0,1,1,2,2,3,4,9,5] integer_set = set(integer_list) integer_set = {1,2}

dir(Class) # lists methods in a class where Class is the name of the class.

Slicing Lists

My_list=[1,2] 
Lists are mutable 
l1 = ['apple', 'pear', 'grapes', 'banana'] 
print(l1) 
print(l1[:1])

Slice a list is done with the list[start:stop] notation.
The number start is which index you start your new list from.

Builtin Methods, Classes, and Libraries

F-strings

New in Python 3 are f-strings.

name = "World"
F"Hello, {name}."

List Comprehension

Lets you look inside a list in a list.

Dictionaries

are highly optimized containers with keys and values. Passing a tuple in would the 0 index a key and the 1 value the value. Dictionaries are indicated with curly brackets { }. You can add keys and values to existing dictionaries. You can pull values directly out of the dictionary just by calling the key Dictionary is a 2 column table, where each key has one value. If you want more columns, you need, for example, a list of dictionaries.

Dictionaries are used for Classes, namespaces, and modules.

Example dictionary

my_dict = {'name': 'Adam', 'surname': 'Smith'}

Pulling Data out of containers (lists, tuples, strings)

Indexing is for specific elements slicing is for sub-sequences

a negative index -1 counts from the back, it’s a[len(a)-X] is a[-X]

you can add indexing to your own classes with .getitem

Slicing can also have a stride, so take every second, or ever third item in the third field

Loops

instead of range and len to loop through a loop use the built in enumerate function.

Functions

You can run help and dir on the functions you create yourself and get useful info out. You write the function definition. You name a function so that you can invoke it by calling the name. then you have the block of code that says what the function does. Add a docstring to make the help of the function more useful. Docstrings sit just after the def line in “““triple quotes "””

When naming your own functions “Function names should be lowercase, with words separated by underscores as necessary to improve readability. Variable names follow the same convention as function names. " From https://peps.python.org/pep-0008/

Built in functions

Full list at Python docs

Performance Notes

result += item is faster than result = result+item because it looks for result only once

Files in and Out

The function to open a file is called open

open(filename, mode='r', buffering=-1, encoding = None, errors=None, newline=None, closefd=True, open=None)

To the list of methods available to a file, remember to use dir(OBEJCTTYPE OR OBJECT INSTANCE) Then to get more information about individual methods get the help for the method with in this case help(passwd_file.close)

In Windows use r"C:\test" when needing to use the backspace. r is for raw, because \ is a Python escape character

passwd_file.readline() to read one line or

listoflines = passwd_file.readlines() to read the files into a list

with open('/tmp/names3.txt', 'w') as fout3:
    fout3.write('Ringo\n')

With closes the file after the block is run. This is the same as

fout3 = open('/tmp/names3.txt', 'w')
fout3.write('Ringo\n')
fout3.close()

the write() method is write a string, the writelines() method is to write a list of strings

Argparse Library

Positional arguments are when the position of argument deters it’s function

For ls, -l is an optional arguement that changes how the command behaves. … –help provides information about how to use the ls command. It doesn’t do anything in windows though

Just importing and using argparse gives –h, –help

Argparse.ArgumentParser() has methods:
Add_argument(“argument_name”) Parse_args() : are the arguments that I have added and makes them accessible with the dot notation. Argparse creates a variable for each argument with the name of the argument. The details magic, meaning you don’t need to know them to use argparse at this level.

Into add_argument we can add a help text as a parameter. It’ll look like this .add_argument(“argument_name”, help=“Here I explain how argument_name is used.”)

By default all arguments passed in are treated as strings. We can specify the type with
.add_argument(“argument_name”, help=“Here I explain how argument_name is used.”, type=int)

So instead of relying on position, we want to name our arguments making the order in which are passed irrelevant and require they are named. This is the magic of starting the argument with –
.add_argument(”–now_optional_and_require_name”)

Unicode Fun

Encode a unicode string to a byte string for storage. Then you decode the byte string to be readable.

We have unicode of variable length so that don’t take up so much memory.

For 4 digit unicode characters \u26F4 More than 4 \U0001F6A2where you pad until 8 digits. Or you can write the name \N{FERRY} or \N{ferry}

You can then encode the characters into ASCII or languages, but mostly only for legacy purposes.

Classes

Except keywords and variable names everything is an object. All objects belong to a class. A class defines what objects can hold and what methods can be used to alter them. An instance is an occurrence of a class.

A class consists of, a name, docstring, class attributes, and def statements (functions in classes are called methods). The variable are called attributes. Classes are named using CamelCasing.

There is a special method in classes called __init__(def, id). It is a constructor, that creates the class. The body of the constructor you give the class the unique attributes each instance of it will have. For example, the attributes in the class Car may be: ```init(self, brand, horsepower, zero_to_sixty, doors)

A car class:

class Car:
    def __init__(self,brand,horsepower,zero_to_sixty,price):
        self.brand = brand
        self.horsepower = horsepower
        self.zero_to_sixty = zero_to_sixty
        self.price = price

    # give a string representation of your object
    # instead of memory address such as <__main__.Car object at 0x108e006d0>
    def __repr__(self):
    	return f"Car (brand: {self.brand}, horsepower: {self.horsepower}, Zero to sixty: {self.zero_to_sixty}, Price: {self.price}"

Methods have a self parameter because you are passing the class to the parameter.

Protected methods are methods that are only to be accessed from within the class. Protected methods start with a _. These should not be called from outside the class and may be removed in future versions of the class.

Subclasses and Superclasses

subclasses inherent from and overrides superclass properties.

class SoyLatte(Latte)
creamer = soy_milk

subclasses do not have a __init__ method, Python looks to the super class

Exceptions

There are several ways to deal with exceptions.

  1. “Look before you leap” (LBYL) Can be done with if statements, to check if result is invalid do the else. Not guaranteed to work as variables and external dependencies can change during program execution.
numerator = 10
divisor = 0
if divisor != 0:
    result = numerator / divisor
else: 
    result = None

Need to make my own examples

  1. “Easier to ask for forgiveness than permission (EAFP) This is the try…expect blocks, and the most used method to avoid errors.
numerator = 10
divisor = 0
try:
    result = numerator / divisor
except Zero DivisionError as e:
    result

e is the instance of the error, making it accessible as a variable. Only put one line in the try statement so you can be specific in dealing with the error occurrence.

Server type applications process_input and log_error(e) serve as placeholders

The three error handling guidelines, quoted from IGTP.

  1. gracefully handle errors you know how to handle and can reasonable expect
  2. Do not silence exceptions that you cannot handle or do not reasonably expect
  3. Use global exception handler to gracefully handle unexpected errors

With tuples an exception statement can catch more then one exception type except (TypeError, ValueError):

finally for code that will run whether or not the exception was met. Used for freeing up external resources if the program fails.

Try and finally have a code smell, so look into using context managers

Collections

namedtuple

instead of having to use index values to refer to tuple elements you can use strings

Libraries

To use a library load it into your namespace. You can check if a variable is in your library with >>>'sin' in dir()

Importing a library is the creation of a variable that points to the function in that library. If you import the whole library then the functions are not in your namespace you will need specify that you are using the library. See example below.

import os
os.getcwd() # To show what directory the console is running
os.chdir() # Change the directory you are in

Help also works here, so you can use >>> help(os.getcwd) When importing specific functions, you will not need to specify the library to look in when running the function.

from os import getcwd, chdir

Use parenthesis if importing functions over multiple lines

from os import (getcwd,
... chdir)

You can choose the name of the variable you are using to call the library.

from os import getcwd as cwd
import numpy as np
import pandas as pd

star imports from os import *import all the functions, class definitions and variables in the namespace. This is not recommended because when importing multiple libraries the name may overwrite each other.

Some libraries are nested so you need specify the full path the functions you want to reach

from xml.dom.minidom import parseString

Be careful of circular imports, where modules mutually import one and other.

Modules and Packages

A library is made of Modules that can be grouped into Packages Packages are directories and require an __.init__.py file to be a package.

module names are short, lowercase and may have underscores packages names are short, lowercase and without underscores

You can set a PYTHONPATH if your libraries are not installed and you need to specify where your libraries are.

You can check under which paths your python REPL looks for libraries with

import sys
sys.path

In rare cases you may want to manually add a python library to your program. This can be done by either adding the path to PYTHONPATH before launching or to the sys.path after having run the program

$ PYTHONPATH = /path/to/library python3

or

import sys
sys.path.append('/path/to/library') # You should write the full path, but relative to where you are also works

Datetime

datetime.today() is a datetime.datetime object

different datetime objects can create different objects datetime.datetime and datetime.date are two different object types

can do math with dates

start_time = datetime.today()
initial_end_time = timedelta(hours=1)
start_datetime.replace(hour=8)

what is time.time() ? It gives the time in seconds from the epoch.

PYTZ Library

pytz brings the Olson tz database into Python. This library allows accurate and cross platform timezone calculations using Python 2.4 or higher. Example code of converting datetimes to different timezones

import pytz 
from datetime import datetime 
from pytz import timezone 
 
input_time="2023-03-23T03:00" 
 
utc = pytz.utc 
cet = timezone('Europe/Copenhagen') 
print(cet.zone) 
print(utc.zone) 
fmt = '%Y-%m-%d %H:%M:%S %Z%z' 
start_datetime=datetime.fromisoformat(input_time) 
 
cet_start_datetime=cet.localize(start_datetime) 
utc_start_datetime=cet_start_datetime.astimezone(utc) 
 
print(F"Not pickled:                {start_datetime}") 
print(F"Pickeled with CET datetime: {cet_start_datetime.strftime(fmt)}") 
print(F"Converted to  UTC datetime:  {utc_start_datetime.strftime(fmt)}")

sched I create a schedule with s.enter(10,1,action) then I run the schedule

s.empty() shows if there are any events pending execution s.run() runs the created schedule s.cancel() I can can cancel events, this I can use when on breaks

Python Library Notes

Writing and Sharing Programs

Let’s move out of the REPL and start writing programs allows for complicated tasks to be completed. Here is where an IDE really starts being useful, as well as using version control to keep track of your work. I have always used Git, therefore this guide also uses Git, github to be more specific. Being familiar with version control and an IDE are important skills, however beyond the scope this guide.

Virtual Environments

When working in terminal it is recommended to use virtual environments (venv) as this will not change your base python installation and make it easy to share the libraries needed for a program. My preferred location for python is ~/Library/venvs to keep them out of my repository.

$ python3 -m venv <venv folder>

With a virtual environment you can collect all the installs you made to have your program run into a requirements.txt file. This can then be shared with others so they can recreate your environment and run your programs. This is safer than copying your venv folder over because they may be running a different version of Python or a different OS.

$ pip freeze > requirements.txt # to list your installed libraries in a file
$ pip install -r requirements.txt # to install your libraries from requirements.txt

To activate a virtual environment

MacOS

$ source PATH_TO_VENV_FOLDER/bin/activate

Powershell

./PATH_TO_VENV_FOLDER/source/activate

To update pip on macOS. python3 -m install --upgrade pip

Style Guide

To make reading others guide easier this guide has use the PEP8 style guide. It provides guidelines for how many spaces to use on indents, how to name variables and classes, and a lot more. When sharing with others it is good to agree on a style to make your code legible and not waste time refactoring code between programmer styles.

Parts of a program

From IGTP

#!/usr/bin/env python3 # if file is executable as a script

module docstring imports metadata / globals/ logging implementation if name == ‘main’: # if file is executable as a script argparse

Testing your Code

Types of testing

Automated testing is the opposite of exploratory testing. Exploratory testing is using the program/exploring the program to find errors.

The simplest start of a test is to use an assert test

def test_sum():
    assert sum([1, 2, 3]) == 6, "Should be 6"

def test_sum_tuple():
    assert sum((1, 2, 2)) == 6, "Should be 6"

if __name__ == "__main__":
    test_sum()
    test_sum_tuple()
    print("Everything passed")

To streamline running multiple tests, there are test runners. 3 popular test runners are unittest, nose2 and pytest.

nose2 adds features on top of unittest.

Unittest

To run a unit test python3 unittest_threats.py KingThreatsTestCase This example runs a unit test on the kings_threats function

import unittest
import sys
from piece_threats import king_threats,trim_threatened

class KingThreatsTestCase(unittest.TestCase):
# Pass into test (N,i, current_column, threatened), board_size)
    def test_king_2x2_1(self):
        result = trim_threatened(king_threats(2,2,1,1,{1}), 4)
        self.assertEqual(result, {1,2,3,4})


if __name__ == '__main__':
    unittest.main()

Building Applications

Once the application is written it is time to build it so it can be distributed and run.

Appendix

Zen of Python

Sources