CS3101 Python Lecture 3

1
CS3101 PythonLecture 3
2
Agenda

• Scoping
• Documentation, Coding Practices, Pydoc
• Functions
• Named, optional, and arbitrary arguments
• Generators and Iterators
• Functional programming tools
• lambda, map, filter, reduce
• Regular expressions
• Homework 3

3
Extra credit solution to HW1

• Dynamic programming 15- lines
• determining whether a bill of N dollars is
  satisfiable resolves to whether you can satisfy a
  bill of N J dollars where J is an item on your
  menu
• Create an empty list (knapsack) with N1 entries
• Base case we know we can satisfy a bill of 0
  dollars
• For each item on your menu
• For index 0 to N 1
• If knapsackindex is empty, and knapsackindex
  items cost is not
• We now know how to satisfy this bill, so append
  the current item to a solution list which lives
  at knapsackindex

4
Homework 3, Exercise 1

• Requirements
• 1. Write a program using regular expressions
  retrieve the current weather from a website of
  your choosing. Just the temperature is OK.
• 2. Use that information to suggest a sport to
  play.
• ./sport.py
• Its 36 degrees today. You should ski!

http//www.nytimes.com/weather
5
Homework 3, Exercise 2

• Requirements
• a) Write a program which uses regular expressions
  and URLLIB to print the address of each image on
  Columbias homepage (www.columbia.edu)
• b) Use regular expressions to print the title of
  each of the news stories on the main page
• ./news.py
• ./images.py

6
Scoping

• Local Namespaces / Local scope
• A functions parameters and variables that are
  bound within the function
• Module scope
• Variables outside functions at the module level
  are global
• Hiding
• Inner scope wins if a name conflict occurs
  between a local variable and a global one, the
  local one takes precedence

7
Global statement

• Local scope wins by default
• If within a function you must refer to a global
  variable of the same name, redeclare it first
  with the global keyword
• global identifiers, where identifiers contains
  one or more IDs separated by commas
• Never use global if your function just accesses a
  global variable, only if it rebinds it
• Global in general is poor style, it breaks
  encapsulation, but youll see it out there

8
Closures and Nested Scope

• Using a def statement with another functions body
  defines a nester or inner function
• The parent function is referred to as a the outer
• Nested functions may access outer functions
  parameters - but not rebind them
• This trick can be used to form closures as well
  see in lambdas

9
Closures

• This example adopted from Python in a Nutshell
• def make_adder(augend)
• def add(addend)
• return addendaugent
• return add
• Calling make_adder(7) returns a function that
  accepts a single argument and adds seven to it

10
Namespace resolution

• Name resolutions proceeds as follows
• Local scope (i.e., this function)
• Outer scope (i.e., enclosing functions)
• Module level scope (i.e., global variables)
• Built in scope (i.e., predefined python keywords)
• A word to the wise - do not name your variables
  when there is a danger of conflicting with
  modules your may import
• E.g., open 5 is dangerous if youre using
  file objects, later use of the open method might
  not resolve where you expect!

11
Documentation and Pydoc
def complex(real0.0, imag0.0) """Form a
complex number. Keyword arguments real --
the real part (default 0.0) imag -- the
imaginary part (default 0.0) """ if imag
0.0 and real 0.0 return complex_zero ...

• String literal beginning method, class, or
  module
• One sentence concise summary, followed by a
  blank, followed by detail.
• References
• http//www.python.org/dev/peps/pep-0257/

12
Code is read MANY more times than it is written

• Trust me, its worth it
• First line should be a concise and descriptive
  statement of purpose
• Self documentation is good, but do not repeat the
  method name! (e.g., def setToolTip(text) sets
  the tool tip)
• Next paragraph should describe the method and any
  side effects
• Then arguments

13
Pythons thoughts on documentation

• A Foolish Consistency is the Hobgoblin of Little
  Minds
• http//www.python.org/dev/peps/pep-0008/

14
Functions, returning multiple values

• Functions can return multiple values (of
  arbitrary type), just separate them by commas
• Always reminded me of MATLAB
• def foo()
• return 1,2,3, 4, (5,6)
• myList, myInt, myTuple foo()

15
A word on mutable arguments

• Be cautious when passing mutable data structures
  (lists, dictionaries) to methods - especially if
  theyre sourced from modules that are not your
  own
• When in doubt, either copy or cast them as tuples

16
Semantics of argument passing

• Recall that while functions can not rebind
  arguments, they can alter mutable types
• Positional arguments
• Named arguments
• Special forms (sequence) and (dictionary)
• Sequence
• zero or more positional followed by
• zero or more named
• zero or 1
• zero or 1

17
Positional arguments

• def myFunction(arg1, arg2, arg3, arg4, arg5,
  arg6)
• .....
• Potential for typos
• Readability declines
• Maintenance a headache
• Frequent headache in Java / C (Im sure we can
  all recall some monster functions written by
  colleagues / fellow students)
• We can do better

18
Named arguments

• Syntax identifier expression
• Named arguments specified in the function
  declaration optional arguments, the expression is
  their default value if not provided by the
  calling function
• Two forms
• 1) you may name arguments passed to functions
  even if they are listed positionally
• 2) you may name arguments within a functions
  declaration to supply default values
• Outstanding for self documentation!

19
Named argument example

• def add(a, b)
• return a b
• Equivilent calls
• print add(4,2)
• print add(a4, b2)
• print add(b2, a4)

20
Default argument example

• def add(a4, b2)
• return ab
• print add(b4)
• print add(a2, b4)
• print add(4, b2)

21
Sequence arguments

• Sequence treats additional arguments as iterable
  positional arguments
• def sum(args)
• equivilent to return sum(args)
• sum 0
• for arg in args
• sum arg
• return sum
• Valid calls
• sum(4,2,1,3)
• sum(1)
• sum(1,23,4,423,234)

22
Sequences of named arguments

• dct must be a dictionary whose keys are all
  strings, values of course are arbitrary
• each items key is the parameter name, the value
  is the argument

collects keyword arguments into a
dictionary def foo(args) print
args foo(homerdonut,\ lisa
tofu) 'homer' 'donut', 'lisa' 'tofu'
23
Optional arguments are everywhere
three ways to call the range function up
to range(5) 0, 1, 2, 3, 4 from, up
to range(-5, 5) -5, -4, -3, -2, -1, 0, 1, 2, 3,
4 from, up to , step range(-5, 5, 2) -5, -3,
-1, 1, 3
24
Arbitrary arguments example

• We can envision a max function pretty easily
• idea max2(1,5,3,1)
• gtgtgt 5
• idea max2(a, b, c, d, e)
• gtgtgt e
• def max2(args)
• for arg in args

25
Arbitrary arguments example

• def max1(args)
• best args0
• for arg in args1
• if arg gt best
• best arg
• return best
• def max2(args)
• return sorted(args)0

26
Argument matching rules

• General rule more complicated to the right
• For both calling and definitional code
• All positional arguments must appear first
• Followed by all keyword arguments
• Followed by the form
• And finally

27
Functions as arguments

• Of course we can pass functions are arguments as
  well

• def myCompare(x, y)
• sorted(5, 3, 1, 9, cmpmyCompare)

28
Lambdas

• The closer you can get to mathematics the more
  elegant your programs become
• In addition to the def statement, Python provides
  an expression which in-lines a function similar
  to LISP
• Instead of assigning a name to a function, lambda
  just returns the function itself anonymous
  functions

29
When should you use Lambda

• Lambda is designed for handling simple functions
• Conciseness lambdas can live places defs cannot
  (inside a list literal, or a function call
  itself)
• Elegance
• Limitations
• Not as general as a def limited to a single
  expression, there is only so much you can squeeze
  in without using blocks of code
• Use def for larger tasks
• Do not sacrifice readability
• More important that your work is a) correct and
  b) efficient w.r.t. to people hours

30
Quick examples

• Arguments work just like functions including
  defaults, , and
• The lambda expression returns a function, so you
  can assign a name to it if you wish

foo (lambda a, bsimpson a
b) foo(lisa) lisa simpson foo(bart) bart
simpson
31
More examples

• Embedding lambdas in a list
• myList (lambda x x2), (lambda x x3)
• for func in myList
• print func(2)
• 4
• 8
• Embedding lambdas in a dictionary
• donuts 'homer' (lambda x x 4), 'lisa'
  (lambda x x 0)
• Donutshomer(2)
• 8
• Donutslisa(2)
• 0

32
Multiple arguments

• (lambda x, y x " likes " y)('homer',
  'donuts')
• 'homer likes donuts

33
State

• def remember(x)
• return (lambda y x y)
• foo remember(5)
• print foo
• ltfunction ltlambdagt at 0x01514970gt
• foo(2)
• 7

34
Maps

• One of the most common tasks with lists is to
  apply an operation to each element in the sequence

w/o maps donuts 1,2,3,4 myDonuts for d
in donuts myDonuts.append(d 2) print
myDonuts 2, 4, 6, 8 w maps def more(d)
return d 2 myDonuts map(more, donuts) print
myDonuts 2, 4, 6, 8
35
Map using Lambdas
donuts 1,2,3,4

• def more(d) return d 3
• myDonuts map(more, donuts)
• print myDonuts
• 3, 6, 9, 12

myDonuts map((lambda d d 3), donuts) print
myDonuts 3, 6, 9, 12
36
More maps

• map is smart
• understands functions requiring multiple
  arguments
• operates over sequences in parallel
• pow(2, 3)
• 8
• map(pow, 2, 4, 6, 1, 2, 3)
• 2, 16, 216
• map((lambda x,y x " likes " y),\
• 'homer', 'bart', 'donuts', 'sugar')
• 'homer likes donuts', 'bart likes sugar

37
Functional programming toolsFilter and reduce

• Theme of functional programming
• apply functions to sequences
• Relatives of map
• filter and reduce
• Filter
• filters out items relative to a test function
• Reduce
• Applies functions to pairs of items and running
  results

38
Filter

• range(-5, 5)
• -5, -4, -3, -2, -1, 0, 1, 2, 3, 4
• def isEven(x) return x 2 0
• filter ((isEven, range(-5,5))
• -4, -2, 0, 2, 4
• filter((lambda x x 2 0), range(-5, 5))
• -4, -2, 0, 2, 4

39
Reduce

• A bit more complicated
• By default the first argument is used to
  initialize the tally
• def reduce(fn, seq)
• tally seq0
• For next in seq
• tally fn(tally, next)
• return tally
• FYI More functional tools available

• reduce((lambda x, y x y), \
• 1,2,3,4)
• 10
• import operator
• reduce(operator.add, 1, 2, 3)
• 6

40
List comprehensions revisited combining filter
and map

• Say we wanted to collect the squares of the
  even numbers below 11
• Using a list comprehension
• x 2 for x in range(11) if x 2 0
• 0, 4, 16, 36, 64, 100
• Using map and filter
• map((lambda x x 2), filter((lambda x x 2
  0), range(11)))
• 0, 4, 16, 36, 64, 100
• Easier way, this uses the optional stepping
  argument in range
• x 2 for x in range(0,11,2)
• 0, 4, 16, 36, 64, 100

41
Reading files with list comprehensions

• old way
• lines open(simpsons.csv).readlines()
• homer,donut\n, lisa,brocolli\n
• for line in lines
• line line.strip()
• with a comprehension
• line.strip() for line in open(simpsons.csv).rea
  dlines()
• homer,donut, lisa,brocolli
• with a lambda
• map((lambda line \
• line.strip(), open(simpsons.csv).readlines())

42
Generators and Iterators

• Generators are like normal functions in most
  respects but they automatically implement the
  iteration protocol to return a sequence of values
  over time
• Consider a generator when
• you need to compute a series of values lazily
• Generators
• Save you the work of saving state
• Automatically save theirs when yield is called
• Easy
• Just use yield instead of return

43
Quick example

• def genSquares(n)
• for i in range(N)
• yield i 2
• print gen
• ltgenerator object at 0x01524BE8gt
• for i in genSquares(5)
• print i, then,
• 0 then 1 then 4 then 9 then 16 then

44
Error handling preview

• def gen()
• i 0
• while i lt 5
• i1
• yield i 2
• x gen()
• x.next()
• gtgt gt 1
• x.next()
• gtgtgt 4
• Traceback (most recent call last)
• File "ltpyshell110gt", line 1, in ltmodulegt
• x.next()
• StopIteration

try x.next() except StopIteration print "done
45
5 Minute Exercise

• Begin writing a generator produce primes
• Start with 0
• When you find a prime, yield (return) that value
• Write code to call your generator

def genPrimes() . yield prime def main() g
genPrimes() while True print g.next()
46
Regular Expressions

• A regular expression (re) is a string that
  represents a pattern.
• Idea is to check any string with the pattern to
  see if it matches, and if so where
• REs may be compiled or used on the fly
• You may use REs to match, search, substitute, or
  split strings
• Very powerful a bit of a bear syntactically

47
Quick examples Match vs. Search

• import re
• p re.compile('a-z')
• m pattern.match('donut')
• print m.group(), m.start(), m.end()
• donut 0 5
• m pattern.search('12 donuts are
• \ better than 1')
• print m.group(), m.span()
• donuts (3, 9)

m pattern.match( \ 12 donuts are better \
than 1') if m print m.group() else
print "no match no match
48
Quick examples Multiple hits

• import re
• p re.compile('\d\sdonuts')
• print p.findall('homer has 4 donuts, bart has 2
  donuts')
• '4 donuts', '2 donuts'
• import re
• p re.compile('\d\sdonuts')
• iterator p.finditer('99 donuts on the shelf, 98
  donuts on the shelf...')
• for match in iterator
• print match.group(), match.span()
• 99 donuts (0, 9)
• 98 donuts (24, 33)

49
Re Patterns 1
Pattern Matches
. Matches any character
Matches the start of the string
Matches the end of the string
Matches zero or more cases of the previous RE (greedy match as many as possible)
Matches one or more cases of the previous RE (greedy)
? Matches zero or one case of the previous RE
?, ? Non greedy versions (match as few as possible)
. Matches any character
50
Re Patterns 2
Pattern Matches
\d, \D Matches one digit 0-9 or non-digit 0-9
\s, \S Matches whitespace \t\n\r\f\v or non-whitespace
\w, \W Matches one alphanumeric char (understands Unicode and various locales if set)
\b, \B Matches an empty string, but only at the start or end of a word
\Z Matches an empty string at the end of a whole string
\\ Matches on backslash
m,n Matches m to n cases of the previous RE
Matches any one of a set of characters
Matches either the preceding or following expression
() Matches the RE within the parenthesis
51
Gotchas

• RE punctuation is backwards
• . matches any character when unescaped, or an
  actual . when in the form \.
• and carry regular expression meaning
  unless escaped

52
Quick examples vs. . \b

• . vs .

\b

• The pattern
• Homer.Simpson will match
• HomerSimpson
• Homer Simpson
• Homer Jay Simpson
• The pattern
• Homer.Simpson will match
• Homer Simpson
• Homer Jay Simpson

• The pattern
• r\bHomer\b will find a hit searching
• Homer
• Homer Simpson
• The pattern
• r\bHomer will find a hit searching
• HomerJaySimpson

53
Sets of chars

• Sets of characters are denoted by listing the
  characters within brackets
• abc will match one of a, b, or c
• Ranges are supported
• 0-9 will match one digit
• You may include special sets within brackets
• Such as \s for a whitespace character or \d for a
  digit

p re.compile('HJ') iteratorp.finditer(\ Home
rJaySimpson") for match in iterator print
match.group(), \ match.span() H
(0, 1) J (5, 6)
54
Alternatives s

• A vertical bar matches a pattern on either side

import re p re.compile(HomerSimpson') iterator
p.finditer(HomerJaySimpson") for match in
iterator print match.group(),
match.span() Homer (0, 5) aco (8, 12)
55
RE Substitution

• import re
• line 'Hello World!'
• r re.compile('world', re.IGNORECASE)
• m r.search(line)
• gtgtgt World
• print r.sub('Mars!!', line, 1)
• gtgtgt Hello Mars!!!

56
RE Splitting

• import re
• line 'lots 42 of random 12 digits 77'
• r re.compile('\d')
• l r.split(line)
• print l
• gtgtgt 'lots ', ' of random ', ' digits ', ''

57
Groups ()

• Frequently you need to obtain more information
  than just whether the RE matched or not.
• Regular expressions are often used to dissect
  strings by writing a RE divided into several
  subgroups which match different components of
  interest.

p re.compile('(homer\s(jay))\ssimpson') m
p.match('homer jay simpson') print
m.group(0) print m.group(1) print
m.group(2) homer jay simpson homer jay jay
58
Putting it all together (and optional flags)

• import re
• r re.compile('simpson', re.IGNORECASE)
• print r.search("HomerJaySimpson").group()
• simpson
• r re.compile('(A-Za-z).?(\d)',
  re.MULTILINE)
• iterator r.finditer('Homer is 42\nMaggie is
  6\nBart is 12')
• for match in iterator
• print match.group(1), "was born",
  match.group(2), "years ago
• Homer was born 42 years ago
• Maggie was born 6 years ago
• Bart was born 12 years ago

Discussion Who can explain how this RE works?
59
Finding tags within HTML

• import re
• line 'lttaggtmy eyes! the goggles do \
• nothing.lt/taggt'
• r re.compile('lttaggt(.)lt/taggt')
• m r.search(line)
• print m.group(1)
• gtgtgt my eyes! the goggles do nothing.

60
5 Minute Exercise

• Download the Columbia homepage to disk
• Open it with python
• Use regular expressions to being extracting the
  news

import re line 'lttaggtmy eyes! the goggles do \
nothing.lt/taggt' r
re.compile('lttaggt(.)lt/taggt') m
r.search(line) print m.group(1) gtgtgt my eyes! the
goggles do nothing.