I've been working on a Python library which - for a number of reasons - needs to dynamically alter itself. Essentially, I want it to parse a document and to generate some code based on that parsed file.
So, to start, I tracked down a suitable "Hello world!" unit test in Python.
ast
It turns out that Python's ast module lets me do exactly what I need. I came across some quite useful supplementary documentation on ast. But, to get started, I needed something simpler that those advanced examples. I therefore wrote a "Hello world!" program using ast. Here it is, in case you were looking for that, too.Hello world!
Since I've become test infected, I wanted to structure my "Hello world!" ast program using unit tests.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| __author__ = 'k0emt' | |
| class Greeter: | |
| def __init__(self): | |
| self.message = 'Hello world' | |
| # print self.message |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| __author__ = 'k0emt' | |
| import unittest | |
| from Experiment import Greeter | |
| class MyTestCase(unittest.TestCase): | |
| def test_default_greeting_set(self): | |
| greeter = Greeter() | |
| # this test will fail until you change the Greeter to return this expected message | |
| self.assertEqual(greeter.message, 'Hello world!') | |
| if __name__ == '__main__': | |
| unittest.main() |
So, to start, I tracked down a suitable "Hello world!" unit test in Python.
Hello world! in ast
Then I rewrote the Greeter.py class to use ast. My version constructs an abstract syntax tree for an Assignment. Specifically, it assigns the string "Hello world!" to the variable "m". The code then fixes the locations, compiles the code and executes it dynamically.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import ast | |
| class Greeter: | |
| def __init__(self): | |
| #m = "Hello world!" | |
| assignment = ast.Module(body=[ ast.Assign(targets = [ | |
| ast.Name(id = 'm', ctx = ast.Store())], | |
| value = ast.Str(s="Hello world!")) | |
| ]) | |
| ast.fix_missing_locations(assignment) | |
| co = compile(assignment, "<ast>", "exec") | |
| exec(co) | |
| self.message = m |
Obviously, the above code is a lot more work than simply assigning the string value to the variable directly. But it meant I now had the world's simplest ast program.
For example, here's a snippet of ast code which uses ast to generate an abstract syntax tree to assign an empty type to a variable named "nothing". In other words, equivalent to nothing = ()
First, I worked out to call a function - one not attached to an instance of a class. But to call a method of a class, I needed to understand a bit more about how Python itself is implemented.
And here's a variant where you pass in a value, i.e. equivalent to result = bar("some value")
And it sort of was - I still needed to use ast.Call to invoke the method. But it took me quite a while to figure out how to tell it which class method to call. For example, if I wanted to call
result = self._baz(theResult)
should I pass in a function name of "self._baz"? (I tried that - it didn't work). Eventually, I worked out that self._baz is an attribute of the instance object referred to as "self". In Python, instance objects have two kinds of valid attribute names, data attributes and methods. Which meant that the code to call one method of an instance from another method looks like this:
I had never thought that profoundly about how Python is really implemented behind the scenes. Although many of the Python design decisions are actually quite well documented.
Nothing
Armed with this most basic of unit tests, I was then in a position to work out how to support various other types of code in ast.For example, here's a snippet of ast code which uses ast to generate an abstract syntax tree to assign an empty type to a variable named "nothing". In other words, equivalent to nothing = ()
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Use ast to generate an abstract syntax tree to assign an empty tuple to a variable named "nothing" | |
| # i.e. equivalent to | |
| # nothing = () | |
| import ast | |
| emptyness = ast.Module(body=[ ast.Assign(targets = [ | |
| ast.Name(id = 'nothing', ctx = ast.Store())], | |
| value = ast.Tuple(elts=[], ctx = ast.Load())) | |
| ]) | |
| ast.fix_missing_locations(emptyness) | |
| co = compile(emptyness, "<ast>", "exec") | |
| exec(co) | |
Invoking Methods
One of the hardest things for me to figure out was how to invoke a method of a class.First, I worked out to call a function - one not attached to an instance of a class. But to call a method of a class, I needed to understand a bit more about how Python itself is implemented.
Calling Functions
Here's some Python ast code to call a function _foo() and assign the returned value to a variable called "result", i.e. equivalent to result = foo()
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import ast | |
| # ast code to call a function _foo() and assign the returned value to a variable called "result", i.e. equivalent to | |
| # result = _foo() | |
| def _foo(): | |
| return "It worked!" | |
| assignresult = ast.Module(body=[ ast.Assign(targets = [ | |
| ast.Name(id = 'result', ctx = ast.Store())], | |
| value = ast.Call(func = ast.Name(id='_foo', ctx = ast.Load()), ctx = ast.Load(), args=[], keywords=[])) | |
| ]) | |
| ast.fix_missing_locations(assignresult) | |
| co = compile(assignresult, "<ast>", "exec") | |
| exec(co) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import ast | |
| # ast code to call a function _bar(), pass in a value and assign the returned value to a variable called "result", i.e. equivalent to | |
| # result = _bar("theResult") | |
| def _bar(theStr): | |
| return theStr | |
| assignresult = ast.Module(body=[ ast.Assign(targets = [ | |
| ast.Name(id = 'result', ctx = ast.Store())], | |
| value = ast.Call(func = ast.Name(id='_bar', ctx = ast.Load()), ctx = ast.Load(), args=[ast.Name(id="theResult", ctx = ast.Load())], keywords=[])) | |
| ]) | |
| ast.fix_missing_locations(assignresult) | |
| co = compile(assignresult, "<ast>", "exec") | |
| exec(co) |
In Python, Methods are Attributes of Classes
Having figured out how to call functions and pass parameters to them, I reckoned that calling a method on a class would be similar.And it sort of was - I still needed to use ast.Call to invoke the method. But it took me quite a while to figure out how to tell it which class method to call. For example, if I wanted to call
result = self._baz(theResult)
should I pass in a function name of "self._baz"? (I tried that - it didn't work). Eventually, I worked out that self._baz is an attribute of the instance object referred to as "self". In Python, instance objects have two kinds of valid attribute names, data attributes and methods. Which meant that the code to call one method of an instance from another method looks like this:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import ast | |
| # ast code to call an instance object method and pass in a parameter, i.e. equivalent to | |
| # result = self._baz(theResult)" | |
| class Greeter: | |
| def _baz(self, theStr): | |
| return theStr | |
| def baz(self, theResult): | |
| assignresult = ast.Module(body=[ ast.Assign(targets = [ | |
| ast.Name(id = 'result', ctx = ast.Store())], | |
| value = | |
| ast.Call(func=ast.Attribute(value=ast.Name(id='self', ctx=ast.Load()), attr='_baz', ctx=ast.Load()), | |
| ctx=ast.Load(), | |
| args=[ast.Name(id="theResult", ctx=ast.Load())], | |
| keywords=[])) | |
| ]) | |
| ast.fix_missing_locations(assignresult) | |
| co = compile(assignresult, "<ast>", "exec") | |
| exec(co) | |
| return result |
An ast Short Cut
In the process of working out how to invoke instance object methods, I came up with a general-purpose shortcut. It turns out that - since Python 2.6 - ast has a very handy helper method called "ast.parse()". This - in combination with ast.dump() - will let you very quickly figure out the correct ast pattern to use for a given bit of Python code. For example, here's how to figure out how to invoke an instance object method
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import ast | |
| # ast code to figure out how to call an instance object method and pass in a parameter, i.e. equivalent to | |
| # result = self._baz(theResult)" | |
| class Greeter: | |
| def _baz(self, theStr): | |
| return theStr | |
| def baz(self, theResult): | |
| theTree = ast.parse("result = self._baz(theResult)") | |
| print(ast.dump(theTree)) |
 |
| 005-Syntax by vicdunk https://flic.kr/p/dUdQkm |
