How to play around with Python and Mock

Once upon a time, I started an internship program. On that program, I needed to program in Python and also write tests.

Based on my short experience with Python, I already hated to write tests. I didn’t have enough abstraction to learn to write useful tests. After a really long and painful process of learning, I started to write some tests, but don’t ask me to do TDD, just don’t.

On the world of testing in Python, you will discover a nice tool called Mock.

Mock is a lib that can ‘mock’ anything that you want and be whatever you want on your test.

On this post, I will show you an application of Mock on an FTP/SFTP code that I did this week, and published on my Github.

First I wrote two methods, one to transfer a file via FTP and one for SFTP, the snippet you can see below:

FTP

For the FTP code, I used the FTP library built in on the Python. Like a friend like to say: “Python comes with batteries” =D

def export_via_ftp(local_obj_file, file_name, **config):
    with closing(FTP(
            host=config['host'],
            user=config['user'],
            passwd=config['password'])) as ftp:
        ftp.cwd(config['destination'])
        ftp.storbinary('STOR '+ file_name, local_obj_file)

SFTP

To write the SFTP method I used Paramiko. It is very simple to use as you can see:

    @patch('myftp.myftp.SSHClient', autospec=True)
    def test_sftp(self, mocked_ssh):
        mocked_client = MagicMock()
        mocked_ssh.return_value = mocked_client
        sftp = export_factory('sftp')

        mocked_file = MagicMock()
        sftp(
            local_file=mocked_file,
            file_name='foo',
            host='localhost',
            port='22',
            user='user',
            password='dev',
            destination='/tmp')

        mocked_client.connect.assert_called_once_with(
            hostname='localhost',
                password='dev',
                port=22,
                username='user')

        mocked_client.open_sftp.return_value.putfo.assert_called_once_with(
            mocked_file,
            '/tmp/foo'
        )

Now we can start the interesting part: Let’s test!

My first test will see if my ‘export_via_ftp‘ method is working correctly. As you can see I have 3 methods used inside that function: The FTP constructor, the cwd(To change directory) and storbinary(to do the transfer).

On this case, I need to mock the FTP constructor, because on my tests I don’t have a server up for testing, and also on the CI that you may use you don’t have access to a server too. And all the purpose of unity testing is to check if your code is working properly, not the server that you may need to access.

Part I: @patch the decorator from the mock

Patch is your friend, but you really need to pay attention to what you are patching. In my case, I need to patch the FTP constructor. So you may think that you need to patch this piece of code:

@patch('ftplib.FTP', autospec=True)

But this is wrong. You need to patch where you are calling the method, not the file that has the source code of that method.
For more clarity here is an example:

If you have two files: A and B, and inside B you have the method_b function. Inside file A you import the method_b function and use it. On the test, you may think to patch the ‘B.method_b’, because the method is declared on B. But no, you need to patch were the call is made, so you patch ‘A.method_b’

When you patch using the decorator, you need to pass as a parameter of your test a name for the patch that you did. On my case is mocked_ftp. For every patch, you need to pass a name of it on the parameters.

    def test_ftp(self, mocked_ftp):

You also can use patch inside the test like:

with patch('myftp.myftp.FTP', autospec=True) as mocked_ftp:

However, the most common use that I see around is using the decorator.

Part II: MagicMock

“MagicMock is a subclass of Mock with default implementations of most of the magic methods. You can use MagicMock without having to configure the magic methods yourself.” Font: MagicMock Docs

MagicMock is the part where I said that a mock can be whatever you want. So on this snippet of code:

mocked_file = MagicMock()

I’m mocking my file that will be used on my mocked transfer for my mocked server.

Part III: Testing my Mock
You can’t have a test running without a few asserts. So when you are using mock you have these ones to use with your mock:

assert_called()
assert_called_once()
assert_called_with()
assert_called_once_with()
assert_any_call()

On all my code you will see that I’m using ‘assert_called_once_with()‘. For more information of this asserts check here.
‘assert_called_once_with‘ test for me if my method was called exactly one time and with the determined parameters.

On the case of the ‘return_value‘ key used:
When you use ‘with something as s‘ you put the return of something on s, so when you are mocking you need to get the return of it and call the method that you want to assert.
Note that my use here of mocking is a special case. Sometimes you can use the mock.return_value to receive a value and make tests where you program break. Like, make it return a 404 instead of 200 in an HTTP request. On the below example it is being tested that the server is down but not the data needed.

    @patch.object(Congressperson.objects, 'all')
    @patch('mocks.core.views._is_chamber_of_deputies_on')
    def test_server_is_off_but_not_data(self, server, people):
        server.return_value = False
        people.return_value = None
        resp = self.client.get('/')
        html = resp.content.decode('utf-8')
        self.assertIn('text-danger', html)
        self.assertIn('Sorry', html)

So on my FTP method, I have two asserts, one for the cwd and one for the storbinary, and gladly they two are working like expected.

Part IV: One more detail

On the code of the SFTP test, the only huge difference that you may see is this one:

    mocked_client = MagicMock()
    mocked_ssh.return_value = mocked_client

On this case, I’m using the return of mocked_ssh and adding it to mocked_client so I can use on the asserts without much trouble. I should have a better explanation for this, but I don’t have one now. =/ Just believe that works! haha

Well, this is my mini not so mini tutorial about mocks. I’m sharing this because since I started to use mock I have a lot of difficulties to get to know all the power of this library. Is hard to find good solutions on StackOverflow or good tutorials so you can understand everything that mock can do.

A good friend of mine said once to me that:

“The mock documentation only helps you when you already now how mock works.”

Well, and I lived to discover that is true.

However, I hope that this tutorial can help you on your adventure with mocks and testing in Python.

References:

You can find the source code of my work on my Github repository here.

And the source code of the example of the test_server_is_off_but_not_data you can find here.

That’s all folks!