Strange drama with the run command when running certain commands

[fgimian]

Hey there again, I'm really not sure what's special about this particular command, but it's tripping up the run() function.

Here's my code:

import os
from invoke import task, run

@task
def forge():
    proxy = 'http://192.168.0.1:80/'
    os.environ.update({
        'http_proxy': proxy,
        'https_proxy': proxy
    })

    cmd = 'puppet module install --target-dir /var/puppet/forgey puppetlabs-motd'
    print cmd
    run(cmd)

And the output:

puppet module install --target-dir /var/puppet/forgey puppetlabs-motd
Warning: Setting manifest is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Warning: Setting modulepath is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Warning: Setting config_version is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Notice: Preparing to install into /var/puppet/forgey ...
Notice: Created target directory /var/puppet/forgey
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
Traceback (most recent call last):
  File "./bin/invoke", line 12, in <module>
    invoke.cli.main()
  File "/root/puppet-pusher/lib/invoke/invoke/cli.py", line 438, in main
    dispatch(sys.argv)
  File "/root/puppet-pusher/lib/invoke/invoke/cli.py", line 431, in dispatch
    return executor.execute(*tasks)
  File "/root/puppet-pusher/lib/invoke/invoke/executor.py", line 100, in execute
    task=task, name=name, args=args, kwargs=kwargs, config=config
  File "/root/puppet-pusher/lib/invoke/invoke/executor.py", line 153, in _execute
    result = task(*args, **kwargs)
  File "/root/puppet-pusher/lib/invoke/invoke/tasks.py", line 111, in __call__
    result = self.body(*args, **kwargs)
  File "/root/puppet-pusher/tasks.py", line 14, in forge
    run(cmd)
  File "/root/puppet-pusher/lib/invoke/invoke/__init__.py", line 23, in run
    return Context().run(command, **kwargs)
  File "/root/puppet-pusher/lib/invoke/invoke/context.py", line 53, in run
    return runner_class(context=self).run(command, **kwargs)
  File "/root/puppet-pusher/lib/invoke/invoke/runners.py", line 262, in run
    raise ThreadException(exceptions)
invoke.exceptions.ThreadException:
Saw 1 exceptions within threads (UnicodeEncodeError):


Thread args: {'args': [<functools.partial object at 0x1c86f70>,
          <open file '<stdout>', mode 'w' at 0x7effba142150>,
          [u'\x1b[mNotice: Preparing to install into /var/puppet/forgey ...\x1b[0m\n',
           u'\x1b[mNotice: Created target directory /var/puppet/forgey\x1b[0m\n',
           u'\x1b[mNotice: Downloading from https://forgeapi.puppetlabs.com ...\x1b[0m\n',
           u'\x1b[mNotice: Installing -- do not interrupt ...\x1b[0m\n'],
          False],
 'target': <bound method Local.io of <invoke.runners.Local object at 0x1c91b90>>}

Traceback (most recent call last):

  File "/root/puppet-pusher/lib/invoke/invoke/runners.py", line 82, in run
    super(_IOThread, self).run()

  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)

  File "/root/puppet-pusher/lib/invoke/invoke/runners.py", line 328, in io
    output.write(data)

UnicodeEncodeError: 'ascii' codec can't encode characters in position 19-27: ordinal not in range(128)

This system is using:

• Python 2.6.6
• Puppet 3.6.2
• Invoke (master)

Regular output of the command looks like this:

Warning: Setting manifest is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Warning: Setting modulepath is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Warning: Setting config_version is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Notice: Preparing to install into /var/puppet/forgey ...
Notice: Created target directory /var/puppet/forgey
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/var/puppet/forgey
└── puppetlabs-motd (v1.2.0)

Any ideas?

Cheers
Fotis

[fgimian]

I believe that it's getting confused with the └── characters actually, I'll see if I can write some code that you can use to reproduce this yourself consistently 😄

Edit: OK, here's some simple code that repos the issue 😄

# encoding: utf-8
from invoke import task, run

@task
def output_strange_stuff():
    print '└──'

@task
def breakit():
    run('invoke output_strange_stuff')

Then just run invoke breakit. An invoke invoking another invoke, kinda cool huh? 😄

I presume this is a unicode related thingy. Using ascii is not the safest option if that's what's going on under the hood because it only covers those 128 chars. Latin-1 would be the safest bet if this is fixed in the code since that covers any possible string (it supports 256 chars).

Please let me know your thoughts anyway 😄

Edit 2: I noticed there is a default_encoding parameter that can be passed into run() which I tried but it didn't seem to help unfortunately

[fgimian]

Just one more note to confuse things further. I built Python 2.6.6 using pyenv on my Mac and can't repro the problem there. But I can reproduce it on my server which is a RHEL 6.6 server using the default Python 2.6.6 installation.

I will try to look into it soon too, but it's a bit of a headache to deal with right now :(

[bitprophet]

That is pretty strange, moreso because we actually have that exact sort of character in our integration test suite and that runs fine for me and our CI:

• invoke/integration/main.py

  Lines 70 to 89 in 3b9ade5

  	class funky_characters_in_stdout:
  	def basic_nonstandard_characters(self):
  	os.chdir('_support')
  	# Crummy "doesn't explode with decode errors" test
  	if WINDOWS:
  	cmd = "type tree.out"
  	else:
  	cmd = "cat tree.out"
  	run(cmd, hide='both')
  	def nonprinting_bytes(self):
  	# Seriously non-printing characters (i.e. non UTF8) also don't
  	# asplode
  	run("echo '\xff'", hide='both')
  	def nonprinting_bytes_pty(self):
  	if WINDOWS:
  	return
  	# PTY use adds another utf-8 decode spot which can also fail.
  	run("echo '\xff'", pty=True, hide='both')

• https://github.com/pyinvoke/invoke/blob/3b9ade5d7af8bf093c0f91f2b59a3d7fe904d7cb/integration/_support/tree.out

However, I dug and this looks like the same issue as #242 which remains unsolved but has a good writeup by pfmoore (who was instrumental in setting up the current encoding infrastructure).

Gonna roll this into that one. Thanks for the detailed report!

[bitprophet]

Also, I did just notice the docstring for run is wrong, the kwarg is encoding not default_encoding, ugh. So thanks for that part at least!

[fgimian]

Thanks a lot mate 😄