Mitogenic Puns

I just discovered Mitogen and then massive performances gains you can get. Makes me wonder if there is any reason to not use it?

Looking for some guidance here. Running a simple playbook against 1500+ hosts using the mitogen plugin. Forks are set to 50 and receiving the following error:

ERROR! [mux  9954] 21:01:40.597674 E mitogen.unix: listener: failed to assign identity to PID 11303: [Errno 32] Broken pipe
ERROR! [mux  9954] 21:01:40.710869 E mitogen.unix: listener: failed to assign identity to PID 11302: [Errno 32] Broken pipe
ERROR! [mux  9954] 21:02:58.552806 E mitogen.unix: listener: failed to assign identity to PID 11521: [Errno 32] Broken pipe
ERROR! [mux  9954] 21:03:29.936661 E mitogen.unix: listener: failed to assign identity to PID 11607: [Errno 32] Broken pipe
ERROR! [mux  9954] 21:03:59.034466 E mitogen.unix: listener: failed to assign identity to PID 11659: [Errno 32] Broken pipe

Any troubleshooting help would be appreciated.

EDIT: This is the rout we went with to resolve this in case someone else runs into something similar. Updated the group_vars config to this:

ansible_ssh_common_args: '-o ProxyCommand="ssh -i /.ssh/id_rsa -o StrictHostKeyChecking=no -W [%h]:%p -q {{ username }}@<bastion_host_redacted>"'
ansible_python_interpreter: /usr/bin/python
ansible_timeout: 90
ansible_user: "{{ username }}"
ansible_password: "{{ password }}"
ansible_become_password: "{{ password }}"

Using the same prompts mentioned below. This use case is working in our situation. We are doing a ssh-copy-id to all of the bastions we require for using ansible with.

ORIGINAL POST:

Been looking at this issue for the last couple of days and I believe I am stuck and trying to understand mitogen integration with ansible. Currently testing this through docker.

/ansible # ansible --version
ansible 2.8.17
  config file = /ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.2 (default, Jul 18 2020, 19:35:03) [GCC 9.2.0]

Mitogen version is 0.2.9.

group_vars file:

ansible_ssh_common_args: '-o ProxyCommand="ssh -o StrictHostKeyChecking=no -W %h:%p -q {{ username }}@<bastion_removed>" -o ConnectionAttempts=10'
ansible_python_interpreter: /usr/bin/python
ansible_timeout: 90
ansible_password: "{{ password }}"
ansible_become_password: "{{ password }}"

playbook prompts:

  vars_prompt:
    - name: username
      prompt: What is your associate ID?
      private: no
    - name: password
      prompt: What is your password?
      unsafe: yes

I can run the playbook locally with no issues, but when using docker, I am having troubles. I am using a different version of ansible/mitogen locally, but I have removed mitogen in my ansible.cfg and still receiving the same error. I am trying to understand what may be missing from my ansible_ssh_common_args or if there is something else missing thats not passing in the correct ssh info.

Error for reference (IP's have been removed):

[mux  121] 17:00:56.180423 D mitogen.parent: failing connection ssh.<VPN> due to PasswordError('SSH password is incorrect')
[mux  121] 17:00:56.183629 D mitogen: BootstrapProtocol(ssh.<VPN>): 

Hello,

Is there anyone setup Mitogen on AWX?

I try to create a folder strategy_plugins same place as my playbook debug.yml

From the installation guide:

[defaults]
strategy_plugins = /path/to/mitogen-0.2.9/ansible_mitogen/plugins/strategy
strategy = mitogen_linear

I downloaded the tar file and extract files mitogen_free.py, mitogen_host_pinned.py, mitogen_linear.py, mitogen.py, __init__.py to strategy_plugins

---
- name: DEBUG
  hosts: debug
  strategy: mitogen_linear
  vars:
  tasks:
     - name: test
      debug:
        msg: "{{ ansible_inventory }}"

But when executing the playbook template on AWX, I have this error:

PLAYBOOK: debug.yml ************************************************************
1 plays in playbooks/debug.yml
PLAY [DEBUG] *******************************************************************
ERROR! Unexpected Exception, this is probably a bug: No module named 'ansible_mitogen'
the full traceback was:
Traceback (most recent call last):
  File "/usr/bin/ansible-playbook", line 123, in <module>
    exit_code = cli.run()
  File "/usr/lib/python3.6/site-packages/ansible/cli/playbook.py", line 127, in run
    results = pbex.run()
  File "/usr/lib/python3.6/site-packages/ansible/executor/playbook_executor.py", line 169, in run
    result = self._tqm.run(play=play)
  File "/usr/lib/python3.6/site-packages/ansible/executor/task_queue_manager.py", line 223, in run
    strategy = strategy_loader.get(new_play.strategy, self)
  File "/usr/lib/python3.6/site-packages/ansible/plugins/loader.py", line 552, in get
    self._module_cache[path] = self._load_module_source(name, path)
  File "/usr/lib/python3.6/site-packages/ansible/plugins/loader.py", line 525, in _load_module_source
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/…

If I extracting tar file ansible_mitogen folder to strategy_plugins it will complain

ERROR! Invalid play strategy specified: mitogen_linear

callback_plugins folder in the same place of playbook is working fine. I'm not sure about strategy_plugins

Could you please help?

Edit 1:

Since the error message No module named 'ansible_mitogen' if I put mitogen_linear.py

I don't want to step on anybody's toes here, but the amount of non-dad jokes here in this subreddit really annoys me. First of all, dad jokes CAN be NSFW, it clearly says so in the sub rules. Secondly, it doesn't automatically make it a dad joke if it's from a conversation between you and your child. Most importantly, the jokes that your CHILDREN tell YOU are not dad jokes. The point of a dad joke is that it's so cheesy only a dad who's trying to be funny would make such a joke. That's it. They are stupid plays on words, lame puns and so on. There has to be a clever pun or wordplay for it to be considered a dad joke.

Again, to all the fellow dads, I apologise if I'm sounding too harsh. But I just needed to get it off my chest.

I was kind of curious, as this was something I heard about a year or two ago. It's basically a python strategy plugin for Ansible that lets it create connections more in parallel, and in a more persistent manner with each host.

Since then, we've moved our AWX install over to k8s on Rancher, so large jobs can benefit from multiple AWX instances and be sliced. Are any of you guys using Mitogen still? Is it worth using? Has anyone used it with Tower or AWX?

Do your worst!

I'm surprised it hasn't decade.

For context I'm a Refuse Driver (Garbage man) & today I was on food waste. After I'd tipped I was checking the wagon for any defects when I spotted a lone pea balanced on the lifts.

I said "hey look, an escaPEA"

No one near me but it didn't half make me laugh for a good hour or so!

Edit: I can't believe how much this has blown up. Thank you everyone I've had a blast reading through the replies 😂

It really does, I swear!

Hi Everyone,

Is it possible to use Ansible pipelining feature and mitogen at a same time?

Buenosdillas

They’re on standbi

Pilot on me!!