Mitogen Puns

I just discovered Mitogen and then massive performances gains you can get. Makes me wonder if there is any reason to not use it?

Looking for some guidance here. Running a simple playbook against 1500+ hosts using the mitogen plugin. Forks are set to 50 and receiving the following error:

ERROR! [mux  9954] 21:01:40.597674 E mitogen.unix: listener: failed to assign identity to PID 11303: [Errno 32] Broken pipe
ERROR! [mux  9954] 21:01:40.710869 E mitogen.unix: listener: failed to assign identity to PID 11302: [Errno 32] Broken pipe
ERROR! [mux  9954] 21:02:58.552806 E mitogen.unix: listener: failed to assign identity to PID 11521: [Errno 32] Broken pipe
ERROR! [mux  9954] 21:03:29.936661 E mitogen.unix: listener: failed to assign identity to PID 11607: [Errno 32] Broken pipe
ERROR! [mux  9954] 21:03:59.034466 E mitogen.unix: listener: failed to assign identity to PID 11659: [Errno 32] Broken pipe

Any troubleshooting help would be appreciated.

EDIT: This is the rout we went with to resolve this in case someone else runs into something similar. Updated the group_vars config to this:

ansible_ssh_common_args: '-o ProxyCommand="ssh -i /.ssh/id_rsa -o StrictHostKeyChecking=no -W [%h]:%p -q {{ username }}@<bastion_host_redacted>"'
ansible_python_interpreter: /usr/bin/python
ansible_timeout: 90
ansible_user: "{{ username }}"
ansible_password: "{{ password }}"
ansible_become_password: "{{ password }}"

Using the same prompts mentioned below. This use case is working in our situation. We are doing a ssh-copy-id to all of the bastions we require for using ansible with.

ORIGINAL POST:

Been looking at this issue for the last couple of days and I believe I am stuck and trying to understand mitogen integration with ansible. Currently testing this through docker.

/ansible # ansible --version
ansible 2.8.17
  config file = /ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.2 (default, Jul 18 2020, 19:35:03) [GCC 9.2.0]

Mitogen version is 0.2.9.

group_vars file:

ansible_ssh_common_args: '-o ProxyCommand="ssh -o StrictHostKeyChecking=no -W %h:%p -q {{ username }}@<bastion_removed>" -o ConnectionAttempts=10'
ansible_python_interpreter: /usr/bin/python
ansible_timeout: 90
ansible_password: "{{ password }}"
ansible_become_password: "{{ password }}"

playbook prompts:

  vars_prompt:
    - name: username
      prompt: What is your associate ID?
      private: no
    - name: password
      prompt: What is your password?
      unsafe: yes

I can run the playbook locally with no issues, but when using docker, I am having troubles. I am using a different version of ansible/mitogen locally, but I have removed mitogen in my ansible.cfg and still receiving the same error. I am trying to understand what may be missing from my ansible_ssh_common_args or if there is something else missing thats not passing in the correct ssh info.

Error for reference (IP's have been removed):

[mux  121] 17:00:56.180423 D mitogen.parent: failing connection ssh.<VPN> due to PasswordError('SSH password is incorrect')
[mux  121] 17:00:56.183629 D mitogen: BootstrapProtocol(ssh.<VPN>): 

Hello,

Is there anyone setup Mitogen on AWX?

I try to create a folder strategy_plugins same place as my playbook debug.yml

From the installation guide:

[defaults]
strategy_plugins = /path/to/mitogen-0.2.9/ansible_mitogen/plugins/strategy
strategy = mitogen_linear

I downloaded the tar file and extract files mitogen_free.py, mitogen_host_pinned.py, mitogen_linear.py, mitogen.py, __init__.py to strategy_plugins

---
- name: DEBUG
  hosts: debug
  strategy: mitogen_linear
  vars:
  tasks:
     - name: test
      debug:
        msg: "{{ ansible_inventory }}"

But when executing the playbook template on AWX, I have this error:

PLAYBOOK: debug.yml ************************************************************
1 plays in playbooks/debug.yml
PLAY [DEBUG] *******************************************************************
ERROR! Unexpected Exception, this is probably a bug: No module named 'ansible_mitogen'
the full traceback was:
Traceback (most recent call last):
  File "/usr/bin/ansible-playbook", line 123, in <module>
    exit_code = cli.run()
  File "/usr/lib/python3.6/site-packages/ansible/cli/playbook.py", line 127, in run
    results = pbex.run()
  File "/usr/lib/python3.6/site-packages/ansible/executor/playbook_executor.py", line 169, in run
    result = self._tqm.run(play=play)
  File "/usr/lib/python3.6/site-packages/ansible/executor/task_queue_manager.py", line 223, in run
    strategy = strategy_loader.get(new_play.strategy, self)
  File "/usr/lib/python3.6/site-packages/ansible/plugins/loader.py", line 552, in get
    self._module_cache[path] = self._load_module_source(name, path)
  File "/usr/lib/python3.6/site-packages/ansible/plugins/loader.py", line 525, in _load_module_source
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/…

If I extracting tar file ansible_mitogen folder to strategy_plugins it will complain

ERROR! Invalid play strategy specified: mitogen_linear

callback_plugins folder in the same place of playbook is working fine. I'm not sure about strategy_plugins

Could you please help?

Edit 1:

Since the error message No module named 'ansible_mitogen' if I put mitogen_linear.py

I was kind of curious, as this was something I heard about a year or two ago. It's basically a python strategy plugin for Ansible that lets it create connections more in parallel, and in a more persistent manner with each host.

Since then, we've moved our AWX install over to k8s on Rancher, so large jobs can benefit from multiple AWX instances and be sliced. Are any of you guys using Mitogen still? Is it worth using? Has anyone used it with Tower or AWX?

Hi Everyone,

Is it possible to use Ansible pipelining feature and mitogen at a same time?

I already know yes it a proprietary stack and its expensive, but what about the ingredients? Nexus: 1,250mg - Aniracetam, CDP Choline, Phosphatidylserine, Pycnogenol. Mitogen: Vitamin C 500mg, Vitamin B5 500mg, 660mg - R-Lipoic Acid, Sulbutiamine, Idebenone, BioPQQ. These guys (Smart Drug Smarts) seem to be pretty informed and researched.

The nucleus says to them "you guys have come a long way from that axon terminal, how'd you find your way here?" Protein kinase A says "well, it would have been tough, but luckily I brought a MAP."

^^I'm ^^sorry.