Hacking ansible

1. Hacking Ansible make it do more ansiblefest 10/2014

2. Who am I? programmer DBA system administrator tinkerer devops tech-janitor ____________ < irc lurker > ------------ , , /( )` ___ / | /- _ `-/ ' (// / / / | ` O O ) / | `-^--'`< ' (_.) _ ) / `.___/` / `-----' / <----. __ / __ <----|====O)))==) ) /==== <----' `--' `.__,' | | / ______( (_ / ______ ,' ,-----' | `--{__________) /

3. ________________ / It runs a TASK on a HOST / ---------------- ^__^ (oo)_______ (__) )/ ||----w | || || What is ansible? ● Configuration management? ● Automation platform? ● Release management? ● Orchestration system?

4. The Guts: ansible internal objects ● inventory: defines my targets ● runner: actually does the work ● connection: how to get to my targets ● playbook: all plays in current invocation ○ play ■ host: where to do it ■ task: “it” aka “what to do” ● callback: show me what was done

5. Example hack - the core --- a/lib/ansible/playbook/__init__.py +++ b/lib/ansible/playbook/__init__.py @@ -346,6 +346,8 @@ class PlayBook(object): run_hosts=hosts ) + runner.module_vars.update({'play_hosts': hosts}) + if task.async_seconds == 0: results = runner.run() else:

6. Example hack - the core ● 10 second patch to add play_hosts ● 3h to find correct spot to patch ● learned a lot about playbook/host iteration ● I did not take “serial:” into account ● runner rewrite in progress

7. The plugin system

8. PluginLoader ... lookup_loader = PluginLoader( 'LookupModule', 'ansible.runner.lookup_plugins', C.DEFAULT_LOOKUP_PLUGIN_PATH, 'lookup_plugins' ) vars_loader = PluginLoader( 'VarsModule', 'ansible.inventory.vars_plugins', C.DEFAULT_VARS_PLUGIN_PATH, 'vars_plugins' ) lib/ansible/utils/plugins.py __________________ / all plugins except inventory / ------------------ ,__, (oo)____ (__) ) ||--|| *

9. Plugins ● Library: host tasks/actions/modules ● Action: master side tasks/actions/modules ● Cache: fact caching ● Callback: play output ● Connection: host connections

10. Plugins ● Shell: what shell to use to execute tasks ● Lookup: master side info lookup ● Vars: variable imports ● Inventory: aside from inventory scripts ● Filter: jinja2 filters for data modification ● Doc Fragment: shared docs for library

11. Library/Action - May 2013 (72) add_host,debug,get_url,mount,postgresql_user,slurp,apt, django_manage,git,mysql_db,rabbitmq_parameter, subversion,apt_key,easy_install,group,mysql_user, rabbitmq_plugin,supervisorctl,apt_repository,ec2,group_by, nagios,rabbitmq_user,svr4pkg,assemble,ec2_facts,hg,ohai, rabbitmq_vhost,sysctl,async_status,ec2_vol,ini_file,opkg, raw,template,async_wrapper,facter,libr,pacman,script,uri, authorized_key,fail,lineinfile,pause,seboolean,user, cloudformation,fetch,lvol,ping,selinux,virt,command,file, macports,pip,service,wait_for,copy,fireball,mail,pkgin,setup, yum,cron,gem,mongodb_user,postgresql_db,shell,zfs

12. Library/Action - October 2014 (175) a10_server,a10_service_group,a10_virtual_server,accelerate,acl,add_host,airbrake_deployment,alternatives,apache2_module, apt,apt_key,apt_repository,apt_rpm,assemble,assert,async_status,at,authorized_key,azure,bigip_facts,bigip_monitor_http, bigip_monitor_tcp,bigip_node,bigip_pool,bigip_pool_member,bigpanda,boundary_meter,bzr,campfire,capabilities,cloudformation, command,composer,copy,cpanm,cron,datadog_event,debconf,debug,digital_ocean,digital_ocean_domain,digital_ocean_sshkey, django_manage,dnsimple,dnsmadeeasy,docker,docker_image,easy_install,ec2,ec2_ami,ec2_ami_search,ec2_asg,ec2_eip, ec2_elb,ec2_elb_lb,ec2_facts,ec2_group,ec2_key,ec2_lc,ec2_metric_alarm,ec2_scaling_policy,ec2_snapshot,ec2_tag,ec2_vol, ec2_vpc,ejabberd_user,elasticache,facter,fail,fetch,file,filesystem,fireball,firewalld,flowdock,gc_storage,gce,gce_lb,gce_net, gce_pd,gem,get_url,getent,git,github_hooks,glance_image,group,group_by,grove,hg,hipchat,homebrew,homebrew_cask, homebrew_tap,hostname,htpasswd,include_vars,ini_file,irc,jabber,jboss,jira,kernel_blacklist,keystone_user,layman, librato_annotation,lineinfile,linode,lldp,locale_gen,logentries,lvg,lvol,macports,mail,modprobe,mongodb_user,monit,mount,mqtt, mysql_db,mysql_replication,mysql_user,mysql_variables,nagios,netscaler,newrelic_deployment,nexmo,nova_compute, nova_keypair,npm,ohai,open_iscsi,openbsd_pkg,openvswitch_bridge,openvswitch_port,opkg,osx_say,ovirt,pacman,pagerduty, pause,ping,pingdom,pip,pkgin,pkgng,pkgutil,portage,portinstall,postgresql_db,postgresql_privs,postgresql_user, quantum_floating_ip,quantum_floating_ip_associate,quantum_network,quantum_router,quantum_router_gateway, quantum_router_interface,quantum_subnet,rabbitmq_parameter,rabbitmq_plugin,rabbitmq_policy,rabbitmq_user,rabbitmq_vhost, raw,rax,rax_cbs,rax_cbs_attachments,rax_cdb,rax_cdb_database,rax_cdb_user,rax_clb,rax_clb_nodes,rax_dns,rax_dns_record, rax_facts,rax_files,rax_files_objects,rax_identity,rax_keypair,rax_meta,rax_network,rax_queue,rax_scaling_group, rax_scaling_policy,rds,rds_param_group,rds_subnet_group,redhat_subscription,redis,replace,rhn_channel,rhn_register,riak, rollbar_deployment,route53,rpm_key,s3,script,seboolean,selinux,service,set_fact,setup,shell,slack,slurp,sns,stackdriver,stat, subversion,supervisorctl,svr4pkg,swdepot,synchronize,sysctl,template,twilio,typetalk,ufw,unarchive,uri,urpmi,user,virt, vsphere_guest,wait_for,win_feature,win_get_url,win_group,win_msi,win_ping,win_service,win_stat,win_user,xattr,yum, zabbix_maintenance,zfs,zypper,zypper_repository

13. Filters: Jinja2’ism ● best way to change data ● chainable pipes ● simple to expand ● can hide complexity __________________________ < [hello, world]|join(‘ ‘) > -------------------------- ^__^ (oo)_______ (__) )/ ||----w | || ||

14. Example hack - filter import random def randomize_list(mylist): random.shuffle(mylist) return mylist ... def filters(self): # this maps filter names to functions return { ‘shuffle’: randomize_list, ... } lib/ansible/runner/filter_plugins/core.py

15. Example hack - filter - hosts: all connection: local gather_facts: false vars: - mylist: [1,2,3,4,5] tasks: - debug: msg=”{{item}}” with_items: “{{mylist|shuffle}}”

16. Example hack - filter #>ansible-playbook test.yml -i ‘localhost,’ __________________________ < TASK: debug msg={{item}} > -------------------------- ok: [localhost] => (item=1) => { "item": 1, "msg": "1" } ok: [localhost] => (item=3) => { "item": 3, "msg": "3" } ok: [localhost] => (item=4) => { "item": 4, "msg": "4" } ok: [localhost] => (item=5) => { "item": 5, "msg": "5" } ok: [localhost] => (item=2) => { "item": 2, "msg": "2" } __________________________ < TASK: debug msg={{item}} > -------------------------- ok: [localhost] => (item=3) => { "item": 3, "msg": "3" } ok: [localhost] => (item=2) => { "item": 2, "msg": "2" } ok: [localhost] => (item=5) => { "item": 5, "msg": "5" } ok: [localhost] => (item=2) => { "item": 2, "msg": "2" } ok: [localhost] => (item=1) => { "item": 1, "msg": "1" } __________________________ < TASK: debug msg={{item}} > -------------------------- ok: [localhost] => (item=2) => { "item": 2, "msg": "2" } ok: [localhost] => (item=4) => { "item": 4, "msg": "4" } ok: [localhost] => (item=1) => { "item": 1, "msg": "1" } ok: [localhost] => (item=3) => { "item": 3, "msg": "3" } ok: [localhost] => (item=5) => { "item": 5, "msg": "5" }

17. Lookups ● you are already using them: with_<lookup> ● they execute on the “master” ● a way to access external files and/or data ● normally returns a list

18. Example hack - lookup lib/ansible/runner/lookup_plugins/etcd.py class LookupModule(object): def __init__(self, basedir=None, **kwargs): self.basedir = basedir self.etcd = etcd() # initializes etcd class def run(self, terms, inject=None, **kwargs): terms = utils.listify_lookup_plugin_terms(terms, self.basedir, inject) if isinstance(terms, basestring): terms = [ terms ] ret = [] for term in terms: key = term.split()[0] value = self.etcd.get(key) # gets the data from etcd class ret.append(value) return ret

19. Example hack - lookup lib/ansible/runner/lookup_plugins/etcd.py class etcd(): # simplified version def __init__(self, url=ANSIBLE_ETCD_URL): self.url = url self.baseurl = '%s/v1/keys' % (self.url) def get(self, key): url = "%s/%s" % (self.baseurl, key) r = urllib2.urlopen(url) data = r.read() item = json.loads(data) value = item['value'] return value _______________ / written by Jan-Piet Mens / --------------- ,__, (oo)____ (__) ) ||--|| *

20. Example hack - lookup #>ansible all -m debug -a ‘msg={{lookup(“etcd”,”key1”)}}’ -i ‘localhost,’ localhost | success >> { "msg": "value1" } #>ansible all -m debug -a ‘msg={{lookup(“etcd”,”key2”)}}’ -i ‘localhost,’ localhost | success >> { "msg": "value2" }

21. Example hack - callback callback_plugins/profile_tasks.py class CallbackModule(object): # simplified example def playbook_on_task_start(self, name, is_conditional): _______________ / written by Jharrod LaFon / --------------- ,__, (oo)____ if self.current is not None: # Record the total time of the previous self.stats[self.current] = time.time() - self.stats[self.current] self.current = name # Record the start time of the current task self.stats[self.current] = time.time() def playbook_on_stats(self, stats): # Sort the tasks by their running time ... (__) ) ||--|| * results = sorted(self.stats.items(), key=lambda value: value[1], reverse=True) for name, elapsed in results: # Print the timings print "{0:-<70}{1:->9}".format('{0} '.format(name),'{0:.02f}s'.format(elapsed))

22. Example hack - callback #>ansible-playbook test.yml -i ‘localhost,’ # re-running the previous filter/shuffle playbook __________________________ < TASK: debug msg={{item}} > -------------------------- ok: [localhost] => (item=2) => { "item": 2, "msg": "2" } ... ok: [localhost] => (item=4) => { "item": 4, "msg": "4" } PLAY RECAP ******************************************************************** debug msg={{item}} ------------------------------------------------------ 0.02s localhost : ok=1 changed=0 unreachable=0 failed=0

23. Example hack - cache import exceptions class BaseCacheModule(object): def get(self, key): lib/ansible/cache/base.py raise exceptions.NotImplementedError def set(self, key, value): raise exceptions.NotImplementedError def keys(self): raise exceptions.NotImplementedError def contains(self, key): raise exceptions.NotImplementedError def delete(self, key): raise exceptions.NotImplementedError def flush(self): raise exceptions.NotImplementedError def copy(self): raise exceptions.NotImplementedError ________________ / I abandoned it | and Josh Drake | revived it / ---------------- / ___ / // / / (( O O )) / // / | | / | | | | | | | | | o | | | | | |m| |m|

24. Example hack - cache from ansible.cache.base import BaseCacheModule class CacheModule(BaseCacheModule): def __init__(self, *args, **kwargs): self._cache = {} def get(self, key): return self._cache.get(key) def set(self, key, value): self._cache[key] = value def keys(self): return self._cache.keys() def contains(self, key): return key in self._cache def delete(self, key): del self._cache[key] def flush(self): lib/ansible/cache/memory.py

25. Example hack - cache from redis import StrictRedis class CacheModule(BaseCacheModule): # simplified version self._cache = StrictRedis(*connection) ... def get(self, key): value = self._cache.get(self._make_key(key))# make_key creates correct prefix # guard against the key not being removed from the zset if value is None: self.delete(key) raise KeyError return json.loads(value) def set(self, key, value): value2 = json.dumps(value) if self._timeout > 0: # a timeout of 0 is handled as meaning 'never expire' self._cache.setex(self._make_key(key), int(self._timeout), value2) else: self._cache.set(self._make_key(key), value2) self._cache.zadd(self._keys_set, time.time(), key) ... lib/ansible/cache/redis.py

26. Tests: test/ ● sadly the presenter sucks at test coverage ● no really, tests are good, regressions! ● assert: the test module ● destructive, non destructive, integration ● ‘make tests’, runs unit tests

27. Before you submit ● update documentation (it is also in the repo) ● create tests, assert! (also in the repo) ● prepare a clear usage/example to post in the PR ● its a dialog, be ready to make your ‘use case’ ● patience … … … … ● not everything belongs in core

SlideShare for iOS

Hacking ansible

by bcoca

on Oct 15, 2014

Statistics

Views

Actions

1 Embed 14

Accessibility

Categories

Upload Details

Usage Rights

Report content

Hacking ansible Presentation Transcript