Ansible + OpenStack: No Panda-monium here!

If, like myself, you are a frequent reader of technology-related articles on the internet, you may have seen this article come across your radar last Friday afternoon. Especially if the panda was as eyecatching for you as it was for myself.

Written by the nice folks over at The New Stack, the article speculates that one of the reasons for Ansible’s popularity in the world of containers is, quite simply, the popularity of Ansible in general. Citing results from the most recent OpenStack user survey, the author states:

Ansible made up a 14-point deficit compared to six months ago to become virtually tied with Puppet as the leading way to deploy/configure OpenStack clusters. Why this is the case, we’re not sure. However, one reason may be that Ansible’s online community is particularly strong.

Not sure why Ansible is growing so fast in popularity with OpenStack users? This must be why the Panda is so puzzled. Right?

Let me clear that up with a list of answers.

From a panda. (Incidentally, credit to Máirín Duffy, who actually created both of these pandas in her design work for the Fedora Project community. Super coincidence!)

Simplicity

Simplicity, ease of use, and a low learning curve are characteristics frequently cited as reasons folks have chosen Ansible — and in the ever-so-slightly (:D) complex world of OpenStack, the ability to abstract away some of that complexity gives users (admins, cloud consumers, I hate the word “users”…) more time to work on other things.

Well. You know what LeVar Burton would say here.

Naveen Joy of Cisco elaborated on this particular point in his talk at the OpenStack Summit earlier this year in Austin, Texas, titled “Kubernetes Automation on OpenStack Using Ansible,” before following up with the long list of benefits.

“We did not want complexity in our automation. Kubernetes and OpenStack are complex for a reason;  they provide a lot of functions and a lot of knobs that you can customize and tune, APIs, a number of projects, variables that you can customize. But automation framework — we wanted it to be simple. We didn’t want to learn another manifest language to get this going. So, Ansible, we looked at it, and said, here’s the way to go…”

That said — making the deployment and management of an OpenStack cloud a little bit easier isn’t the only way in which simplicity matters. The simplicity of using Ansible with OpenStack, or really, Ansible with *anything*, also endears it to many folks — including Major Hayden of Rackspace, who explained further during his session in Austin, “Automated Security Hardening with OpenStack-Ansible.”

“The reason we love it is because there’s no agent required – there’s nothing actually that you have to install on the server to use Ansible with it. So you can deploy from your laptop, you can deploy from a bastion server, you can deploy from Jenkins, you can deploy from wherever you like. But you don’t have to go in on the other end and install ruby, and install 56 other things — you just have to make sure the other end has Python, which, if you’re running OpenStack… I certainly hope you have that installed, otherwise your installation is going to go in a bad direction.”

Versatility

Want to know one of my great “cloud” pet peeves? It’s that people say “users” all the time, and “users” can imply all types of different audiences to different folks. The operators? The sysadmins? The actual consumer of the cloud who just wants to do things on it? Who knows who we’re talking about!

What I do know is this: Users of OpenStack clouds — at any of those levels — are able to use Ansible. It’s useful to all of those groups. And, possibly even more importantly: this usefulness applies to Ansible in use cases everywhere. And in a lot of cases — people aren’t making a move to Ansible because it’s awesome with OpenStack. They’re coming as “users” of new OpenStack deployments and Ansible is *already what they’re using*, because it is so versatile.

Spencer Smith from Solinea explains their reasoning, as part of the consulting side of their company, for using Ansible in deploying Kubernetes, in this clip of their talk from OpenStack Summit.

“And I should say that we chose Ansible mainly because our clients are already using Ansible. They’ve developed expertise internally — so we didn’t want to compromise that, but still wanted to give them a way to deploy Kubernetes.”

Community

Let’s take a look at this snippet from the article: “Why this is the case, we’re not sure. However, one reason may be that Ansible’s online community is particularly strong.”

Well. I suppose it would be pretty easy for us to pat ourselves on the back and say, “Yep! More than 17 thousand stars on GitHub! Over 1400 people in #ansible on IRC! Over 2200 unique contributors to Ansible and its modules! Nearly 180 meetups with 35 thousand members around the world!” — and throw in a quaint mic drop gif, and walk away.

But the truth is: the combination of Ansible and OpenStack isn’t awesome, and thus, growing in use, because of the strength of the Ansible community. It is because of the work and collaboration and self-promotion of the group of folks who care about Ansible AND OpenStack — even without a formalized, central hub.

Here’s just a quick list of a few of the pockets of OpenStack + Ansible work:

(Note: I am super lazy at linking. But lucky for you, I have links to nearly all the projects listed below captured right here, which you can reference from this day forward!)

• OpenStack “Big Tent” projects using Ansible: Kolla (OpenStack services in Docker containers). Openstack-ansible (OpenStack in LXC containers). Bifrost (Ansible + Ironic for bare metal). Openstack-ansible-security (automated deployment of security enhancements, based on the Security Technical Implementation Guide from the United States government).
• OpenStack & Ansible users sharing their magic outside of the big tent: Ursula. Folks at Cisco. HP’s Helion. The fine humans at Catalyst Cloud (who, by the way, really get open source).  And plenty of other awesome examples by individuals and companies and projects are out there. 
• OpenStack community members contribute and maintain OpenStack-related code inside of Ansible. Ansible’s OpenStack modules (more than 30 of them!) allow users / operators / consumers of OpenStack clouds to perform various functions on or in or to their cloud. These modules are literally used every single day by OpenStack’s infrastructure team to utilize OpenStack’s cloud, which, unsurprisingly, is an OpenStack cloud; also unsurprisingly, the maintainers for these modules in Ansible are largely members of OpenStack’s infrastructure team.
• OpenStack is built with help from Ansible. OpenStack’s code is continuously tested and integrated prior to any commit — ensuring that small bits of bad code aren’t breaking life for the developers in a gigantic, incredibly busy project.  How do they do it? With the help of Zuul, an automation system built by OpenStack’s infrastructure team, for the OpenStack community, and used by more than 50 other open source communities as well. How does Zuul do this magic? It used to be with Jenkins. It’s now performed with Ansible. 

The truth is — none of these, alone, are the single reason for the growth of Ansible usage in conjunction with OpenStack. And it’s not just community, or Ansible’s simplicity, or versatility. It’s all of it, working together, compounding upon their collective successes.

Circling Back

Those of us who have spent time thinking Super Heady Deep Thoughts about open source and communities of practice have heard of “the virtuous circle” — a term which has many interpretations to be found on the internets (go figure!). But I find that this one, from this report on Sustainability in Open Source Commons, is spot on.  The author describes the virtuous circle as: “where good initial products attract users, which then potentially attract a new developer, which leads to more improvements. Our research clearly shows that successful projects have a potentially significant user community and that this user community drives project continuity.”

For the Ansible community, this pattern isn’t just limited to “Ansible + OpenStack.” I’d argue that it is repeating itself all over the place with a variety of Ansible + projects and/or practices. Containers. Working with orchestration tools like Kubernetes. AWS. Windows. Networking and its glorious SDN/NFV/TLA future. I could go on and on.

Ansible’s users and contributors consistently blow my mind in their usage of Ansible: how they are making it useful with other open source projects, and improving their day to day workflow. That they do so while also collaborating and sharing with other users and building communities of practice around Ansible EVERYWHERE just leads to more involvement, and more improvements, and more usage of Ansible all around. And enables those of us working for Ansible to do what we’ve always done: Follow our users — because, as users, they know what’s important and useful to them far more than we could ever dream to.

Which really means that Ansible and OpenStack — or Ansible and container orchestration — or Ansible and networking — isn’t thriving simply because of “the virtuous circle.”

It’s that the Ansible community has turned that circle into more of a virtuous venn diagram.

 

Ansible Extras Modules + YOU: How you can help. (It’s easier now!)

If you are a caring user of Ansible, and you meet any of the following criteria, this post is for you — because you can help to improve the quantity and quality of modules in Ansible Extras.

• You are a user of, or contributor to, Ansible Extras modules
• There is a pull request for an Extras module that you have been anxiously waiting to see merged (yours, or someone else’s!)
• You’ve been looking for a way to contribute to the Ansible community
• You are looking for fun and constructive ways to procrastinate doing other things you should be doing

In short: Our improved “new extras modules” review process is now in place, and any new Extras module can be reviewed for inclusion by any user of Ansible who cares to see that module be included.

Want to see a few of the modules that need love? Scroll down to the end!

***

Folks who keep an eye on the various Ansible repositories have probably noticed that your friendly neighborhood Ansible community team (that’s myself and Greg DeKoenigsberg) have been digging through a pretty sizable backlog of issues and pull requests, primarily in the Extras and Core modules repos. We’ve been doing this with a few things in mind; obviously, getting caught up as best as possible, but more importantly, making sure that the contributions of community members are being acknowledged and acted on. We value Ansible’s community members tremendously, and the last thing we want is for their hard work to be unused — or worse, for those people to feel demoralized and not contribute in the future.

With this in mind, we have been not only catching up on checking the status of each and every outstanding issue or pull request in the Extras repository — we’ve also been reviewing ways to reduce what is essentially “Github issues debt” (a play on the term “technical debt“). One of the main issues we’ve identified is, quite simply, bottlenecks in process. While the addition of myself to Greg’s team (formerly a team of one, now a team of two!) is helping with the day-to-day tending and triage of new and existing issues, which was a bottleneck in itself, the previous review process for new modules had a short list of approved reviewers, who not only have lives and are sometimes busy, but also didn’t always have the domain expertise with the technologies enabled in new modules.

And thus: A new process has been born. I encourage you to read the details, particularly if you are interested in helping with reviews, or are already contributing to Ansible, which were outlined by Greg on Friday on the ansible-project and ansible-devel mailing lists. That said, here are the important highlights:

1. Any caring Ansible user can review new Extras modules.
2. 2 +1 votes, and no -1 votes, will result in the new module being merged into Extras. More specifically, a +1 vote to the module working as expected (meaning: you have tested the module in good faith) — and a +1 vote verifying that the module follows the Ansible module guidelines.

Finally, as we outlined in the above referenced mail: this process is based on trust. Trust that users are testing these in good faith, and ensuring that the guidelines are being followed; trust that the submitters of these new modules are willing and able to maintain these new modules over time, and respond to issues and pull requests in a timely fashion.

Want to help?

You have my undying gratitude! And the endless thanks, I’m sure, of the module contributor. Here are some links to get you started:

• As mentioned previously — Greg’s post to the mailing list outlining new PR process for new modules in Extras (the how, and the why.)
• Reviewers file for Extras on GitHub: Outlines basic process.
• Module guidelines.
• General contributor information for Ansible Extras
• Module development documentation, including module testing information.
• Ansible test-module code, and associated documentation.
• More testing info that may be of use:
  • Unit testing Ansible Modules, part 1 and part 2 (courtesy of @linuxsamba)
  • Debugging an Ansible module (courtesy of Graham Hay)

Help get these Extras modules mooo-ving!

While this is in no way a full list of all outstanding new modules that need reviews, it is a list of those that aren’t already under review, require revision, etc. Some of them are entirely new; others are new modules that have had some review and revisions made, but are now stalled for lack of new review, and can now be approved under the new process. If you’re feeling particularly motivated and want to see if something you’re interested in is making progress or needs help — the full list of unreviewed or in-progress new Extras modules can be seen here.

And, yes: Honeybadger is first. Because I know you give a… ahem. Hoot. 🙂

Honeybadger: module to notify Honeybadger.io about app deployments.

Database Stuff:

• Couchbase: module to create and delete Couchbase buckets, using the Couchbase REST API.
• CouchDB: module for handling CouchDB users
• mssql_db: module to create, delete databases and execute sql scripts. (This is basically the mssql equivalent of mysql_db module.)

Docker: docker_facts module to return information about running containers

Nexus: This adds a module for pulling artifacts from a Nexus repository.

Windows (Yes, Windows.)

• win-zip module: Zips file or directory on a Windows node
• win_s3 module: module for downloading/uploading to/from S3 on the Windows node machine. Uses the AWS_SDK for Powershell.

AWS / EC2 / S3 / Redshift

• module redshift handles creating, deleting, modifying, and facts gathering of Redshift clusters
• module redshift_subnet_group handles creating, modifying, and deleting of Redshift Subnet Groups 
• AWS Redshift security groups: https://github.com/ansible/ansible-modules-extras/pull/293
• ec2_vpc_peering:  Module for creating VPC peering connections. https://github.com/ansible/ansible-modules-extras/pull/245
• s3_website: module to manage an s3 bucket as a website
• s3_website_redirect_rule: module to manage s3 bucket website redirect rules
• elastic_beanstalk: module for creating, terminating, and restarting AWS Elastic Beanstalk environments, including support for blue-green deployments using CNAME swapping. 
• elasticache_parameter_group: module to enable creation / deletion / modification of the cache_parameter_group.

Sensu: sensu_subscription manages Sensu subscriptions of the Sensu client running on a machine. Note: this goes well with the already-existing sensu_check module, which manages sensu checks and allows you to specify all possible options (including the correct types) — if you’re using this module, you’re probably a great candidate to review sensu_subscription!

nfsexport: module for working with entries in /etc/exports (or nfs exports file in an otherwise specified location).

Apache Kafka: The kafka_topics module creates new topics in Kafka, or modifies existing ones (though only increasing the partition count is supported in Kafka). Topics can be operated on one at a time, or in a group, to save on starting up the JVM for each topic to check its current state.

git-flow: Adds hooks for executing git-flow commands. Git-flow is a collection of Git extensions to provide high-level repository operations for Vincent Driessen’s branching model.

openweather fact gathering: module to use the openweathermap API to retrieve the current weather at a location.

Webfaction: module to gather facts from Webfaction, including facts about applications, websites, databases, and domains.

ZFS: a module for managing ZFS admin privileges.

ProfitBricks: module to create or restore a volume snapshot. 

Interesting Utilities:

• blockinfile: module to insert/update/remove a text block surrounded by marker lines in a file, in the similar manner as lineinfile module.
• blockfromfile: module for extracting values (via regexp pattern) from an existing file on remote nodes. Should fail silently by default (or loudly if desired), when said file doesn’t exist.
• Deploy-helper: this module was formed to help lower the boilerplate needed to deploy web-projects.

PagerDuty: pagerduty_service allows you to create, update, disable, and delete services in PagerDuty. You can configure webhooks for the services, and prompt PagerDuty to regenerate service keys for API type services.

CoreOS / fleet:

• fleet_facts module which gathers information about CoreOS clusters
• fleet_unit module which manages CoreOS systemd services via Fleet

OpenVZ:

• openvz: manage OpenVZ containers
• openvz_exec: execute commands in OpenVZ container