×
Become a PowerToFly VIP
For less than one coffee a week, receive exclusive content, access to live chats with female thought leaders, and more when you sign up to become a VIP!
Tech Talk

Continuous Deployment for Docker: Lessons Learned

First Hand Experience on the Path to Continuous Deployment for Docker Containers

Authors note: this is a summary of my session at DevOps Summit NY on June 8, 2017. For details on the session along with slides, visithttp://datagridsys.com/nomorescripts.

Being experienced startup folks we are not scared of bleeding edge tech. For instance, in 2005 we made the jump to virtual machines when most companies hadn't even heard of virtualization, in order to create our cloud platform, AppLogic. This time around we were already very familiar with containers, including many of the limitations, and believed they offered a lot of advantages in speed and cost so we decided to go with Docker running on AWS.

Our engineers have written a lot of apps including both interactive user apps and infrastructure level services and we're pretty adept at creating stateless systems where that's possible. Unfortunately, some of the third party software we needed to use, especially open source, wasn't written with stateless in mind.

For us combining Dev and Ops wasn't a choice, but a necessity. We couldn't afford another hire to run ops. Nonetheless we're fanatics for DevOps and automating everything we can in build, test and deploy. In our case the role of the DevOps guru fell on Pavel, one of our senior engineers. He's well versed in systems architecture as well as most of the systems that needed to be touched by the automation.

Introducing VCTR

The app we were bringing out is called Victor (VCTR for Vulnerabilities and Configuration Tracking and Reporting), a free online service that scans containers and VMs for upgrades that fix critical vulnerabilities. You can check out VCTR here.

As we progressed through development some things worked exceptionally well. New tools including Slack made collaboration among a dispersed team very easy. Build and test tools including Jenkins worked as expected though they were a bit harder to set up than they should be. Docker, despite some minor issues, was more mature than we expected. As you probably have guessed, though, there's a "however" coming. In our case, that "however" was that deployments were still complicated and error prone.

As we got started our deployment was managed part-time by Pavel, our senior engineer. After all, the literature available suggested he'd just be writing a few short pipeline scripts. At first this worked great, but since change is inevitable, things got complicated. VCTR began expanding as new features were integrated and Pavel's day job became wrangling an unruly set of deployment scripts. Of course, as he spent time on deployment he wasn't contributing to building VCTR. As Pavel got sucked further and further into script hell with each new change, he noticed he'd fallen into a pattern:

  1. understand changes -> understand current app -> write script
  2. test script -> debug -> deploy
  3. fail -> debug again -> redeploy
  4. manual rollback...
  5. give up

The options were minimal. Either we continue to fail through this deployment until we got it right, or find another solution to the problem

Deployment Pain

Despite deployment issues hindering our velocity we needed to push forward with VCTR so we started user engagement while looking for solutions. We assumed we'd find a new tool or pattern in time, but things took an unexpected turn. While implementing VCTR we found other folks having similar deployment issues. Bringing up a new user with VCTR requires installing a small piece of code in the container or VM. That process, not surprisingly, happens in a deployment script and in most cases inserting that one change caused a failure in deployment. In one instance, while on a video call with a user, we watched as a five minute change to one script turned into a two hour debugging marathon involving more than half a dozen scripts. The only folks who didn't have errors after making the change had actually recognized deployment as a significant issue and built their own system for it. Systems, so significant in the amount of time they took to build that they were given names and teams to maintain them. However, those systems were tied to their company's specific architecture and tool chain.

The heart of the problem is that despite articles to the contrary, containers and microservices have dependencies. Some dependencies are simple. For instance, some software still have start order dependencies. Others require configuration differences based on the environment they're running in. Still other require logging in and executing commands after startup. Some dependencies you create yourself. As an example, changing an API requires updates to both the providing and calling services. You might require all updates to be backward compatible, but that's just moving the work to what may be an even more complicated arena. Lastly, what happens when something fails? How do you find out what failed? How do you get back to a functional state?

Despite mounting evidence we didn't initially recognize the pattern. One day over lunch, as the discussion turned from VCTR to deployment issues we realized we'd seen similar problems before when we first tackled cloud deployment. Furthermore, we knew we could solve this. We decided to build a continuous deployment system.

Lessons Learned

We obviously learned a ton about what not to do when setting up a new containerized application. Here's our conclusion of the good and bad of containers, microservices and DevOps:

The good

  • Containers, though not perfect, have a rich enough ecosystem of tools to be used in production.
  • Microservices architectures are great but don't try to make everything stateless.
  • DevOps is the new norm, but you need to go beyond just assigning an engineer the title or merging the folks onto a single team. Ops must take on a new role defining use cases and Dev must embrace Ops.
  • Get your tool chain in place from the start. We tried to hack our deployments and as such didn't see the problem for what it was until we were knee deep in it.

The bad

  • Deployment remains hard because it's where software meets the real world.
  • Deployment requires a system once you get past a few containers. If you don't adopt one, you're going to end up building one - whether you realize it or not.
  • Scripts work fine for small apps that don't get updated often, but aren't up to the task for continuous deployment for apps that contain more than a few containers or microservices, especially those that are updated frequently.

The Result - Continuous Deployment System

We identified key challenge patterns that both ourselves and others faced during deployment and turned them into solutions. One important difference to the other deployment systems we had seen was that we made sure to build for varied environments and architectures. We didn't want this tool to just work for us or rather just the current pattern we were working in. We needed a true system to grow with us. Here are some of the main requirements of our Continuous Deployment system:

  • Integrate with popular tool chain elements
  • Recognize code and architecture updates
  • Provide for identifying dependencies
  • Provide for various operating environments
  • Automatically deploy authorized updates to operating environments
  • Automatically correct for common errors to return proper operation
  • Provide meaningful logs and error messages
  • Be operable from either UI, API or completely autonomously

We built this deployment system and it's now deploying VCTR on a regular basis. It's also in production at InnoScale and being tested by a handful of DevOps teams who felt the same pain as we did. We're offering the system for free to teams for up to 15 containers.

To learn more about Skopos - our continuous deployment system visit http://datagridsys.com/skopos/.

Originally article on Ulitzer


Deloitte

Deloitte + PowerToFly: An Exclusive Evening With Women Finance Leaders

Join Deloitte M&A and PowerToFly (a women-run organization that introduces select women to executives at major corporations) for an invite-only event on February 28th. The evening will kick off with a panel discussion featuring Deloitte's women financial leaders, and will conclude with an opportunity to network with your peers and the Deloitte team over food and drinks.

The event will be held on Thursday, February 28th from 6:00pm to 8:00pm at 30 Rockefeller Center, 32nd Floor, New York, NY.

Agenda (Subject to Change):

  • 6:00pm - Check-In & Networking over cocktails and light food
  • 6:20pm - Kick-Off from PowerToFly
  • 6:25pm - Welcome & Introduction from Susan Dettmar, M&A Consultative Services Practice Leader at Deloitte
  • 6:35pm - A Behind the Scenes Look at Deloitte with Nadia Orawski, Principal, Deloitte Consulting
  • 6:45pm - Panel Discussion featuring Deloitte's Women Leaders in Finance
  • 7:00pm - Audience Q&A
  • 7:20pm - Networking continues over beverages and light food

Deloitte is hiring, however, you don't need to be looking for new opportunities to attend.

About our Events: All RSVP'd attendees are welcome, regardless of race, color, religion, national origin, gender identity, pregnancy, physical or mental disability, or age. If you require accommodation to fully participate in this event, please email hi@powertofly.com, and we will contact you to discuss your specific needs.

Unfortunately, PowerToFly and Deloitte cannot admit outside recruiters to this particular event. Please email hi@powertofly.com if you have any questions about this policy.

About Deloitte: Deloitte is committed to recruiting, developing, and promoting a diverse workforce while providing unique opportunities across our businesses. Deloitte has been recognized by Working Mothers as one of the 100 best companies for multicultural women. Their benefits include up to 16 weeks of Paid Family Leave, pension plans, and gym & health subsidies.

Event Audience: This event is a great fit for a number of roles including Finance Managers, Integration and Execution Managers, and M&A Strategists.

Better Companies

Audible, Microsoft & HomeAway Leaders On Building Diverse Talent Pipelines

Presented by PowerToFly and Glassdoor

Register today to join this FREE webinar we are co-hosting with Glassdoor!

PowerToFly and Glassdoor have partnered to present a webinar on February 6th, where top talent leaders will share how they're building diverse talent pipelines.

Join leaders from Audible, HomeAway and Microsoft, who will discuss their current D&I strategies and share advice on how you can build your own diverse pipelines.

Moderated by Katharine Zaleski, PowerToFly's CoFounder & President, speakers will include:

  • Jamy Barton, Sr. Director, Tech Diversity at Audible
  • Lyndsi McNaughton, Sr. Global Sourcing Programs Lead - D&I at HomeAway
  • Luci Gomes, Global Director of Talent Acquisition Americas & Commercial, Marketing & Consumer Teams at Microsoft

The webinar will take place on Tuesday, February 6th from 1:00pm to 2:00pm EST.

The second half of this webinar will be dedicated to audience Q&A, and our panel is looking forward to taking your questions. If you have a particular question or topic you'd like us to explore, let us know on the registration form and we'll be taking questions in real time on the webinar too!

About our webinars: All RSVP'd attendees are welcome, regardless of race, color, religion, national origin, gender identity, pregnancy, physical or mental disability, or age.

Should You Start Looking for a New Job?

Take this quiz from our career coach to find out!

I'm 26 and I'm already on my fourth job. My dad is 60... and he's on his second. Clearly, we both fit the Boomer and Millennial stereotypes rather well. Thankfully, though, he's never judged me for "job hopping."

Instead, we've discussed our experiences and realized that he's as scared to quit his job as I am to stay at mine. No matter how old you are, deciding whether you should stick things out or move on isn't easy.

So, to help you organize your thoughts, we put together a "Should I quit my job?" quiz with the help of our career coach, Heather Coll (a member of the much less controversial Gen X).

These questions will help you figure out if it's time to move on - from your current role, company, or both. Because no matter what point you're at in your career, or which generation you're from, you deserve to be happy at the place where you spend most of your waking hours!!!



How'd you do? If you're ready to start job hunting and need help putting your best foot forward, Heather can help you with everything from crafting the perfect cover letter to nailing your interview. Or, if you're still not sure about your next steps, she can help you figure out how much longer you should stay where you are, and how to start looking for the perfect job once you decide to leave. Just don't feel like you have to decide by yourself, and don't let anyone judge you for your decisions - whether they're a cranky Baby Boomer, an entitled millennial, or part of some generation no one's even heard of anyway. You do you!

Microsoft Corporation

Meet Microsoft’s Women Tech Leaders

If you are a Dallas based woman in tech and you'd like to attend our event, please contact us at hi@powertofly.com to be considered for an invite.

Microsoft has partnered with PowerToFly to present a one-night-only evening for women in tech. The night will include a discussion with several of Microsoft's leaders, many of whom are Modern Apps Consultants, Secure Infrastructure Consultants, and Premiere Field Engineers. There will also be plenty of time to network with Microsoft leaders and top women in your field.

The event will take place on Tuesday, February 5th from 5:30pm to 7:30pm at House of Blues, 2200 N. Lamar Street, Dallas.

Agenda (Subject to Change)

  • 5:30pm - Check-In and Networking over Food & Drinks
  • 6:10pm - Introduction by PowerToFly
  • 6:15pm - Keynote Address
  • 6:25pm - Panel Discussion featuring from Microsoft's women leaders including Luci Gomes - Global Director of Talent Acquisition Americas, Services, Commercial, Marketing & Consumer Teams
  • 6:35pm - Audience Q&A
  • 6:45pm - Networking Continues

If you are interested and available, Microsoft will be asking some attendees to interview for open roles on Wednesday, February 6th, the day after the event. Follow up interviews will take between three to four hours of your time and will be comprised of two technical interviews and one hiring manager interview. Let us know when you register if you are available and if so, if you'd prefer the morning or the afternoon. If you are not available on 2/6, Microsoft and PowerToFly will work with you to find an alternate follow up date.

Microsoft is invested in women-focused organizations and providing support to women once they are employees at Microsoft. Their employee network Women@Microsoft reaches over twenty thousand people worldwide. Microsoft's ongoing diversity partnerships include the Anita Borg Institute, National Center for Women & Information Technology (NCWIT), and MentorNet.

Visit Microsoft's page on PowerToFly to learn more about their open roles, benefits and diversity initiatives.

About our Events: All RSVP'd attendees are welcome, regardless of race, color, religion, national origin, gender identity, pregnancy, physical or mental disability, or age. If you require assistance to fully participate in this event, please email hi@powertofly.com, and we will contact you to discuss your specific needs.

Unfortunately, PowerToFly and Microsoft cannot admit outside recruiters to this particular event. Please email hi@powertofly.com if you have any questions about this policy.

Women & Diversity in Hollywood: A Look at 2018

In Spite of #MeToo, Women Directed Fewer Films in 2018 than in 2017

2018 was a banner year for women in politics, with more first-time congresswomen elected than ever before and the most women in Congress ever. Unfortunately, in spite of #MeToo and Time's Up, it wasn't a similarly good year for women in Hollywood. A study by the USC Annenberg Inclusion Initiative shows that women actually directed less films in 2018 than they did in 2017.

READ MORE Show less
GET EMAIL UPDATES FROM POWERTOFLY
COMPANIES THAT CARE
FOLLOW POWERTOFLY
© Rebelmouse 2018