One of the biggest misconceptions I hear from laypeople about AI is the belief that machine learning models get smarter as they interact with the world.
My friend told me a story about a bug that cost his company millions of dollars and illustrates why this perception is wrong. When his boss asked him to investigate a spike in user churn, he realized that a change in a feature definition for recently registered users had broken his model. New users were leaving because his model had recommended random items to them for nearly a month.
The issue isn't just data bugs — the static machine learning models we learned to train in school are downright fragile in the real world. Your inventory forecasting model might break because COVID-19 changes shopping behavior. Your sales conversion model might break because a new marketing campaign succeeded. Your autonomous vehicle might even break because you don't have enough kangaroos in the training set. Static models degrade because real world data is almost never static — your training set isn't representative of the real world for long.
So, as hard as it is for us ML PhDs to admit, most of the work happens after you train the model. After grad school, as I spent more time with teams building ML-powered products, I realized that the best ones don't see their job as training static models. Instead, they aim to build a continual learning system, a piece of infrastructure that can adapt the model to a continuously evolving stream of data.
At Gantry, we want to help teams build ML-powered products that improve as they interact with the world, much like laypeople believe they do already. We think the key to doing so is to transition from training static models to building continual learning systems.
From static models to continual learning systemsIn static ML, you are given a dataset and your job is to iterate on the learning algorithm until the model performs well. Continual ML adds dataset curation as an outer loop to the iteration process.
Continual learning systems adapt models to the evolving world by repeatedly retraining them on newly curated production data. For example, if you retrain your model every month on a sliding window of data, then you have built a simple continual learning system.
Your continual learning system only succeeds if the performance benefit from retraining outweighs the cost. So the more valuable your model, the more quickly your system needs to react to degradations in performance and the more carefully it needs to test the new model to make sure it truly performs better. The more expensive it is to retrain, the more intelligently your system should curate the next dataset using techniques like active learning.
When it does succeed, a continual learning system can cause your product to improve as it interacts with the world. As you collect new data, you can trigger a data flywheel effect, where more data leads to a better model, which improves your product, allowing you to acquire more users and data, closing the flywheel.
Despite the benefits of continual learning systems, few teams have implemented them successfully. ML practitioners in the Full Stack Deep Learning alumni network cited the components of the continual learning system — post-deployment monitoring, maintenance, and retraining — as the most challenging aspects of the ML lifecycle.
So, why has this transition been difficult?
Continual learning has an evaluation problem
My friend's team reacted to their expensive data pipeline bug by building a suite of tools to evaluate the performance of their models more granularly. Catching the degradation in performance for new users quickly would have allowed them to mitigate it by retraining or rolling back the data pipeline. Similarly, for most of us, the challenge with continual learning isn't building the system, it's understanding model performance well enough to operate it.
To see why, let's view the continual learning system through the lens of control systems. Think of a control system as any machine, like a plane, that adapts to a constantly evolving environment to achieve its goal. The plane's pilot flies it by operating a control loop. As conditions change, the pilot adjusts the flight path with actuators (knobs, pedals, etc), observes the effect of the adjustment by reading instruments, and uses the observation to decide on the next adjustment.
We operate our continual learning systems in a similar way. We adjust the system when we ingest, label, or featurize data points, or when we train a new model or promote a different version to production.
Thanks to an explosion in MLOps tooling, in 2021 we can actuate the continual learning system entirely using open source libraries. We can ingest data using Apache Kafka, label it using LabelStudio, and featurize it using Feast. We can retrain the model using a ML library like PyTorch and an orchestrator like Airflow, and deploy it using a serving library like KFServing or Cortex.
At Gantry, we think what's missing are the instruments. Few ML teams today have visibility into model performance required to make decisions about when to retrain, what data to retrain on, and when a new model is ready to go into production. ML teams are operating the continual learning system by feel. Instead, we should be operating them by metrics.
Where do we go from here?
Understanding the performance of your model well enough to adjust your continual learning system is a challenging organizational, systems, and algorithms problem.
- Stakeholders have different priorities for the model. How can you align the team on what evaluation metrics to use?
- The raw data needed to compute the metrics are generated across your system, including the training infrastructure, serving infrastructure, application code, and other parts of the business. How can you build the data infrastructure necessary to view your metrics in one place, especially at scale?
- Ground truth feedback is delayed, noisy, and expensive. How should you approximate your metrics with limited labels?
At Gantry, we're building a tool called an evaluation store that serves as a single source of truth for your model performance metrics. You connect the evaluation store to your raw data sources (e.g., your production model) and define metrics and the data slices on which to compute them. The eval store computes the metrics for you and exposes them to downstream systems so you can use them to make operational decisions.
Here at Zapier, we are ecstatic to announce that our very own Maggie Roque, Director of Diversity, Inclusion, Belonging, and Equity (DIBE), has been recognized by Untapped as a ‘Walk the Walk’ Award winner, alongside the top DEIB leaders of 2022! 🎉
The purpose of Untapped's Top DEIB Leaders of 2022 'Walk the Walk' is to recognize the people challenging the status quo and pushing #DEIB forward in their organizations and communities.
At #Zapier, we believe in DIBE as DNA. As a result of Maggie's leadership, Zapier now has year-by-year goals with supporting initiatives — including how we will measure success for each of our DIBE pillars: people, product, and the public good. Maggie has refined our DIBE strategy in alignment with Zapier’s mission, values, and business strategy. Her work helps ensure all Zapiens, especially our leaders, are equipped with the knowledge, tools, and resources to make it a consistent and standard way of operating. She has also provided mentorship to our employee resource groups, which include BIPOC of Zapier, Women of Zapier, and Prizm (LGBTQIA+).
No matter what, at Zapier we can count on Maggie to align and empower our team toward tremendous DIBE impact as we pursue our mission to make automation work for everyone. We are so proud of you, Maggie! Congratulations on this well-deserved award!
Learn more about Untapped and the 2022 award recipients here: https://lnkd.in/dQfa6Ygw
💎Nestlé’s manufacturing excellence team is growing. The team supports Nestlé USA factories that produce bakery sweets brands including Toll House, Libby's and Carnation, and Nestlé Professional Brands which supply food service operations. Watch the video to the end to apply and begin your career there!
📼The manufacturing excellence team seeks someone passionate about driving world-class manufacturing through continuous improvement methodologies. Jennifer Watson and Taylar Marshall, Senior Managers, give you all the information you need to join their team.
📼Join the manufacturing excellence team if you are a go-getter, someone who takes the initiative to establish cross-functional teams to eliminate losses. This also means you should be highly collaborative with a variety of people and have a curious mindset about how things are manufactured. If you fill these requirements, don’t hesitate to apply!
📼The manufacturing excellence team unlocks career path opportunities throughout different functions, locations, and brands across Nestlé USA. Jenny Watson shares her own experience: her career has included roles in three different functions: manufacturing excellence, manufacturing, and operations strategy. She was based out of three different locations: Springville, Utah, Solon, Ohio, and Medford, Wisconsin across four different categories. The opportunities at Nestlé are truly endless!
Inside The Manufacturing Excellence Team
This team is driving continuous improvement and project management routines in the Toll House factory to contribute to the overall expected business results in the bakery and sweets category. It is a boots-on-the-ground team that tries to solve complex problems with a focus on people development and operator capability building. No day is the same in their team!
🧑💼 Are you interested in joining Nestlé USA? They have open positions! To learn more, click here.
Get to Know Jennifer Watson and Taylar Marshall
More About Nestlé USA
Nestlé USA has been nourishing a growing world for generations. No matter where you work within the Nestlé organization, you’ll discover new opportunities to grow while you help them inspire healthier lives, support local communities, do what’s right for the planet, and make an impact.
From September 12-15, 2022, PowerToFly hosted a four-day virtual event, featuring a three day summit and single day virtual job fair.
To kick off the event, attendees had the opportunity to partake in a one-hour guided networking session followed by three full days of fireside chats and panels where they were able to listen and ask questions to experts and thought leaders across multiple industries.
Featured Summit Topics Included:
- The Art & Science of How to Clarify Your Best Fit Career Path
- Going Back to the Drawing Board: How to Navigate Major Career Shifts
- Pulling Back the Curtain: Understanding What’s Happening Behind the Scenes In the Hiring Process
- 4 Ways to Get Your Foot in the Door to a New Career
- Nailing the Basics: How to Grow with Intention and Purpose
- How to Break Into a New Industry Without Starting Over
Companies We Hosted At The Job Fair:
- Bank of America | Hiring for: Senior Financial Analysts, Business Bankers, Senior Technology Managers, and more!
- ScienceLogic | Hiring for: Technical Support Engineers, Chief Marketing Officers, Product Managers, Executive Assistants, and more!
- PowerToFly | Hiring for: Global DEIB Strategist & Trainers, Account Executives, Support Specialists, Events Specialists, and more!
Thank you for joining 4 Ways to Get Your Foot in the Door to a New Career with Flatiron School Career Coach Betsy Kent! In case we weren’t able to get to your question in the Q&A, or if you thought of additional questions after we wrapped, here are two ways you can contact the Flatiron School Admissions team directly:
- Schedule a casual 10-minute chat with a Flatiron School Admissions rep
- Email us at email@example.com
Attending information sessions, panels, and workshops is the best way to get a sneak peek into what studying at Flatiron School is like — so don't miss what else is coming up! You can find a list of our events HERE.
Starting out as a viral trend on TikTok, the phrase “quiet quitting” has since taken over headlines everywhere from NPR to the Harvard Business Review. But what, exactly, is quiet quitting — and why are so many business leaders getting this so-called “crisis” wrong??
What is quiet quitting?
Per Psychology Today, “quiet quitting” isn’t actually quitting in the two-week notice sense of the word. It’s when employees keep doing their job, but only do the work that’s in their job description or covered by their explicit responsibilities. No going above and beyond. No late hours. No taking on extra projects that don’t come with extra remuneration.
Gallup similarly defines the trend as employees who are “not engaged” at work — people who “do the minimum required and are psychologically detached from their job.” Per their research, that’s a full 50% of the American workforce.
Why quiet quitting isn’t actually a crisis
As a burgeoning attitude toward work, quiet quitting makes perfect sense. With the challenges and stresses of the last few years impacting all workers — but especially working parents, people of color, women, and other marginalized groups — employees are looking for ways to set boundaries, disengage from work, and find working rhythms that work for them and their lives.
And that’s something companies should be supporting. Employers’ responsibility, after all, isn’t to slap a Band-Aid on the problems that are driving quiet quitting in order to get productivity metrics up. It’s to create the conditions for employees to succeed, with work that can be accomplished within reasonable working hours, and to incentivize and tangibly reward any engagement that goes beyond quiet-quitting levels.
It’s time we got this clear. Quiet quitting was never the crisis. Expecting employees to go above and beyond at work in order to maybe stand a shot at a pay raise and promotion next year was.
If you want to ensure your company culture is creating opportunities for folks to feel truly engaged, we’ve rounded up the steps to take below.
8 things your company needs to do to stop facilitating quiet quitting
Quiet quitting doesn’t mean that employees don’t want to work. It means that everyone — employees and employers alike — are recognizing, more than ever, that the workplace can and should be evolving to meet the needs of everyone involved in making work happen. Here are some ways that companies can ensure they are doing that, sourced from McKinsey research on burnout and engagement:
1. Hold your leadership accountable.
Culture is set by the people on the ground, and you need to know that your managers and leaders are creating a culture that’s supportive of mental health. This looks like incorporating mental health questions into regular employee satisfaction surveys, so you have data to track, and including the management of employee well-being as part of how leaders are evaluated and compensated. It also means getting rid of toxic leaders.
2. Destigmatize mental health and boundaries.
Most employers know that stigma exists at work, despite best intentions to fight it. But when employees are afraid to ask for help with mental health needs or to request accommodations so they can do their best work, everyone suffers. Companies can work to destigmatize the issue by highlighting senior leaders’ own experiences with mental health. Vulnerability can help promote psychological safety, as can rewarding employees for setting boundaries and using mental health and wellness benefits.
3. Evolve the kind of benefits you offer.
45% of people who have recently left their jobs said that their care responsibilities were a big part of their decision. Do the benefits your company offers reflect that reality? For instance — if employees must be on-site, can you offer on-site childcare? If not, do you offer a childcare stipend? Do you know what issues they are most struggling with, and are you responding?
4. Promote sustainable working hours.
Do your employees need to be at work — whether online or at the office — from 9 a.m. to 5 p.m.? Or can they set those hours to fit their own schedules? Do you have flexible work policies that are available to everyone, no matter their level of seniority? Hybrid work can facilitate unfair treatment when policies aren’t clear and universally applicable.
5. Provide opportunities for employees to build social ties.
Another reason employees are disengaged at the office? Lack of social support. It can be hard to make connections over video calls and chat, especially for new employees or those who haven’t worked remotely before. Investing in team building can help give employees access to social connections that make their work more meaningful over time.
6. Enable right-size workloads.
As employment has ebbed and flowed over the pandemic, and especially now during the Great Resignation, many companies are finding themselves short-staffed. But piling more work on the people who have stayed isn’t a sustainable solution — it just speeds up their own burnout. Creating
7. Facilitate upskilling and reskilling at work.
Per the McKinsey study linked above, employers who offer reskilling and upskilling opportunities end up with more engaged employees. It pays off for everyone involved: giving employees the chance to laterally move into a different job in order to learn a new set of skills can predict employee retention 250% more than compensation can, for instance.
8. Strengthen your commitment to DEIB.
Employees don’t want to work somewhere they don’t feel like they belong. McKinsey calls out five key action areas when it comes to making a DEIB commitment real: ensuring representation, holding leadership accountable, increasing transparency (like with analytics on promotions and pay), tackling issues with a zero-tolerance policy, and embracing intersectionality.