BOSTON (16 November 2021) – The U.S. Air Force’s Platform One and Kessel Run signed a memorandum recently, titled ‘We Believe’ to continue to collaborate and support one another, their ally organizations and their end users.
The software principles memo serves as a nod to each platform’s working relationship in the Department of Defense as the premier software factories in the military software ecosystem. Both organizations have a similar ethos, culture and purpose; the memo, signed by their respective commands, was also drafted as a way to formalize collaboration efforts and blur the lines between each service, helping to prevent siloing and duplication of efforts in the DoD.
“Kessel Run and Platform One are both such huge thought leaders and cultural leaders for the department, starting with Kessel Run smuggling DevSecOps into the DOD, and continuing with Platform One leading the way to Kubernetes, a common repo, and a desire to bring the entire community together and leverage common enterprise services,” said Lauren Knausenberger, Chief Information Officer of the Air Force. “Together, these teams are moving the department forward by deploying awesome warfighting software and giving more and more Airmen the opportunity to learn agile processes. It’s a huge shift to the way we execute our mission and we need more of it.”
Platform One and Kessel Run’s respective mottos of “Only You Can Prevent Waterfall” and “Ideas Over Rank” are synthesized into key points such as “collaboration over mandates” and “flexibility is the key to staying relevant.” Key points in the memo highlight how the future of DevSecOps and agile software development depend on collaboration toward identical goals, together.
Some calls-to-action in the memo even elicit cooperation from other software teams to help build collaboration across multiple programs, as well as contributing to open source and prioritizing talent pipelines.
Kessel Run, officially known as the Air Force Life Cycle Management Center, Detachment 12, has a proven track record in enhancing efficiency, saving cost and modernizing the way the Air Force operates with its agile software development. With its user-centered approach to development, Kessel Run’s mission is to rapidly deliver combat capabilities to warfighters and revolutionize the Air Force software acquisition process.
“This statement solidifies our commitment to synergize our efforts in modernizing the Air Force with the software solutions we need today to win tomorrow,” said Col. Brian Beachkofski, Commander of Kessel Run. “Platform One and Kessel Run are building the future together.”
Platform One is a software factory focused on merging top talent from other software factories across the Department of Defense. Platform One assists software factories by helping them focus on building mission applications. They provide: collaboration and cybersecurity tools, source code repositories, artifact repositories, development tools, DevSecOps as a service, and more.
“Tomorrow’s conflict will begin with the weapon systems we’ve acquired, but will be won by our ability to adapt faster than our adversary,” said Lt. Col. Brian Viola, Commander of Platform One. “I look forward to our continued collaboration with Kessel Run and laying the foundation necessary to field at the speed of relevance, at enterprise scale.”Air Force Life Cycle Management Center Detachment 12, Kessel Run Media and Communications Engagement Email: media@kr.af.mil
2018, when Google employees found out about their company’s involvement in Project Maven, a controversial US military effort to develop AI to analyze surveillance video, they weren’t happy. Thousands protested. “We believe that Google should not be in the business of war,” they wrote in a letter to the company’s leadership. Around a dozen employees resigned. Google did not renew the contract in 2019.
Project Maven still exists, and other tech companies, including Amazon and Microsoft, have since taken Google’s place. Yet the US Department of Defense knows it has a trust problem. That’s something it must tackle to maintain access to the latest technology, especially AI—which will require partnering with Big Tech and other nonmilitary organizations.
A new survey shows the controversial systems are poised to play an even bigger role in federal business.
In a bid to promote transparency, the Defense Innovation Unit, which awards DoD contracts to companies, has released what it calls “responsible artificial intelligence” guidelines that it will require third-party developers to use when building AI for the military, whether that AI is for an HR system or target recognition.
The guidelines provide a step-by-step process for companies to follow during planning, development, and deployment. They include procedures for identifying who might use the technology, who might be harmed by it, what those harms might be, and how they might be avoided—both before the system is built and once it is up and running.
“There are no other guidelines that exist, either within the DoD or, frankly, the United States government, that go into this level of detail,” says Bryce Goodman at the Defense Innovation Unit, who coauthored the guidelines.
The work could change how AI is developed by the US government, if the DoD’s guidelines are adopted or adapted by other departments. Goodman says he and his colleagues have given them to NOAA and the Department of Transportation and are talking to ethics groups within the Department of Justice, the General Services Administration, and the IRS.
The purpose of the guidelines is to make sure that tech contractors stick to the DoD’s existing ethical principles for AI, says Goodman. The DoD announced these principles last year, following a two-year study commissioned by the Defense Innovation Board, an advisory panel of leading technology researchers and businesspeople set up in 2016 to bring the spark of Silicon Valley to the US military. The board was chaired by former Google CEO Eric Schmidt until September 2020, and its current members include Daniela Rus, the director of MIT’s Computer Science and Artificial Intelligence Lab.
Yet some critics question whether the work promises any meaningful reform.
During the study, the board consulted a range of experts, including vocal critics of the military’s use of AI, such as members of the Campaign for Killer Robots and Meredith Whittaker, a former Google researcher who helped organize the Project Maven protests.
Whittaker, who is now faculty director at New York University’s AI Now Institute, was not available for comment. But according to Courtney Holsworth, a spokesperson for the institute, she attended one meeting, where she argued with senior members of the board, including Schmidt, about the direction it was taking. “She was never meaningfully consulted,” says Holsworth. “Claiming that she was could be read as a form of ethics-washing, in which the presence of dissenting voices during a small part of a long process is used to claim that a given outcome has broad buy-in from relevant stakeholders.”
If the DoD does not have broad buy-in, can its guidelines still help to build trust? “There are going to be people who will never be satisfied by any set of ethics guidelines that the DoD produces because they find the idea paradoxical,” says Goodman. “It’s important to be realistic about what guidelines can and can’t do.”
For example, the guidelines say nothing about the use of lethal autonomous weapons, a technology that some campaigners argue should be banned. But Goodman points out that regulations governing such tech are decided higher up the chain. The aim of the guidelines is to make it easier to build AI that meets those regulations. And part of that process is to make explicit any concerns that third-party developers have. “A valid application of these guidelines is to decide not to pursue a particular system,” says Jared Dunnmon at the DIU, who coauthored them. “You can decide it’s not a good idea.”
One Defense Department advisor suggests that “constructive engagement” will be more successful than opting out.
Margaret Mitchell, an AI researcher at Hugging Face, who co-led Google’s Ethical AI team with Timnit Gebrubefore both were forced out of the company, agrees that ethics guidelines can help make a project more transparent for those working on it, at least in theory. Mitchell had a front-row seat during the protests at Google. One of the main criticisms employees had was that the company was handing over powerful tech to the military with no guardrails, she says: “People ended up leaving specifically because of the lack of any sort of clear guidelines or transparency.”
For Mitchell, the issues are not clear cut. “I think some people in Google definitely felt that all work with the military is bad,” she says. “I’m not one of those people.” She has been talking to the DoD about how it can partner with companies in a way that upholds their ethical principles.
She thinks there’s some way to go before the DoD gets the trust it needs. One problem is that some of the wording in the guidelines is open to interpretation. For example, they state: “The department will take deliberate steps to minimize unintended bias in AI capabilities.” What about intended bias? That might seem like nitpicking, but differences in interpretation depend on this kind of detail.
Monitoring the use of military technology is hard because it typically requires security clearance. To address this, Mitchell would like to see DoD contracts provide for independent auditors with the necessary clearance, who can reassure companies that the guidelines really are being followed. “Employees need some guarantee that guidelines are being interpreted as they expect,” she says.
Though most executives recognize the importance of breaking down silos to help people collaborate across boundaries, they struggle to make it happen. That’s understandable: It is devilishly difficult. Think about your own relationships at work—the people you report to and those who report to you, for starters. Now consider the people in other functions, units, or geographies whose work touches yours in some way. Which relationships get prioritized in your day-to-day job?
We’ve posed that question to managers, engineers, salespeople, and consultants in companies around the world. The response we get is almost always the same: vertical relationships.
But when we ask, “Which relationships are most important for creating value for customers?” the answers flip. Today the vast majority of innovation and business-development opportunities lie in the interfaces between functions, offices, or organizations. In short, the integrated solutions that most customers want—but companies wrestle with developing—require horizontal collaboration.
The value of horizontal teamwork is widely recognized. Employees who can reach outside their silos to find colleagues with complementary expertise learn more, sell more, and gain skillsfaster. Harvard’s Heidi Gardner has found that firms with more cross-boundary collaboration achieve greater customer loyalty and higher margins. As innovation hinges more and more on interdisciplinary cooperation, digitalization transforms business at a breakneck pace, and globalization increasingly requires people to work across national borders, the demand for executives who can lead projects at interfaces keeps rising.
Our research and consulting work with hundreds of executives and managers in dozens of organizations confirms both the need for and the challenge of horizontal collaboration. “There’s no doubt. We should focus on big projects that call for integration across practices,” a partner in a global accounting firm told us. “That’s where our greatest distinctive value is developed. But most of us confine ourselves to the smaller projects that we can handle within our practice areas. It’s frustrating.” A senior partner in a leading consulting firm put it slightly differently: “You know you should swim farther to catch a bigger fish, but it is a lot easier to swim in your own pond and catch a bunch of small fish.”
One way to break down silos is to redesign the formal organizational structure. But that approach has limits: It’s costly, confusing, and slow. Worse, every new structure solves some problems but creates others. That’s why we’ve focused on identifying activities that facilitate boundary crossing. We’ve found that people can be trained to see and connect with pools of expertise throughout their organizations and to work better with colleagues who think very differently from them. The core challenges of operating effectively at interfaces are simple: learning about people on the other side and relating to them. But simple does not mean easy; human beings have always struggled to understand and relate to those who are different.
Leaders need to help people develop the capacity to overcome these challenges on both individual and organizational levels. That means providing training in and support for four practices that enable effective interface work.
1. Develop and Deploy Cultural Brokers
Fortunately, in most companies there are people who already excel at interface collaboration. They usually have experiences and relationships that span multiple sectors, functions, or domains and informally serve as links between them. We call these people cultural brokers. In studies involving more than 2,000 global teams, one of us—Sujin—found that diverse teams containing a cultural broker significantly outperformed diverse teams without one. (See “The Most Creative Teams Have a Specific Type of Cultural Diversity,”HBR.org, July 24, 2018.) Companies should identify these individuals and help them increase their impact.
Cultural brokers promote cross-boundary work in one of two ways: by acting as a bridge or as an adhesive.
A bridge offers himself as a go-between, allowing people in different functions or geographies to collaborate with minimal disruption to their day-to-day routine. Bridges are most effective when they have considerable knowledge of both sides and can figure out what each one needs. This is why the champagne and spirits distributor Moët Hennessy España hired two enologists, or wine experts, to help coordinate the work of its marketing and sales groups, which had a history of miscommunication and conflict. The enologists could relate to both groups equally: They could speak to marketers about the emotional content (the ephemeral “bouquet”) of brands, while also providing pragmatic salespeople with details on the distinctive features of products they needed to win over retailers. Understanding both worlds, the enologists were able to communicate the rationale for each group’s modus operandi to the other, allowing marketing and sales to work more synergistically even without directly interacting. This kind of cultural brokerage is efficient because it lets disparate parties work around differences without investing in learning the other side’s perspective or changing how they work. It’s especially valuable for one-off collaborations or when the company is under intense time pressure to deliver results.
Employees who can reach outside their silos learn more and sell more.
Adhesives, in contrast, bring people together and help build mutual understanding and lasting relationships. Take one manager we spoke with at National Instruments, a global producer of automated test equipment. He frequently connects colleagues from different regions and functions. “I think of it as building up the relationships between them,” he told us. “If a colleague needs to work with someone in another office or function, I would tell them, ‘OK, here’s the person to call.’ Then I’d take the time to sit down and say, ‘Well, let me tell you a little bit about how these guys work.’” Adhesives facilitate collaboration by vouching for people and helping them decipher one another’s language. Unlike bridges, adhesives develop others’ capacity to work across a boundary in the future without their assistance.
Company leaders can build both bridging and adhesive capabilities in their organizations by hiring people with multifunctional or multicultural backgrounds who have the strong interpersonal skills needed to build rapport with multiple parties. Because it takes resilience to work with people across cultural divides, firms should also look for a growth mindset—the desire to learn and to take on challenges and “stretch” opportunities.
In addition, leaders can develop more brokers by giving people at all levels the chance to move into roles that expose them to multiple parts of the company. This, by the way, is good training for general managers and is what many rotational leadership-development programs aim to accomplish. Claudine Wolfe, the head of talent and development at the global insurer Chubb, maintains that the company’s capacity to serve customers around the world rests on giving top performers opportunities to work in different geographies and cultivate an international mindset. “We give people their critical development experiences steeped in the job, in the region,” she says. “They get coaching in the cultural norms and the language, but then they live it and internalize it. They go to the local bodega, take notice of the products on the shelves, have conversations with the merchant, and learn what it really means to live in that environment.”
Matrix organizational structures, in which people report to two (or more) groups, can also help develop cultural brokers. Despite their inherent challenges (they can be infuriatingly hard to navigate without strong leadership and accountability), matrices get people used to operating at interfaces.
We’re not saying that everyone in your organization needs to be a full-fledged cultural broker. But consciously expanding the ranks of brokers and deploying them to grease the wheels of collaboration can go a long way.
2. Encourage People to Ask the Right Questions
It’s nearly impossible to work across boundaries without asking a lot of questions. Inquiry is critical because what we see and take for granted on one side of an interface is not the same as what people experience on the other side.
Indeed, a study of more than 1,000 middle managers at a large bank that Tiziana conducted with Bill McEvily and Evelyn Zhang of the University of Toronto and Francesca Gino of Harvard Business School highlights the value of inquisitiveness in boundary-crossing work. It showed that managers with high levels of curiosity were more likely to build networks that spanned disconnected parts of the company.
But all of us are vulnerable to forgetting the crucial practice of asking questions as we move up the ladder. High-achieving people in particular frequently fail to wonder what others are seeing. Worse, when we do recognize that we don’t know something, we may avoid asking a question out of (misguided) fear that it will make us look incompetent or weak. “Not asking questions is a big mistake many professionals make,” Norma Kraay, the managing partner of talent for Deloitte Canada, told us. “Expert advisers want to offer a solution. That’s what they’re trained to do.”
Leaders can encourage inquiry in two important ways—and in the process help create an organization where it’s psychologically safe to ask questions.
Be a role model.
When leaders show interest in what others are seeing and thinking by asking questions, it has a stunning effect: It prompts people in their organizations to do the same.
Asking questions also conveys the humility that more and more business leaders and researchers are pointing to as vital to success. According to Laszlo Bock,Google’s former senior vice president of people operations, humble people are better at bringing others together to solve tough problems. In a fast-changing business environment, humility—not to be confused with false modesty—is simply a strength. Its power comes from realism (as in It really is a complex, challenging world out there; if we don’t work together, we don’t stand a chance).
Gino says one way a leader can make employees feel comfortable asking questions is by openly acknowledging when he or she doesn’t know the answer. Another, she says, is by having days in which employees are explicitly encouraged to ask “Why?” “What if…?” and “How might we…?” (See “The Business Case for Curiosity,” HBR, September–October 2018.)
Teach employees the art of inquiry.
Training can help expand the range and frequency of questions employees ask and, according to Hal Gregersen of the MIT Leadership Center, can reinvigorate their sense of curiosity. But some questions are better than others. And if you simply tell people to raise more questions, you might unleash interrogation tactics that inhibit rather than encourage the development of new perspectives. As MIT’s Edgar Schein explains in his book Humble Inquiry,questions are the secret to productive work relationships—but they must be driven by genuine interest in understanding another’s view.
It’s also important to learn how to request information in the least biased way possible. This means asking open-ended questions that minimize preconceptions, rather than yes-or-no questions. For instance, “What do you see as the key opportunity in this space?” will generate a richer dialogue than “Do you think this is the right opportunity to pursue?”
As collaborations move forward, it’s helpful for team leaders or project managers to raise queries that encourage others to dive more deeply into specific issues and express related ideas or experiences. “What do you know about x?” and “Can you explain how that works?” are two examples. These questions are focused but neither limit responses nor invite long discourses that stray too far from the issue at hand.
How you process the answers also matters. It’s natural, as conversations unfold, to assume you understand what’s being said. But what people hear is biased by their expertise and experiences. So it’s important to train people to check whether they’re truly getting their colleagues’ meaning, by using language like “This is what I’m hearing—did I miss anything?” or “Can you help me fill in the gaps?” or “I think what you said means the project is on track. Is that correct?”
Finally, periodic temperature taking is needed to examine the collaborative process itself. The only way to find out how others are experiencing a project or relationship is by asking questions such as “How do you think the project is going?” and “What could we do to work together more effectively?”
3. Get People to See the World Through Others’ Eyes
Leaders shouldn’t just encourage employees to be curious about different groups and ask questions about their thinking and practices; they should also urge their people to actively consider others’ points of view. People from different organizational groups don’t see things the same way. Studies (including research on barriers to successful product innovation that the management professor Deborah Dougherty conducted at Wharton) consistently reveal that this leads to misunderstandings in interface work. It’s vital, therefore, to help people learn how to take the perspectives of others. One of us, Amy, has done research showingthat ambitious cross-industry innovation projects succeed when diverse participants discover how to do this. New Songdo, a project to build a city from scratch in South Korea that launched a decade ago, provides an instructive example. Early in the effort, project leaders brought together architects, engineers, planners, and environmental experts and helped them integrate their expertise in a carefully crafted learning process designed to break down barriers between disciplines. Today, in striking contrast to other “smart” city projects, New Songdo is 50% complete and has 30,000 residents, 33,000 jobs, and emissions that are 70% lower than those of other developments its size.
In a study of jazz bands and Broadway productions, Brian Uzzi of Northwestern University foundthat leaders of successful teams had an unusual ability to assume other people’s viewpoints. These leaders could speak the multiple “languages” of their teammates. Other research has shown that when members of a diverse team proactively take the perspectives of others, it enhances the positive effect of information sharing and increases the team’s creativity.
Creating a culture that fosters this kind of behavior is a senior leadership responsibility. Psychological research suggests that while most people are capable of taking others’ perspectives, they are rarely motivated to do so. Leaders can provide some motivation by emphasizing to their teams how much the integration of diverse expertise enhances new value creation. But a couple of other tactics will help:
Organize cross-silo dialogues.
Instead of holding one-way information sessions, leaders should set up cross-silo discussions that help employees see the world through the eyes of customers or colleagues in other parts of the company. The goal is to get everyone to share knowledge and work on synthesizing that diverse input into new solutions. This happens best in face-to-face meetings that are carefully structured to allow people time to listen to one another’s thinking. Sometimes the process includes customers; one consulting firm we know started to replace traditional meetings, at which the firm conveyed information to clients, with a workshop format designed to explore questions and develop solutions in collaboration with them. The new format gives both the clients and the consultants a chance to learn from each other.
One of the more thoughtful uses of cross-silo dialogue is the “focused event analysis” (FEA) at Children’s Minnesota. In an FEA people from the health system’s different clinical and operational groups come together after a failure, such as the administration of the wrong medication to a patient. One at a time participants offer their take on what happened; the goal is to carefully document multiple perspectives beforetrying to identify a cause. Often participants are surprised to learn how people from other groups saw the incident. The assumption underlying the FEA is that most failures have not one root cause but many. Once the folks involved have a multifunctional picture of the contributing factors, they can alter procedures and systems to prevent similar failures.
Hire for curiosity and empathy.
You can boost your company’s capacity to see the world from different perspectives by bringing on board people who relate to and sympathize with the feelings, thoughts, and attitudes of others. Southwest Airlines, which hires fewer than 2% of all applicants,selects people with empathy and enthusiasm for customer service, evaluating them through behavioral interviews (“Tell me about a time when…”) and team interviews in which candidates are observed interacting.
4. Broaden Your Employees’ Vision
You can’t lead at the interfaces if you don’t know where they are. Yet many organizations unwittingly encourage employees to never look beyond their own immediate environment, such as their function or business unit, and as a result miss out on potential insights employees could get if they scanned more-distant networks. Here are some ways that leaders can create opportunities for employees to widen their horizons, both within the company and beyond it:
Bring employees from diverse groups together on initiatives.
As a rule, cross-functional teams give people across silos a chance to identify various kinds of expertise within their organization, map how they’re connected or disconnected, and see how the internal knowledge network can be linked to enable valuable collaboration.
At one global consulting firm, the leader of the digital health-care practice used to have its consultants speak just to clients’ CIOs and CTOs. But she realized that that “unnecessarily limited the practice’s ability to identify opportunities to serve clients beyond IT,” she says. So she began to set up sessions with the entire C-suite at clients and brought in consultants from across all her firm’s health-care practices—including systems redesign, operations excellence, strategy, and financing—to provide a more integrated look at the firm’s health-care innovation expertise.
Those meetings allowed the consultants to discover the connections among the practices in the health-care division, identify the people best positioned to bridge the different practices, and see novel ways to combine the firm’s various kinds of expertise to meet clients’ needs. That helped the consultants spot value-generating opportunities for services at the interfaces between the practices. The new approach was so effective that, in short order, the leader was asked to head up a new practice that served as an interface across all the practices in the IT division so that she could replicate her success in other parts of the firm.
Urge employees to explore distant networks.
Employees also need to be pushed to tap into expertise outside the company and even outside the industry. The domains of human knowledge span science, technology, business, geography, politics, history, the arts, the humanities, and beyond, and any interface between them could hold new business opportunities. Consider the work of the innovation consultancy IDEO. By bringing design techniques from technology, science, and the arts to business, it has been able to create revolutionary products, like the first Apple mouse (which it developed from a Xerox PARC prototype into a commercial offering), and help companies in many industries embrace design thinking as an innovation strategy.
The tricky part is finding the domains most relevant to key business goals. Although many innovations have stemmed from what Abraham Flexner, the founding director of the Institute for Advanced Study, called “the usefulness of useless knowledge,”businesses can ill afford to rely on open-ended exploratory search alone. To avoid this fate, leaders can take one of two approaches:
A top-down approach works when the knowledge domains with high potential for value creation have already been identified. For example, a partner in an accounting firm who sees machine learning as key to the profession’s future might have an interested consultant or analyst in her practice take online courses or attend industry conferences about the technology and ask that person to come back with ideas about its implications. The partner might organize workshops in which the junior employee shares takeaways from the learning experiences and brainstorms, with experienced colleagues, potential applications in the firm.
You can’t lead at the interfaces if you don’t know where they are.
A bottom-up approach is better when leaders have trouble determining which outside domains the organization should connect with—a growing challenge given the speed at which new knowledge is being created. Increasingly, leaders must rely on employees to identify and forge connections with far-flung domains. One approach is to crowdsource ideas for promising interfaces—for example, by inviting employees to propose conferences in other industries they’d like to attend, courses on new skill sets they’d like to take, or domain experts they’d like to bring in for workshops. It’s also critical to give employees the time and resources to scan external domains and build connections to them.
Breaking Down Silos
In today’s economy everyone knows that finding new ways to combine an organization’s diverse knowledge is a winning strategy for creating lasting value. But it doesn’t happen unless employees have the opportunities and tools to work together productively across silos. To unleash the potential of horizontal collaboration, leaders must equip people to learn and to relate to one another across cultural and logistical divides. The four practices we’ve just described can help.
Not only is each one useful on its own in tackling the distinct challenges of interface work, but together these practices are mutually enhancing: Engaging in one promotes competency in another. Deploying cultural brokers who build connections across groups gets people to ask questions and learn what employees in other groups are thinking. When people start asking better questions, they’re immediately better positioned to understand others’ perspectives and challenges. Seeing things from someone else’s perspective—walking in his or her moccasins—in turn makes it easier to detect more pockets of knowledge. And network scanning illuminates interfaces where cultural brokers might be able to help groups collaborate effectively.
Over time these practices—none of which require advanced degrees or deep technical smarts—dissolve the barriers that make boundary-crossing work so difficult. When leaders create conditions that encourage and support these practices, collaboration across the interface will ultimately become second nature.
How to Ask Good Questions
COMMON PITFALLSEFFECTIVE INQUIRYStart with yes-or-no questions.Start with open-ended questions that minimize preconceptions. (“How are things going on your end?” “What does your group see as the key opportunity in this space?”)Continue asking overly general questions (“What’s on your mind?”) that may invite long off-point responses.As collaborations develop, ask questions that focus on specific issues but allow people plenty of room to elaborate. (“What do you know about x?” “Can you explain how that works?”)Assume that you’ve grasped what speakers intended.Check your understanding by summarizing what you’re hearing and asking explicitly for corrections or missing elements. (“Does that sound right—am I missing anything?” “Can you help me fill in the gaps?”)Assume the collaboration process will take care of itself.Periodically take time to inquire into others’ experiences of the process or relationship. (“How do you think the project is going?” “What could we do to work together more effectively?”)
Sujin Jang is an assistant professor of organisational behaviour at INSEAD. Her research focuses on global teams and the challenges of working across cultures.
The U.S. Food and Drug Administration (FDA), Health Canada, and the United Kingdom’s Medicines and Healthcare products Regulatory Agency (MHRA) have jointly identified 10 guiding principles that can inform the development of Good Machine Learning Practice (GMLP). These guiding principles will help promote safe, effective, and high-quality medical devices that use artificial intelligence and machine learning (AI/ML).
Artificial intelligence and machine learning technologies have the potential to transform health care by deriving new and important insights from the vast amount of data generated during the delivery of health care every day. They use software algorithms to learn from real-world use and in some situations may use this information to improve the product’s performance. But they also present unique considerations due to their complexity and the iterative and data-driven nature of their development.
These 10 guiding principles are intended to lay the foundation for developing Good Machine Learning Practice that addresses the unique nature of these products. They will also help cultivate future growth in this rapidly progressing field.
The 10 guiding principles identify areas where the International Medical Device Regulators Forum (IMDRF), international standards organizations, and other collaborative bodies could work to advance GMLP. Areas of collaboration include research, creating educational tools and resources, international harmonization, and consensus standards, which may help inform regulatory policies and regulatory guidelines.
We envision these guiding principles may be used to:
Adopt good practices that have been proven in other sectors
Tailor practices from other sectors so they are applicable to medical technology and the health care sector
Create new practices specific for medical technology and the health care sector
As the AI/ML medical device field evolves, so too must GMLP best practice and consensus standards. Strong partnerships with our international public health partners will be crucial if we are to empower stakeholders to advance responsible innovations in this area. Thus, we expect this initial collaborative work can inform our broader international engagements, including with the IMDRF.
We welcome your continued feedback through the public docket (FDA-2019-N-1185) at Regulations.gov, and we look forward to engaging with you on these efforts. The Digital Health Center of Excellence is spearheading this work for the FDA. Contact us directly at Digitalhealth@fda.hhs.gov, software@mhra.gov.uk, and mddpolicy-politiquesdim@hc-sc.gc.ca.
Guiding Principles
Multi-Disciplinary Expertise Is Leveraged Throughout the Total Product Life Cycle: In-depth understanding of a model’s intended integration into clinical workflow, and the desired benefits and associated patient risks, can help ensure that ML-enabled medical devices are safe and effective and address clinically meaningful needs over the lifecycle of the device.
Good Software Engineering and Security Practices Are Implemented: Model design is implemented with attention to the “fundamentals”: good software engineering practices, data quality assurance, data management, and robust cybersecurity practices. These practices include methodical risk management and design process that can appropriately capture and communicate design, implementation, and risk management decisions and rationale, as well as ensure data authenticity and integrity.
Clinical Study Participants and Data Sets Are Representative of the Intended Patient Population: Data collection protocols should ensure that the relevant characteristics of the intended patient population (for example, in terms of age, gender, sex, race, and ethnicity), use, and measurement inputs are sufficiently represented in a sample of adequate size in the clinical study and training and test datasets, so that results can be reasonably generalized to the population of interest. This is important to manage any bias, promote appropriate and generalizable performance across the intended patient population, assess usability, and identify circumstances where the model may underperform.
Training Data Sets Are Independent of Test Sets:Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition, and site factors, are considered and addressed to assure independence.
Selected Reference Datasets Are Based Upon Best Available Methods:Accepted, best available methods for developing a reference dataset (that is, a reference standard) ensure that clinically relevant and well characterized data are collected and the limitations of the reference are understood. If available, accepted reference datasets in model development and testing that promote and demonstrate model robustness and generalizability across the intended patient population are used.
Model Design Is Tailored to the Available Data and Reflects the Intended Use of the Device: Model design is suited to the available data and supports the active mitigation of known risks, like overfitting, performance degradation, and security risks. The clinical benefits and risks related to the product are well understood, used to derive clinically meaningful performance goals for testing, and support that the product can safely and effectively achieve its intended use. Considerations include the impact of both global and local performance and uncertainty/variability in the device inputs, outputs, intended patient populations, and clinical use conditions.
Focus Is Placed on the Performance of the Human-AI Team: Where the model has a “human in the loop,” human factors considerations and the human interpretability of the model outputs are addressed with emphasis on the performance of the Human-AI team, rather than just the performance of the model in isolation.
Testing Demonstrates Device Performance during Clinically Relevant Conditions: Statistically sound test plans are developed and executed to generate clinically relevant device performance information independently of the training data set. Considerations include the intended patient population, important subgroups, clinical environment and use by the Human-AI team, measurement inputs, and potential confounding factors.
Users Are Provided Clear, Essential Information:Users are provided ready access to clear, contextually relevant information that is appropriate for the intended audience (such as health care providers or patients) including: the product’s intended use and indications for use, performance of the model for appropriate subgroups, characteristics of the data used to train and test the model, acceptable inputs, known limitations, user interface interpretation, and clinical workflow integration of the model. Users are also made aware of device modifications and updates from real-world performance monitoring, the basis for decision-making when available, and a means to communicate product concerns to the developer.
Deployed Models Are Monitored for Performance and Re-training Risks are Managed: Deployed models have the capability to be monitored in “real world” use with a focus on maintained or improved safety and performance. Additionally, when models are periodically or continually trained after deployment, there are appropriate controls in place to manage risks of overfitting, unintended bias, or degradation of the model (for example, dataset drift) that may impact the safety and performance of the model as it is used by the Human-AI team.
The principles identify areas in which international standards organizations and other collaborative groups could advance what the agency calls Good Machine Learning Practice.
The U.S. Food and Drug Administration released a list of “guiding principles” this week aimed at helping promote the safe and effective development of medical devices that use artificial intelligence and machine learning.
The FDA, along with its U.K. and Canadian counterparts, said the principles are intended to lay the foundation for Good Machine Learning Practice.
“As the AI/ML medical device field evolves, so too must GMLP best practice and consensus standards,” said the agency regarding the principles.
WHY IT MATTERS
As the FDA notes, AI and ML technologies have the potential to radically expand the healthcare industry – but their complexity also presents unique considerations.
The 10 guiding principles identify points at which international standards organizations and other collaborative bodies, including the International Medical Device Regulators Forum, could work to advance GMLP.
The agency says stakeholders can use the principles to tailor and adopt good practices from other sectors to be used in the health tech sector, as well as to create new specific practices.
The principles are:
The total product life cycle uses multidisciplinary expertise.
The model design is implemented with good software engineering and security practices.
Participants and data sets represent the intended patient population.
Training data sets are independent of test sets.
Selected reference data sets are based upon best available methods.
Model design is tailored to the available data and reflects intended device use.
Focus is placed on the performance of the human-AI team.
Testing demonstrates device performance during clinically relevant conditions.
Users are provided clear, essential information.
Deployed models are monitored for performance, and retraining risks are managed.
“Areas of [international] collaboration include research, creating educational tools and resources, international harmonization, and consensus standards, which may help inform regulatory policies and regulatory guidelines,” said FDA officials.
At a virtual meeting of the agency’s Center for Devices and Radiological Health and Patient Engagement Advisory Committee last October, Bakul Patel, director of the Digital Health Center of Excellence, emphasized the importance of balancing innovation with patient protection.
“We all know there are some constraints, because of location or the amount of information available, about the cleanliness of the data,” said Patel at the time. “That might drive inherent bias. We don’t want to set up a system where we figure out, after the product is out in the market, that it is missing a certain type of population, or demographic, or other aspect that we have accidentally not realized.”
ON THE RECORD
“Strong partnerships with our international public health partners will be crucial if we are to empower stakeholders to advance responsible innovations in this area,” said FDA officials as they unveiled the new principles. “Thus, we expect this initial collaborative work can inform our broader international engagements, including with the IMDRF.”
Computation is the driving technological force behind many modern social and economic shifts. To better understand how computers will impact critical areas of our lives, we explore how they’re being used today and what forces will shape their future.
By Chuck Buck Original story posted on: October 25, 2021
The Senate Appropriations Committee’s vote is seen as a victory for AHIMA and other leading healthcare organizations.
There was a hint of expectation when Katherine Lusk, chair and president of the American Health Information Management Association (AHIMA), recently teased Talk-Ten-Tuesdays audience members regarding a pending piece of legislation for a national patient identification program.
The Patient ID Now coalition, of which AHIMA is a founding member, continues to make an impact on the problem of patient misidentification. “We’re proud of the Framework for a National Strategy on Patient Identity that Patient ID Now released earlier this year,” Lusk said.
Later that same day, on Oct. 19, the U.S. Senate’s Appropriations Committee removed a longstanding prohibition to fund a national patient identifier program. AHIMA, along with a coalition of other healthcare organizations, had rallied around the cause.
According to the bill’s text, released by AHIMA, it “drops prohibition on using funding to develop a unique patient health identifier for each individual’s health information. The longstanding ban has been a barrier for health institutions to reliably share information about patients, and during the COVID-19 pandemic, for health entities to effectively trace contacts and track immunizations.”
“By joining the bipartisan movement to remove barriers to accurate patient identification, the Senate has taken a firm step towards protecting patient safety, patient privacy, and supporting efforts to address patient identification issues exacerbated by the COVID-19 pandemic,” AHIMA said in a news release posted on its website. “Patient ID Now has highlighted challenges caused by patient misidentification in the healthcare sector’s response to the pandemic. These challenges include thousands of duplicate records created during the vaccination registration process and disruptions in vaccine availability at provider sites because of inaccurate patient documentation.”
Lusk said AHIMA continues to promote the importance of protecting and securing health information. She told Talk Ten Tuesdays audience members that the Association had recently launched its own AHIMA dHealth™, a seal-of-approval program that helps digital health companies demonstrate how they protect patient health information.
healthcarereimagined 