healthcarereimagined

Task Force Lima continues to gain momentum across a variety of pursuits in its ambitious, 18-month plan to ensure the Pentagon can responsibly adopt, implement and secure powerful, still-maturing generative artificial intelligence technologies.

Department of Defense leadership formed that new hub in August ​​within the Chief Digital and AI Office’s (CDAO) Algorithmic Warfare Directorate. Its ultimate mission is to set and steer the enterprise’s path forward with the emerging field of generative AI and associated large language models, which yield (convincing but not always correct) software code, images and other media following human prompts. 

Such capabilities hold a lot of promise, but also complex challenges for the DOD — including many that remain unseen. 

“Task Force Lima has three phases: the ‘learn phase,’ an ‘accelerate phase’ and a ‘guide phase.’ The ‘learn phase’ is where we are performing, for lack of a better word, inventories of what is the demand signal for generative AI across the department. That includes projects that are ongoing, to projects that we think should go forward, to projects that we would like to learn more about. And so, we submitted that as an inquiry to the department — and we’ve received a volume of use cases around 180 that go into many different categories and into many different mission areas,” Task Force Lima Mission Commander Navy Capt. M. Xavier Lugo told DefenseScoop.

In a recent interview, the 28-year Naval officer-turned AI acceleration lead, briefed DefenseScoop about what’s to come with those under-review use cases, a recent “Challenge Day,” and future opportunities and events the task force is planning.

180-plus instances

During his first interview with DefenseScoop back in late September, Lugo confirmed that the task force would be placing an explicit emphasis on enabling generative AI in “low-risk mission areas.”

“That is still the case. However, some of what has evolved from that is they’re not all theoretical. For some of these use cases, there are units that have already started working with those particular technologies and they’re integrating [them] into their workflows. That’s when we’re going to switch from the ‘learn phase’ into the ‘accelerate phase,’ which is where we will partner with the use cases that are ongoing,” Lugo told DefenseScoop in the most recent interview.

At a Pentagon press briefing about the state of AI last week, Deputy Defense Secretary Kathleen Hicks confirmed that the department launched Task Force Lima because it is “mindful of the potential risks and benefits offered by large language models” (LLMs) and other associated generative AI tools.

“Candidly, most commercially available systems enabled by large language models aren’t yet technically mature enough to comply with our DOD ethical AI principles — which is required for responsible operational use. But we have found over 180 instances where such generative AI tools could add value for us with oversight like helping to debug and develop software faster, speeding analysis of battle damage assessments, and verifiably summarizing texts from both open-source and classified datasets,” Hicks told reporters.

The deputy secretary noted that “not all of these use cases” that the task force is exploring are notional.

Some Defense Department components started looking at generative AI even before ChatGPT and similar products “captured the world’s attention,” she said. And a few department insiders have “even made their own models,” by isolating and fine-tuning foundational models for a specific task with clean, reliable and secure DOD data.

“While we have much more evaluating to do, it’s possible some might make fewer factual errors than publicly available tools — in part because, with effort, they can be designed to cite their sources clearly and proactively. Although it would be premature to call most of them operational, it’s true that some are actively being experimented with and even used as part of people’s regular workflows — of course, with appropriate human supervision and judgment — not just to validate, but also to continue improving them,” Hicks said. 

Lugo offered an example of those more non-theoretical generative AI use cases that have already been maturing within DOD.

“As you can imagine, the military has a lot of policies and publications, [tactics, techniques, and procedures, or TTPs], and all sorts of documentation out there for particular areas — let’s say in the human resources area, for example. So, one of those projects would be how do I interact with all those publications and policies that are out there to answer questions that a particular person may have on how to do a procedure or a policy?” he told DefenseScoop.

Among its many responsibilities, one that the CDAO leadership has charged Task Force Lima with is coming up with acceptability criteria and a maturity model for each use case or groups of use cases encompassing generative AI. 

“So, if we say we need an acceptability criteria of a particular value for a capability of summarization for LLMs, let’s say just as an example, then we need a model that matches that and that has that type of maturity in that particular capability. This is analogous to the self-driving vehicle maturity models and how you can have a different level of maturity in a self-driving vehicle for different road conditions. So, in our case the road conditions will be our acceptability criteria, and the model being able to meet that acceptability will be that maturity model,” Lugo explained.

‘Put me in, coach!’

Soon, the Lima team will start collecting information needed to inform its specific deliverables, including new test-and-evaluation frameworks, mitigation techniques, risk assessments and more.

“That output that we get during the ‘accelerate phase’ will be the input for the ‘guide phase,’ which is our last phase where we compile the deliverables to the CDAO Council so they can then make a determination into policy,” Lugo explained.

The task force does not have authority to officially publish guidance on generative AI deployments in DOD, but members previously made recommendations to the CDAO’s leadership that were approved to advise defense components in their efforts. The task force drafted that interim LLM guidance, but due to its classification level it has not been disseminated widely.

“That guidance [included that] any service can publish its own guidance that is more restrictive than the one that [the Office of the Secretary of Defense] publishes,” Lugo said. 

The Navy offered its version of interim guardrails on generative AI and LLMs in September. Shortly after that, the Space Force transmitted a memo that put a temporary pause on guardians’ use of web-based generative AI tools like ChatGPT for its workforce — specifically citing data security concerns.

“Did I learn about the Space Force guidance before it went out? Yes. Would I have had any reason to try to modify that? No,” Lugo told DefenseScoop.

“Space Force — like any other service — has the right to pursue guidance that is even more restrictive than the guidance that is provided by the policy. So, I just want to be clear that they have autonomy to publish their own guidance. At Task Force Lima, we are coordinating with the services — and they understand our viewpoints, and we understand our viewpoints, and there is no conflict on viewpoints here,” he added.

And although it might make sense for one military branch to ban certain uses on a non-permanent basis to address data and security concerns, Lugo noted that doesn’t mean the task force should not be cautiously experimenting with models that are publicly accessible, in order to learn more about them.

In his latest interview with DefenseScoop, the task force chief also stated that his team is “not trying to do this in a vacuum.”

“We are definitely not only working with DOD, but we are working with industry and academia — and actually any organization that is interested in generative AI, they can reach out to us. There’s plenty of work, and there’s plenty of areas of involvement,” Lugo said.

“Also, I want to make sure that just because we are interacting with industry, that doesn’t take us out of the industry-agnostic, technology-agnostic hat. I am always ensuring that we keep that, because that’s what keeps us as an honest broker of this technology,” he added.

Lugo’s currently leading a core team of roughly 15 personnel. But he’s also engaging with a still-growing “expanded team” of close to 500 points of contacts associated with the task force’s activities and aims. To him, those officials are essentially on secondary duty, or a support function to his unit.

“We’re getting more people interested. Now, those 500 people — I’ve got everything from people watching from the bleachers, to personnel saying, ‘Hey, put me in coach!’ So, I’ve got a broad spectrum,” Lugo said.

Nearly 250 people attended a recent “Challenge Day” that the CDAO hosted to connect with industry and academic partners about the challenges associated with implementing generative AI within the DOD.

“There’s a lot of interest in the area, but there’s not that many companies in it. So what we saw was that it’s not just the normal names that you would hear on a day to day basis — but there’s also a lot of companies interested in integrating models. There’s companies that are not necessarily known for LLMs or generative AI, but they are known for other types of integration in the data space and in the AI space. So that was good, because that means that there’s a good pool of talent that will be working on the challenges that we have submitted to industry,” Lugo said.

According to Lugo, the cadre has received more than 120 responses to the recent request for informationreleased to the public to garner input on existing generative AI use cases and critical technical obstacles that accompany its emergence.

The RFI is about learning “what are the insights out there, what are the approaches to solving these particular challenges that we have. And as we compile that information, we will then go ahead and do a more formal solicitation through the proper processes,” he said.

On Nov. 30, industry and academic partners will have an additional opportunity this year to meet with Task Force Lima at the CDAO Industry Day. And down the line during the CDAO’s first in-person symposium — which is set to take place Feb. 20-22 in Washington — an entire track will be dedicated to Task Force Lima and generative AI.

Attendee registration opened in October, and the office is now accepting submissions for potential speakers at that event.

“I’m very optimistic that the challenges that we have submitted will be addressed — and hopefully corrected — by some innovative techniques,” Lugo told DefenseScoop.

Article link: https://www.linkedin.com/posts/defensescoop_inside-task-force-limas-exploration-of-180-activity-7127423756221718528-byMQ?

Foreign or malicious actors can get what they need to target military personnel and their families for blackmail, disinfo, and more, Duke researchers find.

ALEXANDRA KELLEY | 

NOVEMBER 7, 2023 05:00 AM ET

A new report found that it is easy to buy personal data on U.S. servicemembers online, where it can cost as little as one cent to obtain records through data brokers.

Conducted by researchers at Duke University, the study Data Brokers and the Sale of Data on U.S. Military Personnel examined the availability of sensitive data for U.S. military personnel, including names, home addresses, emails and specific branch information being sold on third-party data-broker platforms.

After scraping or buying data from hundreds of data broker sites, researchers explored how freely available military service information could pose national security threats.

Researchers found only minimal identity verification protocols when buying potentially sensitive data online. 

“We found a lack of robust controls when asking some data brokers about buying data on the U.S. military and when actually purchasing data from some data brokers, such as identity verification, background checks or detective controls to ascertain our intended uses for the purchased data,” the report reads.

Researchers were able to purchase demographic characteristics including religious practices, health information and financial data of thousands of both active-duty members and veterans. Some datasets available for sale were so specific that they could list the office of service, such as the U.S. Marine Corps and Pentagon Force Protection Agency. 

While the U.S. has grappled with the absence of federal law governing data broker practices, the report places unregulated data sales online into a national security context. 

“The inconsistencies of controls when purchasing sensitive, non-public, individually identified data about active-duty members of the military and veterans extends to situations in which data brokers are selling to customers who are outside of the United States,” the report reads. 

The report concluded with policy recommendations focused on passing legislation that would bring regulatory guardrails to online user data privacy. 

Despite multiple efforts, Congress has still not managed to pass a national data privacy law. Lawmakers chimed in following the release of the report, with Sen. Ron Wyden, D-Ore., reiterating the call for comprehensive privacy legislation.

“The researchers findings should be a sobering wake-up call for policy makers that the data broker industry is out of control and poses a serious threat to U.S. national security,” Wyden said in an emailed statement to Nextgov/FCW. “As I have been warning for years, consumer privacy is a national security issue. The United States needs a comprehensive solution to protect Americans’ data from unfriendly nations rather than focusing on ineffective Band-Aids like banning TikTok.”

On the House side, Rep. Frank Pallone, D-N.J., echoed Wyden’s national security concerns.

“These findings are yet another terrible example of the harms posed by the data broker industry and underscore the need to pass comprehensive national privacy legislation and regulate data brokers,” Pallone said in a statement. “Congress needs to pass legislation that minimizes the amount of information that can be collected on Americans, cracks down on the abuses of data brokers and provides consumers with the tools they need to protect their information.”

Article link: https://www.nextgov.com/cybersecurity/2023/11/data-active-duty-servicemembers-available-purchase-online-report-says/391811/

• Rachel Longhurst
• Woojin Choi

November 06, 2023

Summary.

Traditional project management skills, such as project governance or project management methodology, aren’t sufficient to meet changing organizational needs. Gartner recently surveyed 373 project management leaders to identify the “next generation” skills — from organizational awareness to financial acumen — that have a disproportionate impact on performance. They also identified three future-focused project manager roles: the teacher, the fixer, and the orchestrator — all of which highlight the uniquely human aspects of project management that go beyond performing discrete, repetitive tasks.

The future of the project manager role has been hotly debated as a number of trends shift organizational dynamics:

• Gartner research has found that businesses are increasingly adopting agile development and product management models, with 44% of work now delivered through agile methods and 39% through product models.
• Traditional project management activities, such as validating requirements, maintaining scope and measuring benefits, have become the domain of autonomous delivery teams (i.e., multidisciplinary teams accountable for business outcomes) such as scrum and fusion teams.
• Moreover, recent technological advancements — most notably, in generative AI — mean many project manager tasks, such as resource utilization tracking and business case creation, can be successfully automated.

Two popular agile reference manuals — the Scrum Guide and SAFe Reference Guide — omit the project manager role altogether.

Yet, a recent global Gartner survey suggests that project manager is actually expected to be one of the fastest-growing project management office (PMO) roles across the next two to three years. If project managers aren’t going anywhere, how can they continue to provide value in a changing context?

10 Next-Generation Skills Project Managers Need

Project managers have a decisive role to play in this new environment, where autonomous delivery teams must address complex challenges, such as overcoming organizational silos, managing hidden interdependencies, and realizing cross-team efficiencies. While AI offers promising benefits for project management, machines will never be able to replicate the uniquely human aspects of the job, such as relationship building and stakeholder management.

However, traditional project manager skills, such as project governance and project management methodology, won’t be sufficient to deliver on these changing organizational needs. Gartner predicts that by 2026, two-thirds of project managers’ skills and roles will be redesigned to meet the needs of their new operating context.

To understand the skills today’s project managers need to succeed, we surveyed 373 project management leaders and identified 10 “next-generation” skills that have a disproportionate impact on performance. Project managers proficient in these skills were found to be 1.4 times more effective at achieving key business and functional outcomes. What’s more, our research found these skills were far more impactful than organizational tenure or formal project management certifications. The next-generation skills include:

• Organizational awareness
• Data acumen
• Cross-functional collaboration
• Decision making
• Willingness to explore and adopt new technology
• Financial acumen
• Process and framework expertise (i.e., business process improvement, agile, organizational change management, risk)
• Customer centricity
• Growth mindset
• The ability to coach and motivate their teams

Organizational awareness, cross-functional collaboration, and customer centricity are especially important in the context of agile and product-centric delivery, where the nature of work involves bridging organizational silos and delivering enterprise-wide value. For example, creating and maintaining a mobile app requires coordination across many different business units and internal functions like finance and supply chain in order to provide a seamless customer experience. Project managers are at the intersection of a growing number of customers and stakeholders. As such, they must operate with a wider purview and more strategically to maximize business value.

Moreover, data acumen and digital adoption are two skills that we expect will only become more critical over time. We’ve arrived at a juncture where the rate at which organizations are amassing data and the rate of technological advancements are outpacing the average employee’s capacity to leverage them.

Three Future-Focused Project Manager Roles

Gartner has identified three critical roles project managers can play to meet organizations’ future needs: teacher, fixer, and orchestrator. At times, project managers may need to play a mix of roles at the same time to meet the needs of different parts of the organization.

The Teacher: For organizations early in their digital journey.

A teacher project manager helps bolster the competencies distributed delivery teams need to succeed. They’re particularly skilled in coaching and motivating individuals and teams; project management processes and frameworks; and adopting new technologies. They also have strong communication skills that enable them to effectively coach stakeholders on a wide variety of complex concepts, for example, regulatory and compliance activities.

This style of project manager is particularly valuable for organizations with product owners who are relatively immature in their role. Organizations without a strict 1-to-1 ratio of product owners to delivery teams can also benefit from teacher project managers who can more easily move across multiple teams to ensure they adopt an uncompromising focus on shared accountability and enterprise outcomes, rather than arguing over local ownership or outputs.

The Fixer: For organizations seeking to boost their delivery efficiency.

Organizations that need support identifying, resolving, and mitigating challenges in workflows and complex portfolios can benefit from the fixer project manager role.

Fixer project managers are adept at cross-functional collaboration, decision-making, and financial acumen. The fixer can creatively address complex problems, identify and manage risks at both a project and a cross-portfolio level, and operate in complex portfolios.

Organizations that are undergoing digital business transformation or shifting operating models can reap benefits from the fixer project manager. So, too, can organizations with a significant amount of cross-silo dependency who see delivery execution suffering from a lack of engagement and alignment with business partners.

The Orchestrator: For organizations that need improved cross-functional coordination. 

Enterprise digital transformation initiatives need project managers who can manage high levels of complexity and support delivery teams in a resource-constrained environment. Orchestrator project managers are true stewards of the organization’s resources and insights, ensuring that delivery of work is aligned and correctly prioritized.

Orchestrator project managers are expertly skilled at data acumen, customer centricity and organizational awareness. They collate diverse information and insights, align it with strategic imperatives, and translate into executable actions. The orchestrator also has strong organizational awareness and cross-functional stakeholder management skills, with specific experience operating in significantly complex portfolios, especially those with dispersed stakeholders.

The organizations that most benefit from the orchestrator focus on customer-facing initiatives. Organizations that are challenged by capacity management or resourcing decisions across silos can also gain significant advantages.

Developing project managers to exhibit a different set of skills and play new roles won’t happen overnight. To set project managers up for success in today’s environment, organizations need to invest in training and development initiatives that focus on the 10 next-generation skills.

What’s notable about the next-generation skills is not that they’re new in and of themselves; rather, it’s that they collectively highlight the uniquely human aspects of project management that go beyond performing discrete, repetitive project management tasks.

Yes, project managers still need to know how to create business cases and generate reports. However, it’s their sound judgment in decision making — whether that involves people, data, or technology — and their ability to make an impact beyond themselves through their coaching and relationship building that truly differentiates the next-generation project manager.

Article link: https://hbr.org/2023/11/what-the-next-generation-of-project-management-will-look-like?

• Rachel Longhurst is a director within the Gartner IT Leaders and Tech Professionals research practice advising clients on strategic portfolio management, including project and portfolio management and application portfolio management.
• WCWoojin Choi is a senior principal within the Gartner IT Leaders and Tech Professionals research practice advising clients on strategic portfolio management.

The new strategy was produced by the Chief Digital and AI Office (CDAO) and unveiled by Deputy Defense Secretary Kathleen Hicks on Thursday.

BYBRANDI VINCENT NOVEMBER 2, 2023

Pentagon leaders released a new, long-awaited strategic plan that revamps and redefines how U.S. military and defense components adopt and deploy crucial data, analytics and AI capabilities — particularly as they prepare for higher-tech conflicts down the line.

The 2023 Department of Defense Data, Analytics, and AI Adoption Strategy was produced by the Chief Digital and AI Office (CDAO) and unveiled by Deputy Defense Secretary Kathleen Hicks at a Pentagon press briefing on Thursday.

“Increasingly over the last dozen years, advances in machine learning have heralded and accelerated new generations of AI breakthroughs — with much of the innovation happening outside of the DOD and government. And so our task in DOD is to adopt these innovations wherever they can add the most military value,” she told reporters.

Senior CDAO officials, including its chief Craig Martell, have been teasingthis newly launched strategy — which was slated for publication by the end of the summer — repeatedly in recent months.

It follows two foundational guiding documents meant to drive AI use across the DOD. The first enterprise-wide AI strategy was launched in 2018, and Pentagon officials disseminated the first revised version in 2021.

This new 26-page AI adoption strategy “not only builds on DOD’s prior AI and data strategies — but also includes updates to account for recent industry advances in federated environments, decentralized data management, generative AI and more,” Hicks said.  

She confirmed the department’s hope that this updated guidance will help all components and services speedily realize critical unfolding AI-aligned efforts, including those related to Joint All-Domain Command and Control (JADC2) and the new Replicator initiative, which aims to counter China by fielding thousands of autonomous systems in multiple domains within the next 2 years.

The strategy includes sections spotlighting new key outcomes the department is eyeing to enable with AI, the associated goals officials will pursue and the high-priority areas where they plan to fully embrace these rapidly evolving technologies. Specific goals listed involve removing policy barriers, investing in interoperable and federated infrastructure, improving data management and growing AI talent, among others. 

“The state of AI in DOD is not a short story, nor is it static. We must keep doing more — safely and swiftly — given the nature of strategic competition with the [People’s Republic of China], our pacing challenge. At the same time, we benefit from a national position of strength. And our own uses grow stronger every day, and we will be keeping up the momentum ensuring we make the best possible use of AI technology responsibly and at speed,” Hicks said.

During a call with reporters following Hicks’ briefing, Martell also echoed this intention of Pentagon leadership to keep a flexible and fluid approach when it comes to unleashing AI and machine learning for the military in the coming years. 

“Technologies evolve. Things are going to change next week, next year, next decade — and what wins today might not win tomorrow. Rather than identify a handful of AI-enabled warfighting capabilities that will beat our adversaries, our strategy outlines the approach to strengthening the organizational environment within which people can continuously deploy data analytics capabilities for enduring decision advantage,” Martell explained.

He and other officials on the call told reporters that the CDAO team is crafting new “iterative” implementation guidance that will accompany the strategy. They expect to release it in the next few months. 

That plan will look a bit different from traditional implementation guides, Martell said, because “each of the services have wildly different needs, they’re at wildly different points in their journey and they have wildly different infrastructure.” 

Notably, this refreshed strategy also comes as DOD leadership is heeding warnings from top technology and national security experts about the dire need for the U.S. military to be “AI-ready” by 2025.

During the Pentagon press briefing, Hicks told DefenseScoop that the 2023 AI adoption strategy marks “a key piece of how we get there.”

Another major contributing element to DOD becoming AI-ready to match its adversaries in that time frame, in her view, is on-time congressional appropriations and predictable resourcing to advance ongoing work.

“Absent any predictability in our funding streams, it’s very hard for us to be able to project [DOD’s AI readiness] with accuracy,” Hicks told DefenseScoop.

Article link: https://defensescoop.com/2023/11/02/pentagon-redefines-its-overarching-plan-to-accelerate-data-and-ai-adoption/?

Researchers from Tsinghua University, China, have developed an all-analog photoelectronic chip that combines optical and electronic computing to achieve ultrafast and highly energy-efficient computer vision processing, surpassing digital processors.

Computer vision is an ever-evolving field of artificial intelligence focused on enabling machines to interpret and understand visual information from the world, similar to how humans perceive and process images and videos.

It involves tasks such as image recognition, object detection, and scene understanding. This is done by converting analog signals from the environment into digital signals for processing by neural networks, enabling machines to make sense of visual information. However, this analog-to-digital conversion consumes significant time and energy, limiting the speed and efficiency of practical neural network implementations.

The proposed all-analog photoelectronic chip, as detailed in the research, addresses this limitation by combining photonic and electronic computing in a single chip, offering a groundbreaking solution for high-speed and energy-efficient visual data processing. The findings of the study are published in Nature, along with a Research Briefing summarizing the work.

Dr. Jiamin Wu, one of the authors of the study, explained to Phys.org why they focused on the hardware side of things, saying, “Our team, motivated by enhancing the real-world impact of AI advancements, has long been dedicated to developing efficient hardware solutions for AI execution.”

Best of both worlds

Combining optical and electronic analog computing modules in the study is a pivotal aspect as it allows the researchers to harness the benefits of both light (in the form of photons) and electrons in an all-analog manner.

In doing so, the researchers have addressed the practical limitations of photonic (light-based) computing, such as the complicated implementation of optical nonlinearity, considerable power consumption of ADCs, and vulnerability to noises and system errors.

“An optical computing module that implements a diffractive neural network is first used to extract information and reduce data dimensionality in a highly parallel way,” explained Dr. Wu. This process is highly efficient and allows information to be extracted from high-resolution light fields.

“The output of the optical computing module is then received by a photodiode array to generate light-induced photocurrents. These are directly used for further computation in the electronic analog domain,” he continued. This seamless conversion allows for the creation of intricate network structures, enhancing overall task performance.

The module further analyzes the light-generated electric currents. Notably, it doesn’t require converting analog signals into digital ones. This flexibility in electronic circuits enables adaptive and reconfigurable training methods, which are essential for real-world performance improvements.

The researchers were able to successfully design an integrated photoelectronic processor called an “all-analog chip combining electronic and light computing,” or ACCEL.

“By utilizing the intrinsic nonlinearity of photoelectric effect and data processing in the analog electronic field without analog-to-digital conversion, the proposed all-analog photoelectronic chip achieves energy efficiency and computing speed that are several orders of magnitude higher than those of state-of-the-art digital processor,” said Dr. Wu.

Putting it to the test

The researchers conducted a series of tests to test the ACCEL’s classification accuracies in various tasks, including recognizing handwritten numbers, distinguishing clothing items, and interpreting cursive writing.

It displayed the ability to classify high-resolution images within 72 nanoseconds, a feat that defies the limits of conventional processing. Astonishingly, the ACCEL consumes 4 million times less energy than a top-of-the-line GPU, even though it is more than 3,000 times faster.

But the ACCEL chip doesn’t stop there. Its adaptability extends to incoherent light sources, making it a versatile solution with applications beyond the expected.

“Compared to high-performance GPUs, our all-analog photoelectronic chip is three orders of magnitude faster and six orders of magnitude more energy-efficient. This makes it suitable for high-speed processing in applications like industrial assembly lines and autonomous driving.”

“Moreover, thanks to its exceptional computing efficiency and minimal energy demands, our chip could bring a new era for portable systems such as wearable devices for health monitoring, where the system is traditionally powered by a battery and the life-span of the device has been severely constrained due to the limited energy source,” said Dr. Wu.

Future work

The researchers acknowledge that while the all-analog photoelectronic demonstrated high power and efficiency, there’s still room for improvement.

“Though the ACCEL achieved fast computing speed and high energy efficiency, there is still room for the improvement of the processing capability of this chip,” explained Dr. Wu.

In the future, the researchers hope to explore more efficient architectures with photoelectronic computing to handle more extensive computer vision tasks and extend this technology to new artificial intelligence algorithms like large language models (LLMs).

This ongoing research aims to push the boundaries of analog photonic technology for future advancements.

Article link: https://techxplore-com.cdn.ampproject.org/c/s/techxplore.com/news/2023-10-future-ai-hardware-scientists-unveil.amp

By ROSS WILKERSOCTOBER 30, 2023 02:38 PM ET

The General Services Administration is putting the security and stability of government supply chains front and center in this planned procurement.

The General Services Administration has given industry its first draft of a potential 10-year, $919.7 million blanket purchase agreement for acquiring supply chain risk illumination software tools and related analytic support services.

GSA envisions the Supply Chain Risk Illumination Professional Tools and Services, or SCRIPTS, BPA as helping the agency bolster its ability to mitigate the risks of fraud, abuse and other adversaries’ actions against supply chains the U.S. government relies on.

Nov. 14 is the due date for questions and comments on the draft request for quotes. An industry day is scheduled for Dec. 12, and registration closes on Nov. 8, GSA said in a Friday notice to Sam.gov.

The security and stability of supply chains has been front-and-center over the past three years for essentially all sectors including public sector, given the well-documented shortages of people and parts needed to make products and systems.

While the situation has shown more improvement over recent months, GSA’s unveiling of this SCRIPTS BPA is an indication that the government wants to better position itself for responding to and helping mitigate future disruptions.

GSA designed the BPA as being available to both defense and civilian agencies seeking new illumination tools to evaluate vendors, build up cyber hygiene and gain visibility into foreign investments that may present risks to supply chains.

SCRIPTS will be a multiple-award contract with an initial five-year base period followed by a single option for five additional years. A portion of the BPA will be set aside for small businesses.

GSA will compete the BPA through its Multiple Award Schedules program for acquiring commercially oriented products and services.

The agency also encourages parties that are interested in participating in the SCRIPTS BPA to submit an e-offer via the FASt Lane process no later than Dec. 31.

Article link: https://washingtontechnology.com/contracts/2023/10/gsa-previews-its-plan-919m-supply-chain-monitoring-software-buy/391614/

Two studies find “self-supervised” models, which learn about their environment from unlabeled data, can show activity patterns similar to those of the mammalian brain.

Anne Trafton | MIT News

Publication Date: October 30, 2023

To make our way through the world, our brain must develop an intuitive understanding of the physical world around us, which we then use to interpret sensory information coming into the brain.

How does the brain develop that intuitive understanding? Many scientists believe that it may use a process similar to what’s known as “self-supervised learning.” This type of machine learning, originally developed as a way to create more efficient models for computer vision, allows computational models to learn about visual scenes based solely on the similarities and differences between them, with no labels or other information.

A pair of studies from researchers at the K. Lisa Yang Integrative Computational Neuroscience (ICoN) Center at MIT offers new evidence supporting this hypothesis. The researchers found that when they trained models known as neural networks using a particular type of self-supervised learning, the resulting models generated activity patterns very similar to those seen in the brains of animals that were performing the same tasks as the models.

The findings suggest that these models are able to learn representations of the physical world that they can use to make accurate predictions about what will happen in that world, and that the mammalian brain may be using the same strategy, the researchers say.

“The theme of our work is that AI designed to help build better robots ends up also being a framework to better understand the brain more generally,” says Aran Nayebi, a postdoc in the ICoN Center. “We can’t say if it’s the whole brain yet, but across scales and disparate brain areas, our results seem to be suggestive of an organizing principle.”

Nayebi is the lead author of one of the studies, co-authored with Rishi Rajalingham, a former MIT postdoc now at Meta Reality Labs, and senior authors Mehrdad Jazayeri, an associate professor of brain and cognitive sciences and a member of the McGovern Institute for Brain Research; and Robert Yang, an assistant professor of brain and cognitive sciences and an associate member of the McGovern Institute. Ila Fiete, director of the ICoN Center, a professor of brain and cognitive sciences, and an associate member of the McGovern Institute, is the senior author of the other study, which was co-led by Mikail Khona, an MIT graduate student, and Rylan Schaeffer, a former senior research associate at MIT.

Both studies will be presented at the 2023 Conference on Neural Information Processing Systems (NeurIPS) in December.

Modeling the physical world

Early models of computer vision mainly relied on supervised learning. Using this approach, models are trained to classify images that are each labeled with a name — cat, car, etc. The resulting models work well, but this type of training requires a great deal of human-labeled data.

To create a more efficient alternative, in recent years researchers have turned to models built through a technique known as contrastive self-supervised learning. This type of learning allows an algorithm to learn to classify objects based on how similar they are to each other, with no external labels provided.

“This is a very powerful method because you can now leverage very large modern data sets, especially videos, and really unlock their potential,” Nayebi says. “A lot of the modern AI that you see now, especially in the last couple years with ChatGPT and GPT-4, is a result of training a self-supervised objective function on a large-scale dataset to obtain a very flexible representation.”

These types of models, also called neural networks, consist of thousands or millions of processing units connected to each other. Each node has connections of varying strengths to other nodes in the network. As the network analyzes huge amounts of data, the strengths of those connections change as the network learns to perform the desired task.

As the model performs a particular task, the activity patterns of different units within the network can be measured. Each unit’s activity can be represented as a firing pattern, similar to the firing patterns of neurons in the brain. Previous work from Nayebi and others has shown that self-supervised models of vision generate activity similar to that seen in the visual processing system of mammalian brains.

In both of the new NeurIPS studies, the researchers set out to explore whether self-supervised computational models of other cognitive functions might also show similarities to the mammalian brain. In the study led by Nayebi, the researchers trained self-supervised models to predict the future state of their environment across hundreds of thousands of naturalistic videos depicting everyday scenarios.    

“For the last decade or so, the dominant method to build neural network models in cognitive neuroscience is to train these networks on individual cognitive tasks. But models trained this way rarely generalize to other tasks,” Yang says. “Here we test whether we can build models for some aspect of cognition by first training on naturalistic data using self-supervised learning, then evaluating in lab settings.”

Once the model was trained, the researchers had it generalize to a task they call “Mental-Pong.” This is similar to the video game Pong, where a player moves a paddle to hit a ball traveling across the screen. In the Mental-Pong version, the ball disappears shortly before hitting the paddle, so the player has to estimate its trajectory in order to hit the ball.

The researchers found that the model was able to track the hidden ball’s trajectory with accuracy similar to that of neurons in the mammalian brain, which had been shown in a previous study by Rajalingham and Jazayeri to simulate its trajectory — a cognitive phenomenon known as “mental simulation.” Furthermore, the neural activation patterns seen within the model were similar to those seen in the brains of animals as they played the game — specifically, in a part of the brain called the dorsomedial frontal cortex. No other class of computational model has been able to match the biological data as closely as this one, the researchers say.

“There are many efforts in the machine learning community to create artificial intelligence,” Jazayeri says. “The relevance of these models to neurobiology hinges on their ability to additionally capture the inner workings of the brain. The fact that Aran’s model predicts neural data is really important as it suggests that we may be getting closer to building artificial systems that emulate natural intelligence.”

Navigating the world

The study led by Khona, Schaeffer, and Fiete focused on a type of specialized neurons known as grid cells. These cells, located in the entorhinal cortex, help animals to navigate, working together with place cells located in the hippocampus.

While place cells fire whenever an animal is in a specific location, grid cells fire only when the animal is at one of the vertices of a triangular lattice. Groups of grid cells create overlapping lattices of different sizes, which allows them to encode a large number of positions using a relatively small number of cells.

In recent studies, researchers have trained supervised neural networks to mimic grid cell function by predicting an animal’s next location based on its starting point and velocity, a task known as path integration. However, these models hinged on access to privileged information about absolute space at all times — information that the animal does not have.                               

Inspired by the striking coding properties of the multiperiodic grid-cell code for space, the MIT team trained a contrastive self-supervised model to both perform this same path integration task and represent space efficiently while doing so. For the training data, they used sequences of velocity inputs. The model learned to distinguish positions based on whether they were similar or different — nearby positions generated similar codes, but further positions generated more different codes.    

“It’s similar to training models on images, where if two images are both heads of cats, their codes should be similar, but if one is the head of a cat and one is a truck, then you want their codes to repel,” Khona says. “We’re taking that same idea but applying it to spatial trajectories.”

Once the model was trained, the researchers found that the activation patterns of the nodes within the model formed several lattice patterns with different periods, very similar to those formed by grid cells in the brain.

“What excites me about this work is that it makes connections between mathematical work on the striking information-theoretic properties of the grid cell code and the computation of path integration,” Fiete says. “While the mathematical work was analytic — what properties does the grid cell code possess? — the approach of optimizing coding efficiency through self-supervised learning and obtaining grid-like tuning is synthetic: It shows what properties might be necessary and sufficient to explain why the brain has grid cells.”

The research was funded by the K. Lisa Yang ICoN Center, the National Institutes of Health, the Simons Foundation, the McKnight Foundation, the McGovern Institute, and the Helen Hay Whitney Foundation.

Article link: https://news.mit.edu/2023/brain-self-supervised-computational-models-1030

The directive is part of an executive order on the “Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence.”

BYJON HARPER

OCTOBER 30, 2023

President Biden on Monday signed an executive order that, among other things, would require the Department of Defense to conduct a pilot aimed at finding ways to use AI to protect national security networks.

The EO on the “Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence,” gives the Pentagon 180 days to conduct the pilot.

The secretary of defense and the secretary of homeland security “shall, consistent with applicable law, each develop plans for, conduct, and complete an operational pilot project to identify, develop, test, evaluate, and deploy AI capabilities, such as large-language models, to aid in the discovery and remediation of vulnerabilities in critical United States Government software, systems, and networks,” the directive states.

Within the next 270 days, the two secretaries must each deliver a report to the White House on the results of actions taken “pursuant to the plans and operational pilot projects … including a description of any vulnerabilities found and fixed through the development and deployment of AI capabilities and any lessons learned on how to identify, develop, test, evaluate, and deploy AI capabilities effectively for cyber defense,” according to the EO.

The Defense Department has already set up a group known as Task Force Lima to look at potential use cases for generative AI tools such as large language models. More broadly, the Pentagon’s Chief Digital and AI Office is focused on helping deploy artificial intelligence capabilities across the department. A number of other DOD agencies and offices are also involved in artificial intelligence and cybersecurity efforts, and it wasn’t immediately clear which DOD organizations will execute the pilot that Biden directed.

Biden highlighted cybersecurity concerns during remarks at the White House on Monday before he signed the order.

“In the wrong hands, AI can make it easier for hackers to exploit vulnerabilities in the software that makes our society run. That’s why I’m directing the Department of Defense and Department of Homeland Security — both of them — to develop game-changing cyber protections that will make our computers and our critical infrastructure more secure than it is today,” he said.

Additionally, in the next six months the Pentagon chief must also deliver a report about ways to fix “gaps in AI talent” related to national defense, including recommendations for addressing challenges in the DOD’s ability to hire certain noncitizens; streamlining processes for certain noncitizens to access classified information through “Limited Access Authorization” at department labs; the appropriate use of enlistment authority under 10 U.S.C. 504(b)(2) for experts in artificial intelligence and other critical and emerging technologies; and ways that DOD and DHS can “enhance the use of appropriate authorities for the retention of certain noncitizens of vital importance to national security,” according to the directive.

Meanwhile, White House officials have been tasked with overseeing the development of a national security memorandum that will provide guidance to the Pentagon, intelligence community and other relevant agencies on the continued adoption of AI capabilities for national security missions, including as it relates to AI assurance and risk-management practices for use cases that “may affect the rights or safety of United States persons and, in appropriate contexts, non-United States persons,” per the EO.

Notably, the memo will also include directives to address the potential adversarial use of AI systems “in ways that threaten the capabilities or objectives of the Department of Defense or the Intelligence Community, or that otherwise pose risks to the security of the United States or its allies and partners.”

That document is expected to be delivered to the president within the next 270 days.

Article link: https://defensescoop.com/2023/10/30/biden-tasks-pentagon-to-carry-out-new-ai-pilot-for-cyber-defense/?

Experts say its emphasis on content labeling, watermarking, and transparency represents important steps forward.

By Tate Ryan-Mosley &Melissa Heikkilä

October 30, 2023

MIT Technology Review Explains: Let our writers untangle the complex, messy world of technology to help you understand what’s coming next. You can read more from the series here.

The US has set out its most sweeping set of AI rules and guidelines yet in an executive orderissued by President Joe Biden today. The order will require more transparency from AI companies about how their models work and will establish a raft of new standards, most notably for labeling AI-generated content.

The goal of the order, according to the White House, is to improve “AI safety and security.” It also includes a requirement that developers share safety test results for new AI models with the US government if the tests show that the technology could pose a risk to national security. This is a surprising move that invokes the Defense Production Act, typically used during times of national emergency.

The executive order advances the voluntary requirements for AI policy that the White House set back in August, though it lacks specifics on how the rules will be enforced. Executive orders are also vulnerable to being overturned at any time by a future president, and they lack the legitimacy of congressional legislation on AI, which looks unlikely in the short term.  

“The Congress is deeply polarized and even dysfunctional to the extent that it is very unlikely to produce any meaningful AI legislation in the near future,” says Anu Bradford, a law professor at Columbia University who specializes in digital regulation.

Nevertheless, AI experts have hailed the order as an important step forward, especially thanks to its focus on watermarking and standards set by the National Institute of Standards and Technology (NIST). However, others argue that it does not go far enough to protect people against immediate harms inflicted by AI.

Here are the three most important things you need to know about the executive order and the impact it could have. 

What are the new rules around labeling AI-generated content? 

The White House’s executive order requires the Department of Commerce to develop guidance for labeling AI-generated content. AI companies will use this guidance to develop labeling and watermarking tools that the White House hopes federal agencies will adopt. “Federal agencies will use these tools to make it easy for Americans to know that the communications they receive from their government are authentic—and set an example for the private sector and governments around the world,” according to a fact sheet that the White House shared over the weekend. 

The hope is that labeling the origins of text, audio, and visual content will make it easier for us to know what’s been created using AI online. These sorts of tools are widely proposed as a solution to AI-enabled problems such as deepfakes and disinformation, and in a voluntary pledge with the White House announced in August, leading AI companies such as Google and Open AI pledged to develop such technologies.

The trouble is that technologies such as watermarks are still very much works in progress. There currently are no fully reliable ways to label text or investigate whether a piece of content was machine generated. AI detection tools are still easy to fool. 

The executive order also falls short of requiring industry players or government agencies to use these technologies.

On a call with reporters on Sunday, a White House spokesperson responded to a question from MIT Technology Review about whether any requirements are anticipated for the future, saying, “I can imagine, honestly, a version of a call like this in some number of years from now and there’ll be a cryptographic signature attached to it that you know you’re actually speaking to [the White House press team] and not an AI version.” This executive order intends to “facilitate technological development that needs to take place before we can get to that point.”

The White House says it plans to push forward the development and use of these technologies with the Coalition for Content Provenance and Authenticity, called the C2PA initiative. As we’ve previously reported, the initiative and its affiliated open-source communityhas been growing rapidly in recent months as companies rush to label AI-generated content. The collective includes some major companies like Adobe, Intel, and Microsoft and has devised a new internet protocol that uses cryptographic techniques to encode information about the origins of a piece of content.

The coalition does not have a formal relationship with the White House, and it’s unclear what that collaboration would look like. In response to questions, Mounir Ibrahim, the cochair of the governmental affairs team, said, “C2PA has been in regular contact with various offices at the NSC [National Security Council] and White House for some time.”

The emphasis on developing watermarking is good, says Emily Bender, a professor of linguistics at the University of Washington. She says she also hopes content labeling systems can be developed for text; current watermarking technologies work best on images and audio. “[The executive order] of course wouldn’t be a requirement to watermark, but even an existence proof of reasonable systems for doing so would be an important step,” Bender says.

Will this executive order have teeth? Is it enforceable? 

While Biden’s executive order goes beyond previous US government attempts to regulate AI, it places far more emphasis on establishing best practices and standards than on how, or even whether, the new directives will be enforced.

The order calls on the National Institute of Standards and Technology to set standards for extensive “red team” testing—meaning tests meant to break the models in order to expose vulnerabilities—before models are launched. NIST has been somewhat effective at documenting how accurate or biased AI systems such as facial recognition are already. In 2019, a NIST study of over 200 facial recognition systems revealed widespread racial bias in the technology.

However, the executive order does not require that AI companies adhere to NIST standards or testing methods. “Many aspects of the EO still rely on voluntary cooperation by tech companies,” says Bradford, the law professor at Columbia.

The executive order requires all companies developing new AI models whose computational size exceeds a certain threshold to notify the federal government when training the system and then share the results of safety tests in accordance with the Defense Production Act. This law has traditionally been used to intervene in commercial production at times of war or national emergencies such as the covid-19 pandemic, so this is an unusual way to push through regulations. A White House spokesperson says this mandate will be enforceable and will apply to all future commercial AI models in the US, but will likely not apply to AI models that have already been launched. The threshold is set at a point where all major AI models that could pose risks “to national security, national economic security, or national public health and safety” are likely to fall under the order, according to the White House’s fact  sheet. 

The executive order also calls for federal agencies to develop rules and guidelines for different applications, such as supporting workers’ rights, protecting consumers, ensuring fair competition, and administering government services. These more specific guidelines prioritize privacy and bias protections.

“Throughout, at least, there is the empowering of other agencies, who may be able to address these issues seriously,” says Margaret Mitchell, researcher and chief ethics scientist at AI startup Hugging Face. “Albeit with a much harder and more exhausting battle for some of the people most negatively affected by AI, in order to actually have their rights taken seriously.”

What has the reaction to the order been so far? 

Major tech companies have largely welcomed the executive order. 

Brad Smith, the vice chair and president of Microsoft, hailed it as “another critical step forward in the governance of AI technology.” Google’s president of global affairs, Kent Walker, said the company looks “forward to engaging constructively with government agencies to maximize AI’s potential—including by making government services better, faster, and more secure.”

“It’s great to see the White House investing in AI’s growth by creating a framework for responsible AI practices,” said Adobe’s general counsel and chief trust officer, Dana Rao. 

The White House’s approach remains friendly to Silicon Valley, emphasizing innovation and competition rather than limitation and restriction. The strategy is in line with the policy priorities for AI regulation set forth by Senate Majority Leader Chuck Schumer, and it further crystallizes the lighter touch of the American approach to AI regulation. 

However, some AI researchers say that sort of approach is cause for concern. “The biggest concern to me in this is it ignores a lot of work on how to train and develop models to minimize foreseeable harms,” says Mitchell.

Instead of preventing AI harms before deployment—for example, by making tech companies’ data practices better—the White House is using a “whack-a-mole” approach, tackling problems that have already emerged, she adds.  

The highly anticipated executive order on artificial intelligence comes two days before the UK’s AI Safety Summit and attempts to position the US as a global leader on AI policy. 

It will likely have implications outside the US, adds Bradford. It will set the tone for the UK summit and will likely embolden the European Union to finalize its AI Act, as the executive order sends a clear message that the US agrees with many of the EU’s policy goals.

“The executive order is probably the best we can expect from the US government at this time,” says Bradford.

Article link: https://www-technologyreview-com.cdn.ampproject.org/c/s/www.technologyreview.com/2023/10/30/1082678/three-things-to-know-about-the-white-houses-executive-order-on-ai/amp/

Correction: A previous version of this story had Emily Bender’s title wrong. This has now been corrected. We apologize for any inconvenience.

A vector database is designed to store, manage and index massive quantities of high-dimensional vector data efficiently. These databases are rapidly growing in interest to create additional value for generative artificial intelligence (AI) use cases and applications. According to Gartner, by 2026, more than 30 percent of enterprises will have adopted vector databases to ground their foundation models with relevant business data.¹ 

Unlike traditional relational databases with rows and columns, data points in a vector database are represented by vectors with a fixed number of dimensions, clustered based on similarity. This design enables low latency queries, making them ideal for AI-driven applications.

Vector databases vs. traditional databases

The nature of data has undergone a profound transformation. It’s no longer confined to structured information easily stored in traditional databases. Unstructured data is growing 30 to 60 percent year over year, comprising social media posts, images, videos, audio clips and more.² Typically, if you wanted to load unstructured data sources into a traditional relational database to store, manage and prepare for AI, the process is labor-intensive and far from efficient, especially when it comes to new generative use cases such as similarity search. Relational databases are great for managing structured and semi-structured datasets in specific formats, while vector databases are best suited for unstructured datasets through high-dimensional vector embeddings.

What are vectors?

Enter vectors. Vectors are arrays of numbers that can represent complex objects like words, images, videos and audio, generated by a machine learning(ML) model. High-dimensional vector data is essential to machine learning, natural language processing (NLP) and other AI tasks. Some examples of vector data include: 

• Text: Think about the last time you interacted with a chatbot. How do they understand natural language? They rely on vectors which can represent words, paragraphs and entire documents, that are converted via machine learning algorithms. 
• Images: Image pixels can be described by numerical data and combined to make up a high-dimensional vector for that image. 
• Speech/Audio: Like images, sound waves can also be broken down into numerical data and represented as vectors, enabling AI applications such as voice recognition. 

What are vector embeddings?

The volume of unstructured datasets your organization needs for AI will only continue to grow, so how do you handle millions of vectors? This is where vector embeddings and vector databases come into play. These vectors are represented in a continuous, multi-dimensional space known as an embedding, which are generated by embedding models, specialized to convert your vector data into an embedding. Vector databases serve to store and index the output of an embedding model. Vector embeddings are a numerical representation of data, grouping sets of data based on semantic meaning or similar features across virtually any data type.  

For example, take the words “car” and “vehicle.” They both have similar meanings even though they are spelled differently. For an AI application to enable effective semantic search, vector representations of “car” and “vehicle” must capture their semantic similarity. When it comes to machine learning, embeddings represent high-dimensional vectors that encode this semantic information. These vector embeddings are the backbone of recommendations, chatbots and generative apps like ChatGPT.

Vector database vs graph database

Knowledge graphs represent a network of entities such as objects or events and depicts the relationship between them. A graph database is a fit-for-purpose database for storing knowledge graph information and visualizing it as a graph structure. Graph databases are built on nodes and edges that represent the known entities and complex relationships between them, while vector databases are built on high-dimensional vectors. As a result, graph databases are preferred for processing complex relationships between data points while vector databases are better for handling different forms of data such as images or videos.

How vector embeddings and vector databases work

Enterprise vector data can be fed into an embedding model such as IBM’s watsonx.ai models or Hugging Face (link resides outside ibm.com), which are specialized to convert your data into an embedding by transforming complex, high-dimensional vector data into numerical forms that computers can understand. These embeddings represent the attributes of your data used in AI tasks such as classification and anomaly detection.

Vector storage

Vector databases store the output of an embedding model algorithm, the vector embeddings. They also store each vector’s metadata, which can be queried using metadata filters. By ingesting and storing these embeddings, the database can then facilitate fast retrieval of a similarity search, matching the user’s prompt with a similar vector embedding.

Vector indexing

Storing data as embeddings isn’t enough. The vectors need to be indexed to accelerate the search process. Vector databases create indexes on vector embeddings for search functionality. The vector database indexes vectors using a machine learning algorithm. Indexing maps vectors to new data structures that enable faster similarity or distance searches, such as nearest neighbor search between vectors.

Similarity search based on querying or prompting

Querying vectors can be done via calculations measuring the distance between vectors using algorithms, such as nearest neighbor search. This measuring can be based on various similarity metrics such as cosine similarity, used by that index to measure how close or distant those vectors are. When a user queries or prompts an AI model, an embedding is computed using the same embedding model algorithm. The database calculates distances and performs similarity calculations between query vectors and vectors stored in the index. They return the most similar vectors or nearest neighbors according to the similarity ranking. These calculations support various machine learning tasks such as recommendation systems, semantic search, image recognition and other natural language processing tasks.

Vector databases and retrieval augmented generation (RAG)

Enterprises are increasingly favoring retrieval augmented generation (RAG)approach in generative AI workflows for its faster time-to-market, efficient inference and reliable output, particularly in key use cases such as customer care and HR/Talent. RAG ensures that the model is linked to the most current, reliable facts and that users have access to the model’s sources, so that its claims can be checked for accuracy. RAG is core to our ability to anchor large language models in trusted data to reduce model hallucinations. This approach relies on leveraging high-dimensional vector data to enrich prompts with semantically relevant information for in-context learning by foundation models. It requires effective storage and retrieval during the inference stage, which handles the highest volume of data. Vector databases excel at efficiently indexing, storing and retrieving these high-dimensional vectors, providing the speed, precision and scale needed for applications like recommendation engines and chatbots.

Enterprises are increasingly favoring retrieval augmented generation (RAG)approach in generative AI workflows for its faster time-to-market, efficient inference and reliable output, particularly in key use cases such as customer care and HR/Talent. RAG ensures that the model is linked to the most current, reliable facts and that users have access to the model’s sources, so that its claims can be checked for accuracy. RAG is core to our ability to anchor large language models in trusted data to reduce model hallucinations. This approach relies on leveraging high-dimensional vector data to enrich prompts with semantically relevant information for in-context learning by foundation models. It requires effective storage and retrieval during the inference stage, which handles the highest volume of data. Vector databases excel at efficiently indexing, storing and retrieving these high-dimensional vectors, providing the speed, precision and scale needed for applications like recommendation engines and chatbots.

Advantages of vector databases

While it’s clear that vector database functionality is rapidly growing in interest and adoption to enhance enterprise AI-based applications, the following benefits have also demonstrated business value for adopters: 

Speed and performance: Vector databases use various indexing techniques to enable faster searching. Vector indexing along with distance-calculating algorithms such as nearest neighbor search, are particularly helpful with searching for relevant results across millions if not billions of data points, with optimized performance. 

Scalability: Vector databases can store and manage massive amounts of unstructured data by scaling horizontally, maintaining performance as query demands and data volumes increase.

Cost of ownership: Vector databases are a valuable alternative to training foundation models from scratch or fine-tuning them. This reduces the cost and speed of inferencing of foundation models.

Flexibility: Whether you have images, videos or other multi-dimensional data, vector databases are built to handle complexity. Given the multiple use cases ranging from semantic search to conversational AI applications, the use of vector databases can be customized to meet your business and AI requirements. 

Long term memory of LLMs: Organizations can start with a general-purpose models like IBM watsonx.ai’s Granite series models, Meta’s Llama-2 or Google’s Flan models, and then provide their own data in a vector database to enhance the output of the models and AI applications critical to retrieval augmented generation. 

Data management components: Vector databases also typically provide built-in features to easily update and insert new unstructured data.

Considerations for vector databases and your data strategy

There is a breadth of options when it comes to choosing a vector database capability to meet your organization’s data and AI needs.

Types of vector databases

There are a few alternatives to choose from.

• Standalone, proprietary vector databases such as Pinecone
• Open-source solutions such as weaviate or milvus, which provide built-in RESTful APIs and support for Python and Java programming languages
• Platforms with vector database capabilities integrated, coming soon to IBM watsonx.data
• Vector database/search extensions such as PostgreSQL’s open source pgvector extension, providing vector similarity search capabilities

Integration with your data ecosystem

Vector databases should not be considered as standalone capabilities, but rather a part of your broader data and AI ecosystem. Many offer APIs, native extensions or can be integrated with your databases. Since they are built to leverage your own enterprise data to enhance your models, you must also have proper data governance and security in place to ensure the data with which you are grounding these LLMs can be trusted. 

This is where a trusted data foundation plays an important role in AI, and that starts with your data and how it’s stored, managed and governed before being used for AI. Central to this is adata lakehouse, one that is open, hybrid and governed, suchIBM watsonx.data, part of thewatsonxAI data platform that fits seamlessly into a data fabric architecture. For example, IBM watsonx.data, is built to access, catalog, govern and transform all of your structured, semi-structured and unstructured data and metadata. You can then leverage this governed data and watsonx.data’s integrated vector database capabilities (tech preview Q4, 2023) for machine learning and generative AI use cases.

When vector indexing is not optimal

Using a vector store and index is well suited for applications that are based on facts or fact-based querying. For example, asking about a company’s legal terms last year or extracting specific information from complex documents. The set of retrieval context you would get would be those that are most semantically similar to your query through embedding distance. However, if you want to get a summary of topics, this doesn’t lend itself well to a vector index. In this case you would want the LLM to go through all of the different possible contexts on that topic within your data. Instead, you may use a different kind of index, such as a list index rather than a vector index, since a vector index would only fetch the most relevant data.

Use Cases of Vector Databases

The applications of vector databases are vast and growing. Some key use cases include:

Semantic search: Perform searches based on the meaning or context of a query, enabling more precise and relevant results. As not only words but phrases can be represented as vectors, semantic vector search functionality understands user intent better than general keywords.

Similarity search and applications: Find similar images, text, audio or video data with ease, for content retrieval including advanced image and speech recognition, natural language processing and more.

Recommendation engines: E-commerce sites, for instance, can use vector databases and vectors to represent customer preferences and product attributes. This enables them to suggest items similar to past purchases based on vector similarity, enhancing user experience and increasing retention.Conversational AI: Improving virtual agent interactions by enhancing the ability to parse through relevant knowledge bases efficiently and accurately to provide real-time contextual answers to user queries, along with the source documents and page numbers for reference.

Vector database capabilities

watsonx.ai

A next generation enterprise studio for AI builders to build, train, validate, tune and deploy both traditional machine learning and new generative AI capabilities powered by foundation models. Build a Q&A resource from a broad internal or external knowledge base with the help of AI tasks in watsonx.ai, such as retrieval augmented generation.

Learn more

IBM Cloud® Databases for PostgreSQL-

Our PostgreSQL database-as-a-service offering lets teams spend more time building with high availability, backup orchestration, point-in-time-recovery (PITR) and read replica with ease. PostgreSQL offers pgvector, an open-source vector extension that will be able to be configured with IBM Cloud PostgreSQL extensions (coming soon), providing vector similarity search capabilities.

Learn more

IBM Cloud Databases for Elasticsearch

Our Elasticsearch database-as-a-service comes with a full-text search engine, which makes it the perfect home for your unstructured text data. Elasticsearch also support various forms of semantic (link resides outside ibm.com) similarity search. It supports dense vectors (link resides outside ibm.com) for exact nearest neighbor search, but it also provides built-in AI models to compute sparse vectors and conduct advanced similarity search (link resides outside ibm.com).

Learn more

Article link: https://www.ibm.com/topics/vector-database?