From Relational Databases to AI: An Insurance Data Modernization Journey

Jeff Needham, Luca Napoli, and Jack Yallop
March 14, 2024 | Updated: April 19, 2024
#genAI

Imagine you’re a data architect, a developer, or a data engineer at an insurance company. Management has asked you and your team to build a new AI claim adjustment system, a customer-facing LLM-powered chatbot, and an application to streamline the underwriting process.

However, doing so is far from straightforward due to the challenges you face on a daily basis. The bulk of your time is spent navigating your company’s outdated legacy systems, which were built in the 1970s and 1980s. Some of these legacy platforms were written in COBOL and CICS, and today very few people on your team know how to develop and maintain those technologies. Moreover, the data models you work with are another source of frustration. Every interaction with them is a reminder of the intricate structures that have evolved over time, making data manipulation and analysis a nightmare.

In sum, legacy systems are preventing your team—and your company—from innovating and keeping up with both your industry and customer demands.

Whether you’re trying to modernize your legacy systems to improve operational efficiency, or to boost developer productivity, or if you want to build AI-powered apps that integrate with large language models (LLMs), MongoDB has a solution for that. In this post, we’ll walk you through a journey that starts with a relational data model refactored into MongoDB collections, vectorization and querying of unstructured data and, finally, retrieval augmented generation (RAG): asking large language models (LLMs) questions about data in natural language.

Identifying, modernizing, and storing the data

Our journey starts with an assessment of the data sources we want to work with. As shown below, we can bucket the data into three different categories:

  • Structured legacy data: Tables of claims, coverages, billings, and more. Is your data locked in rigid relations schemas? This tutorial is a step-by-step guide on how to migrate a real-life insurance relational model with the help of MongoDB Relational Migrator, refactoring 21 tables to only five MongoDB collections.

  • Structured data (JSON): You might have files of policies, insurance products, or forms in JSON format. Check out our docs to learn how to insert those into a MongoDB collection.

  • Unstructured data (PDFs, Audios, Images, etc.): If you need to create and store a numerical representation (vector embedding) of, for instance, claim-related photos of accidents or PDFs of policy guidelines, you can have a look at this blog that will walk you through the process of generating embeddings of pictures of car crashes and persisting them alongside existing fields in a MongoDB collection.

Diagram of different types of data being stored in MongoDB. On the left, buckets of Structured Legacy Data, Structured Data, and Unstructured data are presented. Structured Legacy Data connects to the MongoDB Relational Migrator, before flowing into the Converged AI Data Store. Structured Data flows directly to the AI Data Store. And Unstructured Data connects to an Embedding Model, which then flows into the AI Data Store utilizing vectors.
Figure 1: Storing different types of data into MongoDB

Regardless of the original format or source, our data has finally landed into MongoDB Atlas into what we call a Converged AI Data Store, which is a platform that centrally integrates and organizes enterprise data, including vectors, that enable the development of ML- and AI-powered applications.

Accessing, experimenting and interacting with the data

It’s time to put the data to work. The Converged AI Data Store unlocks a plethora of use cases and efficiency gains, both for the business and for developers. The next step of the journey is about the different ways we can interact with our data:

  • Database and Full Text Search: Learn how to run database queries, start from the basics and move up to advanced features such as facets, fuzzy search, autocomplete, highlighting, and more with Atlas Search.

  • Vector Search: We can finally leverage unstructured data. The Image Search blog we mentioned earlier also explains how to create a Vector Search index and run vector queries against embeddings of photos.

  • RAG: Combining Vector Search and the power of LLMs, it is possible to interact in natural language with our data (see Figure 2 below), asking complex questions and getting detailed answers. Follow this tutorial to become a RAG expert.

Diagram of RAG where we combine custom data with the LLM. In step 1, the user prompts in natural language through the embedding model. Step 2, the vectorized prompt is used to retrieve context through the Converged AI Data Store. Step 3, the LLM is prompted with context and original question. Step 4, the user receives the answer.
Figure 2: Retrieval augmented generation (RAG) diagram where we dynamically combine our custom data with the LLM to generate reliable and relevant outputs

Having explored all the different ways we can ask questions of the data, we made it to the end of our journey. You are now ready to modernize your company’s systems and finally be able to keep up with the business’ demands.

What will you build next?

If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources:

← Previous

利用生成式人工智能和 MongoDB 应对网络安全的最大挑战

在不断变化的网络安全环境中，企业面临着众多挑战，需要利用尖端技术提供创新解决方案。 最紧迫的问题之一是网络威胁日益复杂，包括恶意软件、勒索软件和网络钓鱼攻击，这些攻击越来越难以检测和缓解。 此外，数字基础设施的快速扩张扩大了攻击面，使安全团队更难监控和保护每个入口和出口点。 另一个重大挑战是缺少熟练的网络安全专业人员（据独立调查估计，全球缺口约为 400 万1），这使得许多组织容易受到攻击。 这些挑战凸显了对先进技术的需求，这些技术可以增强人类保护数字资产和数据的努力。 生成式AI有何帮助？ 生成式人工智能 ( gen AI ) 已成为应对这些网络安全挑战的强大工具。 通过利用大型语言模型 ( LLM ) 在现有数据集的基础上生成新数据或模式，生成式人工智能可以在多个关键领域提供创新解决方案： 强化威胁检测和响应 生成式人工智能可用于模拟网络威胁，包括复杂的恶意软件和网络钓鱼攻击。 这些模拟有助于训练机器学习模型，以更准确地检测新的和不断演变的威胁。 此外，生成式人工智能可以帮助开发实时对威胁做出反应的自动响应系统。 虽然这永远不会消除对人工监督的需求，但可以减少人工干预和劳累，从而更快地缓解攻击。 例如，在适当的监督下，它可以自动为易受攻击的系统打补丁，或调整防火墙规则以阻止攻击载体。 这种自动快速反应能力对于减少零日漏洞尤为重要，因为从发现漏洞到攻击者利用漏洞之间的窗口很短。 从安全事件事后分析中汲取可操作的经验教训 在网络安全事件发生后，进行彻底的事后分析对于了解事件的经过、原因以及今后如何防止类似事件的发生至关重要。 在这一过程中，生成式人工智能可以综合和汇总多种来源的复杂数据（日志、网络流量和安全警报等），发挥关键作用。 通过分析这些数据，生成式人工智能可以识别可能导致安全漏洞的模式和异常，从而提供由于信息量和复杂性而可能被人类分析师忽视的见解。 此外，它还可以生成全面的报告，突出显示关键发现、诱发因素和潜在漏洞，从而简化事后分析过程。 这种能力不仅能加快恢复和学习过程，还能使组织实施更有效的补救策略，最终加强其网络安全态势。 生成用于深度模型训练的合成数据 用于培训网络安全系统的真实数据短缺，这是一个重大障碍。生成式人工智能可以创建真实的合成数据集，反映真实的网络流量和用户行为，而不会暴露敏感信息。 这种合成数据可用于训练检测系统，在不损害隐私或安全的情况下提高其准确性和有效性。 自动检测网络钓鱼 网络钓鱼仍然是最常见的攻击载体之一。 生成式人工智能可以分析网络钓鱼电子邮件和网站中的模式，生成能够高精度预测和检测网络钓鱼尝试的模型。 通过将这些模型集成到电子邮件系统和网络浏览器中，组织可以自动过滤掉网络钓鱼内容，保护用户免受潜在威胁。 综合考虑：机遇与风险 生成式人工智能有望实现复杂流程的自动化、加强威胁检测和响应、提供对网络威胁的更深入了解，从而改变网络安全实践。随着业界不断将生成式人工智能融入网络安全战略，我们必须对这项技术的道德使用和滥用潜力保持警惕。 尽管如此，它在加强数字防御方面所带来的好处是毋庸置疑的，因此成为应对网络威胁的持久战中的宝贵资产。 MongoDB 如何提供帮助？ 有了 MongoDB，您的开发团队就能以任何规模更快地构建和部署强大、正确和差异化的实时网络防御系统。 要了解 MongoDB 如何做到这一点，请考虑 AI 技术堆栈包含三层： 底层计算 (GPU) 和 LLM 微调模型的工具以及用于上下文学习和对训练模型进行推理的工具 人工智能应用程序和相关最终用户体验 MongoDB 在堆栈的第二层运行。 它使客户能够将自己的专有数据带到任何计算基础设施上运行的任何 LLM，以构建生成式人工智能驱动的网络安全应用程序。 为此，MongoDB 解决采用生成式人工智能保障网络安全时最棘手的问题。 MongoDB Atlas 将运营数据、非结构化数据和矢量数据安全地统一在一个完全托管的多云平台中，避免了在不同系统之间复制和同步数据的需要。 MongoDB 基于文档的架构还允许开发团队轻松地对应用程序数据和矢量嵌入之间的关系进行建模。 这样就可以更深入、更快速地分析和见解与安全相关的数据。 图 1： 在统一的 API 和开发者数据平台中，MongoDB Atlas 汇集了构建现代网络安全应用程序所需的所有数据服务。 MongoDB 的开放式架构与丰富的 AI 开发者框架、LLM 和嵌入式提供商的生态系统相集成。这与我们业界领先的多云功能相结合，使您的开发团队能够灵活快速地行动，避免在这个快速发展的领域中被任何特定的云提供商或 AI 技术限制。 请查看我们的 AI 资源页面，了解有关使用 MongoDB 构建 AI 驱动的应用的更多信息。 将生成式人工智能和 MongoDB 应用于现实世界的网络安全应用 威胁情报 ExTrac 利用 AI 驱动的分析技术和 MongoDB Atlas，通过分析数千个来源的数据来预测公共安全风险。该平台最初帮助西方政府预测冲突，现在正扩展到企业的声誉管理等方面。 MongoDB 的文档数据模型使 ExTrac 能够高效管理复杂数据，增强实时威胁识别。 Atlas Vector Search 有助于增强语言模型，并管理文本、图像和视频的矢量嵌入，从而加快功能开发。这种方法使 ExTrac 能够利用 MongoDB 的灵活性和强大功能，有效地为客户建立趋势模型、追踪不断变化的叙事和预测风险，从而处理任何形状和结构的数据。在 ExTrac 案例研究中了解更多信息。 网络安全评估 VISO TRUST 利用 AI 简化对第三方网络风险的评估，使复杂的供应商安全信息能够快速获取，以便做出明智的决策。 VISO TRUST 的平台利用 Amazon Bedrock 和 MongoDB Atlas，实现了供应商安全尽职调查的自动化，大大减少了安全团队的工作量。 其 AI 驱动的方法涉及人工智能，可对安全文档进行分类、检测组织并预测人工智能中的安全控制位置。 MongoDB Atlas 为密集检索系统提供文本嵌入，通过检索增强生成 ( RAG ) 提高 LLM 的准确性，提供即时、可操作的安全见解。 通过创新地使用技术，VISO TRUST 能够提供快速、可扩展的网络风险评估，为 InstaCart 和 Upwork 等企业大大减少了工作量和时间。 MongoDB 灵活的文档数据库和 Atlas Vector Search 在管理和查询海量数据方面发挥了关键作用，支持 VISO TRUST 提供全面网络风险情报的使命。 在 Viso Trust 案例研究中了解更多信息。 开始使用的步骤 由 LLM 驱动的生成式人工智能，辅以编码为矢量嵌入的操作数据，为网络安全领域带来了许多新的可能性。 如果您想进一步了解这项技术及其可能性，请查看我们的 Atlas Vector Search Learning Byte 。在短短 10 分钟内，您将大致了解不同的使用案例以及如何开始。 1 1 Hill, M. （2023 年 4 月 10 日）。 尽管进行了大规模的招聘活动，但网络安全劳动力缺口仍达 400 万。 CSO。

March 13, 2024
Next →

Building AI With MongoDB: Integrating Vector Search And Cohere to Build Frontier Enterprise Apps

Cohere is the leading enterprise AI platform, building large language models (LLMs) which help businesses unlock the potential of their data. Operating at the frontier of AI, Cohere’s models provide a more intuitive way for users to retrieve, summarize, and generate complex information. Cohere offers both text generation and embedding models to its customers. Enterprises running mission-critical AI workloads select Cohere because its models offer the best performance-cost tradeoff and can be deployed in production at scale. Cohere’s platform is cloud-agnostic. Their models are accessible through their own API as well as popular cloud managed services, and can be deployed on a virtual private cloud (VPC) or even on-prem to meet companies where their data is, offering the highest levels of flexibility and control. Cohere’s leading Embed 3 and Rerank 3 models can be used with MongoDB Atlas Vector Search to convert MongoDB data to vectors and build a state-of-the-art semantic search system. Search results also can be passed to Cohere’s Command R family of models for retrieval augmented generation (RAG) with citations. Check out our AI resource page to learn more about building AI-powered apps with MongoDB. A new approach to vector embeddings It is in the realm of embedding where Cohere has made a host of recent advances. Described as “AI for language understanding,” Embed is Cohere’s leading text representation language model. Cohere offers both English and multilingual embedding models, and gives users the ability to specify the type of data they are computing an embedding for (e.g., search document, search query). The result is embeddings that improve the accuracy of search results for traditional enterprise search or retrieval-augmented generation. One challenge developers faced using Embed was that documents had to be passed one by one to the model endpoint, limiting throughput when dealing with larger data sets. To address that challenge and improve developer experience, Cohere has recently announced its new Embed Jobs endpoint . Now entire data sets can be passed in one operation to the model, and embedded outputs can be more easily ingested back into your storage systems. Additionally, with only a few lines of code, Rerank 3 can be added at the final stage of search systems to improve accuracy. It also works across 100+ languages and offers uniquely high accuracy on complex data such as JSON, code, and tabular structure. This is particularly useful for developers who rely on legacy dense retrieval systems. Demonstrating how developers can exploit this new endpoint, we have published the How to use Cohere embeddings and rerank modules with MongoDB Atlas tutorial . Readers will learn how to store, index, and search the embeddings from Cohere. They will also learn how to use the Cohere Rerank model to provide a powerful semantic boost to the quality of keyword and vector search results. Figure 1: Illustrating the embedding generation and search workflow shown in the tutorial Why MongoDB Atlas and Cohere? MongoDB Atlas provides a proven OLTP database handling high read and write throughput backed by transactional guarantees. Pairing these capabilities with Cohere’s batch embeddings is massively valuable to developers building sophisticated gen AI apps. Developers can be confident that Atlas Vector Search will handle high scale vector ingestion, making embeddings immediately available for accurate and reliable semantic search and RAG. Increasing the speed of experimentation, developers and data scientists can configure separate vector search indexes side by side to compare the performance of different parameters used in the creation of vector embeddings. In addition to batch embeddings, Atlas Triggers can also be used to embed new or updated source content in real time, as illustrated in the Cohere workflow shown in Figure 2. Figure 2: MongoDB Atlas Vector Search supports Cohere’s batch and real time workflows. (Image courtesy of Cohere) Supporting both batch and real-time embeddings from Cohere makes MongoDB Atlas well suited to highly dynamic gen AI-powered apps that need to be grounded in live, operational data. Developers can use MongoDB’s expressive query API to pre-filter query predicates against metadata, making it much faster to access and retrieve the more relevant vector embeddings. The unification and synchronization of source application data, metadata, and vector embeddings in a single platform, accessed by a single API, makes building gen AI apps faster, with lower cost and complexity. Those apps can be layered on top of the secure, resilient, and mature MongoDB Atlas developer data platform that is used today by over 45,000 customers spanning startups to enterprises and governments handling mission-critical workloads. What's next? To start your journey into gen AI and Atlas Vector Search, review our 10-minute Learning Byte . In the video, you’ll learn about use cases, benefits, and how to get started using Atlas Vector Search.

April 25, 2024