Key Moments
Prioritizing AI/ML Projects in Your Organization
DeepLearning.AIWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AI/ML projects fail due to lack of organizational commitment, understanding, and proper data governance, leading to a need for structured evaluation criteria before embarking on implementation.
Key Insights
AI/ML projects often fail due to a lack of understanding of AI capabilities, treating them as standard software projects, and insufficient organizational commitment, with only a small fraction making it to production.
Key criteria for evaluating AI/ML projects include business value (revenue/savings), cost of serving, time to market, build vs. buy decisions, competitive landscape, data availability, labeling effort, feature engineering needs, security, privacy, and ethical considerations.
JP Morgan approaches AI prioritization by considering both financial impacts (revenue, cost reduction) and non-financial impacts like client and employee experience, reputation, and crucial cultural change, especially in early adoption phases.
A portfolio approach is recommended for AI/ML projects, spanning near, medium, and long horizons, to mitigate risk, as multiple projects are expected to fail.
Metrics for AI/ML projects evolve through three stages: model metrics (accuracy, error rates), performance metrics (latency, throughput, cost), and customer usage metrics (feedback, adoption rates).
JP Morgan has established an Explainable AI (XAI) center of excellence with over 20 people, recognizing its importance for regulatory compliance and increasing model adoption, even using techniques like SHAP.
Common pitfalls in AI/ML project implementation
Many organizations rush into AI/ML projects without proper planning, leading to failure. Key reasons for these failures include a fundamental lack of understanding about what AI can and cannot do, treating AI projects like traditional software development cycles, and insufficient organizational commitment. Projects are often run as side initiatives with limited investment and without well-rounded teams that include data engineering, ML ops, and software engineering support. Furthermore, the critical prerequisite of adequate and suitable training data is often underestimated or not fully understood. Models are not always right, and the journey from building a model to a production-ready product is complex and iterative, differing significantly from standard software development lifecycles. Misaligned expectations and a misunderstanding of AI's experimental, iterative nature contribute significantly to project failure.
Technical challenges in data acquisition and management
A significant technical hurdle for AI/ML projects is the availability of quality training data. Acquiring real-world data from production databases can be restricted by access limitations, security concerns, and privacy issues, often requiring explicit customer consent and navigating legal complexities. Data labeling is another major challenge, as raw data is frequently not labeled or poorly labeled, necessitating manual annotation efforts. The data used may also lack proper use-case coverage, especially for general AI products aiming to serve multiple industries. Finally, a lack of coherent data governance mechanisms means training data is often managed ad-hoc, uncataloged, and poorly maintained, further hindering robust ML processes.
Establishing AI/ML project evaluation criteria
To mitigate failure and increase success probability, a structured approach to evaluating AI/ML project ideas is essential. Kumaran Ponnambalam from Cisco outlines several key criteria. The first set focuses on business value, including potential revenue generation, cost savings, and overall business impact, alongside an estimate of the serving cost and risk modeling. Time to market is also crucial, differentiating between projects that can be delivered in months versus those requiring longer efforts. A build-versus-buy decision is vital, assessing the availability of off-the-shelf technologies and open-source models, and determining the extent of in-house development needed. The competitive landscape should also be considered to ensure a strategic advantage. These criteria help in making informed decisions about which projects to pursue.
Data-centric evaluation and building blocks
Beyond business value, a critical set of evaluation criteria revolves around data. This includes assessing the type and availability of training data, any access restrictions or customer permission requirements, and the effort and cost associated with labeling and feature engineering. Security, privacy, and ethical considerations for data usage and model predictions are paramount, as is the cost of data storage and management, requiring proper cataloging and governance. The 'building blocks' criteria assess the availability of pre-existing algorithms, pre-trained models, and libraries, as well as cloud-based AI services (e.g., from AWS or Google). Evaluating the tools, techniques, and automation capabilities for the ML lifecycle (MLOps, experiment tracking, model management, serving) and the ease of integrating the model into the broader solution are also key.
Resource assessment and portfolio management
The 'people' criteria for evaluating AI/ML projects extend beyond data scientists to include data engineers, software engineers, MLOps/DevOps specialists, and potentially hardware resources like GPUs. Kumaran emphasizes that for a team of ten, only two or three might be data scientists. Andrea Stefanucci from JP Morgan highlights the importance of considering both financial and non-financial impacts, such as client and employee experience, reputation, and cultural change, when prioritizing projects. He uses an impact-feasibility matrix, identifying 'quick wins' and 'big bets,' while avoiding low-impact, high-cost projects. Both speakers advocate for a portfolio approach, managing multiple projects across different time horizons (near, medium, long-term) to spread risk, as some projects are bound to fail.
JP Morgan's strategic approach to AI scaling
Andrea Stefanucci explains JP Morgan's evolution from prioritizing individual use cases to transforming entire business areas with AI at their core, aiming for a multiplier effect. This involves establishing foundational enablers like centralized and decentralized AI teams, creating cross-cutting reusable capabilities to prevent duplication, and setting up the right ecosystem with robust infrastructure for rapid experimentation and data access. Attracting top AI talent is crucial, as is educating the business on AI possibilities through traditional courses, showcases, and newsletters. Transparency throughout the multi-year AI adoption journey, tracking progress from current state to future transformed state, is also key. JP Morgan focuses on common capabilities like intelligent client interaction and explainability, and uses examples like transforming the client onboarding process to illustrate their end-to-end transformation strategy.
Metrics, explainability, and the evolution of AI techniques
Metrics for AI/ML projects evolve through distinct stages. Initially, the focus is on model accuracy and error rates. As models scale for production, metrics shift to latency, throughput, CPU/memory usage, and cost. Finally, customer usage metrics, direct feedback, and adoption rates become paramount. Explainable AI (XAI) is a significant focus, with JP Morgan establishing a dedicated center of excellence. XAI is vital for regulatory compliance (e.g., explaining credit rejections) and enhances model adoption by providing insights into model reasoning. While classical techniques like regression and decision trees offer inherent explainability, advanced neural networks, though powerful, often require techniques like SHAP for interpretation. Reinforcement learning is currently more prevalent in research for simulation environments rather than widespread business application. Both real and synthetic data are used, with synthetic data generated to overcome scarcity or privacy concerns.
Hiring and developing AI talent
Hiring and developing AI talent requires looking beyond technical skills. Kumaran emphasizes the need for continuous learners who can adapt to rapidly changing AI technologies and individuals comfortable with uncertainty and experimentation, capable of quickly changing direction based on iterative results. A complementary skill set within the team is essential, covering data science, MLOps, and cloud technologies. Andrea highlights the importance of providing the right environment for experimentation, access to data, and fostering fast, iterative feedback loops with the business. JP Morgan looks for talent across a spectrum including NLP, deep learning, reinforcement learning, and explainability, recognizing that both specialists and versatile individuals who can think holistically about integrating AI solutions are valuable.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Organizations
●Books
●Concepts
●People Referenced
Prioritizing AI/ML Projects: Key Considerations
Practical takeaways from this episode
Do This
Avoid This
Common Questions
AI/ML projects often fail due to a lack of organizational commitment, unclear understanding of AI capabilities, insufficient investment, absence of well-rounded teams, poorly defined metrics, and critical issues with training data availability and quality. Misaligned expectations with traditional software development lifecycles also contribute significantly.
Topics
Mentioned in this video
Founder of AI Fund, which backs Fourth Brain.
Principal Engineer at Cisco, focusing on AI in emerging technologies. He is also an author on LinkedIn Learning and a big data enthusiast.
Director of Product and Curriculum at Fourth Brain, and the host of the event.
Head of AI Strategy and Product Management at JP Morgan AI Research, with extensive experience in financial services.
The parent company of JP Morgan, discussed in the context of moving towards an AI-first mindset.
A consulting firm where Andrea Stefanucci previously worked.
An organization where Kumaran Ponnambalam is a principal engineer, focusing on AI in emerging technologies and incubation.
A consulting firm where Andrea Stefanucci previously worked.
A financial services firm, parent company of JP Morgan Chase, moving towards an AI-first mindset. Andrea Stefanucci leads AI strategy and product management there.
A company that offers AI services like sentiment analysis, text-to-speech, and speech-to-text that can be leveraged for AI projects.
Mentioned as a provider of serving technology for AI models.
A technology mentioned in relation to Kumaran Ponnambalam's courses on LinkedIn Learning.
Amazon Web Services, a provider of AI services like sentiment analysis, text-to-speech, and speech-to-text that can be leveraged for AI projects.
A container orchestration system mentioned as part of the technical skill set needed for ML Ops and DevOps teams.
A technique used for explaining AI models, mentioned as a starting point for explainable AI efforts.
A platform where Kumaran Ponnambalam has courses on ML, AI, and big data.
A containerization platform mentioned as part of the technical skill set needed for ML Ops and DevOps teams.
A popular open-source machine learning framework used by Cisco.
A tool for serving PyTorch models, mentioned as an example of how frameworks are catching up in capabilities.
An online community platform where questions can be posted for the event.
Graphics Processing Units, expensive resources often needed for deep learning models, which can impact the cost-effectiveness of AI projects.
A popular open-source machine learning framework used by Cisco, competing closely with TensorFlow.
A technology mentioned in relation to Kumaran Ponnambalam's courses on LinkedIn Learning.
A popular open-source library for machine learning.
A project management tool where defects are logged when AI models fail to predict correctly, highlighting a misunderstanding of AI project lifecycles.
An organization focused on helping ML practitioners advance their careers through cohort-based courses, offering discounts for MLOps and machine learning engineer programs.
Collaborated with Fourth Brain for the event and offers subscription links and surveys for future event feedback.
A consulting firm where Andrea Stefanucci previously worked.
More from DeepLearningAI
View all 104 summaries
26 minBuild Your Own App In Just 30 Minutes! Full Course with Andrew Ng
29 minAI Dev 26 x SF | Paul Everitt: The Shift to Agentic Engineering
26 minAI Dev 26 x SF | Manos Koukoumidis & Stefan Webb: VibeML: Build your AI model in hours, not months
26 minAI Dev 26 x SF | João Moura: Building Recurring, Governed, and Embedded Enterprise Workflows
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free