Is Apache Spark Too Costly? An Amazon Engineer Tells His Story

The New Stack Podcast

Player FM - Internet Radio Done Right

521 subscribers

DevOps

on yıl önce eklendi

İçerik The New Stack Podcast and The New Stack tarafından sağlanmıştır. Bölümler, grafikler ve podcast açıklamaları dahil tüm podcast içeriği doğrudan The New Stack Podcast and The New Stack veya podcast platform ortağı tarafından yüklenir ve sağlanır. Birinin telif hakkıyla korunan çalışmanızı izniniz olmadan kullandığını düşünüyorsanız burada https://tr.player.fm/legal özetlenen süreci takip edebilirsiniz.

<div class="span index">1</div> <span><a class="" data-remote="true" data-type="html" href="/series/the-final-flight-of-captain-forrester">The Final Flight of Captain Forrester</a></span>

1
The Final Flight of Captain Forrester

Abonelikten çık

8 gün önce8d ago

Abonelikten çık

Aylık

A woman’s search for her father—a pilot who disappeared during a mission in Vietnam—collides with the fight over what we owe those who never returned from war. From the producers of "America's Girls," and hosted by Texas Monthly writer Josh Alvarez, the show debuts April 2025. Texas Monthly Audio subscribers get early access to the show, plus bonus episodes and more subscriber-only audio. Visit texasmonthly.com/audio to learn more.

1 yıl önce 25:26

MP3•Bölüm sayfası

Is Apache Spark too costly? Amazon Principal Engineer Patrick Ames tackled this question during an interview with The New Stack Makers, sharing insights into transitioning from Spark to Ray for managing large-scale data. Ames, described as a "go-to" engineer for exabyte-scale projects, emphasized a goal-driven approach to solving complex engineering problems, from simplifying daily chores to optimizing software solutions.

Initially, Spark was chosen at Amazon for its simplicity and open-source flexibility, allowing efficient merging of data with minimal SQL code. The team leveraged Spark in a decoupled architecture over S3 storage, scaling it to handle thousands of jobs daily. However, as data volumes grew to hundreds of terabytes and beyond, Spark’s limitations became apparent. Long processing times and high costs prompted a search for alternatives.

Enter Ray—a unified framework designed for scaling AI and Python applications. After experimentation, Ames and his team noted significant efficiency improvements, driving the shift from Spark to Ray to meet scalability and cost-efficiency needs.

Learn more from The New Stack about Apache Spark and Ray:

Amazon to Save Millions Moving From Apache Spark to Ray

How Ray, a Distributed AI Framework, Helps Power ChatGPT

Join our community of newsletter subscribers to stay on top of the news and at the top of your game.

890 bölüm

#DevOps #Software Development #Tech #News #Tech News #Technology #Thenewstack #New Stack #Development #Joab Jackson #Alex Williams #Software Engineering #Heather Joslyn #Cloud Native #Kubernetes #Open Source #Darry Taft

Is Apache Spark Too Costly? An Amazon Engineer Tells His Story

The New Stack Podcast

521 subscribers

published 1 yıl önce

Paylaş

MP3•Bölüm sayfası

Learn more from The New Stack about Apache Spark and Ray:

Amazon to Save Millions Moving From Apache Spark to Ray

How Ray, a Distributed AI Framework, Helps Power ChatGPT

Join our community of newsletter subscribers to stay on top of the news and at the top of your game.

890 bölüm

Tüm bölümler

1
Container Security and AI: A Talk with Chainguard's Founder 20:51

1 day önce20:51

20:51

In this episode of The New Stack Makers , recorded at KubeCon + CloudNativeCon Europe, Alex Williams speaks with Ville Aikas, Chainguard founder and early Kubernetes contributor. They reflect on the evolution of container security, particularly how early assumptions—like trusting that users would validate container images—proved problematic. Aikas recalls the lack of secure defaults, such as allowing containers to run as root, stemming from the team’s internal Google perspective, which led to unrealistic expectations about external security practices. The Kubernetes community has since made strides with governance policies, secure defaults, and standard practices like avoiding long-lived credentials and supporting federated authentication. Aikas founded Chainguard to address the need for trusted, minimal, and verifiable container images—offering zero-CVE images, transparent toolchains, and full SBOMs. This security-first philosophy now extends to virtual machines and Java dependencies via Chainguard Libraries. The discussion also highlights the rising concerns around AI/ML security in Kubernetes, including complex model dependencies, GPU integrations, and potential attack vectors—prompting Chainguard’s move toward locked-down AI images. Learn more from The New Stack about Container Security and AI Chainguard Takes Aim At Vulnerable Java Libraries Clean Container Images: A Supply Chain Security Revolution Revolutionizing Offensive Security: A New Era With Agentic AI Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

1
Kelsey Hightower, AWS's Eswar Bala on Open Source's Evolution 37:52

6 gün önce37:52

37:52

In a candid episode of The New Stack Makers , Kubernetes pioneer Kelsey Hightower and AWS’s Eswar Bala explored the evolving relationship between enterprise cloud providers and open source software at KubeCon+CloudNativeCon London. Hightower highlighted open source's origins as a grassroots movement challenging big vendors, and shared how it gave people—especially those without traditional tech credentials—a way into the industry. Recalling his own journey, Hightower emphasized that open source empowered individuals through contribution over credentials. Bala traced the early development of Kubernetes and his own transition from building container orchestration systems to launching AWS’s Elastic Kubernetes Service (EKS), driven by growing customer demand. The discussion, recorded at KubeCon + CloudNativeCon Europe, touched on how open source is now central to enterprise cloud strategies, with AWS not only contributing but creating projects like Karpenter, Cedar, and Kro. Both speakers agreed that open source's collaborative model—where companies build in public and customers drive innovation—has reshaped the cloud ecosystem, turning former tensions into partnerships built on community-driven progress. Learn more from The New Stack about the relationship between enterprise cloud providers and open source software: The Metamorphosis of Open Source: An Industry in Transition The Complex Relationship Between Cloud Providers and Open Source How Open Source Has Turned the Tables on Enterprise Software Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

1
The Kro Project: Giving Kubernetes Users What They Want 21:51

8 gün önce21:51

21:51

In a rare show of collaboration, Google, Amazon, and Microsoft have joined forces on Kro — the Kubernetes Resource Orchestrator — an open source, cloud-agnostic tool designed to simplify custom resource orchestration in Kubernetes. Announced during KubeCon + CloudNativeCon Europe, Kro was born from strong customer demand for a Kubernetes-native solution that works across cloud providers without vendor lock-in. Nic Slattery, Product Manager at Google and Jesse Butler, Principal Product Manager, AWS shared with The New Stack that unlike many enterprise products, Kro didn’t stem from top-down strategy but from consistent customer "pull" experienced by all three companies. It aims to reduce complexity by allowing platform teams to offer simplified interfaces to developers, enabling resource requests without needing deep service-specific knowledge. Kro also represents a unique cross-company collaboration, driven by a shared mission and open source values. Though still in its alpha stage, the project has already attracted 57 contributors in just seven months. The team is now focused on refining core features and preparing for a production-ready release — all while maintaining a narrowly scoped, community-first approach. Learn more from The New Stack about KRO: One Mighty kro; One Giant Leap for Kubernetes Resource Orchestration Kubernetes Gets a New Resource Orchestrator in the Form of Kro Orchestrate Cloud Native Workloads With Kro and Kubernetes Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

1
OpenSearch: What’s Next for the Search and Analytics Suite? 20:10

13 gün önce20:10

20:10

OpenSearch has evolved significantly since its 2021 launch, recently reaching a major milestone with its move to the Linux Foundation. This shift from company-led to foundation-based governance has accelerated community contributions and enterprise adoption, as discussed by NetApp’s Amanda Katona in a New Stack Makers episode recorded at KubeCon + CloudNativeCon Europe. NetApp, an early adopter of OpenSearch following Elasticsearch’s licensing change, now offers managed services on the platform and contributes actively to its development. Katona emphasized how neutral governance under the Linux Foundation has lowered barriers to enterprise contribution, noting a 56% increase in downloads since the transition and growing interest from developers. OpenSearch 3.0, featuring a Lucene 10 upgrade, promises faster search capabilities—especially relevant as data volumes surge. NetApp’s ongoing investments include work on machine learning plugins and developer training resources. Katona sees the Linux Foundation’s involvement as key to OpenSearch’s long-term success, offering vendor-neutral governance and reassuring users seeking openness, performance, and scalability in data search and analytics. Learn more from The New Stack about OpenSearch: Report: OpenSearch Bests ElasticSearch at Vector Modeling AWS Transfers OpenSearch to the Linux Foundation OpenSearch: How the Project Went From Fork to Foundation Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

1
Kong’s AI Gateway Aims to Make Building with AI Easier 21:05

20 gün önce21:05

21:05

AI applications are evolving beyond chatbots into more complex and transformative solutions, according to Marco Palladino, CTO and co-founder of Kong. In a recent episode of The New Stack Makers, he discussed the rise of AI agents, which act as "virtual employees" to enhance organizational efficiency. For instance, AI can now function as a product manager for APIs—analyzing documentation, detecting inaccuracies, and making corrections. However, reliance on AI agents brings security risks, such as data leakage and governance challenges. Organizations need observability and safeguards, but developers often resist implementing these requirements manually. As GenAI adoption matures, teams seek ways to accelerate development without rebuilding security measures repeatedly. To address these challenges, Kong introduced AI Gateway, an open-source plugin for its API Gateway. AI Gateway supports multiple AI models across providers like AWS, Microsoft, and Google, offering developers a universal API to integrate AI securely and efficiently. It also features automated retrieval-augmented generation (RAG) pipelines to minimize hallucinations. Palladino emphasized the need for consistent security in AI infrastructure, ensuring developers can focus on innovation while leveraging built-in protections. Learn more from The New Stack about Kong’s AI Gateway Kong: New ‘AI-Infused’ Features for API Management, Dev Tools From Zero to a Terraform Provider for Kong in 120 Hours Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

1
What’s the Future of Platform Engineering? 26:44

27 gün önce26:44

26:44

Platform engineering was meant to ease the burdens of Devs and Ops by reducing cognitive load and repetitive tasks. However, building internal development platforms (IDPs) has proven challenging. Despite this, Gartner predicts that by 2026, 80% of software engineering organizations will have a platform team. In a recent New Stack Makers episode, Mallory Haigh of Humanitec and Nathen Harvey of Google discussed the current state and future of platform engineering. Haigh emphasized that many organizations rush to build IDPs without understanding why they need them, leading to ineffective implementations. She noted that platform engineering is 10% technical and 90% cultural change, requiring deep introspection and strategic planning. AI-driven automation, particularly agentic AI, is expected to shape platform engineering’s future. Haigh highlighted how AI can enhance platform orchestration and optimize GPU resource management. Harvey compared platform engineering to generative AI—both aim to reduce toil and improve efficiency. As AI adoption grows, platform teams must ensure their infrastructure supports these advancements. Learn more from The New Stack about platform engineering: Platform Engineering on the Brink: Breakthrough or Bust? Platform Engineers Must Have Strong Opinions The Missing Piece in Platform Engineering: Recognizing Producers Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

1
AI Agents are Dumb Robots, Calling LLMs 28:31

5 weeks önce28:31

28:31

AI agents are set to transform software development, but software itself isn’t going anywhere—despite the dramatic predictions. On this episode of The New Stack Makers, Mark Hinkle, CEO and Founder of Peripety Labs, discusses how AI agents relate to serverless technologies, infrastructure-as-code (IaC), and configuration management. Hinkle envisions AI agents as “dumb robots” handling tasks like querying APIs and exchanging data, while the real intelligence remains in large language models (LLMs). These agents, likely implemented as serverless functions in Python or JavaScript, will automate software development processes dynamically. LLMs, leveraging vast amounts of open-source code, will enable AI agents to generate bespoke, task-specific tools on the fly—unlike traditional cloud tools from HashiCorp or configuration management tools like Chef and Puppet. As AI-generated tooling becomes more prevalent, managing and optimizing these agents will require strong observability and evaluation practices. According to Hinkle, this shift marks the future of software, where AI agents dynamically create, call, and manage tools for CI/CD, monitoring, and beyond. Check out the full episode for more insights. Learn more from The New Stack about emerging trends in AI agents: Lessons From Kubernetes and the Cloud Should Steer the AI Revolution AI Agents: Why Workflows Are the LLM Use Case to Watch Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

1
Goodbye SaaS, Hello AI Agents 30:02

6 weeks önce30:02

30:02

The transition from SaaS to Services as Software with AI agents is underway, necessitating new orchestration methods similar to Kubernetes for containers. AI agents will require resource allocation, workflow management, and scalable infrastructure as they evolve. Two key trends are driving this shift: Data Evolution – From spreadsheets to AI agents, data has progressed through relational databases, big data, predictive analytics, and generative AI. Computing Evolution – Starting from mainframes, the journey has moved through desktops, client servers, web/mobile, SaaS, and now agentic workflows. Janakiram MSV, an analyst, notes on this episode of The New Stack Makers that SaaS depends on data—without it, platforms like Salesforce and SAP lack value. As data becomes more actionable and compute more agentic, a new paradigm emerges: Services as Software. AI agents will automate tasks previously requiring human intervention, like emails and sales follow-ups. However, orchestrating them will be complex, akin to Kubernetes managing containers. Unlike deterministic containers, AI agents depend on dynamic, trained data, posing new enterprise challenges in memory management and infrastructure. Learn more from The New Stack about evolution to AI agents: How AI Agents Are Starting To Automate the Enterprise Can You Trust AI To Be Your Data Analyst? Agentic AI is the New Web App, and Your AI Strategy Must Evolve Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

1
How Generative AI Is Reshaping the SDLC 21:42

7 weeks önce21:42

21:42

Amazon Q Developer is streamlining the software development lifecycle by integrating AI-powered tools into AWS. In an interview at AWS in Seattle, Srini Iragavarapu, director of generative AI Applications and Developer Experiences at AWS, discussed how Amazon Q Developer enhances the developer experience. Initially focused on inline code completions, Amazon Q Developer evolved by incorporating generative AI models like Amazon Nova and Anthropic models, improving recommendations and accelerating development. British Telecom reported a 37% acceptance rate for AI-generated code. Beyond code completion, Amazon Q Developer enables developers to interact with Q for code reviews, test generation, and migrations. AWS also developed agentic frameworks to automate undifferentiated tasks, such as upgrading Java versions. Iragavarapu noted that internally, AWS used Q Developer to migrate 30,000 production applications, saving $260 million annually. The platform offers code generation, testing suites, RAG capabilities, and access to AWS custom chips, further flattening the SDLC by automating routine work. Listen to The New Stack Makers for the full discussion. Learn more from The New Stack about Amazon Q Developer: Amazon Q Developer Now Handles Your Entire Code Pipeline Amazon Q Apps: AI-Powered Development for All Amazon Revamps Developer AI With Code Conversion, Security Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

1
OAuth Works for AI Agents but Scaling is Another Question 25:36

8 weeks önce25:36

25:36

Maya Kaczorowski noticed that AI identity and AI agent identity concerns were emerging from outside the security industry, rather than from CISOs and security leaders. She concluded that OAuth, the open standard for authentication, already serves the purpose of granting access without exposing passwords. Kaczorowski, a respected technologist and founder of Oblique, a startup focused on self-serve access controls, recently wrote about OAuth and AI agents and shared her insights on this episode of The New Stack Makers. She noted that developers see AI agents as extensions of themselves, granting them limited access to data and capabilities—precisely what OAuth is designed to handle. The challenges with AI agent identity are vast, involving different approaches to authentication, such as those explored by companies like AuthZed. While existing authorization models like RBAC or ABAC may still apply, the real challenge lies in scale. The exponential growth of AI-related entities—from users to LLMs—could mean even small organizations manage hundreds of thousands of agents. Future solutions must accommodate this massive scale efficiently. For the full discussion, check out The New Stack Makers interview with Kaczorowski. Learn more from The New Stack about OAuth requirements for AI Agents: OAuth 2.0: A Standard in Name Only? AI Agents Are Redefining the Future of Identity and Access Management Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

1
LLMs and AI Agents Evolving Like Programming Languages 28:08

9 weeks önce28:08

28:08

The rise of the World Wide Web enabled developers to build tools and platforms on top of it. Similarly, the advent of large language models (LLMs) allows for creating new AI-driven tools, such as autonomous agents that interact with LLMs, execute tasks, and make decisions. However, verifying these decisions is crucial, and critical reasoning may be a solution, according to Yam Marcovitz, tech lead at Parlant.io and CEO of emcie.co. Marcovitz likens LLM development to the evolution of programming languages, from punch cards to modern languages like Python. Early LLMs started with small transformer models, leading to systems like BERT and GPT-3. Now, instead of mere text auto-completion, models are evolving to enable better reasoning and complex instructions. Parlant uses "attentive reasoning queries (ARQs)" to maintain consistency in AI responses, ensuring near-perfect accuracy. Their approach balances structure and flexibility, preventing models from operating entirely autonomously. Ultimately, Marcovitz argues that subjectivity in human interpretation extends to LLMs, making perfect objectivity unrealistic. Learn more from The New Stack about the evolution of LLMs: AI Alignment in Practice: What It Means and How to Get It Agentic AI: The Next Frontier of AI Power Make the Most of AI Agents: Tips and Tricks for Developers Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

1
Writing Code About Your Infrastructure? That's a Losing Race 31:21

10 weeks önce31:21

31:21

Adam Jacob, CEO of System Initiative, discusses a shift in infrastructure automation—moving from writing code to building models that enable rapid simulations and collaboration. In The New Stack Makers, he compares this approach to Formula One racing, where teams use high-fidelity models to simulate race conditions, optimizing performance before hitting the track. System Initiative applies this concept to enterprise automation, creating a model that understands how infrastructure components interact. This enables fast, multiplayer feedback loops, simplifying complex tasks while enhancing collaboration. Engineers can extend the system by writing small, reactive JavaScript functions that automate processes, such as transforming existing architectures into new ones. The platform visually represents these transformations, making automation more intuitive and efficient. By leveraging models instead of traditional code-based infrastructure management, System Initiative enhances agility, reduces complexity, and improves DevOps collaboration. To explore how this ties into the concept of the digital twin, listen to the full New Stack Makers episode. Learn more from The New Stack about System Initiative: Beyond Infrastructure as Code: System Initiative Goes Live How System Initiative Treats AWS Components as Digital Twins System Initiative Code Now Open Source Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

1
OpenTelemetry: What’s New with the 2nd Biggest CNCF Project? 30:14

11 weeks önce30:14

30:14

Morgan McLean, co-founder of OpenTelemetry and senior director of product management at Splunk, has long tackled the challenges of observability in large-scale systems. In a conversation with Alex Williams on The New Stack Makers , McLean reflected on his early frustrations debugging high-scale services and the need for better observability tools. OpenTelemetry, formed in 2019 from OpenTracing and OpenCensus, has since become a key part of modern observability strategies. As a Cloud Native Computing Foundation (CNCF) incubating project, it’s the second most active open source project after Kubernetes, with over 1,200 developers contributing monthly. McLean highlighted OpenTelemetry’s role in solving scaling challenges, particularly in Kubernetes environments, by standardizing distributed tracing, application metrics, and data extraction. Looking ahead, profiling is set to become the fourth major observability signal alongside logs, tracing, and metrics, with general availability expected in 2025. McLean emphasized ongoing improvements, including automation and ease of adoption, predicting even faster OpenTelemetry adoption as friction points are resolved. Learn more from The New Stack about the latest trends in Open Telemetry: What Is OpenTelemetry? The Ultimate Guide Observability in 2025: OpenTelemetry and AI to Fill In Gaps Honeycomb.io’s Austin Parker: OpenTelemetry In-Depth Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

1
What’s Driving the Rising Cost of Observability? 24:55

12 weeks önce24:55

24:55

Observability is expensive because traditional tools weren’t designed for the complexity and scale of modern cloud-native systems, explains Christine Yen, CEO of Honeycomb.io. Logging tools, while flexible, were optimized for manual, human-scale data reading. This approach struggles with the massive scale of today’s software, making logging slow and resource-intensive. Monitoring tools, with their dashboards and metrics, prioritized speed over flexibility, which doesn’t align with the dynamic nature of containerized microservices. Similarly, traditional APM tools relied on “magical” setups tailored for consistent application environments like Rails, but they falter in modern polyglot infrastructures with diverse frameworks. Additionally, observability costs are rising due to evolving demands from DevOps, platform engineering, and site reliability engineering (SRE). Practices like service-level objectives (SLOs) emphasize end-user experience, pushing teams to track meaningful metrics. However, outdated observability tools often hinder this, forcing teams to cut back on crucial data. Yen highlights the potential of AI and innovations like OpenTelemetry to address these challenges. Learn more from The New Stack about the latest trends in observability: Honeycomb.io’s Austin Parker: OpenTelemetry In-Depth Observability in 2025: OpenTelemetry and AI to Fill In Gaps Observability and AI: New Connections at KubeCon Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

1
How Oracle Is Meeting the Infrastructure Needs of AI 27:28

13 weeks önce27:28

27:28

Generative AI is a data-driven story with significant infrastructure and operational implications, particularly around the rising demand for GPUs, which are better suited for AI workloads than CPUs. In an episode of The New Stack Makers recorded at KubeCon + CloudNativeCon North America, Sudha Raghavan, SVP for Developer Platform at Oracle Cloud Infrastructure, discussed how AI’s rapid adoption has reshaped infrastructure needs. The release of ChatGPT triggered a surge in GPU demand, with organizations requiring GPUs for tasks ranging from testing workloads to training large language models across massive GPU clusters. These workloads run continuously at peak power, posing challenges such as high hardware failure rates and energy consumption. Oracle is addressing these issues by building GPU superclusters and enhancing Kubernetes functionality. Tools like Oracle’s Node Manager simplify interactions between Kubernetes and GPUs, providing tailored observability while maintaining Kubernetes’ user-friendly experience. Raghavan emphasized the importance of stateful job management and infrastructure innovations to meet the demands of modern AI workloads. Learn more from The New Stack about how Oracle is addressing the GPU demand for AI workloads with its GPU superclusters and enhancing Kubernetes functionality: Oracle Code Assist, Java-Optimized, Now in Beta Oracle’s Code Assist: Fashionably Late to the GenAI Party Oracle Unveils Java 23: Simplicity Meets Enterprise Power Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

Player FM'e Hoş Geldiniz!

Player FM şu anda sizin için internetteki yüksek kalitedeki podcast'leri arıyor. En iyi podcast uygulaması ve Android, iPhone ve internet üzerinde çalışıyor. Aboneliklerinizi cihazlar arasında eş zamanlamak için üye olun.