Just Enough Data | Newsletter 15
Live with Astronomer: Simplified DAG authoring with the Astro SDK
In this “Live with Astronomer” session, we’ll give an overview of the Astro SDK and show how it makes writing ELT pipelines easy and efficient. We’ll cover everything you need to get started using this open source tool in just a few minutes.
When: May 2, 2023 at 2pm EDT
DATA+AI Summit Generation AI
Large Language Models (LLM) are taking AI mainstream. Join the premier event for the global data community to understand their potential and shape the future of your industry with data and AI.
San Francisco, Moscone Center
June 26 - 29, 2023
Horizontally scaling Kafka consumers with rendezvous hashing
How we used rendezvous hashing to horizontally scale Kafka consumers to support hundreds of concurrent topics with fewer connections, thus lowering our infrastructure costs.
Spark Connect Available in Apache Spark 3.4
In Apache Spark 3.4, Spark Connect introduced a decoupled client-server architecture that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol. The separation between client and server allows Spark and its open ecosystem to be leveraged from everywhere. It can be embedded in modern data applications, in IDEs, Notebooks and programming languages.
Databricks and Hugging Face integrate Apache Spark for faster AI model building
Databricks and Hugging Face have collaborated to introduce a new feature that allows users to create a Hugging Face dataset from an Apache Spark data frame. This new integration provides a more straightforward method of loading and transforming data for artificial intelligence (AI) model training and fine-tuning. Users can now map their Spark data frame into a Hugging Face dataset for integration into training pipelines.
ChatGPT Prompt Engineering for Developers
In ChatGPT Prompt Engineering for Developers, you will learn how to use a large language model (LLM) to quickly build new and powerful applications. Using the OpenAI API, you’ll be able to quickly build capabilities that learn to innovate and create value in ways that were cost-prohibitive, highly technical, or simply impossible before now.
Mastering AI-Powered Product Development: Introducing Promptimize for Test-Driven Prompt Engineering
Promptimize, a Python toolkit designed to measure success and outcomes while doing prompt engineering.
LEARNING IS THE ONLY WAY TO GROW.
Ajith Shetty
Bigdata Engineer — Bigdata, Analytics, Cloud and Infrastructure.
Medium Subscribe✉️ ||More blogs📝||LinkedIn📊||Profile Page📚||Git Repo👓