The Data Engineering team is responsible for Slack’s data lake, analytics dashboards, and other data services. The team’s mission is to empower users to leverage data to make decisions quickly, accurately, and easily. Slack’s data lake grew in size from sub-petabyte to over 100 petabytes in recent years and it now spans millions of tables. As the complexity of managing this data grew, so did a diverse team of Slack engineers dedicated to supporting the ecosystem.
We have strong female representation among engineers in Data at Slack. Our Data Engineering culture celebrates diversity in perspectives and experiences. As data complexity intensifies, having a mosaic of creative problem solvers from different backgrounds is the key to navigating intricate challenges with agility and insight.
Let’s dive into the personal stories of women who are redefining Slack’s data landscape.
- Listen to Jessica, a recent hire, as she navigates the complexities of Pinot and Tableau technology.
- Follow Nilanjana, Ramya, Shrushti, and Nathalie, seasoned engineers, as they lead large-scale engineering projects using technologies like Spark, Merlin, DataHub, and Secor.
- Lastly, hear from Suzanna, Lakshmi, and Beate, female engineering leaders, as they shed light on the landscape of growth and opportunities for women in data engineering.
Hi, I’m Jessica Stewart, and I’m a Senior Software Engineer on the Data Orchestration team. I joined in May 2023.
My team oversees the internal implementations of data workflows using Apache Airflow and Apache Pinot. Operating within our primary cluster, which stores terabytes of data, we maintain a system that boasts sub-second query latency and a nearly 99.95% query success rate service-level agreement (SLA). This datastore serves us internally by supporting tools for employees and externally as the backbone for Slack’s user-facing analytics dashboards.
A current technical challenge we face revolves around migrating from a virtual machine setup to a cloud-native Kubernetes infrastructure. Virtual machines mean higher infrastructure costs and maintenance overhead, so we were eager to increase efficiency with a containerized set-up. We have an internal Slack Kubernetes platform that does not fully support Helm charts or Kubernetes services, so we had to make some customizations to the open-source solution. In order to maintain performance requirements, we configured a customer networking set-up that integrates Pinot with our internal Kubernetes platform. Furthermore, working with Pinot allows us to delve into both software and infrastructure layers. We’ve developed custom Python tooling that wraps around Pinot to standardize operations, and we’ve streamlined data ingestion through Airflow pipelines. On the infrastructure side, we’ve automated deployments and maintenance tasks using Ansible, Kubernetes, and Jenkins.
When interviewing with Slack, my prospective co-workers were supportive and kind, and this continues to be the case as we work on projects, handle incidents, and plan future work. I have the opportunity to own initiatives, work cross-functionally and broaden my skill set — all while working with co-workers I trust and enjoy.
Hi, I’m Nilanjana Mukherjee, and I’m a Staff Software Engineer on the Metrics Foundations team. I joined in October 2021.
My team is responsible for generating actionable datasets and metrics for data-driven decision-making. We also maintain high data accuracy and desired landing time SLAs.
Joining the Data Engineering team two years ago has been an incredible experience with opportunities and challenges. Amidst my involvement in many data modeling projects, I led a pivotal strategic initiative: a migration of workloads from Hive to Spark 3. Being one of the earliest adopters of Spark at Slack made it difficult but rewarding. I became the Spark subject matter expert to over 40 teams across Slack.
The journey was challenging but significantly contributed to my personal and professional development. I owe much of my success to the resources and unwavering support from my managers and team members. Recently, I was thrilled to be promoted from Senior Engineer to Staff Engineer — a recognition of my growth, and a testament to the opportunities for advancement at Slack in data engineering.
Hi, I’m Ramya Sundaresan, and I’m a Senior Software Engineer on the Metrics Foundations team. I joined in May 2022.
I’ve had the opportunity to collaborate with two data teams during my tenure at Slack: the Data Ingestion team and the Metrics Foundations team. On the Data Ingestion team, we were entrusted with the critical task of collecting, processing, and ingesting data into our data warehouse from a variety of sources. This included extracting data from Slack’s Vitess application database, managing customized log pipelines, and integrating data retrieval pipelines from Google Sheets. Within the Metrics Foundations team, I found common ground with Nilanjana as we focused on similar areas of interest and expertise.
My contributions to Slack’s data engineering ecosystem include implementing Apache Iceberg within Kafka Connect clusters, orchestrating the migration of Airflow to Kubernetes from AWS EC2 (Amazon Web Services Elastic Compute Cloud) instances, and migrating jobs to Spark3 on AWS EMR6 (Elastic MapReduce) clusters. We heavily leverage AWS. Transitioning from Spark 2 on AWS EMR 5 to Spark 3 upgrades on AWS EMR 6 was a strategic endeavor. It was in line with a company-wide objective to establish a unified tech stack and diminish reliance on legacy systems like Hive and MapReduce. I led multiple teams through automated workflows and comprehensive documentation resulting in a confident and successful migration. We achieved our migration goal in under a year using parallel build pipelines and rigorous testing. This unlocked better performance and fortified our systems with improved compatibility, interoperability, and long-term support.
My journey as a software engineer at Slack has given me opportunities to lead, innovate, and contribute meaningfully to my team’s goals. Our on-call rotation and incident management prioritizes team member well-being with organization and automation. My work gives me a lot of satisfaction, but I also need to acknowledge a delightful constant in my life — my toddler son, who often includes the colorful Slack logo whenever he doodles, and reminds me of the joys beyond code.
Hi, I’m Shrushti Patel, and I’m a Senior Software Engineer on the Data Infrastructure team. I joined in August 2020.
My team owns all the infrastructure and services required to deliver reliable and timely data. Being part of this team, I have worked on services like AWS EMR, Airflow, Trino, Secor, and Ranger to manage infrastructure.
When I joined, Slack had a data ingestion setup reliant on Secor for transferring Kafka data to S3 hosted on EC2. This was causing issues when we tried to incorporate new data topics. My team determined that Secor fell short in terms of industry-standard recognition, ongoing development, and support for emerging data formats. I assumed responsibility for this Secor setup, and I spearheaded its migration to Bedrock (an internally-developed Kubernetes framework). This transition simplified the addition of new topics, harnessed the benefits of Kubernetes, and resulted in cost savings through resource optimization.
To further enhance our data infrastructure, we are migrating from Secor to Kafka Connect: a widely-adopted industry standard for streaming systems. Kafka Connect offers out-of-the-box support for state management, fault tolerance, and scaling. Unlike Secor, Kafka Connect operates as an abstraction and leverages connectors (executable JARs) within the Kafka-Connect ecosystem. This strategic shift is moving us towards real-time streaming capabilities. The work aligns with our commitment to industry standards, ongoing development, and adaptability to evolving data formats.
My time with Slack’s Data Infrastructure team has been a transformative journey. I have delved into mentorship, engaging projects, and a culture of continuous learning. Slack’s commitment to mobility has enriched my growth by providing me the opportunity to work with all the teams under data and cross-organizational teams like Cloud and Observability. When I returned from maternity leave, the team supported me by providing context about ongoing projects and onboarding me back into the flow. This experience showcased Slack’s compassion by creating an environment where professional excellence is nurtured alongside empathy for personal milestones.
Hi, I’m Nathalie Kaligirwa, a Senior Software Engineer on the Metrics Platform & Governance team. My team builds scalable and standardized tools to enhance the data experience. I joined in November 2021.
The past two years have been eventful. I had a healthy baby, took six months of maternity leave, and worked on complex projects with multiple stakeholders and distributed teams. I aimed to balance personal growth with work commitments by aligning tasks with my energy level and busy schedule.
Metrics standardization is a major ongoing initiative at Slack, and it encompasses projects like Merlin, a framework for abstracting metrics and wide tables creation. During parenthood, I focused on specific components within this initiative that would be manageable during a time of increased personal demands. This allowed me to contribute to the larger vision of the project while balancing new parent responsibilities. For example, I updated a service to dynamically add new metrics created with Merlin and eliminated the need for manual migration. I then switched gears to work on improving the search experience. This set the groundwork for a new metadata service that provides lineage across multiple data tools and scalable search.
Whether I was consolidating metrics, learning to change diapers, integrating data services, putting together a nursery, working on metadata services, or navigating a compromised sleep schedule, I grew professionally and personally. Looking back, I am proud of my work and grateful for my team’s support.
As Slack’s Data Engineering leadership, we are committed to mentorship and providing opportunities to learn and grow. The team is also committed to mobility — we encourage internal transfers for professional growth and diversification of skills. Slack’s continuous learning philosophy not only enriches individual careers but also creates a dynamic and collaborative workplace.
We model Slack’s value of compassion in our support for team members taking maternity leave. The team is organized to make reintegration after a hiatus a seamless experience. We genuinely care not just about our teammate’s professional performance but also their personal milestones.
As leaders at Slack, we actively nurture an environment where women in data engineering thrive and reach their full potential. We do this through:
- Visibility: We amplify women’s voices by encouraging them to take on leadership roles and speak at conferences. We celebrate their achievements and showcase them as role models through participation in industry events.
- Mentorship: We offer both formal and informal mentorship programs where we connect experienced women leaders with mentees for guidance and support. This fosters confidence, leadership skills and a strong network of peers.
- Advocacy: We actively challenge biases and advocate for fair practices in all aspects of our work environment.
- Empowerment: We support flexible work arrangements, diversity and inclusion initiatives, and access to training resources. We create a welcoming and inclusive space for all.
Embracing diversity: our commitment to inclusion
Our personal experiences at Slack demonstrate the impact of these inclusion efforts. This is the Slack we envisioned: a place where ideas are heard, contributions are valued, and individual journeys are fueled by a collaborative and supportive culture. Our success is evident in the growing number of women choosing to join our teams and in the legacy of compassion and inclusion we are proud to cultivate.
Interested in joining our Data Engineering team? Apply now
