|
| 1 | +--- |
| 2 | +title: "Google Summer of Code 2025 - Beam ML Vector DB/Feature Store integrations" |
| 3 | +date: 2025-09-26 00:00:00 -0400 |
| 4 | +categories: |
| 5 | + - blog |
| 6 | + - gsoc |
| 7 | +aliases: |
| 8 | + - /blog/2025/09/26/gsoc-25-ml-connectors.html |
| 9 | +authors: |
| 10 | + - mohamedawnallah |
| 11 | + |
| 12 | +--- |
| 13 | +<!-- |
| 14 | +Licensed under the Apache License, Version 2.0 (the "License"); |
| 15 | +you may not use this file except in compliance with the License. |
| 16 | +You may obtain a copy of the License at |
| 17 | +
|
| 18 | +http://www.apache.org/licenses/LICENSE-2.0 |
| 19 | +
|
| 20 | +Unless required by applicable law or agreed to in writing, software |
| 21 | +distributed under the License is distributed on an "AS IS" BASIS, |
| 22 | +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 23 | +See the License for the specific language governing permissions and |
| 24 | +limitations under the License. |
| 25 | +--> |
| 26 | + |
| 27 | +## What Will I Cover In This Blog Post? |
| 28 | + |
| 29 | +I have three objectives in mind when writing this blog post: |
| 30 | + |
| 31 | +- Documenting the work I've been doing during this GSoC period in collaboration |
| 32 | +with the Apache Beam community |
| 33 | +- A thoughtful and cumulative thank you to my mentor and the Beam Community |
| 34 | +- Writing to an older version of myself before making my first ever contribution |
| 35 | +to Beam. This can be helpful for future contributors |
| 36 | + |
| 37 | +## What Was This GSoC Project About? |
| 38 | + |
| 39 | +The goal of this project is to enhance Beam's Python SDK by developing |
| 40 | +connectors for vector databases like Milvus and feature stores like Tecton. These |
| 41 | +integrations will improve support for ML use cases such as Retrieval-Augmented |
| 42 | +Generation (RAG) and feature engineering. By bridging Beam with these systems, |
| 43 | +this project will attract more users, particularly in the ML community. |
| 44 | + |
| 45 | +## Why Was This Project Important? |
| 46 | + |
| 47 | +While Beam's Python SDK supports some vector databases, feature stores and |
| 48 | +embedding generators, the current integrations are limited to a few systems as |
| 49 | +mentioned in the tables down below. Expanding this ecosystem will provide more |
| 50 | +flexibility and richness for ML workflows particularly in feature engineering |
| 51 | +and RAG applications, potentially attracting more users, particularly in the ML |
| 52 | +community. |
| 53 | + |
| 54 | +| Vector Database | Feature Store | Embedding Generator | |
| 55 | +|----------------|---------------|---------------------| |
| 56 | +| BigQuery | Vertex AI | Vertex AI | |
| 57 | +| AlloyDB | Feast | Hugging Face | |
| 58 | + |
| 59 | +## Why Did I Choose Beam As Part of GSoC Among 180+ Orgs? |
| 60 | + |
| 61 | +I chose to apply to Beam from among 180+ GSoC organizations because it aligns |
| 62 | +well with my passion for data processing systems that serve information |
| 63 | +retrieval systems and my core career values: |
| 64 | + |
| 65 | +- **Freedom:** Working on Beam supports open-source development, liberating |
| 66 | +developers from vendor lock-in through its unified programming model while |
| 67 | +enabling services like [Project Shield](https://projectshield.withgoogle.com/landing) to protect free |
| 68 | +speech globally |
| 69 | + |
| 70 | +- **Innovation:** Working on Beam allows engagement with cutting-edge data |
| 71 | +processing techniques and distributed computing paradigms |
| 72 | + |
| 73 | +- **Accessibility:** Working on Beam helps build open-source technology that |
| 74 | +makes powerful data processing capabilities available to all organizations |
| 75 | +regardless of size or resources. This accessibility enables projects like |
| 76 | +Project Shield to provide free protection to media, elections, and human rights |
| 77 | +websites worldwide |
| 78 | + |
| 79 | +## What Did I Work On During the GSoC Program? |
| 80 | + |
| 81 | +During my GSoC program, I focused on developing connectors for vector databases, |
| 82 | +feature stores, and embedding generators to enhance Beam's ML capabilities. |
| 83 | +Here are the artifacts I worked on and what remains to be done: |
| 84 | + |
| 85 | +| Type | System | Artifact | |
| 86 | +|----------------|--------|----------| |
| 87 | +| Enrichment Handler | Milvus | [PR #35216](https://github.com/apache/beam/pull/35216) <br> [PR #35577](https://github.com/apache/beam/pull/35577) <br> [PR #35467](https://github.com/apache/beam/pull/35467) | |
| 88 | +| Sink I/O | Milvus | [PR #35708](https://github.com/apache/beam/pull/35708) <br> [PR #35944](https://github.com/apache/beam/pull/35944) | |
| 89 | +| Enrichment Handler | Tecton | [PR #36062](https://github.com/apache/beam/pull/36062) | |
| 90 | +| Sink I/O | Tecton | [PR #36078](https://github.com/apache/beam/pull/36078) | |
| 91 | +| Embedding Gen | OpenAI | [PR #36081](https://github.com/apache/beam/pull/36081) | |
| 92 | +| Embedding Gen | Anthropic | To Be Added | |
| 93 | + |
| 94 | +Here are side-artifacts that are not directly linked to my project: |
| 95 | +| Type | System | Artifact | |
| 96 | +|------|--------|----------| |
| 97 | +| AI Code Review | Gemini Code Assist | [PR #35532](https://github.com/apache/beam/pull/35532) | |
| 98 | +| Enrichment Handler | CloudSQL | [PR #34398](https://github.com/apache/beam/pull/34398) <br> [PR #35473](https://github.com/apache/beam/pull/35473) | |
| 99 | +| Pytest Markers | GitHub CI | [PR #35655](https://github.com/apache/beam/pull/35655) <br> [PR #35740](https://github.com/apache/beam/pull/35740) <br> [PR #35816](https://github.com/apache/beam/pull/35816) | |
| 100 | + |
| 101 | +For more granular contributions, checking out my |
| 102 | +[ongoing Beam contributions](https://github.com/apache/beam/pulls?q=is%3Apr+author%3Amohamedawnallah). |
| 103 | + |
| 104 | +## How Did I Approach This Project? |
| 105 | + |
| 106 | +My approach centered on community-driven design and iterative implementation, |
| 107 | +Originally inspired by my mentor's work. Here's how it looked: |
| 108 | + |
| 109 | +1. **Design Document**: Created a comprehensive design document outlining the |
| 110 | +proposed ML connector architecture |
| 111 | +2. **Community Feedback**: Shared the design with the Beam developer community |
| 112 | +mailing list for review |
| 113 | +3. **Iterative Implementation**: Incorporated community feedback and applied |
| 114 | +learnings in subsequent pull requests |
| 115 | +4. **Continuous Improvement**: Refined the approach based on real-world usage |
| 116 | +patterns and maintainer guidance |
| 117 | + |
| 118 | +Here are some samples of those design docs: |
| 119 | + |
| 120 | +| Component | Type | Design Document | |
| 121 | +|-----------|------|-----------------| |
| 122 | +| Milvus | Vector Enrichment Handler | [[Proposal][GSoC 2025] Milvus Vector Enrichment Handler for Beam](https://lists.apache.org/thread/4c6l20tjopd94cqg6vsgj20xl2qgywtx) | |
| 123 | +| Milvus | Vector Sink I/O Connector | [[Proposal][GSoC 2025] Milvus Vector Sink I/O Connector for Beam](https://lists.apache.org/thread/cwlbwnhnf1kl7m0dn40jrqfsf4ho98tf) | |
| 124 | +| Tecton | Feature Store Enrichment Handler | [[Proposal][GSoC 2025] Tecton Feature Store Enrichment Handler for Beam](https://lists.apache.org/thread/7ynn4r8b8b1c47ojxlk39fhsn3t0jrd1) | |
| 125 | +| Tecton | Feature Store Sink I/O Connector | [[Proposal][GSoC 2025] Tecton Feature Store Sink I/O Connector for Beam](https://lists.apache.org/thread/dthd3t6md9881ksvbf4v05rxnlj1fgvn) | |
| 126 | + |
| 127 | + |
| 128 | +## Where Did Challenges Arise During The Project? |
| 129 | + |
| 130 | +There were 2 places where challenges arose: |
| 131 | + |
| 132 | +- **Running Docker TestContainers in Beam Self-Hosted CI Environment:** The main |
| 133 | +challenge was that Beam runs in CI on Ubuntu 20.04, which caused compatibility |
| 134 | +and connectivity issues with Milvus TestContainers due to the Docker-in-Docker |
| 135 | +environment. After several experiments with trial and error, I eventually tested |
| 136 | +with Ubuntu latest (which at the time of writing this blog post is Ubuntu 25.04), |
| 137 | +and no issues arose. This version compatibility problem led to the container |
| 138 | +startup failures and network connectivity issues |
| 139 | + |
| 140 | +- **Triggering and Modifying the PostCommit Python Workflows:** This challenge |
| 141 | +magnified the above issue since for every experiment update to the given |
| 142 | +workflow, I had to do a round trip to my mentor to include those changes in the |
| 143 | +relevant workflow files and evaluate the results. I also wasn't aware that |
| 144 | +someone can trigger post-commit Python workflows by updating the trigger files |
| 145 | +in `.github/trigger_files` until near the middle of GSoC. I discovered there is |
| 146 | +actually a workflows README document in `.github/workflows/README.md` that was |
| 147 | +not referenced in the `CONTRIBUTING.md` file at the time of writing this post |
| 148 | + |
| 149 | +## How Did This Project Start To Attract Users in the ML Community? |
| 150 | + |
| 151 | +It is observed that after we had a Milvus Enrichment Handler PR before even |
| 152 | +merging, we started to see community-driven contributions like |
| 153 | +[this one that adds Qdrant](https://github.com/apache/beam/pull/35686). Qdrant |
| 154 | +is a competitor to Milvus in the vector space. This demonstrates how |
| 155 | +the project's momentum and visibility in the ML community space attracted |
| 156 | +contributors who wanted to expand the Beam ML ecosystem with additional vector |
| 157 | +database integrations. |
| 158 | + |
| 159 | +## How Did This GSoC Experience Working With Beam Community Shape Me? |
| 160 | + |
| 161 | +If I have to boil it down across three dimensions, they would be: |
| 162 | + |
| 163 | +- **Mindset:** Before I was probably working in solitude making PRs about new |
| 164 | +integrations with mental chatter in the form of fingers crossed, hoping that |
| 165 | +there will be no divergence on the design. Now I can engage people I am working |
| 166 | +with through design docs, making sure my work aligns with their vision, which |
| 167 | +potentially leads to faster PR merges |
| 168 | +- **Skillset:** It was one year before contributing to Beam where I wrote |
| 169 | +professionally in Python, so it was a great opprtunity to brush up on my Python |
| 170 | +skills and seeing how some design patterns are used in practice, like the query |
| 171 | +builder pattern seen in CloudSQL Vector Ingestion in the RAG package. I also |
| 172 | +learned about vector databases and feature stores, and also some AI |
| 173 | +integrations. I also think I got a bit better than before in root cause analysis |
| 174 | +and filtering signals from noise in long log files like PostCommit Python |
| 175 | +workflows |
| 176 | +- **Toolset:** Learning about Beam Python SDK, Milvus, Tecton, Google CloudSQL, |
| 177 | +OpenAI and Anthropic text embedding generators, and lnav for effective log file |
| 178 | +navigation, including their capabilities and limitations |
| 179 | + |
| 180 | +## Tips for Future Contributors |
| 181 | + |
| 182 | +If I have to boil them down to three, they would be: |
| 183 | + |
| 184 | +- **Observing:** Observing how experienced developers in the Beam dev team |
| 185 | +work—how their PRs look, how they write design docs, what kind of feedback they |
| 186 | +get on their design docs and PRs, and how you can apply it (if feasible) to |
| 187 | +avoid getting the same feedback again. What kind of follow-up PRs do they create |
| 188 | +after their initial ones? How do they document and illustrate their work? What |
| 189 | +kind of comments do they post when reviewing other people's related work? Over |
| 190 | +time, you build your own mental model and knowledge base on how the ideal |
| 191 | +contribution looks in this area. There is a lot to learn and explore in an |
| 192 | +exciting, not intimidating way |
| 193 | +- **Orienting:** Understanding your place in the ecosystem and aligning your |
| 194 | +work with the project's context. This means grasping how your contribution fits |
| 195 | +into Beam's architecture and roadmap, identifying your role in addressing |
| 196 | +current gaps, and mapping stakeholders who will review, use, and maintain your |
| 197 | +work. Most importantly, align with both your mentor's vision and the community's |
| 198 | +vision to ensure your work serves the broader goals |
| 199 | +- **Acting:** Acting on feedback from code reviews, design document discussions, |
| 200 | +and community input. This means thoughtfully addressing suggested changes in a |
| 201 | +way that moves the discussion forward, addressing concerns raised by |
| 202 | +maintainers, and iterating on your work based on community guidance. Being |
| 203 | +responsive to feedback, asking clarifying questions when needed, and |
| 204 | +demonstrating that you're incorporating the community's input into your |
| 205 | +contributions given that it is aligned with the project direction |
| 206 | + |
| 207 | +## Who Do I Want To Thank for Making This Journey Possible? |
| 208 | + |
| 209 | +If I have to boil them down to three, they would be: |
| 210 | + |
| 211 | +- **My Mentor, Danny McCormick:** I wouldn't hesitate to say that Danny is the |
| 212 | +best mentor I have worked with so far, given that I have worked with several |
| 213 | +mentors. What makes me say that: |
| 214 | + - **Generosity:** Danny is very generous with his time, feedback, and |
| 215 | + genuinely committed to reviewing my work on a regular basis. We have weekly |
| 216 | + 30-minute sync calls over almost 21 weeks (5 months) since the official |
| 217 | + community bonding period, where he shares with me his contextual expertise and |
| 218 | + addresses any questions I may have with openness to extend time if needed and |
| 219 | + flexible about skipping calls when there was no agenda |
| 220 | + - **Flexibility:** When I got accepted to GSoC, after a few days I also got |
| 221 | + accepted to a part-time internship that I had applied to before GSoC, while |
| 222 | + also managing my last semester in my Bachelor of Computer Science, which was |
| 223 | + probably the hardest semester. During our discussion about working capacity, |
| 224 | + Danny was very flexible regarding that, with more emphasis on making progress, |
| 225 | + which encouraged me to make even more progress. I have also never felt there |
| 226 | + are very hard boundaries around my project scope—I felt there was an area to |
| 227 | + explore that motivated me to think of and add some side-artifacts to Beam, |
| 228 | + e.g., adding Gemini Code Assist for AI code review |
| 229 | + - **Proactivity**: Danny was very proactive in offering support and help |
| 230 | + without originally asking, e.g., making Beam Infra tickets that add API keys |
| 231 | + to unblock my work |
| 232 | +- **Beam Community:** From my first ever contribution to Beam [adding FlattenWith and Tee examples to the playground](https://github.com/apache/beam/issues/32840#issuecomment-2424055627), |
| 233 | +I was welcomed with open arms and felt encouraged to make more contributions. |
| 234 | +Also, for their valuable comments on my design documents on the dev mailing list |
| 235 | +as well as the PRs |
| 236 | +- **Google:** I would like to genuinely thank Google for introducing me to open |
| 237 | +source in [GSoC 2023](https://summerofcode.withgoogle.com/archive/2023/projects/u7Y9S6sc) |
| 238 | +and giving me a second chance to interact with Apache Beam through GSoC 2025. |
| 239 | +Without it, I probably wouldn't be here writing this blog post, nor would I have |
| 240 | +this fruitful experience |
| 241 | + |
| 242 | +## What's Next? |
| 243 | + |
| 244 | +I am now focusing on helping move the remaining artifacts in this project scope |
| 245 | +from the in-progress state to the merging state. After this, I would love to |
| 246 | +keep my contributions alive in Beam Python and Go SDK, to name a few. I would |
| 247 | +also love to connect with you all on my |
| 248 | +[LinkedIn](https://www.linkedin.com/in/mohamedawnallah/) and |
| 249 | +[GitHub](https://github.com/mohamedawnallah). |
| 250 | + |
| 251 | +## References |
| 252 | +- [Google Summer of Code Project Listing](https://summerofcode.withgoogle.com/programs/2025/projects/X32yGjqz) |
| 253 | +- [Original GSoC Proposal](https://docs.google.com/document/d/1YOeK3jb94kSOUxucfqeZL0pkRI08dYljV_4v5SH5i5U/edit?usp=sharing) |
| 254 | +- [GSoC 2025 Tracking Issue](https://github.com/apache/beam/issues/35046) |
0 commit comments