Missing community members

[dbeavon]

How likely is it that the project would die if Microsoft stopped supporting it?

I think HD Insight and Synapse (and Azure Databricks to a lesser degree) are the monetized platforms that benefit from the investments into this project.

But I've recently learned that Azure Synapse Analytics is considering the removal of .Net in Spark 3.3 (see https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-33-runtime )

The reason I ask is because I think we've lost some of the full-time contributors to this project,
people like Rahul Potharaju @rapoth and Terry Kim @imback82 See:
https://www.databricks.com/session/introducing-net-bindings-for-apache-spark

(I suspect there are reasons why we've lost them, and why they cannot contribute.
This is probably based on the policies of their current employers. )

I'm hoping this project has enough momentum that it will keep moving ahead one way or another. However based on history, there have been other implementations of .Net in Spark that didn't survive (see https://github.com/microsoft/Mobius )

Thoughts?

[GeorgeS2019]

Mobius is precursor to the existing .NET for Spark :-)

[dbeavon]

True. And there may be another successor to this project some day.

My point is that it may be a very bumpy road if our full-time maintainers have left. I'm trying to understand whether Microsoft will still keep some skin in the game, or if the community can no longer count on that.... IE. What does it look like if Microsoft doesn't want to actively push this project forward anymore?

[GeorgeS2019]

Check out this discussion

[dbeavon]

@GeorgeS2019 Thanks for that link. I'm not very familiar with the process of contributing to github projects. Unfortunately I don't have lots of bandwidth to help much either. But I can probably do some minimal things, like test new releases on my local machine and what-not.

Another thing that makes me hesitant is if Microsoft were to stand in the way of this project. If they don't benefit financially from the results of this project, but another Spark platform does, then they might not be eager to see the project actually succeed.

For example, I think it would slow us down in a big way if we can't update the nuget package when we need to. I think Microsoft is the gatekeeper for that.

BTW, I saw your issue about these docs...
MicrosoftDocs/azure-docs#105466

In regards to your question over there, I believe the PM who had worked on this project in the past was Michael Rys. I think he has moved on (maybe working on cosmos db). I would guess that there is not any Microsoft PM who is currently involved in this project . I'm not sure. If you find out, please let me know!

[GeorgeS2019]

@dbeavon

I posted these questions on documentation as IT IS SO FUNDAMENTAL to understand that if these documents are not up-to-date, naturally, they will be fewer customers who would believe in the technologies promoted by these documents.

To understand Synapse, it is important to understand the history of technologies that are being re-purposed to constitute Synapse analytics.

I am here as I could not see a future where .NET technologies are irrelevant in big data analytics. .NET for Spark, which has been championed, developed, and maintained over many years is key to that future :-)