Replies: 4 comments 7 replies
-
|
It seems to run well. We would like to see 3.3 support. That said the biggest limitation is that a lot of important libraries (e.g. Sedona) don’t have bindings for .NET. |
Beta Was this translation helpful? Give feedback.
-
|
I would also suggest this is a classic example of a project that got started with a lot of fanfare but it was left without adequate support. Probably would have been better if Databricks or Apache supported it vs the .NET Foundation |
Beta Was this translation helpful? Give feedback.
-
|
The reason I ask is because I'm being told by the PG that they are eliminating .Net support from Spark 3.3 in Synapse Analytics. For anyone who has invested a lot of effort in this technology, this feels like a real kick in the jewels. I'm not sure if the decision is final but the PG has updated their documentation pages to say that .Net has been removed. Link: ... ideally they would qualify this by saying .Net 6 is still under consideration. Or perhaps they could offer a mechanism for allowing customers to continue running .Net Core 3.1 (albeit without full support). This would not be any different than how customers run this in various other cluster environments like on Azure Databricks.
|
Beta Was this translation helpful? Give feedback.
-
|
@Niharikadutta @MikeRys I agree that the current language on the synapse page definitely leaves the impression the project is dead |
Beta Was this translation helpful? Give feedback.
-
|
.NET 6 is now actively working. Please check the c# notebooks for Synapse, which I hope will be restored soon once the .NET6 migration is done. |
Beta Was this translation helpful? Give feedback.
-
What do you base your hopes on? Have you spoken to the PG? Do you work for Microsoft? Are you on the Synapse product team? I appreciate your optimism. But it is 100% contrary to the actual language used by the Synapse PG. I think some pessimism is in order at the moment. Either way, it would be nice if we could receive some authoritative/official communication from the Synapse folks themselves (People like Martin Lee and @MikeRys ) Here is some actual language about .Net from the Synapse PG. ... I think the best thing for this community is to identify all of the alternative Spark platforms that will take the place of Synapse for hosting our .Net workloads. Perhaps we can gather votes to identify the best platform. Everyone originally thought that Synapse was the premium hosting platform for .Net but it is clearly not. I have been running .Net workloads on Azure Databricks for some time. The .Net performance suffers over there (because of a non-standard driver node). And I once thought the support was pretty poor over there (on Azure Databricks). And I thought Synapse would offer first-class support for .Net. ... But Synapse cannot hope to be be any better than Azure Databricks in the long run, given that they seem to be removing .Net altogether in Spark 3.3. It is too bad because Synapse had something truly special going on with the C# notebook experience (.Net interactive with Spark). I think they got impatient with the fact that this stuff takes a very, very long time to monetize. Also I suspect the engineering team for .Net on Synapse has been dissolved. (I would rather not be starting rumors myself. I have not yet confirmed this, but it seems likely based on linkedin profiles and such.) |
Beta Was this translation helpful? Give feedback.
-
|
Not related to Microsoft, totally independent. Thanks for providing much of the background I need to "TRY" lobby for support. I hope MORE users could do the same and I would not be surprised to much extent I may end up with your observations Would it be more effective if we gather more feedback under ONE issue rather then discussion? |
Beta Was this translation helpful? Give feedback.
-
|
@GeorgeS2019 Can you show me where you think someone "deleted ALL C# .NET for Spark samples for Synapse Jan 2023"? I briefly mentioned this on a call with the Microsoft folks, but then later I noticed that the folder you shared has not been updated/modified for several years. Remember git will give a complete history of changes, along with the relevant timestamps. Your screenshot doesn't prove that anything was recently deleted.... Also, I don't think the removal of .Net from the Synapse platform is an "issue" which the opensource community can actually fix. Unfortunately the future path of Synapse is 100% Microsoft's (misguided) decision. The future path of the Synapse platform is somewhat disconnected from this community project. Even if Microsoft totally removed .Net from Synapse Spark, you would still be able to deploy your .Net batch jobs to another Spark offerings in Azure (like Azure Databricks or HD Insight). Of course the notebook experience is missing in those environments, but perhaps you can get your notebooks working on-premise so they can help you during your development. (I typically don't use a notebook/REPL experience for production workloads anyway, but it is extremely helpful while doing development with small subsets of data.) |
Beta Was this translation helpful? Give feedback.
-
For respect, I only show the commit which I tracked down before it is deleted commit d23df9706444ad26aa0f48fd27fe9d10d61aada8 (HEAD) My main interest is to see .NET for Spark support continue. Not interested in what has happened. :-) |
Beta Was this translation helpful? Give feedback.
-
|
There are a few options moving forwards, it would be best if we could persuade the dotnet team to support us. The first step is to get AS MUCH FEEDBACK as possible while WE get a more holistic understanding of the status. Your feedback has been valuable. |
Beta Was this translation helpful? Give feedback.
-
The notebook is based on .Interactive. The notebook team is experimenting with new features. More language interoperability through variable sharing. e.g. python and c# Why I am optimistic there are a few options moving forwards. |
Beta Was this translation helpful? Give feedback.
-
Has anyone had any luck using .Net for Spark on Synapse?
Michael (@MikeRys ) said that there was some usage in the past:
See #809.
However, that message is about two years old. It is possible that .Net for Spark is less popular than it was a couple years ago. I'm not having a great customer-support experience while using .Net for Spark on Synapse. Based on the level of support, I'm starting to think that I'm the only Microsoft customer using .Net on Synapse. Is that even possible?
Personally, I love this technology, and wish it was available to me twenty years ago when I first read Ralph Kimball's Data Warehouse Toolkit. Building data warehouses with Spark is a totally different experience than building them without Spark. That book would have been primarily a Spark book, if the Spark technology had been around at the time.
Having said that, I have heard that there may be some unhappy news about .Net on Synapse-Spark. An announcement is likely in the near future, when Apache Spark 3.3 is GA. (Currently it is preview, and currently .Net is still listed as one of the components of Spark 3.3.)
https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-33-runtime
I don't want to spread any gossip until an authoritative announcement is made. But I am hoping to know if anyone else has invested as much effort as we have in .Net. In my case, I totally rebuilt all our Scala pipelines to run in .Net on Synapse-Spark.
Beta Was this translation helpful? Give feedback.
All reactions