How-to explore a Kiara table in Jupyter?

[MariellaCC]

Is there a recommended approach to let users explore a Kiara table in Jupyter or some steps that could be implemented (e.g. from my side) to move towards this goal?
Ideally, users may need to sort and filter rows and navigate the data frame.

[makkus]

There is no recommended approach as far as I am aware, we'd have to come up with one?

If you are happy with the way Jupyter does it for pandas frame, I'll add a method that forwards the Jupyter 'magic' method for the kiara KiaraTable class to one for a dynamically created Pandas dataframe, so end users will see the same thing as they would see for an instance of that.

If you want more, there are options as well, but would involve quite a bit more work. A starting point would probably be https://ipython.readthedocs.io/en/stable/config/integrating.html -- but it can get arbitrarily complicated depending on what exactly you want to achieve. Also, I have no real experience here, I have never actually implemented my own Jupyter widgets, so I'm not sure how all this is reliant on the notebook api, or the lab one, etc.

[MariellaCC]

I labelled this issue as a "discussion" so that we can work towards a recommended approach via the discussion.

I have a question concerning the temporary code workaround. Here is an extract of this code you provided:

df = self.to_pandas_dataframe()
result = {}
result["text/html"] =  df._repr_html_()

Is there currently a possibility, via the workaround code (for display testing purposes to help find what could be useful before potential implementation later), of having several outputs like result["text/html"] to let users choose between several preview options (e.g. specify the number of cols to be output, whether to output head or tail) ?

[makkus]

No idea, to be honest. I'd guess not. You'll have to look up how the _repr_html_ and _repro_mimebundle_ are implemented in the Jupyter source code, and in which cases which is going to be called. I guess it could be controlled via environment variable, or calling a rendering method directly. How that is all wired within Jupyter, I have no idea, sorry.

[MariellaCC]

What I meant is: would there be a way for users to choose the value that is rendered by Kiara in the notebook:

for example:

df = self.to_pandas_dataframe()
df_head = df.head()
df_tail = df.tail()

result = {}

result["text/html1"] =  df_head._repr_html_()
result["text/html2"] =  df_tail._repr_html_()

I can figure out the df_head._repr_html_() parts myself, but what I do not know is what would be possible to test in Kiara (only with the workaround code; this is just to play with ideas). Alternatively, I can simply convert the pyarrow table output from Kiara module to Pandas from within the notebook and play with ideas from there, it's not a problem.

[makkus]

Ah, sorry, I misunderstood. This is probably a good as any example for what I was talking about, in terms of the kiara API by itself not being sufficient to have a good user experience when using it from Jupyter.

What we could do here is record a requirement (or multiple) around how users should be able to preview values (this would be specific to the table datatype, but I'd guess we'd have to do the same exercise for all the other datatypes). Then I can add helper methods to the KiaraTable class if we deem it a good idea (we might not want to clutter it with too many methods, but that would be a case-by-case decision). In this instance, we could for example add a head and tailmethod, analog what the DataFrame class has, that would return the head/tail of a KiaraTable, again as KiaraTable, and the preview would work as it does with the DataFrame class.

So, it's possible, but whether or not we would want to support it would be a decision the product owner of the Jupyter client product would determine.

[MariellaCC]

Ok great, thanks. Then I will experiment by converting to pandas for now, just for playing with ideas and getting feedback from the team on what may be needed/useful or not.