Adding a condition to evaluations before logging?

[Aleks-Petkov]

Hi! I'm using the python client and it is working very well, but I'm experiencing one issue that I would love to resolve. Namely, when evaluating some very long list (~30k elements), nvim freezes completely as it attempts to log it to the log buffer. My question is if/how one could log only e.g. the first 10 elements if the evaluated expression is a list over a given size?

Sorry if this has been asked before or if it is a trivial question, I'm very new to nvim in general. Any help would be greatly appreciated!

[Olical]

So the Clojure client has this because the REPL protocol does the trimming for us, I have a feeling a couple of other REPLs do too but not all of them, so this is quite a common problem.

Right now I think all you can do is wrap your calls in a call to some function like take which will only return the first n items from a given list, that way you can cap your output on expressions that you know will return a lot of items.

The was we could "fix" this in the Python client specifically would be to wrap every evaluation you perform in more code that will check the result type and try to trim it down if it's too big, but then we also need a way to say "ignore that rule, just give me everything even if it's slow".

I also worry that by introducing wrapping code we'll potentially break some code you're trying to evaluate in unintended ways. Although maybe it's worth the risk... what we'd have to do is extend this code so we had two paths, one for wrapped with a limiter and one without (which is what we have right now).

conjure/fnl/conjure/client/python/stdio.fnl

Lines 114 to 122 in dfb9f75

	(defn- get-exec-str
	[s]
	(.. "import base64\nexec(base64.b64decode('" (b64.encode s) "'))\n"))
	(defn- prep-code [s]
	(let [python-expr (str-is-python-expr? s)]
	(if python-expr
	(.. s "\n")
	(get-exec-str s))))

Then we'd need to update the mappings or add some config so we either use the truncating version or non-truncating, it'd probably have to be two mappings, or maybe an option you can easily set to true while working if you absolutely need to see more data. The Clojure client has limits that you can raise or remove for example if you need to turn off nREPL truncation for whatever reason.

What do you think, would it be easy to write some Python that would look at a value, truncate it if it's too big then return that truncated result, maybe with some warnings attached to show that data had been truncated? That's what I'd struggle with, writing the Python to do that nicely with all data types we can think of. I suppose it'd mostly be lists and dictionaries?

[Aleks-Petkov]

Thanks for the reply! A truncation function in Python sounds doable enough! When I have time (which I suspect won't be anytime soon) I could write it and re-open this discussion if it isn't implemented by anyone else until then, and perhaps get some help on how to actually use the truncation function in the client.

[russtoku]

To expand this discussion a bit, would a

truncation function in Python

be a replacement of sys.__displayhook__ which is the default implementation for sys.displayhook?

The Python version 3.8 docs for sys.displayhook show a pseudo-code implementation of sys.displayhook. That could be used as the basis for a truncation function which might use reprlib.

reprlib was introduced in Python version 3.2 and provides an alternate repr that shortens the displayed representation of various objects. For example, a large list

>>> alist
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]

would be rendered like:

>>> reprlib.aRepr.repr(alist)
'[0, 1, 2, 3, 4, 5, ...]'

The truncation function could provide some global variable that toggles between the different representations. This toggle could be flipped by a key mapping for the Conjure Python client.

Another detail would involve how to add the truncation function to the Python REPL that the Conjure Python client is connected to.

[russtoku]

I experimented with reprlib in Python 3.8 and copied the pseudo-code from the docs.

import sys, builtins, reprlib                                                            
                                                                                         
def my_displayhook(value):                                                               
  if value is None:                                                                      
    return                                                                               
  builtins._ = None                                                                      
  text = reprlib.repr(value)                                                             
  try:                                                                                   
    sys.stdout.write(text)                                                               
  except UnicodeEncodeError:                                                             
    bytes = text.encode(sys.stdout.encoding, 'backslashreplace')                         
    if hasattr(sys.stdout, 'buffer'):                                                    
      sys.stdout.buffer.write(bytes)                                                     
    else:                                                                                
      text = bytes.decode(sys.stdout.encoding, 'strict')                                 
      sys.stdout.write(text)                                                             
  sys.stdout.write("\n")                                                                 
  builtins._ = value

Here's a go at replacing sys.displayhook:

>>> save_dh = sys.displayhook                                                            
>>> sys.displayhook = my_displayhook                                                     
                                                                                         
# Create variables with long output representations.                                     
>>> alist = [i for i in range(50)]                                                       
>>> adict  = {i : chr(65+i) for i in range(26)}                                          
                                                                                         
# Now with new displayhook.                                                              
>>> alist                                                                                
[0, 1, 2, 3, 4, 5, ...]                                                                  
>>> adict                                                                                
{0: 'A', 1: 'B', 2: 'C', 3: 'D', ...}                                                    
                                                                                         
# Return to regular displayhook.                                                         
>>> sys.displayhook = save_dh

It would be possible to create a key mapping in Neovim that would toggle sys.displayhook between the two displayhooks.