getting rid of readline everywhere (more fun with coproc)

[bkerin]

I tried again with your advice on building bash with multiple coproc and it worked :). So I though maybe it might be time to get rid of readline everywhere, or at least everywhere I spend a lot of time. So here is a start at this for sqlite3:

sqliteServerMagicMarker='sqliteServerMAaaaaGiiICMarker'
sqliteServerMarkerSelect="select \"$sqliteServerMagicMarker\";"
function startSqliteServerCoproc {
  coproc sqliteServer { sqlite3 -interactive $@; }
  # Get and pass greeting lines from sqlite3 through to reassure user
  local reply_line
  read -r reply_line <&"${sqliteServer[0]}"
  echo $reply_line
  read -r reply_line <&"${sqliteServer[0]}"
  echo $reply_line
}
function sendToSqliteServer {
  local awn=${@//$'\n'/}  # Args Without Newlines
  local maybe_sc=
  # Don't automatically semicolonize .directive commands
  if [[ ! $awn =~ ^\. ]]; then
    maybe_sc=';'
  fi
  {
    echo "$awn $maybe_sc";
    echo "select \"$sqliteServerMagicMarker\";";
  } >&"${sqliteServer[1]}";
}
function readFromSqliteServer {
  local reply_line
  read -r reply_line <&"${sqliteServer[0]}"
  [[ $reply_line =~ ^sqlite\> ]] || return 1   # Eat prompt and echoed command
  while true; do
    read -r reply_line <&"${sqliteServer[0]}"
    [ "$reply_line" == "sqlite> $sqliteServerMarkerSelect" ] && break
    echo "$reply_line"
  done;
  while true; do
    read -r reply_line <&"${sqliteServer[0]}"
    [[ $reply_line =~ ^$sqliteServerMagicMarker ]] && break
  done;
}
function sl {
  sendToSqliteServer "$@" && readFromSqliteServer;
}

It works nicely now can say e.g.:

$ startSqliteServerCoproc /tmp/mydb.sqlite3
$ sl '
select * from foo
'

Next would be some sort of latching mode or at least a binding type thingy for sl at start of line that types the '' for me :)

[bkerin]

Mostly as an illustration of the difficulties here's an update on the above effort. It's got not-quite-working sql completion or not-quite-working latched sql mode but not both together :)

It does seem like ble.sh could usefully expand into adjacent readline-using territory pretty easily but more knowledge of it's innards than I've got would be decidedly useful.

sqliteServerDBFile=

function sqliteServerUpdateCompletionWords {
  [ -n "$sqliteServerDBFile" ] || return 1
  [ -r "$sqliteServerDBFile" ] || return 2
  # FIXME: unfortunately neither bash nor blesh let this work on anything
  # other than a word at the very start of a single-quoted string or HERE doc
  # :( I guess some complete -F or something on ble.sh side would be needed
  complete -r sl 2>/dev/null
  complete                                                                    \
    -W "$(~/.helper_scripts/sqlite3_db_completion_words $sqliteServerDBFile)" \
    sl
}

sqliteServerMagicMarker='sqliteServerMAaaaaGiiICMarker'
sqliteServerMarkerPrinter=".print $sqliteServerMagicMarker"

function startSqliteServerCoproc {
  coproc sqliteServer { sqlite3 -interactive $@; }
  # WARNING: comment these out again to get better notice what's up with
  # coprocesses (but then C-q etc. don't work as usual anymore)
  disown $sqliteServer_PID
  # Get and pass greeting lines from sqlite3 through to reassure user
  local reply_line
  read -r reply_line <&"${sqliteServer[0]}"
  echo $reply_line
  read -r reply_line <&"${sqliteServer[0]}"
  echo $reply_line

  # The first non-option argument must be the db file
  local dbf=
  for arg in $@; do
    if [[ $arg =~ ^[^-] ]]; then
      sqliteServerDBFile=$arg
    fi
  done

  sqliteServerUpdateCompletionWords || return 1;
}

function sendToSqliteServer {

  local awn=${@//$'\n'/}  # Args Without Newlines

  local maybe_sc=
  # Don't automatically semicolonize .directive commands
  if [[ ! $awn =~ ^\. ]]; then
    maybe_sc=';'
  fi

  {
    echo "$awn $maybe_sc";
    echo "$sqliteServerMarkerPrinter";
  } >&"${sqliteServer[1]}";
}

function readFromSqliteServer {
  local reply_line

  read -r reply_line <&"${sqliteServer[0]}"
  [[ $reply_line =~ ^sqlite\> ]] || return 1   # Eat prompt and echoed command

  while true; do
    read -r reply_line <&"${sqliteServer[0]}"
    [ "$reply_line" == "sqlite> $sqliteServerMarkerPrinter" ] && break
    echo "$reply_line"
  done

  while true; do
    read -r reply_line <&"${sqliteServer[0]}"
    [[ $reply_line =~ ^$sqliteServerMagicMarker ]] && break
  done
}

function sl {
  # This is the function to use to enter queries on the command line
  sendToSqliteServer "$@" && readFromSqliteServer
  sqliteServerUpdateCompletionWords
}

where the completion word probe script sqlite3_db_completion_words is just:

#!/usr/bin/perl -w

# Given an sqlite3 db file, probe it with some sqlite3 commands to determine
# a set of completion words of interest associated with it (table and column
# names) and output that list of words one per line on STDOUT.
#
# The shipping sqlite3 client doesn't seem to consider context at all when
# completing but just lumps all table names and column names (but strangely
# not all virtual table column names) together in a single pool.  If the
# table and column names are sane this works well enough and that's what
# this script is intended to support for contexts other than the sqlite3
# client (excep we also get virtual table column names :).

# vim:foldmethod=marker

# Pragmas and Modules # {{{1

use strict;
use strictures;

use v5.30;
use feature 'signatures';
no warnings qw(experimental::signatures);

use Carp;
use Data::Dumper;

# }}}1

sub sysck ($cmd) # {{{1
{
    # Like system() but the error check is integrated

    not system($cmd) or confess "command '$cmd' returned non-zero";
} # }}}1
sub btck ($cmd) # {{{1
{
    # Like backticks but the error check and chomp() are integrated

    my $result = `$cmd`;
    $? == 0 or confess "backtick command '$cmd' returned non-zero";
    chomp($result);
    return $result;
} # }}}1

@ARGV == 1 or die "wrong number of arguments";

my $dbf = $ARGV[0];   # DataBase File

sub dbc ($cmd) # {{{1
{
  # DataBase Command
  my $po = btck("sqlite3 -line $dbf '$cmd' 2>&1");   # Probe Output
  # The chomp in out btck() might have removed the \n here so we only remove it
  # if it's present and don't complain if it isn't:
  $po =~ s/^-- Loading resources from .*\n?// or confess "$po";
  return $po;
} # }}}1
sub tables () # Ref to list of tables/views/(and indexes/triggers also?) {{{1
{
  my $co = dbc('.tables');   # Command Output
  $co =~ s/\s+/ /g;
  return [ split(' ', $co) ]
} # }}}1
sub columns ($table) # Ref to column names of given table/view) {{{1
{
  my $co = dbc("PRAGMA table_info($table)");
  my @rw = map { m/^\s*name\s*=\s*(\w+)\s*$/ ? $1 : () } split("\n", $co);
  return \@rw;
} # }}}1

my @tables = @{tables()};

my @columns = map { @{columns($_)} } @tables;

my @sorted_unique_words = sort keys %{{map { $_ => 1 } (@tables, @columns)}};

print join("\n", @sorted_unique_words)."\n";

And the latching mode:

# If defined non-empty commands should go to the sqlite coproc
sqliteMode=

function esm { sqliteMode=defined_nonempty; }   # Enter sqliteMode
function lsm { sqliteMode=; }                   # Leave sqliteMode

function my/maybeSetPromptForSqliteMode.hook {
  if [ -n "$sqliteMode" ]; then
    PS1='mySqlite> '
  else
    PS1='$ '
  fi
}
blehook PRECMD=my/maybeSetPromptForSqliteMode.hook

# FIXME: this is pretty pathological in general, and unfinished.  For example
# double quotes presumably don't make it through to sqlite correctly,
# complete currently doesn't work because no leading sl (I think we could
# just nuke all completions and replace with a new command-independent
# set consisting of just the ones currently used for sl command) 'select'
# is a bash keyword that by itself kicks off multi-line mode in ble.sh
# (which is sort of convenient :) but SELECT isn't (though with the newline
# replacement here multi-line mode can be forced with a string literal with
# only a newline), etc.  Modifying _ble_edit_exe_BASH_COMMAND_eval looks
# pretty unsuppored and risky.  ble/widget/vi_imap/accept-single-line-or
# (and similar stuff for vi_nmap) could be changed to make multi-line mode
# the default, and maybe intelligently terminate if a trailing semicolon is
# found (though really probably better to just require C-j as elsewhere in
# ble.sh) I think a wide swath of ble.sh would need to be rewired to make
# this work well.
function my/maybeChangeCommandToSendToSqliteServer.hook {
  if [ -n "$sqliteMode" ]; then
    local ibc="$_ble_edit_exec_BASH_COMMAND_eval";   # Impending "BASH_COMMAND"
    ibc="${ibc//$'\n'/ }"   # Replace newlines with spaces
    local space=$' \t'
    local rex='^['${space}']*lsm['${space}']*$'
    if [[ $ibc =~ $rex ]]; then
      sqliteMode=
      return 0
    fi
    _ble_edit_exec_BASH_COMMAND_eval='sl '"\"$ibc\""
  fi
}
blehook PREEXEC=my/maybeChangeCommandToSendToSqliteServer.hook

[akinomyoga]

function sqliteServerUpdateCompletionWords {
  [ -n "$sqliteServerDBFile" ] || return 1
  [ -r "$sqliteServerDBFile" ] || return 2
  # FIXME: unfortunately neither bash nor blesh let this work on anything
  # other than a word at the very start of a single-quoted string or HERE doc

What is the actual command line where this doesn't work? For example, I think the following wouldn't work:

$ sl 'SELECT col[TAB]

but I guess the following will still work as far as the column names are generated by sqlite3_db_completion_words:

$ sl SELECT col[TAB]

  # :( I guess some complete -F or something on ble.sh side would be needed

To make the first version (the completion inside a single argument e.g. 'SELECT col[TAB]') work, in a complete -F function, you need to split the content of the current word (e.g. 'SELECT col') into the sub-words therein (e.g. SELECT & col) and perform the completion for the sub-word (e.g. col).

  complete -r sl 2>/dev/null
  complete                                                                    \
    -W "$(~/.helper_scripts/sqlite3_db_completion_words $sqliteServerDBFile)" \
    sl
}

The expansions in the option argument of -W will be evaluated when the completion is attempted. For example, you can specify it in the following way (please notice that $(...) is enclosed within single quotations as '$(...)') so you do not need to update the completion settings every time there is a change in the database:

complete -W '$(~/.helper_scripts/sqlite3_db_completion_words "$sqliteServerDBFile")' sl

---

# FIXME: this is pretty pathological in general, and unfinished.

I actually have a similar idea with gnuplot. Gnuplot is a graph plotting program and can be used interactively. I often use gnuplot for my work, but it is troublesome to frequently switch between the Bash prompt and the Gnuplot prompt. Also, I would like to use the enhanced line editor for the Gnuplot commands. I implemented ble/util/bgproc in #302 (comment), but I actually already had a plan to implement ble/util/bgproc as a part of the gnuplot implementation.

My naive idea is to switch the mode for the command line starting with a configurable prefix. For example, if we configure the prefix as +, when the command line is accepted by RET,

$ + plot 'data.txt' u 1:($2*10**2) w l

it would be processed by sending the string plot 'data.txt' u 1:($2+sqrt()) w l to the background Gnuplot process.

I think your sqlite3 case can also be processed in the same way as my gnuplot plan. Or maybe you are now thinking of the mode where any command lines are processed with the background server.

# For example
# double quotes presumably don't make it through to sqlite correctly,

You can use ble/string#quote-command as described later.

# complete currently doesn't work because no leading sl (I think we could
# just nuke all completions and replace with a new command-independent
# set consisting of just the ones currently used for sl command) 'select'
# is a bash keyword that by itself kicks off multi-line mode in ble.sh
# (which is sort of convenient :) but SELECT isn't (though with the newline
# replacement here multi-line mode can be forced with a string literal with
# only a newline), etc.

In the gnuplot plan, the prefix + would switch the syntax used to analyze the command line for highlighting. The current command-based completion is tightly connected to the Bash syntax. We instead should use the Gnuplot syntax to determine the completion context when we attempt completions for the + command lines.

I think we can also process completions for your sqlite3 case in a similar way. You first define the sqlite3 syntax and then provide the completion context with an appropriate completion source. The provided completion source would generate appropriate words under the context of the current cursor position in the command line. But currently, there is no documentation about how to define a new syntax and/or language under ble/syntax:CustomLanguage/*.

# Modifying _ble_edit_exe_BASH_COMMAND_eval looks
# pretty unsuppored and risky.

Right, _ble_edit_exe_BASH_COMMAND_eval is the current implementation detail and might be changed in the future. Actually, bleopt internal_exec_type is related, but this is also undocumented. You are supposed to define a shell function ble-edit/exec:sqliteMode/process, which processes _ble_edit_exec_lines in a way similar to ble-edit/exec:gexec/process. But again, internal_exec_type and _ble_edit_exec_lines are internal stuff and might be changed in the future.

# ble/widget/vi_imap/accept-single-line-or
# (and similar stuff for vi_nmap) could be changed to make multi-line mode
# the default, and maybe intelligently terminate if a trailing semicolon is
# found (though really probably better to just require C-j as elsewhere in
# ble.sh)

The termination of the command line is supposed to be judged by ble/syntax:$_ble_syntax_lang/is-complete (though currently _ble_syntax_lang is fixed to be bash, and thus ble/syntax:bash/is-complete is hardcoded in ble/widget/accept-single-line-or).

# I think a wide swath of ble.sh would need to be rewired to make
# this work well.
function my/maybeChangeCommandToSendToSqliteServer.hook {
  if [ -n "$sqliteMode" ]; then
    local ibc="$_ble_edit_exec_BASH_COMMAND_eval";   # Impending "BASH_COMMAND"
    ibc="${ibc//$'\n'/ }"   # Replace newlines with spaces
    local space=$' \t'
    local rex='^['${space}']*lsm['${space}']*$'
    if [[ $ibc =~ $rex ]]; then
      sqliteMode=
      return 0
    fi
    _ble_edit_exec_BASH_COMMAND_eval='sl '"\"$ibc\""

You can use ble/string#quote-command for proper quoting (in case the word might contain an arbitrary character).

local ret
ble/string#quote-command sl "$ibc"
_ble_edit_exec_BASH_COMMAND_eval=$ret

  fi
}
blehook PREEXEC=my/maybeChangeCommandToSendToSqliteServer.hook

[bkerin]

Regarding completion yes it's the single-quote version (and HERE docs) that don't work as is. I think these must work as the additional load of thinking about shell metachars is too much (at least for me) while considering sql query. I'll try with complete -F later.

Regarding prefix (e.g. + or sl) I now think you're right it's the way to go here (versus e.g. latched sqlite mode). Latched mode is just a habit from the existing client and the first thing one wants to do with better integration is to e.g. pipe to grep (in theory possible from db side of course but for whatever reason it often seems simpler when investigating to use an after-the-fact filter). The prefix sl is not much to type or remember, is all up-front, and is often present in history during the refining process anyway. I expect most of this is true of gnuplot and probably most other REPL with commands complicated enough to be use cases here.

Regarding pipes I couldn't figure out how to get them to work with bash coproc because it closes the pipe ends in children. So I've now tried the util.bgproc you made and it works nicely :) The only issue is for some reason it seems to make the final exit from a ble.sh shell quite a lot slower. If there's some way to tell bgproc to die faster I didn't find it.

One other note on a different topic: I didn't know about the nice comments in e.g. util.bgproc because I just looked at installed sources where they're apparently compiled out for speed. As looking at/instrumenting installed scripts is a pretty common way to start with devel for scripts it might be worth compiling in a header comment directing user to original sources, or maybe even a mode to include commented version for easy reading while instrumenting in-place (unfortunately I can't read the Japanese so it might not help me much but maybe others :)

[bkerin]

... I changed the interval of the bgproc termination check to a shorter one in commit 8d623c1 (where the interval was originally 1 sec). Does this change the behavior in your setup?

No. In my case it seems related to an issue that first hit for me with linux mint: when I exit a terminal that has any background program running from it, it takes the extra time to exit. My previous linux on the same laptop didn't do that.

I think the terminal you use also affects the behavior. I'm not sure how each terminal judges the termination of the session, but there are possibly many ways to detect, for example, by looking at whether the pty masters are still alive, by detecting the termination of the shell process, or by checking the processes that belong to the session of the shell. Depending on the way the terminal choose, I think the behavior can be different. What are your terminals in Linux Mint and in the previous Linux?

You're right it's the terminal. I checked several of them and they break down as follows:

Slow to exit when backgrounded process:

•  gnome-terminal
  

•  xfce4-terminal
  

•  terminator
  

•  tilda
  

•  tilix
  

•  lxterminal
  

Not slow when backgrounded process:

•  xterm
  

•  rxvt
  

•  st (suckless.org)
  

•  yakuake        
  

•  terminology                                                           
  

If it doesn't change, does the behavior change if you specify the option kill-timeout=0 to ble/util/bgproc#open? [ Note: `kill-timeout=0' sends SIGTERM to the background process. This may forcibly stop the background process and cause problems. ]

Neither of these make any difference either. The explicit send of .exit I would expect to resolve the issue for sure for the sqlite process in particular,
so I wonder if it's the supervisor process you mention above that might be triggering this issue?

That was my initial suspicion and already added a workaround for that in 8d623c1 as I replied in my previous reply. The background process that checks the termination of the bgproc process immediately exits when it detects that all the bgproc processes have terminated (including the child processes that are started by the main bgproc process). If that commit doesn't change the behavior, it is either the problem is unrelated to the background process or there is some problem in tha background process.

I checked the bgproc#post .exit again, and it looks like I screwed up last time because it does seem to prevent the problem. Sorry for this mistake.

Hmm, or another possibility might be there are even other processes started by the bgproc processes that are ignoring SIGTERM and surviving for a long time.

May I ask what is the order of the delay length? Is it about ten seconds or more? Or is it about a few seconds? Or is it less than a second? Does the problem occur with any setup of bgproc or with a specific setup (such as pipes or sqlite3)?

It's about 2 seconds. Any existing background process seems to cause the issue including both the sqlite3 server and perlRegexServer (and vim, browser etc.).

Armed with your previous advice on how to use onstop and realization that I messed up the previous test on sqlite3 server I tried this for the perlRegexServer:

function ble/util/bgproc/onstop:perlRegexServer {
  kill ${perlRegexServer_bgproc[4]}
}

and this seems to do the job. Looking quickly at #stop it appears that it works by first closing the pipes with the expectation that this will end up killing the underlying process. I guess this is this expected to happen via SIGPIPE? I guess this must be the clean and normal way, but for myself I wouldn't generally have any expectation that an abandoned child would be killed cleanly and might prefer the apparently quicker SIGTERM (at least in the absence of an onstop handler).

Out of curiosity, does you setup exhibit a difference in time required to exit when you exit a terminal from which a still-existing other process (e.g. vim, browser) has been started? As I mentioned this irritating behavior is new for me, maybe it's not universal.

I now tried different setups. XTerm and my terminal closes its window relatively quickly. However, GNU screen doesn't close the window soon. It waits for several seconds when there is a running background process. ssh will not close forever as far as there is a background process (unless the user press C-c). It depends on the terminal as I expected.

Yes. So of course none of this is really ble.sh fault and it's impacted only in a very minor way with a different performance in exit speed depending on terminal emulator if an (as yet generally unused) library module is used. It seems a bit fussy to worry over a 2 second exit speed change but that sort of thing can be significant in UI and bgproc is otherwise very nice and offers quite detailed control in it's options. It might be worth an option to clean up the processes with an immediate SIGTERM, or perhaps just a paragraph in the util.bgproc.sh comments mentioning the issue and possible solution.

[akinomyoga]

Looking quickly at #stop it appears that it works by first closing the pipes with the expectation that this will end up killing the underlying process.

Yes, that is the primary expectation of the ble/util/bgproc implementation.

However, if there are other background programs that inherit and hold the file descriptors, the bgproc process will not receive any errors until all the other such processes terminate. I think this is somewhat related or similar to the deadlock situation that Chet worried about for MULTIPLE_COPROCS.

I guess this is this expected to happen via SIGPIPE?

Yes. More precisely, I think there are two cases. If the background process waits for the next request from the main Bash process when the pipes are closed, the background process will read EOF and then considers that the session is closed. If the background process is trying to write something to the pipe in response when the pipes are closed, the background process will receive `SIGPIPE. I guess the former would be more likely to happen in real situations.

It might be worth an option to clean up the processes with an immediate SIGTERM, or perhaps just a paragraph in the util.bgproc.sh comments mentioning the issue and possible solution.

kill-timeout=0 that I mentioned in #309 (reply in thread) is supposed to be the one.

#309 (reply in thread)
If it doesn't change, does the behavior change if you specify the option kill-timeout=0 to ble/util/bgproc#open? [ Note: `kill-timeout=0' sends SIGTERM to the background process. This may forcibly stop the background process and cause problems. ]

Doesn't kill-timeout=0 work? Maybe the bgproc process still survives after sending SIGTERM. In that case, the supervisor process will still wait for the termination of the bgproc process for ten seconds, and if the bgproc process doesn't end in ten seconds, the supervisor process sends SIGKILL in the current implementation of bgproc.

There is a possibility that the bgproc process ignores SIGTERM or does not respond to it immediately, for example, for the reason that it masks the signal for some IO. Or maybe for the reason that it installs a signal handler or SIG_IGN to SIGTERM.

Anyway, the background process will be forcibly terminated after ten seconds by SIGKILL. I'm not sure if it is a good idea to immediately send SIGKILL, but maybe I can add an option to change the timeout for SIGKILL. When the process doesn't immediately die on the pipe errors or on SIGTERM, there can be a reason such as that the process needs to clean up states on a disk, etc. Forcibly disabling such a treatment might result in a broken state of the system.

[akinomyoga]

However, if there are other background programs that inherit and hold the file descriptors,

Ah, maybe I now know the cause. If you just execute the command by ble/util/bgproc, the bgproc process (such as sqlite3) is created as a child of the subshell. Of course, the subshell holds the file descriptors and doesn't exit until the child process sqlite3 terminates, so sqlite3 will not receive any pipe errors. This is a kind of deadlock. In my case, I use exec to start the sqlite3 session, which I think is the reason that I didn't experience the deadlock.

You can include exec in the command, but maybe I should add the exec option to ble/util/bgproc#open (Hmm, but that just seems like prepending exec to the specified command string). Note that it cannot be the default setting because exec can only be applied to the external command name with arguments, but the interface of ble/util/bgproc accepts an arbitrary shell command (including compound constructs), and it doesn't receive the command name and the arguments separately.

[bkerin]

Looking quickly at #stop it appears that it works by first closing the pipes with the expectation that this will end up killing the underlying process.

Yes, that is the primary expectation of the ble/util/bgproc implementation.

However, if there are other background programs that inherit and hold the file descriptors, the bgproc process will not receive any errors until all the other such processes terminate. I think this is somewhat related or similar to the deadlock situation that Chet worried about for MULTIPLE_COPROCS.

I guess this is this expected to happen via SIGPIPE?

Yes. More precisely, I think there are two cases. If the background process waits for the next request from the main Bash process when the pipes are closed, the background process will read EOF and then considers that the session is closed. If the background process is trying to write something to the pipe in response when the pipes are closed, the background process will receive `SIGPIPE. I guess the former would be more likely to happen in real situations.

Sounds reasonable. I went through and made a pretty good effort to make sure eof in the perl regex server would result in an immediate exit and this didn't seem to make a difference.

It might be worth an option to clean up the processes with an immediate SIGTERM, or perhaps just a paragraph in the util.bgproc.sh comments mentioning the issue and possible solution.

kill-timeout=0 that I mentioned in #309 (reply in thread) is supposed to be the one.

#309 (reply in thread)
If it doesn't change, does the behavior change if you specify the option kill-timeout=0 to ble/util/bgproc#open? [ Note: `kill-timeout=0' sends SIGTERM to the background process. This may forcibly stop the background process and cause problems. ]

Doesn't kill-timeout=0 work? Maybe the bgproc process still survives after sending SIGTERM. In that case, the supervisor process will still wait for the termination of the bgproc process for ten seconds, and if the bgproc process doesn't end in ten seconds, the supervisor process sends SIGKILL in the current implementation of bgproc.

It doesn't seem to make a difference for this issue. I verified with some instrumentation in bgproc#stop that it sets $close_timeout.

There is a possibility that the bgproc process ignores SIGTERM or does not respond to it immediately, for example, for the
reason that it masks the signal for some IO. Or maybe for the reason that it installs a signal handler or SIG_IGN to SIGTERM.

I don't think it's this since the small perl regex server doesn't change the signal disposition at all and as I said it does seem to die immediately when given kill ${perlRegexServer_bgproc[4]} in the onstop.

Looking at bgprog#stop it doesn't appear that much happens between the onstop hook call and

(ble/util/nohup 'ble/util/bgproc#stop/.kill "-${bgproc[4]}" "$close_timeout"')

I haven't examined the details of that last command (one oddity I notice is it appears to be using a negative pid meaning group signaling?), but I did check with some crude instrumentation that the fd#close calls between are not the issue.

[bkerin]

I think I've found some small bugs in ble/util/conditional-sync that make kill-timeout=0 not work right, I'll submit a patch shortly

[bkerin]

Carrying on I set out to make completion work inside HERE doc in this:

$ sl "$(cat <<'END_HERE'                                                                                                                                                                            
select "foo"
END_HERE
)" | grep foo
"foo"
foo

This finally has the desired properties that it's immune to bash syntax and can be used in pipelines etc. Unfortunately it looks like complete -F functions don't run at all inside HERE documents. I guess this isn't too shocking. So I wondered how to work around
this using ble.sh. I'm not sure where to start, it looks like it might be required to wrap the complete widget?

[bkerin]

Reasonably usable hack for automatic insertion of HERE doc on typing e.g. 'sl ' looks like:

ble-sabbrev sl='sl "$(cat <<'"'"'END_SQL'"'"$'\n'END_SQL$'\n'')"'

function ble/widget/my/magic-space-wrapper { # {{{3

  # Get Current word (word right before cursor)
  local rex='[0-9A-Za-z]+$'  
  [[ ${_ble_edit_str:0:$_ble_edit_ind} =~ $rex ]]
  local cword=${BASH_REMATCH[0]}

  ble/widget/magic-space

  # If $cword is our magic SQL-HERE-doc-initiation thing positon in HERE doc
  if [ "$cword" = "sl" ]; then
    ble/widget/vi_imap/normal-mode
    ble/widget/backward-line
    ble/widget/backward-line
    ble/widget/vi_nmap/insert-mode-at-forward-line
  fi

} # }}}3

ble-bind -m 'vi_imap' -f 'SP' 'my/magic-space-wrapper'

Now the only thing missing compared to sqlite3 client it completion inside the HERE doc

[bkerin]

In https://github.com/bkerin/ble.sh are changes to lib/core-complete.sh that allow completion of sqlite3 meta-commands/tables/columns inside the HERE doc to work (together with functions from my .bashrc). It's a bit of a hack and I haven't come close to understanding all of core-complete.sh, but I have the following theories in which I have varying degrees of confidence:

1. Upstream ble.sh quits parsing inside HERE docs
2. Due to 1, the native bash completion support doesn't end up firing (it's been a while I'm not sure on this one)
3. Native bash completion is pretty useless for this anyway with it's lack of full command line context and non-support for multi-line
4. Due to 1 it's also not possible to splice in a better replacement in the same source location where native bash completion happens due to 1

So I've found a place to splice in custom completion before any of the ble.sh syntax pass happens. Based on past conversations I guess there are probably other places where ble.sh doesn't fully parse and these are likely to be the ones where full context-sensitive completion might be desired. It does seem a shame to invite users to make their own poor reinventions of the sophisticated command detection/expansion/etc. that are already happening in core-complete.sh, but that stuff is pretty hard to fully learn and in
many situations user will be able to do whatever small thing they're imagining well enough for their own purposes from scratch. Because of this and the fact that native bash completion is so limited I think adding a hook something like what's in this branch would be worthwhile (while of course making sure it doesn't get used for regular devel, maybe not even in contrib).

Remaining issues: it would be nice if when an sl command is loaded from history the cursor could be positioned at the point it was when C-j got pushed. Maybe there's already some way to do this though.

[akinomyoga]

In https://github.com/bkerin/ble.sh are changes to lib/core-complete.sh

If you are fine with just calling a specific function for here documents, you do not have to modify ble.sh. I think you can just put the following code in blerc (I haven't tested it actually, so there would need to be some adjustments):

_ble_syntax_bash_complete_check_prefix[_ble_ctx_HERE0]=heredoc
_ble_syntax_bash_complete_check_prefix[_ble_ctx_HERE1]=heredoc
function ble/syntax/completion-context/.check-prefix/ctx:heredoc {
  local p=$((wlen>=0?wbeg:istat))
  ble/syntax/completion-context/.add myHeredocCompleter "$p"
}

function ble/complete/source:myHeredocCompleter {
  customCompleterAdapter # the function you have in your copy of ble.sh
}

• Upstream ble.sh quits parsing inside HERE docs

I'm not sure what exactly you mean by quitting parsing, but ble.sh (specifically core-syntax.sh) does recognize here documents and it is possible to retrieve the information. core-complete.sh references the AST constructed by core-syntax.sh. However, it is true that the contents of here documents are not subject to parsing because there is really no syntax there.

• Due to 1, the native bash completion support doesn't end up firing (it's been a while I'm not sure on this one)

The reason that the native Bash completion is not fired on here documents is unrelated to the parsing of here documents. It's related to the interface. The native Bash completion interface passes contexts through COMP_WORDS, COMP_CWORD, etc. as command line arguments, and existing bash-completion scripts mainly handle COMP_WORDS and COMP_CWORD. There is no way to pass the context of here documents. I don't think it is right to invoke the native bash completion for here documents. It anyway performs the normal argument completion instead of the completion of here documents. There isn't anything I can do because it is the decision by the upstream Bash and unrelated to ble.sh.

However, there is no reason to stick with the native Bash completion interface.

• Native bash completion is pretty useless for this anyway with it's lack of full command line context and non-support for multi-line

Yes.

• Due to 1 it's also not possible to splice in a better replacement in the same source location where native bash completion happens due to 1

This is also an issue, but the position inside the here document can be specified by the completer. You seem to do it by overwriting COMP1, COMP2, COMPS, and COMPV, but I haven't thought deeply about whether it works robustly.

Based on past conversations I guess there are probably other places where ble.sh doesn't fully parse and these are likely to be the ones where full context-sensitive completion might be desired.

What's that? I don't think there is a shell syntax that ble.sh doesn't parse. But, of course, it doesn't parse e.g. inside the arguments which is unspecified by the shell language. For example, it doesn't parse, awk scripts, sed scripts, bc, perl, python, etc. Each completer should take care of it.

---

Remaining issues: it would be nice if when an sl command is loaded from history the cursor could be positioned at the point it was when C-j got pushed. Maybe there's already some way to do this though.

Is that really needed? Is there any existing shell behaving in that way? If we try to support it, we need to record and maintain the cursor positions of all the history entries (of at least the current session), and it seems like too much overhead for just supporting it. Is it so useful that it's worth paying the cost?

Maybe what you want is C-o with preserving the cursor position when performed on the latest history entry.

[akinomyoga]

I agree that trying to sneakily extend the bash completion setup isn't a good idea, but it seems like ble.sh could support something more powerful

Yes, it should be technically possible, but it's not so easy to determine a general, powerful, and reasonable interface unless we consider several use cases and examples of completions. So far, I don't have time to enumerate and collect possible use cases to carefully design the interface, so it's not implemented.

---

And in general the good multi-line support invites users to use formatting and do more at the prompt before going to files, and doing that makes maintaining cursor position desirable.

You seem to try to explain the usefulness of preserving the cursor position of each history entry in the above sentence, but I honestly do not understand how it is useful in the above situation.

Maybe what you want is C-o with preserving the cursor position when performed on the latest history entry.

I took a look but it's not obvious to me what single-command-mode is doing

Ah, sorry, C-o in this case is not single-command-mode but accept-and-next (which is the default behavior of the native Bash) (cf Vi (Vim) editing mode -- wiki). Also, the behavior after "with" is not implemented. If you like it, I might add an option to accept-and-next.

[bkerin]

I agree it isn't easy to tell where custom completion should happen. Before parse is a bit of a shame and I only ended up there because of the parse stopping as discussed, and my not being able to figure out easily how to fix it. But the interface should IMO be simple for people to figure out (but just without the pointless limitations of native bash complete). So it might be that many users trying to do custom completion for some special case wouldn't want to learn how to access everything core-complete.sh and core-syntax.sh provide even if it was thoroughly documented and fixed enough for general use (which obviously might be undesirable).

What I mean about the good multi-line support is that it makes it practical to do more directly from the command line, before resorting to scripts stored in files. Maybe multi-line support sometimes helps with long lines all in bash syntax, but for me at least such commands are a bit rare. If the command gets long it's much more likely I'm trying stuff in something from other language (usually SQL or perl for me). In these cases the command execution is likely to be triggered from inside the command rather than at the end (especially in multi-line case where C-j is needed anyway). The normal development pattern then usually involves successive refinements of the embedded sql or perl, perhaps followed by more changes to the shell command if another filter is is involved or something. But usually you're modifying one or the other at least several times in a row before possibly switching to working on the other again. So in the case of long formatted multi-line commands it would be useful if the editing position was remembered.

I never happened to learn accept-and-next, but I can see how it would be useful when repeating a sequence of commands, and how preserving position would be useful when incrementally modifying such a sequence. But this is a slightly different use case than repeatedly refining a single monolithic command, for which it doesn't look like accept-and-next is useful.

[akinomyoga]

I agree it isn't easy to tell where custom completion should happen. [...]. So it might be that many users trying to do custom completion for some special case wouldn't want to learn how to access everything core-complete.sh and core-syntax.sh provide even if it was thoroughly documented and fixed enough for general use (which obviously might be undesirable).

The general, powerful, and reasonable interface includes simplicity because it's an interface. What I'm talking about here is specifically the interface of the here-document completion. Something you are suggesting in bkerin/ble.sh is to pass the completer just the whole line comp_text and comp_index and to provide no other context information even though it is the completion context of here documents. It's unacceptable. It is a simple interface, but this requires users re-parse comp_text and comp_index to do anything the users want to do even though ble.sh already has the parsing result. At least, the command name that receives the redirections by the here document should be included. At least, the range of the here document should be specified.

---

What I mean about the good multi-line support is that it makes it practical to do more directly from the command line, before resorting to scripts stored in files. [...]. But usually you're modifying one or the other at least several times in a row before possibly switching to working on the other again.

After reading the above, I increasingly think that it's sufficient to modify accept-and-next to maintain the position. You say "you're modifying one or the other at least several times in a row", which shows that it's sufficient to remember one or two positions but there is no need to remember the last positions of every line of the entire command history.

I never happened to learn accept-and-next, but I can see how it would be useful when repeating a sequence of commands, and how preserving position would be useful when incrementally modifying such a sequence. But this is a slightly different use case than repeatedly refining a single monolithic command,

It's exactly a use case of accept-and-next. Have you checked the behavior of accept-and-next when it is used at the newest command line?

A (simple) modification to accept-and-next can't maintain the position of the second last entry, but maintaining the position of the last command already improves the editing flow described above. Even if the last position of the second last command is not recorded, in that case, you can just move to the position. ble.sh provides many widgets (including vim motions, etc). Then, I need to think about how we implement it and whether it's worth implementing. If it's too complicated compared to the benefit, I won't add it to ble.sh. For example, I don't like to add the full history recording of the last positions of entries just for this small behavior.

[bkerin]

The general-purpose expansion and parsing that e.g. core-syntax.sh does is hard, but the particular case parsing that users likely want for their particular cases is usually easy. It's a real question whether users would rather learn any interface you might design rather than just roll their own. Command name is useful and easy to interface, but it's also trivial (without expansion) to parse out, and where do you draw the line? HERE doc boundaries are similarly easy to parse, but if you provide an interface with all such possible bits of shell syntax explicitly broken out I think it would be large and/or complicated.

I didn't check behavior of accept-and-next on newest line, but IIRC in the context of an older command it advances to next command which isn't desirable in my case. I didn't describe the case well unfortunately. It's very often not a matter of just repeating the command, because it's frequently necessary to do some investigation between runs of the command under development to determine it's exact effect. At the least this is often enough the case that accept-and-next would only sometimes work and would be a non-homogeneous approach (I normally reverse search back to the command in question when ready for another attempt).

Maybe nobody else does this pattern and it's not worth it for this alone (though I bet it's actually pretty common), but I think last-position is a useful heuristic in general. Someday in the past I added some vim module that caused it to remember file position and it's a big improvement.

In any case I'm still super happy with ble.sh and will be happy to be in it's constellation of fan-forks :)

[bkerin]

In https://github.com/bkerin/ble.sh are changes to lib/core-complete.sh

If you are fine with just calling a specific function for here documents, you do not have to modify ble.sh. I think you can just put the following code in blerc (I haven't tested it actually, so there would need to be some adjustments):

_ble_syntax_bash_complete_check_prefix[_ble_ctx_HERE0]=heredoc
_ble_syntax_bash_complete_check_prefix[_ble_ctx_HERE1]=heredoc
function ble/syntax/completion-context/.check-prefix/ctx:heredoc {
  local p=$((wlen>=0?wbeg:istat))
  ble/syntax/completion-context/.add myHeredocCompleter "$p"
}

function ble/complete/source:myHeredocCompleter {
  customCompleterAdapter # the function you have in your copy of ble.sh
}

Ok I'll try this later.

• Upstream ble.sh quits parsing inside HERE docs

I'm not sure what exactly you mean by quitting parsing, but ble.sh (specifically core-syntax.sh) does recognize here documents and it is possible to retrieve the information. core-complete.sh references the AST constructed by core-syntax.sh. However, it is true that the contents of here documents are not subject to parsing because there is really no syntax there.

I wouldn't expect any parsing inside HERE of course, but at the moment ble/complete/context:syntax/generate-sources
returns non-zero and stops the entire completion process quite early. IIRC when I looked into it it looked like the command name wasn't getting determined. Maybe this behavior is expected when inside HERE because ble.sh doesn't anticipate doing any completion there, but it would be nice if there was some way to do it for user's that want to.

• Due to 1, the native bash completion support doesn't end up firing (it's been a while I'm not sure on this one)

The reason that the native Bash completion is not fired on here documents is unrelated to the parsing of here documents. It's related to the interface. The native Bash completion interface passes contexts through COMP_WORDS, COMP_CWORD, etc. as command line arguments, and existing bash-completion scripts mainly handle COMP_WORDS and COMP_CWORD. There is no way to pass the context of here documents. I don't think it is right to invoke the native bash completion for here documents. It anyway performs the normal argument completion instead of the completion of here documents. There isn't anything I can do because it is the decision by the upstream Bash and unrelated to ble.sh.

I agree that trying to sneakily extend the bash completion setup isn't a good idea, but it seems like ble.sh could support something more powerful (and appropriate to it's stronger multi-line handling at about the same site? But as things are completion doesn't get that far.

However, there is no reason to stick with the native Bash completion interface.

• Native bash completion is pretty useless for this anyway with it's lack of full command line context and non-support for multi-line

Yes.

• Due to 1 it's also not possible to splice in a better replacement in the same source location where native bash completion happens due to 1

This is also an issue, but the position inside the here document can be specified by the completer. You seem to do it by overwriting COMP1, COMP2, COMPS, and COMPV, but I haven't thought deeply about whether it works robustly.

Based on past conversations I guess there are probably other places where ble.sh doesn't fully parse and these are likely to be the ones where full context-sensitive completion might be desired.

What's that? I don't think there is a shell syntax that ble.sh doesn't parse. But, of course, it doesn't parse e.g. inside the arguments which is unspecified by the shell language. For example, it doesn't parse, awk scripts, sed scripts, bc, perl, python, etc. Each completer should take care of it.

Remaining issues: it would be nice if when an sl command is loaded from history the cursor could be positioned at the point it was when C-j got pushed. Maybe there's already some way to do this though.

Is that really needed? Is there any existing shell behaving in that way? If we try to support it, we need to record and maintain the cursor positions of all the history entries (of at least the current session), and it seems like too much overhead for just supporting it. Is it so useful that it's worth paying the cost?

I'm not sure, but in this setup I'm attempting where slave processes are used to allow ble.sh superior vim keys and multiline support it seems like a logical thing to want, since the topic of refinement is likely to be the material in the contained HERE document. And in general the good multi-line support invites users to use formatting and do more at the prompt before going to files, and doing that makes maintaining cursor position desirable. IIRC there's already a hook to rewrite history as it occurs, so this could be achieved only by interested users if there was a retrieval-time hook as well (and ideally a documented way to set the cursor position).

Maybe what you want is C-o with preserving the cursor position when performed on the latest history entry.

I took a look but it's not obvious to me what single-command-mode is doing

[akinomyoga]

Could you reply in the thread?