-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Transition all size variables to ssize_t
#592
Comments
Avoiding the bigfile.
|
Actually I don't reprod your test case :-(
I don't reprod either with production (ubuntu 22.04)
Indeed the reported wc is completly wrong but it doesn't core dump. |
Strange, this is using a Red Hat supplied package (ksh-1.0.4 from https://kojipkgs.fedoraproject.org//packages/ksh/1.0.4/1.fc38/x86_64/ksh-1.0.4-1.fc38.x86_64.rpm) and this happens on a clean install on a Fedora 38 (Rawhide) container as well as on Fedora 37, and reproduces every time. I’ll try rebuilding from source myself later just to rule out a bad package. |
May be RH setup on your vm has some limit (ulimit -a) not handled by ksh (mem, swap exhaustion) leading to seg fault ? |
I haven’t had a chance to look at it further, but that’s not the issue …
|
I get this one...
But not the bigfile one. Is my bigfile not "big" enough?
Well, SOMETHING happens if I make it big enough..
macOS 12.6.2 on an M1Pro. |
The printf version seems another issue (simple wrap around on buf size), and can easily be avoided. So the real problem is with bigfile loading, I took care to set a bigfile with pure ascii data (no zero filled holes). I still don't have segfault on fresh installed fedora (x86_64), yet I got resource problem with the cat version even though I got plenty of space (and swap space).
This cat: error is strange since I thought I got plenty of space. doing < instead of cat.
No errors, no seg fault. |
On fedora fresh install in a VM guest I got a problem with the 'cat' version of your test case
This is due to ksh93 using /dev/shm to implement pipe (instead of pipe(2)). /dev/shm is implemented on top of tmpfs, and then is limited to 50% of you main memory, in my case my VM guest is 4Gb then /dev/shm is 2Gb, the size where I got the 'No space left on device' though I got plenty of space on my discs.
To get rid of this you can umount this /dev/shm But then pipes ends up in /tmp, still no joy.
To get rid of this one comment out the [mount] section in /usr/lib/systemd/system/tmp.mount and reboot the VM
Note that tmpfs DON'T overflow to the swap device, as I did thought, at some point in time /tmp was a ram disk spilling to the swap device, and then the tmpfs got me by surprise. I think that ksh should catch the error on the reading part of the pipe, i.e after the cat crash) and tell the user that the crash was due to tmpfs usage (i.e points to user to something to look for). I think as well that a ksh option (or ENV var or both) should be used to tell ksh not to use tmpfs in Generally speaking, tmpfs is a bad good idea specially on system with small main memory, still capable of handling huge files, /tmp as always got an history of being filed up, it as always randomly crashed system when mishandled, tmpfs just exacerbate it. Generally, /tmp should handle small short lived files, and those file got to be cleaned up to recover FS space, often this is done by unlink them for auto clean at process exit, and still if not cleaned a reboot may do so, but for system that don't reboot... So bottom line, using tmpfs for long lived file that can be big is not good and is a misuse/abuse of tmpfs and thats what ksh does. The purpose of pipe(2) was to exactly reduce the filesystem usage between 2 process, the consumer would produce a buffer, the consumer would grab it, then tell the consumer it can trash (reuse) its buffer to produce the next buffer, etc... the tmpfs usage to implement ksh pipe is bad because it is equivalent to make the producer produce its whole file, then the consumer read it (not exactly the same, but at least produced data is never cleaned up) One could argue that the tmpfs pipe implementation avoid two buffer copy between userland andkernel, in this case ksh should use some heuristics to decide when to go the tmpfs way when the producer will not produce massive output, for instance $(echo blah) is ok with tmpfs. So before cutting patch on this a discussion must be done to figure out all the option available. |
I’ll look into it a bit more later now that we’re in January, but the system I’ve been testing on has 128GB of memory, so the tmpfs limit isn’t a factor here, but all of this is a good observation.
|
Yes the 'cat bigfile' case failure on 'small' system may be has nothing to do with your segfault situation. The fact is that ubuntu and fedora don't behave the same at the moment on my side. May be ksh is just missing some more helpfull error message, it took me age to converge to tmpfs on the first EDIT: |
Clearly, the problem is due to |
ssize_t
Related (?) problem: AST
|
Reproducer:
bigfile
contains ASCII text; this is ksh93 1.0.4 on a 64-bit Linux system.The text was updated successfully, but these errors were encountered: