Replies: 20 comments 11 replies
-
Hi @pdbourke, It should work. I see:
That's libvips 8.15 on ubuntu 23.10.
You can see the load and save options in the API docs, eg: https://www.libvips.org/API/current/VipsForeignSave.html#vips-tiffsave That's the C API, but the command-line interface is generated automatically, so they are always the same. |
Beta Was this translation helpful? Give feedback.
-
My humble apologies, you are right ... the 16 bits are preserved. I
must have been hallucinating.
Thanks for the link to the load and save options.
PS G:\Murten_stitching> vipsheader .\scroll1images_full_v1_16_40ppmm.tif
.\scroll1images_full_v1_16_40ppmm.tif: 1409196x440000 ushort, 3 bands,
rgb16, tiffload
vips resize scroll1images_full_v1_16_40ppmm.tif
scroll1images_full_v1_16_4ppmm.tif 0.1
PS G:\Murten_stitching> vipsheader .\scroll1images_full_v1_16_4ppmm.tif
.\scroll1images_full_v1_8_4ppmm.tif: 140920x44000 ushort, 3 bands,
rgb16, tiffload
…On Tue, 2 Apr 2024 at 10:14, John Cupitt ***@***.***> wrote:
Hi @pdbourke,
It should work. I see:
$ vipsheader x.tif
x.tif: 1450x2048 ushort, 3 bands, rgb16, tiffload
$ vips resize x.tif y.tif 0.1
$ vipsheader y.tif
y.tif: 145x205 ushort, 3 bands, rgb16, tiffload
That's libvips 8.15 on ubuntu 23.10.
vipsthumbnail will make 8-bit images, maybe you used that? resize will keep the format.
You can see the load and save options in the API docs, eg:
https://www.libvips.org/API/current/VipsForeignSave.html#vips-tiffsave
That's the C API, but the command-line interface is generated automatically, so they are always the same.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
Follow up question, how does one change the bit depth from 16 to 8
using the command line? tif or v files.
…On Tue, 2 Apr 2024 at 10:58, Paul Bourke ***@***.***> wrote:
My humble apologies, you are right ... the 16 bits are preserved. I
must have been hallucinating.
Thanks for the link to the load and save options.
PS G:\Murten_stitching> vipsheader .\scroll1images_full_v1_16_40ppmm.tif
.\scroll1images_full_v1_16_40ppmm.tif: 1409196x440000 ushort, 3 bands,
rgb16, tiffload
vips resize scroll1images_full_v1_16_40ppmm.tif
scroll1images_full_v1_16_4ppmm.tif 0.1
PS G:\Murten_stitching> vipsheader .\scroll1images_full_v1_16_4ppmm.tif
.\scroll1images_full_v1_8_4ppmm.tif: 140920x44000 ushort, 3 bands,
rgb16, tiffload
On Tue, 2 Apr 2024 at 10:14, John Cupitt ***@***.***> wrote:
>
> Hi @pdbourke,
>
> It should work. I see:
>
> $ vipsheader x.tif
> x.tif: 1450x2048 ushort, 3 bands, rgb16, tiffload
> $ vips resize x.tif y.tif 0.1
> $ vipsheader y.tif
> y.tif: 145x205 ushort, 3 bands, rgb16, tiffload
>
> That's libvips 8.15 on ubuntu 23.10.
>
> vipsthumbnail will make 8-bit images, maybe you used that? resize will keep the format.
>
> You can see the load and save options in the API docs, eg:
>
> https://www.libvips.org/API/current/VipsForeignSave.html#vips-tiffsave
>
> That's the C API, but the command-line interface is generated automatically, so they are always the same.
>
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you were mentioned.Message ID: ***@***.***>
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
Found it ... "cast"
…On Sat, 13 Apr 2024 at 13:19, Paul Bourke ***@***.***> wrote:
Follow up question, how does one change the bit depth from 16 to 8
using the command line? tif or v files.
On Tue, 2 Apr 2024 at 10:58, Paul Bourke ***@***.***> wrote:
>
> My humble apologies, you are right ... the 16 bits are preserved. I
> must have been hallucinating.
> Thanks for the link to the load and save options.
>
> PS G:\Murten_stitching> vipsheader .\scroll1images_full_v1_16_40ppmm.tif
> .\scroll1images_full_v1_16_40ppmm.tif: 1409196x440000 ushort, 3 bands,
> rgb16, tiffload
>
> vips resize scroll1images_full_v1_16_40ppmm.tif
> scroll1images_full_v1_16_4ppmm.tif 0.1
>
> PS G:\Murten_stitching> vipsheader .\scroll1images_full_v1_16_4ppmm.tif
> .\scroll1images_full_v1_8_4ppmm.tif: 140920x44000 ushort, 3 bands,
> rgb16, tiffload
>
> On Tue, 2 Apr 2024 at 10:14, John Cupitt ***@***.***> wrote:
> >
> > Hi @pdbourke,
> >
> > It should work. I see:
> >
> > $ vipsheader x.tif
> > x.tif: 1450x2048 ushort, 3 bands, rgb16, tiffload
> > $ vips resize x.tif y.tif 0.1
> > $ vipsheader y.tif
> > y.tif: 145x205 ushort, 3 bands, rgb16, tiffload
> >
> > That's libvips 8.15 on ubuntu 23.10.
> >
> > vipsthumbnail will make 8-bit images, maybe you used that? resize will keep the format.
> >
> > You can see the load and save options in the API docs, eg:
> >
> > https://www.libvips.org/API/current/VipsForeignSave.html#vips-tiffsave
> >
> > That's the C API, but the command-line interface is generated automatically, so they are always the same.
> >
> > —
> > Reply to this email directly, view it on GitHub, or unsubscribe.
> > You are receiving this because you were mentioned.Message ID: ***@***.***>
>
>
>
> --
> Paul Bourke
> paulbourke.net
> ***@***.***
> +61 433338325
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
Thanks, indeed I found out about cast.
What's the difference between using "scale" vs colourspace?
…On Sat, 13 Apr 2024 at 21:17, John Cupitt ***@***.***> wrote:
tldr: you should usually use colourspace, not cast.
cast is very low level -- it just changes the numeric format of the
image, like x = (unsigned char) y in C, except that it's a saturated
cast, so -1 is clipped to 0 and 542 is clipped to 255.
16 bit TIFFs usually use 0 - 65535 for black to white, so if you just use
cast you're likely to get an 8 bit image where everything is white.
libvips tracks things like this in the interpretation field. This
indicates the expected interpretaion of numeric value in the image in terms
of their colour, so an interpretation tag of rgb16 means 0 to 65535.
You can use the colourspace operation to transform between
interpretations. For example:
$ vipsheader k2.tif
k2.tif: 1450x2048 ushort, 3 bands, rgb16, tiffload
$ vips stats k2.tif .csv
0 65535 235599341177 8909839749401991 26445.687542317708 17341.924764814306 71 16 553 412
0 65535 84706726045 3382086049508195 28524.624880455282 18034.655088751002 71 16 553 412
0 65535 77168075407 2917733147847025 25986.016772292565 17528.864677453672 0 16 1107 290
0 65535 73724539725 2610020552046771 24826.420974205281 16203.767489116626 0 16 1107 290
The first row is for all channels and columns are minimum, maximum, sum,
sum of squares, mean, standard deviation, x coordinate of minimum, y
coordinate of minimum, x coordinate of maximum, y coordinate of maximum.
You can see this image has black (0) pixels, white (65535 pixels) and a
mean of 26445.
Now:
$ vips colourspace k2.tif x.tif srgb
$ vipsheader x.tif
x.tif: 1450x2048 uchar, 3 bands, srgb, tiffload
$ vips stats x.tif .csv
0 255 915862337 135026058411 102.80423143408764 67.733090641831609 926 192
886 431
0 255 329405785 51275366075 110.92597824622845 70.442796170792533 926 192
886 431
0 255 299955367 44216939485 101.00867692618534 68.462490060644214 928 192
589 372
0 255 286501185 39533752851 96.478039129849137 63.283569611147875 1049 194
586 368
Now we have an 8 bit image and the mean has dropped to 102 since libvips has scaled everything down.
—
Reply to this email directly, view it on GitHub
<#3918 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABLREFZCRQIURPSKRSVPDR3Y5EH5PAVCNFSM6AAAAABFSJPVMKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMBTGE3DO>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
Yes. Thanks.
It seems I have a problem.
If I apply colourspace as below, it doesn't seem to start, been on 0%
for 10 hours.
vips colourspace .\scroll1images_full_v1_16_40ppmm.tif f:scroll1.v
srgb --vips-progress
vips.exe f:scroll1.v: 1409196 x 440000 pixels, 32 threads, 1409196 x 1
tiles, 640 lines in buffer
vips.exe f:scroll1.v: 0% complete
If I do a resize of the image by 0.1 then a colourspace of that
version completes "in minutes".
vips colourspace .\scroll1images_full_v1_16_4ppmm.tif f:\test8.v srgb
--vips-progress
vips.exe f:\test8.v: 140920 x 44000 pixels, 32 threads, 140920 x 1
tiles, 640 lines in buffer
vips.exe f:\test8.v: done in 75.7s
…On Sat, 13 Apr 2024 at 22:36, John Cupitt ***@***.***> wrote:
scale searches for min and max and rescales the image to fit the range.
Did you find the index and the docs? Look up operations here:
https://www.libvips.org/API/current/func-list.html
And click on one to get eg.:
https://www.libvips.org/API/current/libvips-conversion.html#vips-scale
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
Strange, it should work. You might be hitting some windows limits -- that's a BIG image.
Indeed, and just one of three (and the smallest) making up the whole piece.
This is the project
https://www.epfl.ch/labs/emplus/projects/diagram/
I've completed the scanning and stitching, now there are some
processing stages to bring it all together. In short, three pieces
each 440,000 pixels high and between 1.1 and 1.4 million pixels wide
and originals at 16bits from the PhaseOne iXH camera.
I can share some more details with you if you send me an email
address, I don't want them on a public forum though.
ps: we're also using vips for the iiif files for online viewing.
…--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
I wondered if it might be an internal pixel buffer going over 4gb, so I had a quick go on this laptop with an image of the same width, though not as high:
And both images seem fine. Would you be able to test one of your mmmmonsters with WSL? Install ubuntu, then That might help narrow it down to an issue in the windows build, or in libvips itself. I guess this is libvips 8.15, is that right? |
Beta Was this translation helpful? Give feedback.
-
Off topic, but vipsdisp might be useful for viewing these crazy things, perhaps I've sent you the link before, I don't remember. https://github.com/jcupitt/vipsdisp There's a win build on the "releases" page. |
Beta Was this translation helpful? Give feedback.
-
Don't worry about the 16 bit image I was running the colourspace on, tif to
tif, that all worked.
When converting to .v, where some things start to fail is between the size
of these two files
-a---- 4/14/2024 1:19 AM 1498675245779 scroll3_8.tif
width: 1135360
height: 440000
bands: 3
format: uchar
coding: none
interpretation: srgb
and
-a---- 2/19/2024 11:48 PM 1342369493899
scroll3images_full_v1_40ppmm.tif
width: 1135360
height: 440000
bands: 3
format: uchar
coding: none
interpretation: srgb
I was lead to a false sense of security because earlier testing was done on
the slightly smaller image.
For example, the first one fails with the following command but it works
for the second.
vips copy scroll3_8.tif scroll3_8.v --vips-progress
The reason why I'm slightly nervous now is that the next step is to do some
image warping to fix the left and right seams (I know how to write this
myself in C/C++ based upon the .v files), and then concatenate all three
images together, before saving in iiif.
I guess in the meantime I should learn how to do the .v export, and perhaps
other operations in the C/C++ api. Doesn't look too involved.
…On Mon, 15 Apr 2024 at 19:08, John Cupitt ***@***.***> wrote:
Gosh wow ok. Gulp.
So that's about 1.2TB for the image you are running colourspace on, is
that right? I'll see if I can find a machine here with enough free space
that I can try to reproduce the problem.
—
Reply to this email directly, view it on GitHub
<#3918 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABLREFZR5QHKTYO462N2G7LY5OKKBAVCNFSM6AAAAABFSJPVMKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMJVG43TQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
I'm doing this on the clients Windows box, a well endowed machine.
vips --version
vips-8.14.3
Shall I ask their admin to update to 8.15?
…On Mon, 15 Apr 2024 at 19:37, John Cupitt ***@***.***> wrote:
I wondered if it might be an internal pixel buffer going over 4gb, so I
had a quick go on this laptop with an image of the same width, though not
as high:
***@***.*** ~/pics/huge $ vips colourspace ../k2.jpg x.tif rgb16
***@***.*** ~/pics/huge $ time vips replicate x.tif huge.v 1000 10
real 2m33.134s
user 1m37.369s
sys 2m46.288s
***@***.*** ~/pics/huge $ vipsheader huge.v
huge.v: 1450000x20480 ushort, 3 bands, rgb16, tiffload
***@***.*** ~/pics/huge $ ls -l huge.v
-rw-r--r-- 1 john john 178176002196 Apr 15 10:25 huge.v
***@***.*** ~/pics/huge $ vips colourspace huge.v x.v srgb --vips-progress
vips x.v: 1450000 x 20480 pixels, 16 threads, 1450000 x 1 tiles, 384 lines in buffer
vips x.v: done in 93.7s
And both images seem fine.
Would you be able to test one of your mmmmonsters with WSL? Install
ubuntu, then sudo apt install libvips-tools, and try running your
colourspace there?
That might help narrow it down to an issue in the windows build, or in
libvips itself.
I guess this is libvips 8.15, is that right?
—
Reply to this email directly, view it on GitHub
<#3918 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABLREFYH53LMANADY3OGCQDY5ONUZAVCNFSM6AAAAABFSJPVMKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMJWGA4DO>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
Sorry, the stats for scroll3images_full_v1_40ppmm.tif should be (damn
copy/paste on Anydesk)
width: 1089653
height: 426558
bands: 3
format: uchar
coding: none
interpretation: srgb
…On Mon, 15 Apr 2024 at 19:37, Paul Bourke ***@***.***> wrote:
Don't worry about the 16 bit image I was running the colourspace on, tif
to tif, that all worked.
When converting to .v, where some things start to fail is between the size
of these two files
-a---- 4/14/2024 1:19 AM 1498675245779 scroll3_8.tif
width: 1135360
height: 440000
bands: 3
format: uchar
coding: none
interpretation: srgb
and
-a---- 2/19/2024 11:48 PM 1342369493899
scroll3images_full_v1_40ppmm.tif
width: 1135360
height: 440000
bands: 3
format: uchar
coding: none
interpretation: srgb
I was lead to a false sense of security because earlier testing was done
on the slightly smaller image.
For example, the first one fails with the following command but it works
for the second.
> vips copy scroll3_8.tif scroll3_8.v --vips-progress
The reason why I'm slightly nervous now is that the next step is to do
some image warping to fix the left and right seams (I know how to write
this myself in C/C++ based upon the .v files), and then concatenate all
three images together, before saving in iiif.
I guess in the meantime I should learn how to do the .v export, and
perhaps other operations in the C/C++ api. Doesn't look too involved.
On Mon, 15 Apr 2024 at 19:08, John Cupitt ***@***.***>
wrote:
> Gosh wow ok. Gulp.
>
> So that's about 1.2TB for the image you are running colourspace on, is
> that right? I'll see if I can find a machine here with enough free space
> that I can try to reproduce the problem.
>
> —
> Reply to this email directly, view it on GitHub
> <#3918 (reply in thread)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABLREFZR5QHKTYO462N2G7LY5OKKBAVCNFSM6AAAAABFSJPVMKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMJVG43TQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
8.15 has some nice improvements, but 8.14 ought to work OK, I think.
We'll upgrade to 8.15.2.
If you could ask the admin to put WSL on that'd be great for testing. I currently suspect an issue in the win build of libvips, but I'm often wrong (of course).
We can try that next.
As it happens I'm a Mac person, so I could get them to put a drive in the post.
…--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
I tried vipsdisp, it suffers from exactly the same problem. It's happy
with files below the magic threshold but not those above it.
Consistent at least. ;-)
But it may prove useful for checking things along the way.
And updating the 8.15.2 didn't help.
…On Mon, 15 Apr 2024 at 19:38, John Cupitt ***@***.***> wrote:
Off topic, but vipsdisp might be useful for viewing these crazy things, perhaps I've sent you the link before, I don't remember.
https://github.com/jcupitt/vipsdisp
There's a win build on the "releases" page.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
If you could put WSL on that'd be great for testing. I currently suspect an issue in the win build of libvips, but I'm often wrong (of course).
But not this time. The same copy from tif to v seems to be working
where it failed under Windows.
…--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
Thanks again for your help and we can probably close this discussion for now.
Things seem stable on Ubuntu (wsl on the same windows box as before)
and the Mac.
The problems I was seeing would therefore seem to be Windows related
rather than fundamentally vips.
I've put some C/C++ programs together to do some of the simple things
I need, that and the command line utilities should cover requirements
for this project.
I'm hoping, anyway.
…On Tue, 16 Apr 2024 at 11:27, Paul Bourke ***@***.***> wrote:
> If you could put WSL on that'd be great for testing. I currently suspect an issue in the win build of libvips, but I'm often wrong (of course).
But not this time. The same copy from tif to v seems to be working
where it failed under Windows.
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
I think there might be a libvips bug, eg.: https://github.com/libvips/libvips/blob/master/libvips/iofuncs/util.c#L555-L574 That's the thing that loops on
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/write?view=msvc-170 ie. A minimal test case would be very helpful. Did you try with a wide but not as high image? |
Beta Was this translation helpful? Give feedback.
-
I made a quick PR: But we need some minimal test cases and more digging. |
Beta Was this translation helpful? Give feedback.
-
I did this
vips crop scroll1_8.tif croptest.tif 0 0 1400000 20000
The original is 1409196x440000 and well above the failure threshold for a
copy under Windows.
The following
vips copy croptest.tif croptest.v
worked under both Ubuntu and Windows. Noting that the following does not
work under Windows but does under Ubuntu.
vips copy scroll1_8.tif scroll1_8_copy.v
So it seems the height, total file size, does matter.
…On Tue, 16 Apr 2024 at 15:45, John Cupitt ***@***.***> wrote:
I think there might be a libvips bug, eg.:
https://github.com/libvips/libvips/blob/master/libvips/iofuncs/util.c#L555-L574
That's the thing that loops on write().
size_t is 64-bits on windows, but looking at MSDN they give it the type:
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/write?view=msvc-170
ie. unsigned int not size_t (!!!??!!1) which I think would cause the
problem you're seeing. Perhaps there's a similar issue with
MapViewOfFile() (the win32 mmap() sort-of equivalent)?
A minimal test case would be very helpful. Did you try with a wide but not
as high image?
—
Reply to this email directly, view it on GitHub
<#3918 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABLREF6YH6ICKTNEME7KBM3Y5S3G3AVCNFSM6AAAAABFSJPVMKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRVGY2DM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
For me it seems to happen at sizes around 1000000x440000
but as Kleis reports, smaller.
And yes, in my case also, around (but not exactly) 2GB written.
…On Tue, 16 Apr 2024 at 18:59, John Cupitt ***@***.***> wrote:
Ah, frustrating :(
Sorry to keep asking, but could you try some other sizes? It'd be very helpful to find a test case that didn't involve having 1.2tb of free disk space!
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
--
Paul Bourke
paulbourke.net
***@***.***
+61 433338325
|
Beta Was this translation helpful? Give feedback.
-
How do I resize a 16bit tiff on the command line?
This
vips resize source.tif destination.tif 0.1
results in a 8 bit image despite the source being 16 bit.
I suspect it is one of those options in square brackets but I haven't been able to find a list of those, yet.
Beta Was this translation helpful? Give feedback.
All reactions