Wish: Add support for focal point

[HRasch]

Hello and thanks for your efforts in this great library. Today i researched on some points in case of responsive images. One interesting feature for a resizing library would be the option to define a focal point. If an image gets resized, this point ensures that the "important" part of a picture is still included. I know there's an option "anchor" that goes in this direction, but support for a focal point would be much better. Here's some info from another Software that supports such an option:

https://docs.imgix.com/tutorials/focal-point-cropping

Thanks.

[lilith]

We call it "gravity" in the JSON API. https://github.com/imazen/imageflow/blob/main/docs/src/json/constrain.md I haven't implemented in the URL API yet, because I'm wondering if we want to support multiple focal points or a rectangle instead of just a point

…

On Sun, May 22, 2022, 1:49 PM Holger Rasch ***@***.***> wrote: Hello and thanks for your efforts in this great library. Today i researched on some points in case of responsive images. One interesting feature for a resizing library would be the option to define a focal point. If an image gets resized, this point ensures that the "important" part of a picture is still included. I know there's an option "anchor" that goes in this direction, but support for a focal point would be much better. Here's some info from another Software that supports such an option: https://docs.imgix.com/tutorials/focal-point-cropping Thanks. — Reply to this email directly, view it on GitHub <#602>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA2LH75MUWQ75X45K6TWEDVLKFUJANCNFSM5WT2ZQ6A> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[HRasch]

My opinion: Support one point and an optional second point. Two points are the easiest way to span a rectangle if needed ;-)

[lilith]

If automatic crop is also specified in a command query, should padding be used to absolutely prevent cropping into the focus rectangle if too large? What do you think is most intuitive?

[HRasch]

I guess it would be more intuitive not to use padding and just crop the focus-area. The focus area is to mark the most important area of the picture. Maybe it‘s possible to „zoom out“ so the focal area keeps untouched while the requested ratio is still valid?

[lilith]

Well, the problem is that zooming out would add padding if the image isn't large enough, right? Ex, if you focused the whole image, then requested any different aspect ratio, there'd be nothing to show.

[HRasch]

I would expect to use the center of the focal area as reference and crop, even if this violates the focal area.

[lilith]

If so, then there's no reason for a 2nd focal point, right? Because it would be no different than simply providing the center of the desired rectangle?

[HRasch]

Yes, finally it would result in the same behavior. The difference would be that if you change the ratio, the focal area would be the center instead of the center of the image. To be honest, I currently solved this issue by using the object-fit together with object-position and css custom-properties. This finally result in larger images than necessary to be transfered. The typical use-case is that you have e.g. a ratio of 16:9 on desktop-size and a ratio of 4:3 on mobile phones. If the focus is not in the center of the image, the important part moves out of screen. If you define a focal area my expected behavior ( if the native resolution is e.g. 1600x1200, focal-area 200:200, 600:600) is that on desktop, where you need a width of 1600px and height of 900px, the center of this area is calculated (400:400), then the required image-area is calculated (0:-50, 1600:850), the area is moved to the origin (0:0, 1600:900). On mobile, where you need a ratio of 4:3, (width: 640px, height: 480px), the area would be 80;160, 720;640. If you have a ratio of 4:3 with a width: 320px, height: 240px, the result would be the area of 240:280, 560:520, what is cropping the focal area unnecessary. In this case the image should be resized to show the complete focal area (172:200, 628:600). Your edge-case, where the focal area is very large and you can‘t resize without exceeding the native image size, the image should be resized to the max and then cropped. Hope my explaination is not too confusing 🥴

[lilith]

There are cases where users want cropping to be prevented, like #594.

We also probably want to offer integration with other features, like face detection: imazen/imageflow-dotnet#44

[HRasch]

I think, you select mode „crop“ to have an image, covering the image area. Adding padding suddenly will lead to more unexpected results than cropping the image if zooming out is not possible anymore because it exceeds the image size.

[iJungleboy]

Hi @lilith

Even though I'm a developer I actually end up using the resizer as an end user a lot. And I'm always surprised as how difficult certain simple looking scenarios end up being, so I'll try to summarize why one point is ok, but not enough...

Note that I like the https://docs.imgix.com/tutorials/focal-point-cropping explanations, because it does cover a lot of aspects. But again, not quite all. In my world, here's what usually should happen:

1. The designer configures the system to make images for headers very-wide , for normal content 16:9, for tall-side-pics maybe 1:2
2. The user adds content to all these predefined things and determines what's important
   It's important to note that the user is usually not qualified to change settings like compression, aspect ratio (would mess up the design) or stuff like that. So he/she can't actually change this.
3. Based on these two givens, the system should be able to do everything as magically as possible

For simplicity, let's assume we have these predefined formats

1. square
2. tall
3. wide
4. 16:9

Part 1: Point or Area depends on Min/Max control

Now IMHO the user should just say what must certainly be in the image, so that automatic cropping (almost) never takes too much away.

With this, I think the designer should be able to specify rules which try to

1. crop as much as possible
2. crop as little as possible

For example, a SQUARE image MINIMUM would still result in something like this:

Whereas SQUARE MAXIMUM would result in something like this:

The imgix tries to solve this using the zoom factor, but that only works if the user is optimizing the image for exactly one specific use case (so it's a great idea, but often wouldn't be enough).

So to summarize part 1

1. If you only want to crop as little as possible, and the formats are mostly similar to the original image size, then 1 point could work.
2. As soon as it should be more automated, we need an area (2 points / rectangle), and a parameter to minimize / maximize cropping

Part 2: More extremes

For wide banners the previous example would fail quickly. So there is either...

...A need for multiple priority areas where the user can determine different priority areas depending on the extremes (this should probably be handled at the CMS level, but could also be handled at the resizer level

which would then result in a banner like

I believe that Part 1 would best be solved at resizer level. Part 2 is probably more diverse in what can be necessary, so for now I think we shouldn't handle it yet, until it's really clear what the requirements are.

[HRasch]

If I understood it correctly, Lilith was talking about the case where it's not possible to keep the focal area together with the ratio like

I would prefer to get the blue rectangle, even if it violates the focal area, instead of a padding

[iJungleboy]

@HRasch Basically what you're describing is certainly correct.

If the resulting bounds would be outside of the possible image, the system should automatically do "the right thing".
What that is could be a url parameter but by default it should result in what you're describing.

[iJungleboy]

Any progress on this? @lilith

[wilz05]

@lilith any updates on this ?

[iJungleboy]

bump?

[jeremy-farrance]

Wow, this was a great read. I've been hoping for this kind of feature+functionality for a long time, but was unaware of these discussions over here on ImageFlow. We recently worked through a few real-world (CMS) edge cases. I am probably repeating stuff above and in #594, but the main pain points are a) having to know the image's width and height (file access is slow, handling many image formats is "dependencies"), and b) maths, which slow me down.

Here is the demo we built and the two main approaches we needed, Target() and Focus(). It always helps to solve a real-world example and not cover "every hypothetical edge case." These covered the majority of our experience-based use cases (we are a website building agency AND present solutions that our users (CMS editors) could understand and use (sometimes with a little trial and error)). Rather than explain, see the linked example below. The two different methods both work on X,Y that are decimal percentages, but Target() can zoom, Focus() will not. I thought I was covering 99% of use-cases, but after reading these two ImageFlow issues, I see I am more in the 96% range. ;)

Target and Focus Examples

Anyhow, after working this through we did put a version of the Focus() solution on 2 production sites and it is working very well. The performance ding is minimized because we are only doing it for 1 image on the page and the client was both impressed and happy with the result.

I hope this gets worked on soon and that there is some way I can help.

---

I don't know if this helps beyond, "show your work" - but in working through this the two key items that made both the think-through and (C#) code reasonable were

1. Having a linear clamp, specify a center percent while restricting the return to inside 0 and the length:

  public static (int Start, int End) CalculatePartialLength(int originalLength, double centerPercent, double partialLengthFraction)
  {
    // Calculate the center point based on the percentage
    int centerPoint = (int)Math.Floor(originalLength * (centerPercent / 100.0));

    // Calculate the half-length of the partial length
    int halfPartialLength = (int)Math.Floor((originalLength * partialLengthFraction) / 2.0);

    // Calculate the start and end points of the partial length
    int startPoint = centerPoint - halfPartialLength;
    int endPoint = centerPoint + halfPartialLength;

    // Clamp the start and end points within the original length
    startPoint = Math.Max(0, startPoint);
    endPoint = Math.Min(originalLength, endPoint);

    // Ensure the length is correct by adjusting the start or end point if necessary
    if (endPoint - startPoint != halfPartialLength * 2)
    {
        if (startPoint == 0)
        {
            endPoint = Math.Min(originalLength, halfPartialLength * 2);
        }
        else if (endPoint == originalLength)
        {
            startPoint = Math.Max(0, originalLength - halfPartialLength * 2);
        }
    }

    // note-to-self: I am left feeling like there is a smarter way to do this or, at least, I should have avoided solving for the crazy hypothetical edge cases
    return (startPoint, endPoint);
  }

2. similar for a rectangle with a target (crop) width and height

public static (int X, int Y)[] CalculateCropCoordinates(int originalWidth, int originalHeight, 
    double focusXPercent, double focusYPercent, 
    int cropWidth, int cropHeight)
  {
      // Ensure crop dimensions do not exceed original image dimensions
      cropWidth = Math.Min(cropWidth, originalWidth);
      cropHeight = Math.Min(cropHeight, originalHeight);

      // Calculate focus point in pixels
      int focusX = (int)(originalWidth * focusXPercent / 100);
      int focusY = (int)(originalHeight * focusYPercent / 100);

      // Calculate top left corner of the crop
      // Reminder, C# v10 has Math.Clamp()
      int topLeftX = Clamp(focusX - cropWidth / 2, 0, originalWidth - cropWidth);
      int topLeftY = Clamp(focusY - cropHeight / 2, 0, originalHeight - cropHeight);

      // Calculate bottom right corner based on the top left corner and the crop dimensions
      var bottomRight = (topLeftX + cropWidth, topLeftY + cropHeight);

      return new[] { (topLeftX, topLeftY), bottomRight };
  }

  // Helper method to clamp a value between a minimum and maximum
  private static int Clamp(int value, int min, int max)
  {
      return Math.Max(min, Math.Min(max, value));
  }

[lilith]

Hmm, so here's my current thought:

&c=x1,y1,x2,y2 as new percentage based crop syntax, nobody seems to use pixels. All coordinates hereafter are %

If &mode=crop:

&c.preserve=x1,y1,x2,y1,x1,y1,x2,y2, etc, allowing multiple rectangles to preserve with decreasing priority. A single point is also permitted, which behaves the same as 'gravity'

&c.gravity=x1,y1

&c.fit=
<various modes that allow or disallow padding, downscaling, upscaling, focusing, focusing between preserved rectangles if 1 or more are impossible without padding, etc - we need to consider what's most intuitive given usage of &zoom and &scale commands and multi-pixel-density picture elements / srcset, since it would be odd to have content changed based on dppx.

I'd love to have help building out a table with these interactions and definining what will make the most sense to people, and which crop fit modes we can omit.

[lilith]

https://docs.imageflow.io/querystring/transforms.html for query command ref

The json api has more extensive fit modes:

https://docs.imageflow.io/json/constrain.html

&c.gravity might need to just be &gravity so it can be used for padding.

&region could exist to add padding/canvas by allowing < 0 and > 1 values, and replace &c... thoughts?

[lilith]

For percentages, should range be 0..100 or 0..1? What seems easiest?

Should &c exist separately from &region ?

[jeremy-farrance]

For ours, I felt most comfortable with 0 .. 100 and optionally allowing up to 2 decimal places. This is because we felt most comfortable communicating this to clients as a percentage.

But since you said "easiest," industry standards have muddied those waters a bit, at least if you do CSS a lot. :)

So I am just stating my preference: 0.00 .. 100.00. I can easily predict that some people will want 4 or more decimal places though.

[jeremy-farrance]

Also, in case you didn't notice, the url params exist for cropxunits and cropyunits - so it you set them both to 100, your crop=x1,y1,x2,y2 are all treated (effectively) as percentages and I've had no problem passing in values with 2 and 4 decimal places. Now that I think about it, this probably influenced my answer above (because I'd already done, it made sense to me, and most importantly: worked).

[lilith]

It's mostly that I don't want &cropxunits=100&cropyunits=100 to be so universally needed.

[lilith]

I've sketched out my idea of the interactions and details - can you all request access and leave comments/edits? This is certainly complex to handle in a way that interacts well with all other features.

https://docs.google.com/spreadsheets/d/1RDWcTka6MfMXsd5tJES4O7UYFLn4sKw7EgFj9Hd97hc/edit?usp=sharing