Add documentation for custom metrics #5430
Conversation
WalkthroughThe pull request introduces a new section titled "Custom evaluation metrics" in the FiftyOne documentation. This section explains how users can add custom metrics to their evaluation runs, detailing their support across all evaluation methods. It includes examples for computing custom metrics via the SDK or directly from the App, as well as a new subsection on "Developing custom metrics," which describes how to implement custom metric operators by subclassing the Changes
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
docs/source/images/app/model-evaluation-custom-metric.pngis excluded by!**/*.png,!**/*.png
📒 Files selected for processing (1)
docs/source/user_guide/evaluation.rst(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: build
🔇 Additional comments (3)
docs/source/user_guide/evaluation.rst (3)
1996-2004: LGTM! Clear introduction to custom metrics.The introduction effectively explains what custom metrics are and points users to example implementations in the fiftyone-plugins repository.
2005-2063: LGTM! Well-structured SDK usage example.The example clearly demonstrates:
- How to use custom metrics via SDK and App
- The syntax for passing metric operators with and without kwargs
- Where to find the results in both SDK and App
2064-2141: LGTM! Comprehensive development guide with excellent example.The development guide:
- Properly introduces the base class
- Provides a well-documented example implementation
- Explains each component (config, parameters, compute, fields)
- Includes helpful comments explaining each method's purpose
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
docs/source/user_guide/evaluation.rst(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: build
🔇 Additional comments (2)
docs/source/user_guide/evaluation.rst (2)
2045-2047: Update the image alt text to match the content.The alt text should accurately reflect that this image shows custom metrics functionality.
- :alt: model-evaluation-custom-metric + :alt: model-evaluation-custom-metrics
2064-2141: LGTM! Well-documented custom metric development guide.The custom metric development section provides a comprehensive guide with clear explanations and a well-documented example implementation.
brimoor
force-pushed
the
docs/custom-metrics
branch
from
266eadc to
68ba964
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
docs/source/release-notes.rst (1)
88-90: LGTM! Consider adding an example.The addition of custom evaluation metrics support is well documented with a reference link. To make it even more helpful for users, consider adding a brief inline example showing the basic usage.
- Added support for defining :ref:`custom evaluation metrics <custom-evaluation-metrics>` and applying them when evaluating models. For example:: import fiftyone as fo # Define custom metric def my_metric(results): # Custom evaluation logic return score # Apply custom metric results = dataset.evaluate_detections( "predictions", "ground_truth", eval_key="eval", method="custom", metric_fn=my_metric )
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
docs/source/images/evaluation/custom-evaluation-metric.pngis excluded by!**/*.png,!**/*.png
📒 Files selected for processing (2)
docs/source/release-notes.rst(1 hunks)docs/source/user_guide/evaluation.rst(3 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: build
🔇 Additional comments (3)
docs/source/user_guide/evaluation.rst (3)
2004-2015: LGTM! Clear and concise introduction.The introduction effectively explains the scope and capabilities of custom metrics in FiftyOne.
2019-2021: Fix the RST link formatting.The link to
metric-examplesis not properly formatted according to RST syntax.-The example below shows how to compute a custom metric from the -`metric-examples <https://github.com/voxel51/fiftyone-plugins/tree/main/plugins/metric-examples>`_ -plugin when evaluating object detections: +The example below shows how to compute a custom metric from the +`metric-examples <https://github.com/voxel51/fiftyone-plugins/tree/main/plugins/metric-examples>`__ +plugin when evaluating object detections:
2080-2082: Update the image alt text.The alt text should accurately reflect the image content about custom evaluation metrics.
- :alt: custom-evaluation-metric + :alt: model-evaluation-custom-metric
What changes are proposed in this pull request?
This PR adds documentation for custom metric operators.
How is this patch tested? If it is not, please explain why.
Only updates the documentation.
Release Notes
Is this a user-facing change that should be mentioned in the release notes?
notes for FiftyOne users.
(Details in 1-2 sentences. You can just refer to another PR with a description
if this PR is part of a larger change.)
What areas of FiftyOne does this PR affect?
fiftyonePython library changesSummary by CodeRabbit