Support custom json and base64 encoders for Token and Parser #301

dcalsky · 2023-04-02T15:01:49Z

This PR adds support for Token and Parser using custom JSON and base64 encoders to encode/decode related data for performance and experimentation purpose.

By implementing the Base64Encoder interface with EncodeToString and DecodeString methods to create a custom base64 encoder, while implementing JSONEncoder with Marshal and Unmarshal methods to create a custom json encoder.

import (
	"github.com/bytedance/sonic"
	asmbase64 "github.com/segmentio/asm/base64"
)

type sonicJsonEncoder struct{}

func (s *sonicJsonEncoder) Marshal(v any) ([]byte, error) {
	return sonic.Marshal(v)
}

func (s *sonicJsonEncoder) Unmarshal(data []byte, v any) error {
	return sonic.Unmarshal(data, v)
}

type asmBase64Encoder struct{}

func (s *asmBase64Encoder) EncodeToString(data []byte) string {
	return asmbase64.StdEncoding.EncodeToString(data)
}

func (s *asmBase64Encoder) DecodeString(data string) ([]byte, error) {
	return asmbase64.RawURLEncoding.DecodeString(data)
}

Then Using WithJSONEncoder and WithBase64Encoder to create a Parser, and using WithTokenJSONEncoder and WithTokenBase64Encoder to create a Token.

// create the token and parser  with json encoder based on sonic and base64 encoder based on asm/base64
token := jwt.NewWithClaims(jwt.SigningMethodHS256, jwt.RegisteredClaims{}, jwt.WithTokenJSONEncoder(&sonicJsonEncoder{}), jwt.WithTokenBase64Encoder(&asmBase64Encoder{}))

parser := jwt.NewParser(jwt.WithJSONEncoder(&sonicJsonEncoder{}), jwt.WithBase64Encoder(&asmBase64Encoder{}))

The disadvantages of this PR are (Any suggestions are welcomed):

~~Introduce the third-party libs into the go module~~
due to almost all of functions and structs are under the same package, the optional methods naming might be confused, like WithJSONEncoder and WithTokenJSONEncoder

Resolves #168

oxisto · 2023-04-02T15:41:32Z

Very nice! I will do a review over the next days, but it’s basically the direction I envisioned. Just some general pointers: We want to avoid the inclusion of any third party libs, even if it’s only in the tests. I would suggest only testing the interface itself in the test without the actual sonic implementation.

dcalsky · 2023-04-02T15:55:33Z

Very nice! I will do a review over the next days, but it’s basically the direction I envisioned. Just some general pointers: We want to avoid the inclusion of any third party libs, even if it’s only in the tests. I would suggest only testing the interface itself in the test without the actual sonic implementation.

I've updated the PR according to your advice.

token_option.go

encoder.go

parser.go

parser_option.go

token.go

alam-chime · 2023-04-21T21:58:11Z

thanks @dcalsky
+1 to this PR

oxisto · 2023-09-13T20:39:50Z

Thanks for the initial work on this @dcalsky. I had another go at this, basically splitting up encoding and decoding and also converting most of the interfaces into function types. This should be pretty good integrated in the existing code base now. It even supports both parsing JSON with and without JSON number (as long as the underlying JSON library supports it).

I wonder whether I could also make our "special" base64 options (strict, allow padding) work with external libraries as well. For now, they are disabled when an external decoder is used.

If we are happy with this approach, I want to finalise docs and examples.

cc @mfridman @dcalsky

oxisto · 2024-02-01T17:10:19Z

@mfridman Do we still want to pursue this? It did gather some interest from the community. If yes, I would do another cleanup pass and add docs and examples/tests.

mfridman · 2024-02-01T17:33:28Z

I'll review this in the next few days.

mfridman · 2024-02-05T14:45:48Z

It seems useful, albeit the number of requests for this was quite low.

I was mainly waiting to see what comes of golang/go#63397. If the native std lib could be improved then bringing your own encoding package becomes unnecessary?

I'm a bit on the fence on this one, but if we get it into a good enough shape I wouldn't block. My main hesitation is all the plumbing we have to do to support this in the current /v5 API.

MikeYast · 2024-07-04T15:19:32Z

Hi,

I am waiting for this change because we are experiencing performance issues with large JWT tokens using the standard JSON parser.

I was mainly waiting to see what comes of golang/go#63397. If the native std lib could be improved then bringing your own encoding package becomes unnecessary?
According to their proposal, they are not going to improve performance significantly (like sonic json parser is doing).

Personally, I am waiting for this change or any other change which allows us to replace standard JSON parser.

Semerokozlyat · 2024-07-04T15:27:59Z

Hello, would like to express an urgent demand in this change to boost up our JWT parsing logic on a highly loaded service.

Co-Authored-By: Christian Banse <[email protected]>

oxisto · 2024-07-04T19:48:31Z

It seems useful, albeit the number of requests for this was quite low.

I was mainly waiting to see what comes of golang/go#63397. If the native std lib could be improved then bringing your own encoding package becomes unnecessary?

I'm a bit on the fence on this one, but if we get it into a good enough shape I wouldn't block. My main hesitation is all the plumbing we have to do to support this in the current /v5 API.

@mfridman seems there is some more interest in this. Can you maybe give this review another go? I think it's not that much plumbing making it work. It is mainly two additional if-clauses in token and parse. The rest are just all optional options.

mfridman · 2024-07-04T20:06:36Z

Thanks for the ping @oxisto, I'll take a pass this weekend. I get the notification but typically lack time during the week for more in-depth reviews.

dcalsky · 2024-07-05T08:09:18Z

@oxisto Almost forget this PR... I will fine-tune this PR to ensure minimum changes today later or tomorrow.

dcalsky · 2024-07-06T05:36:48Z

I have written some example tests using custom decoder and encoder. the new methods are very smooth and easy to use. It's time to merge them into the our main branch.

oxisto · 2024-07-06T07:21:15Z

I have written some example tests using custom decoder and encoder. the new methods are very smooth and easy to use. It's time to merge them into the our main branch.

Thanks! That also got rid of the pesky code coverage warning.

I agree, I like the public interface part of this. I think @mfridman's concern is more about the inner workings of this inside parse/etc. I would abstain from further review comments now (expect small nits) and give this into the hands of @mfridman since I heavily reworked this PR myself so I am not really neutral as a reviewer here :)

oxisto · 2024-07-06T07:23:04Z

encoder.go

+	DecodeString(s string) ([]byte, error)
+}
+
+type StrictFunc[T Base64Encoding] func() T


I wonder if this type really needs to be exported at all

oxisto · 2024-07-06T07:23:30Z

encoder.go

+
+type StrictFunc[T Base64Encoding] func() T
+
+type Stricter[T Base64Encoding] interface {


This at least needs a comment if we want to keep It exported, it could probably also be private I guess

oxisto · 2024-07-06T07:23:45Z

encoder.go

+	Strict() T
+}
+
+func DoStrict[S Base64Encoding, T Stricter[S]](x T) Base64Encoding {


Double-check if this is actually still needed, which I think it is not?

oxisto · 2024-07-06T07:23:55Z

encoder.go

+// unmarshal algorithms.
+type JSONUnmarshalFunc func(data []byte, v any) error
+
+type JSONDecoder interface {


Needs a comment because its a public interface

oxisto · 2024-07-06T07:24:27Z

encoder.go

+	Decode(v any) error
+}
+
+type JSONNewDecoderFunc[T JSONDecoder] func(r io.Reader) T


Needs a comment

oxisto · 2024-07-06T07:25:23Z

parser.go

+	decoding
+}
+
+type decoding struct {


its not private, but still we probably could use some comments here, especially for this outer struct

oxisto · 2024-07-06T07:25:29Z

token.go

+	encoders
+}
+
+type encoders struct {


its not private, but still we probably could use some comments here, especially for this outer struct

oxisto · 2024-07-06T07:31:56Z

parser.go

-		} else {
-			err = json.Unmarshal(claimBytes, &claims)
+	// If `useJSONNumber` is enabled, then we must use a dedicated JSONDecoder
+	// to decode the claims. However, this comes with a performance penalty so


small nit: we only know definitly that the performance penalty occurs when using json.NewDecoder. Third-party libraries might not have that penalty, so we might at least state this like ... a performance penalty (when using the standard library decoder).

Any evidence?

oxisto · 2024-07-06T07:35:41Z

parser_option.go

@@ -121,8 +124,80 @@ func WithPaddingAllowed() ParserOption {
 // WithStrictDecoding will switch the codec used for decoding JWTs into strict
 // mode. In this mode, the decoder requires that trailing padding bits are zero,
 // as described in RFC 4648 section 3.5.
+//
+// Note: This is only supported when using [encoding/base64.Encoding], but not


I think this note is not true anymore, since we are checking whether the library supports a Strict() function. We need to check this.

oxisto · 2024-07-06T07:36:50Z

parser_option.go

+
+// WithJSONDecoder supports a custom JSON decoder to use in parsing the JWT.
+// There are two functions that can be supplied:
+//   - jsonUnmarshal is a [JSONUnmarshalFunc] that is used for the


typo: double "the"

oxisto · 2024-07-06T07:39:45Z

parser_option.go

+	}
+}
+
+// WithBase64Decoder supports a custom Base64 when decoding a base64 encoded


grammar: "a custom Base64 [something missing here]"

oxisto · 2024-07-06T07:41:10Z

token_option.go

@@ -3,3 +3,15 @@ package jwt
 // TokenOption is a reserved type, which provides some forward compatibility,
 // if we ever want to introduce token creation-related options.
 type TokenOption func(*Token)
+
+func WithJSONEncoder(f JSONMarshalFunc) TokenOption {


missing comments here

oxisto · 2024-07-06T07:41:14Z

token_option.go

+	}
+}
+
+func WithBase64Encoder(enc Base64Encoding) TokenOption {


missing comments here

dcalsky · 2024-10-15T02:01:28Z

Do we just miss the code annotation part now, if it is supplied, the PR would pass? @oxisto

MikeYast · 2024-10-19T11:46:48Z

parser.go

+
+	// Choose our JSON decoder. If no custom function is supplied, we use the standard library.
+	var unmarshal JSONUnmarshalFunc
+	if p.jsonUnmarshal != nil {


BTW, why are we determining whether a custom JSON marshaller is present in the Parse() method instead of during the Parser creation? The same applies to other places. Parser.jsonUnmarshal should be initialized during Parser construction, as we're not planning to replace JSON marshallers or base64 encoders on the fly.

MikeYast · 2024-10-19T11:48:22Z

parser.go

@@ -200,18 +227,37 @@ func (p *Parser) ParseUnverified(tokenString string, claims Claims) (token *Toke
 // take into account whether the [Parser] is configured with additional options,
 // such as [WithStrictDecoding] or [WithPaddingAllowed].
 func (p *Parser) DecodeSegment(seg string) ([]byte, error) {
-	encoding := base64.RawURLEncoding
+	var encoding Base64Encoding
+	if p.rawUrlBase64Encoding != nil {


The same comment about stage where custom base64 encoder function set. We can set p.rawUrlBase64Encoding to base64.RawURLEncoding or custom function on Parser creation

I added this as a safe guard if people ever came across the crazy idea to directly create a Parser struct, which is unfortunately possible because we did not hide it behind an interface. But yes, if we assume that in the case you are doing that you REALLY know what you are doing we can do it in the init

Yes, that sounds reasonable.

dcalsky force-pushed the main branch 2 times, most recently from cd88b49 to 7c726ab Compare April 2, 2023 15:54

oxisto reviewed Apr 2, 2023

View reviewed changes

token_option.go Outdated Show resolved Hide resolved

oxisto reviewed Apr 2, 2023

View reviewed changes

oxisto force-pushed the main branch from 7c726ab to 6207357 Compare September 13, 2023 14:44

zhouyiheng.go and others added 7 commits July 4, 2024 21:45

feat: custom json and base64 encoders for Token and Parser

90d315c

Co-Authored-By: Christian Banse <[email protected]>

Slightly better way to handle useNumber

8089d9e

Extract encoders/decoders to extra struct

7a5e5d6

Also supports UseNumber

f0fa303

Support almost all base64 options

7684d3e

Added more Godoc

3ae2a4a

supporting strict() for all libraries

f64f460

oxisto force-pushed the main branch from 441e06d to f64f460 Compare July 4, 2024 19:45

test: supply example tests for custom encoder and decoder

5e2ab08

oxisto reviewed Jul 6, 2024

View reviewed changes

oxisto requested a review from mfridman July 6, 2024 07:26

oxisto reviewed Jul 6, 2024

View reviewed changes

token_option.go

}

}

func WithBase64Encoder(enc Base64Encoding) TokenOption {

Copy link

Collaborator

oxisto Jul 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing comments here

oxisto mentioned this pull request Jul 23, 2024

Add option to change time precision for creation/parsing tokens. #400

Open

oxisto mentioned this pull request Oct 14, 2024

Supporting alternative base64/json encoder/decoder #168

Open

MikeYast reviewed Oct 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support custom json and base64 encoders for Token and Parser #301

Support custom json and base64 encoders for Token and Parser #301

dcalsky commented Apr 2, 2023 •

edited

Loading

oxisto commented Apr 2, 2023

dcalsky commented Apr 2, 2023

alam-chime commented Apr 21, 2023

oxisto commented Sep 13, 2023

oxisto commented Feb 1, 2024

mfridman commented Feb 1, 2024

mfridman commented Feb 5, 2024

MikeYast commented Jul 4, 2024

Semerokozlyat commented Jul 4, 2024

oxisto commented Jul 4, 2024

mfridman commented Jul 4, 2024

dcalsky commented Jul 5, 2024

dcalsky commented Jul 6, 2024

oxisto commented Jul 6, 2024 •

edited

Loading

oxisto Jul 6, 2024

oxisto Jul 6, 2024

oxisto Jul 6, 2024

oxisto Jul 6, 2024

oxisto Jul 6, 2024

oxisto Jul 6, 2024

oxisto Jul 6, 2024

oxisto Jul 6, 2024

dcalsky Jul 8, 2024

oxisto Jul 8, 2024

oxisto Jul 6, 2024

oxisto Jul 6, 2024

oxisto Jul 6, 2024 •

edited

Loading

oxisto Jul 6, 2024

oxisto Jul 6, 2024

dcalsky commented Oct 15, 2024

MikeYast Oct 19, 2024

MikeYast Oct 19, 2024

oxisto Oct 19, 2024 •

edited

Loading

MikeYast Oct 20, 2024


		type StrictFunc[T Base64Encoding] func() T

		type Stricter[T Base64Encoding] interface {

Support custom json and base64 encoders for Token and Parser #301

Are you sure you want to change the base?

Support custom json and base64 encoders for Token and Parser #301

Conversation

dcalsky commented Apr 2, 2023 • edited Loading

oxisto commented Apr 2, 2023

dcalsky commented Apr 2, 2023

alam-chime commented Apr 21, 2023

oxisto commented Sep 13, 2023

oxisto commented Feb 1, 2024

mfridman commented Feb 1, 2024

mfridman commented Feb 5, 2024

MikeYast commented Jul 4, 2024

Semerokozlyat commented Jul 4, 2024

oxisto commented Jul 4, 2024

mfridman commented Jul 4, 2024

dcalsky commented Jul 5, 2024

dcalsky commented Jul 6, 2024

oxisto commented Jul 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oxisto Jul 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcalsky commented Oct 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oxisto Oct 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcalsky commented Apr 2, 2023 •

edited

Loading

oxisto commented Jul 6, 2024 •

edited

Loading

oxisto Jul 6, 2024 •

edited

Loading

oxisto Oct 19, 2024 •

edited

Loading