Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encoding with application/x-www-form-urlencoded #7

Open
yugui opened this issue Apr 19, 2015 · 16 comments · May be fixed by #3711
Open

Encoding with application/x-www-form-urlencoded #7

yugui opened this issue Apr 19, 2015 · 16 comments · May be fixed by #3711

Comments

@yugui
Copy link
Member

yugui commented Apr 19, 2015

Today request/response body is always encoded in JSON.
Should accept application/x-www-form-urlencoded too.

  1. Define a new field in the custom option to define a list of acceptable encoding of request body
  2. Similar field to define the list of possible response body
    • maybe should prefer the same encoding as the request
  3. Make the parameter deserialization/serialization pluggable
@buckhx
Copy link

buckhx commented Jul 24, 2017

Is this supported or still open?

I'm implementing an OAuth2 server and the RFC requires the request Content-Type to be application/x-www-form-urlencoded.

@theRealWardo
Copy link
Contributor

we ended up rewriting application/x-www-form-urlencoded requests as JSON prior to the grpc-gateway receiving them like this:

        formWrapper := func(h http.Handler) http.Handler {
                return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                        log.Printf("Got request: %#v\n", r)
                        if strings.ToLower(strings.Split(r.Header.Get("Content-Type"), ";")[0]) == "application/x-www-form-urlencoded" {
                                log.Println("Rewriting form data as json")
                                if err := r.ParseForm(); err != nil {
                                        http.Error(w, err.Error(), http.StatusBadRequest)
                                        log.Println("Bad form request", err.Error())
                                        return
                                }
                                jsonMap := make(map[string]interface{}, len(r.Form))
                                for k, v := range r.Form {
                                        if len(v) > 0 {
                                                jsonMap[k] = v[0]
                                        }
                                }
                                jsonBody, err := json.Marshal(jsonMap)
                                if err != nil {
                                        http.Error(w, err.Error(), http.StatusBadRequest)
                                }

                                r.Body = ioutil.NopCloser(bytes.NewReader(jsonBody))
                                r.ContentLength = int64(len(jsonBody))
                                r.Header.Set("Content-Type", "application/json")
                        }
                        mux.ServeHTTP(w, r)
                })
        }
        if err := http.ListenAndServe(":8080", formWrapper(mux)); err != nil {
                log.Fatalf("failed to start gateway server on 8080: %v", err)
        }

@rinatio
Copy link

rinatio commented Jan 18, 2018

+1 It would be useful to have it built-in!

@jinleileiking
Copy link

jinleileiking commented Feb 12, 2018

After one afternoon reading the f**king source code.

This solution is not good.

  1. It cannot check the input whether a int or string by pb.
  2. ...

The good solution is:

  1. make post && form use get logic.
  2. change the autogen code:
- func request_Example_GetForm_0(ctx context.Context, marshaler runtime.Marshaler, client ExampleClient, req *http.Request, pathPar
|     var protoReq FormRequest
|     var metadata runtime.ServerMetadata
|      
|        ---------------------------from form data to genrate : req.URL.Query() ------------------------------
|     if  err := runtime.PopulateQueryParameters(&protoReq, req.URL.Query(), filter_Example_GetForm_0); err != nil {
|         return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
|     }
|
|     msg, err := client.GetForm(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
|     return msg, metadata, err
|
| }
  1. change the form data to url.Values then call:

PopulateQueryParameters(msg proto.Message, values url.Values, filter *utilities.DoubleArray)

Evgeniy-L pushed a commit to Evgeniy-L/grpc-gateway that referenced this issue Nov 16, 2018
* add generation second file with  private methods

* update logic

* update README.md

* remove private definitions
@tuhao1020
Copy link

tuhao1020 commented Jul 29, 2021

any update? @yugui

@johanbrandhorst
Copy link
Collaborator

I don't know anyone working on this at the moment. Are you interested in helping add support for this?

@tuhao1020
Copy link

@johanbrandhorst I'll try it after reading the relevant code

@rmilejcz
Copy link

rmilejcz commented Apr 4, 2022

what are the protocols for this in 2022? We are trying to support receiving data from third party APIs (so we cannot control the format of the post request) and we ran into this issue. What are people doing to properly serve x-www-form-urlencoded via gprc-gateway? Is there any option besides writing our own custom marshaler?

I see this pull request but it was not merged, and it references another PR that was merged but it doesn't seem to affect whether or not we can add support for x-www-form-urlencoded requests

@johanbrandhorst
Copy link
Collaborator

I think your only recourse today is a custom marshaler.

@jhowliu
Copy link

jhowliu commented Sep 22, 2023

Below is my custom marshaler for decoding the request in x-www-form-urleencoded format and returning the response in JSON. Hope this can help someone who suffer the same problem.

package marshaler

import (
	"encoding/json"
	"fmt"
	"io"
	"io/ioutil"
	"net/url"

	"github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
	"github.com/grpc-ecosystem/grpc-gateway/v2/utilities"
	"google.golang.org/protobuf/proto"
)

type UrlEncodeMarshal struct {
	runtime.Marshaler
}

// ContentType means the content type of the response
func (u UrlEncodeMarshal) ContentType(_ interface{}) string {
	return "application/json"
}

func (u UrlEncodeMarshal) Marshal(v interface{}) ([]byte, error) {
	return json.Marshal(v)
}

// NewDecoder indicates how to decode the request
func (u UrlEncodeMarshal) NewDecoder(r io.Reader) runtime.Decoder {
	return runtime.DecoderFunc(func(p interface{}) error {
		msg, ok := p.(proto.Message)
		if !ok {
			return fmt.Errorf("not proto message")
		}

		formData, err := ioutil.ReadAll(r)
		if err != nil {
			return err
		}

		values, err := url.ParseQuery(string(formData))
		if err != nil {
			return err
		}

		filter := &utilities.DoubleArray{}

		err = runtime.PopulateQueryParameters(msg, values, filter)

		if err != nil {
			return err
		}

		return nil
	})
}

@mickael-kerjean
Copy link

This works fine. Any chance this could get merge in here? I'm happy to contribute in unit testing as it seems that was missing part in https://github.com/grpc-ecosystem/grpc-gateway/pull/548/files

@johanbrandhorst
Copy link
Collaborator

Yes, we'd be happy to accept a new contribution for this, thank you!

@mickael-kerjean
Copy link

@jhowliu do you prefer to make a PR yourself so I add the tests to your PR or should I create the PR with your code myself?

@jhowliu
Copy link

jhowliu commented Nov 2, 2023

Hi @mickael-kerjean , sorry about waiting for a while. I have made a PR. Feel free to add the tests.

https://github.com/jhowliu/grpc-gateway/tree/feat/support-url-form-encoded-marshaler

@paullen
Copy link

paullen commented Dec 7, 2024

Hi, is there any update on this PR being merged?

@johanbrandhorst
Copy link
Collaborator

Which PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.