Twitter API doesn't look header Accept-Encoding: identity

[mattn_jp]

I'm developing https://github.com/mattn/twty

This worked fine but not working now.

See https://github.com/golang/go/issues/13298

Twitter API doesn't treat "Accept-Encoding: identity". I'm guessing many twitter client that doesn't use browser may not work.

[mattn_jp]

Still Not?

[satoru_takeuchi]

I'm guessing many twitter client that doesn't use browser may not work.

I encountered the same problem when using curl. This problem would happen when web clients use http2 and don't handle Content-Encoding if it don't accept compressed body.

Here is the reproducer.

=============
$ curl -v --get --http2 --header "Accept-Encoding: identity;q=1.0,*;q=0" https://api.twitter.com/1.1/account/verify_credentials.json
...

GET /1.1/account/verify_credentials.json HTTP/1.1
Host: api.twitter.com
User-Agent: curl/7.50.1
Accept: /
Accept-Encoding: identity;q=1.0,*;q=0

• Connection state changed (MAX_CONCURRENT_STREAMS updated)!
  < HTTP/2 400
  < content-encoding: gzip
  < content-length: 87
  < content-type: application/json; charset=utf-8
  < date: Tue, 27 Dec 2016 14:05:00 GMT
  < server: tsa_m
  < set-cookie: guest_id=xxxxxxxxxxxxxxxxxxxxxxxxxxxx
  < strict-transport-security: max-age=631138519
  < x-connection-hash: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  < x-response-time: 102
  <
• Connection #0 to host api.twitter.com left intact
  ^_^@^@^@^@^@^@^@\252VJ-\312/V\262\212\256VJ\316OIU\262224\325Q\312M-.NL^G\362\224\234^RS^T^\KK2R\363J2\223^SK2\363\363^TR^RK^R\365\224jck^@^@^@\377\377^C^@^]W^^\342>^@^@^@s$
  ==============

The response body is encoded with gzip even if I only accept identity. If I use http1.1, this problem doesn't happen.

==============
$ curl -v --get --http1.1 --header "Accept-Encoding: identity;q=1.0,*;q=0" https://api.twitter.com/1.1/account/verify_credentials.json
...

GET /1.1/account/verify_credentials.json HTTP/1.1
Host: api.twitter.com
User-Agent: curl/7.50.1
Accept: /
Accept-Encoding: identity;q=1.0,*;q=0

< HTTP/1.1 400 Bad Request
< content-length: 62
< content-type: application/json; charset=utf-8
< date: Tue, 27 Dec 2016 13:46:59 GMT
< server: tsa_m
< set-cookie: guest_id=xxxxxxxxxxxxxxxxxxxxxxxxx
< strict-transport-security: max-age=631138519
< x-connection-hash: xxxxxxxxxxxxxxxxxxxxxxx
< x-response-time: 101
<
* Connection #0 to host api.twitter.com left intact
{"errors":[{"code":215,"message":"Bad Authentication data."}]}$
===============

• software version

================
$ curl --version
curl 7.50.1 (x86_64-pc-linux-gnu) libcurl/7.50.1 GnuTLS/3.5.7 zlib/1.2.8 libidn/1.33 libssh2/1.7.0 nghttp2/1.17.0 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets
================

[watfordjc]

If you can't handle gzip content, it is probably best to avoid using HTTP/2 unless you know the server follows h2-13 or later where mandatory GZIP compression support in clients was dropped from the spec.

Up until this point the draft HTTP/2 spec stated that clients MUST support GZIP content-encoding and that servers MAY send responses with GZIP encoding regardless of the value of the Accept-Encoding header.

api.twitter.com still supports SPDY/3.1, where clients MUST support GZIP.

Although api.twitter.com advertises h2 support, it looks like they are using a server (and/or a server configuration) that is ignorant of the fact section 9.3 was dropped from the HTTP/2 spec. It could also be SPDY support adds in the GZIP assumption and it might be difficult to remove that assumption from the server code without removing SPDY support.

From HTTPbis New York Interim Minutes, 5 June 2014:

roberto: we need a new error code or new http semantics to force downgrade to http/1.1

will keeps arguing about making sure we don't resolve the issue too soon without having a reasonable option to preserve the performance improvement

end resolution was to remove implicit gzip and google folks are supposed to put up a proposal for coming up with a ratchet / hard fail solution to move first to lock in support for compression, so any intermediary that tries to break it afterward will cause a hard fail.

[andypiper]

Thank you for this - I will share it with the team. Appreciate the detail.

[satoru_takeuchi]

Thank you for your detailed explanation!