Calculating oauth signature using OAuthSignatureCalculator intermittently fails in Netty's validateHeaderValue method

[fbascheper]

We're currently trying to upgrade Dispatch / reboot to the latest AsyncHttpClient 2.0.x version. I've created dispatch/reboot#149 for this. But now one of our tests intermittently fails in OAuthSignatureCalculator from an exception originally thrown by Netty.

We are using scalacheck Properties to generate random values for both the key and token, which are subsequently fed into the OAuthSignatureCalculator, which randomly crashes with AHC 2.0.32 (and the latest 2.1 alpha as well).

I've extracted a test failure on our side and created a test that you can add to org.asynchttpclient.oauth.OAuthSignatureCalculatorTest so you can reproduce our error.

    @Test
    public void testInvalidHeader() throws NoSuchAlgorithmException {

        String keyValue = "\ucb6e\u3b02\ucfb8\u771a\u8f7d\u1525\u8b6f\u8613\u0e59\u5796\uf4a2\uaf17\ua430\u4291\u85ff\ufa3b\u5d71\u0698\u2407\u2596\ued75\u7e6d\u9b47\u3f93\u5bd2\uef0f\u3832\u8a6b\u12d7\ucfc0\ua3ee\ud1f0\u55f7\u95ff\u8fc7\u6063\u5ee1\uc192\u8481\u2e54\ubd86\u9b8a\u6530\ueb35\u6819\ub2ed\ube81\u725f\u5cf7\ua777\uc91c\u0f3a\ubef9\ud129\u8979\u8be0\u3b05\u000c\u375b\u8e4b\u6fcc\ub458\u72f3\u7ee8\u29fb\u68c4\u1af2\u8353\u4fd8\u9e79\uf68b\u8953";
        String tokenValue = "\u54ac\ucc53\u01e5\u6c4b\u1cb2\u2623\u39c5\ua8d5\uea21\uf33f\uf991\u9efa\u5e5e\u23db\u7b38\uf48f\u6570\uc375\u258f\u614e\u77a0\u35a9\u9eaf\u2b0e\u0633\ud4b7\ucff1\u81d7\uf94f\u7abb\u4c97\ub1f2\u9ebf\u78cf\u254a\u63e8\uca37\u479a\u4fd1\uf482\u3bf1\u8405\ue1b4\ua337\uab20\ub8c6\u47c7\uf206\u627d\uc6a1\u5799\u1dc2\u60ee\u23e4\u48fb\ufdc6\u42e5\u84eb\u709b\u641b\u656b\u8420\ue692\u70f6\uc7fc\ucf2c\u1587\uaee4\u3d31\u6977\u08ad";

        ConsumerKey consumer = new ConsumerKey(keyValue, tokenValue);
        RequestToken reqToken = new RequestToken(keyValue, tokenValue);
        OAuthSignatureCalculator calc = new OAuthSignatureCalculator(consumer, reqToken);

        RequestBuilder reqBuilder = new RequestBuilder();
        reqBuilder.setUrl("https://api.dropbox.com/1/oauth/access_token?oauth_token=%EC%AD%AE%E3%AC%82%EC%BE%B8%E7%9C%9A%E8%BD%BD%E1%94%A5%E8%AD%AF%E8%98%93%E0%B9%99%E5%9E%96%EF%92%A2%EA%BC%97%EA%90%B0%E4%8A%91%E8%97%BF%EF%A8%BB%E5%B5%B1%DA%98%E2%90%87%E2%96%96%EE%B5%B5%E7%B9%AD%E9%AD%87%E3%BE%93%E5%AF%92%EE%BC%8F%E3%A0%B2%E8%A9%AB%E1%8B%97%EC%BF%80%EA%8F%AE%ED%87%B0%E5%97%B7%E9%97%BF%E8%BF%87%E6%81%A3%E5%BB%A1%EC%86%92%E8%92%81%E2%B9%94%EB%B6%86%E9%AE%8A%E6%94%B0%EE%AC%B5%E6%A0%99%EB%8B%AD%EB%BA%81%E7%89%9F%E5%B3%B7%EA%9D%B7%EC%A4%9C%E0%BC%BA%EB%BB%B9%ED%84%A9%E8%A5%B9%E8%AF%A0%E3%AC%85%0C%E3%9D%9B%E8%B9%8B%E6%BF%8C%EB%91%98%E7%8B%B3%E7%BB%A8%E2%A7%BB%E6%A3%84%E1%AB%B2%E8%8D%93%E4%BF%98%E9%B9%B9%EF%9A%8B%E8%A5%93");
        Request req = reqBuilder.build();

        calc.calculateAndAddSignature(req, reqBuilder);

    }

This test fails in IntelliJ with the following exception:

java.lang.IllegalArgumentException: Header value contains a prohibited character '\f': OAuth oauth_consumer_key="쭮㬂쾸眚载ᔥ譯蘓๙垖꼗ꐰ䊑藿層嵱ژ␇▖繭魇㾓寒㠲詫዗쿀ꏮ퇰嗷闿过恣廡솒蒁⹔붆鮊攰栙닭뺁牟峷ꝷ줜༺뻹턩襹诠㬅㝛蹋濌둘狳绨⧻棄᫲荓俘鹹襓", oauth_token="쭮㬂쾸眚载ᔥ譯蘓๙垖꼗ꐰ䊑藿層嵱ژ␇▖繭魇㾓寒㠲詫዗쿀ꏮ퇰嗷闿过恣廡솒蒁⹔붆鮊攰栙닭뺁牟峷ꝷ줜༺뻹턩襹诠㬅㝛蹋濌둘狳绨⧻棄᫲荓俘鹹襓", oauth_signature_method="HMAC-SHA1", oauth_signature="sXbS7srKY99BOlV3fwSoiOnox9Y%3D", oauth_timestamp="1495203088", oauth_nonce="j4OjLpGXQAhh9%2FeciI7WlA%3D%3D", oauth_version="1.0"

	at io.netty.handler.codec.http.HttpHeaders.validateHeaderValue(HttpHeaders.java:1222)
	at io.netty.handler.codec.http.DefaultHttpHeaders.set(DefaultHttpHeaders.java:212)
	at org.asynchttpclient.RequestBuilderBase.setHeader(RequestBuilderBase.java:189)
	at org.asynchttpclient.oauth.OAuthSignatureCalculatorInstance.sign(OAuthSignatureCalculatorInstance.java:88)
	at org.asynchttpclient.oauth.OAuthSignatureCalculatorInstance.sign(OAuthSignatureCalculatorInstance.java:71)
	at org.asynchttpclient.oauth.OAuthSignatureCalculator.calculateAndAddSignature(OAuthSignatureCalculator.java:54)
	at org.asynchttpclient.oauth.OAuthSignatureCalculatorTest.testInvalidHeader(OAuthSignatureCalculatorTest.java:118)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:86)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:643)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1128)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at org.testng.TestRunner.privateRun(TestRunner.java:782)
	at org.testng.TestRunner.run(TestRunner.java:632)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
	at org.testng.SuiteRunner.run(SuiteRunner.java:268)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
	at org.testng.TestNG.run(TestNG.java:1064)
	at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:127)

[slandelle]

Thanks!

Could you compare with the token generated by another OAuth1 implementation?
Is this an AHC issue where we fail to generate a proper token, a bug in Netty validation, or a bug in the OAuth spec that generates invalid headers?

[fbascheper]

Do you have a suggestion which other OAuth implementation I can try and compare this with?
I'm guessing that this is not a bug in Netty validation (disallowing \f in a http header), so most likely it's either a spec or implementation issue.

[slandelle]

I guess signpost is the reference in Java: https://github.com/mttkay/signpost

[fbascheper]

OK, I've tried to verify this with signpost and I don't get the exception there, because it encodes the characters differently.

In signpost the auth header looks something like

OAuth oauth_consumer_key="%E6%BA%B5%EC%A5%AD%0C%E6%BF%82%EB%90%AD%EC%AA%AF%EE%86%A1%E4%86%BC%E5%87%A1%E0%AD%BD%E9%90%8D%D4%BA%E9%96%B8%E7%9F%8D%EA%AD%87%E1%96%81%E3%B2%9F%E1%B3%B0%D9%BF%EE%84%B4%E7%B3%8E%E9%90%93%E8%A5%9E%EC%B7%8B%E4%88%AF%E2%BC%84%EB%B9%A0%E2%AF%9C%EC%87%B6%EC%8A%B8%E9%88%A7%EF%AD%9E%E8%9C%AC%EB%97%88%E2%80%A8%E9%88%83%E4%A6%B9%EF%9B%89%E4%81%90%E1%9F%B1%EE%A8%A9", oauth_nonce="4964641412297288570", oauth_signature="%2FViLQgd%2BoV7DHcSTNDWtBT%2F%2BFTM%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1495271569", oauth_token="%E6%BA%B5%EC%A5%AD%0C%E6%BF%82%EB%90%AD%EC%AA%AF%EE%86%A1%E4%86%BC%E5%87%A1%E0%AD%BD%E9%90%8D%D4%BA%E9%96%B8%E7%9F%8D%EA%AD%87%E1%96%81%E3%B2%9F%E1%B3%B0%D9%BF%EE%84%B4%E7%B3%8E%E9%90%93%E8%A5%9E%EC%B7%8B%E4%88%AF%E2%BC%84%EB%B9%A0%E2%AF%9C%EC%87%B6%EC%8A%B8%E9%88%A7%EF%AD%9E%E8%9C%AC%EB%97%88%E2%80%A8%E9%88%83%E4%A6%B9%EF%9B%89%E4%81%90%E1%9F%B1%EE%A8%A9", oauth_version="1.0"

AHC generates a header like

OAuth oauth_consumer_key="쭮㬂쾸眚载ᔥ譯蘓๙垖꼗ꐰ䊑藿層嵱ژ␇▖繭魇㾓寒㠲詫዗쿀ꏮ퇰嗷闿过恣廡솒蒁⹔붆鮊攰栙닭뺁牟峷ꝷ줜༺뻹턩襹诠㬅㝛蹋濌둘狳绨⧻棄᫲荓俘鹹襓", oauth_token="쭮㬂쾸眚载ᔥ譯蘓๙垖꼗ꐰ䊑藿層嵱ژ␇▖繭魇㾓寒㠲詫዗쿀ꏮ퇰嗷闿过恣廡솒蒁⹔붆鮊攰栙닭뺁牟峷ꝷ줜༺뻹턩襹诠㬅㝛蹋濌둘狳绨⧻棄᫲荓俘鹹襓", oauth_signature_method="HMAC-SHA1", oauth_signature="dJoXkYEVcBGjnAWxikjU97HAbDU%3D", oauth_timestamp="1495272341", oauth_nonce="cGIbglq1s6bLVSvUwwpl6w%3D%3D", oauth_version="1.0"

We're still talking randomly generated strings of course.

[fbascheper]

A further investigation in the keyValue / tokenValue in my testcase reveals that the \u000c in the keyValue is causing the problem here. If I remove this character, the exception does not occur. It does not matter if I add tho \u000c to the tokenValue.

[slandelle]

@fbascheper I think I've pinned it. Could you please give the fix a try?

[fbascheper]

LGTM, I've run several thousand tests with the original code (I already worked around it in the dispatch code by using only alphanumeric characters instead of arbitrary strings) and I didn't get any more test failures using a local 2.0.33-snapshot build.
Thanks for the quick turnaround!

[slandelle]

Thanks for your feedback!