TL;DR:
When i add a node into the database that has a added path like "http://localhost/fxs" auth seems to fail and the server returns 401. Without the added path everything works. Even changing different hostname (like test.tld) or port does not produce an error. It seems somehow the url is used in auth but not the hostname?
Long version with steps i tried to find and diagnose the error:
Since #1189 is fixed i was able to run a working local instance of syncstorage_rs. I then tried to deploy syncstorage_rs on a dedicated server with public ip and for added security behind Apache2.4 as proxy. The connection with the tokenserver part (/1.0/sync/1.5) works without problems but every time firefox tries to connect to the syncstorage part (/1.5/{uid}/*) the server returns 401 as status. I suspected that apache as a proxy is somehow causing issues.
After investigating further i could replicate the error in a local setup with the following configs:
firefox-syncstorage.toml:
host="127.0.0.1"
port=8000
database_url = "mysql://sample_user:sample_password@localhost/syncstorage_rs"
master_secret = "mysecret"
human_logs = 1
enable_quota = 0
disable_syncstorage = false
tokenserver.database_url = "mysql://sample_user:sample_password@localhost/tokenserver_rs"
tokenserver.enabled = true
tokenserver.fxa_email_domain = "api.accounts.firefox.com"
tokenserver.fxa_metrics_hash_secret = "mysecret"
tokenserver.fxa_oauth_server_url = "https://oauth.accounts.firefox.com"
tokenserver.test_mode_enabled = false
apache vhost:
<VirtualHost *:80>
ServerName test.tld
ServerAlias test.tld
AllowEncodedSlashes On
ProxyPreserveHost On
<LocationMatch "/fxs">
ProxyPass http://localhost:8000
ProxyPassReverse http://localhost:8000
</LocationMatch>
</VirtualHost>
When i directly connect to syncserver_rs with Firefox ("identity.sync.tokenserver.uri=http://locahost:8000/1.0/sync/1.5") and add a node into the tokenserver_rs database with "node=http://localhost:8000" everyting works as expected.
Here is the relevant part of the log of syncstorage_rs (everything after the "/1.5/13/storage/meta/global" request is removed to shorten the log to be readable):
RUST_LOG="trace" /usr/bin/firefox-syncstorage --config=/etc/firefox-syncstorage.toml
Feb 06 23:13:28.330 INFO Starting 32 workers
Feb 06 23:13:28.339 INFO Starting "actix-web-service-127.0.0.1:8000" service on 127.0.0.1:8000
Feb 06 23:13:28.340 INFO Server running on http://127.0.0.1:8000 (mysql) No quota
Feb 06 23:13:39.429 INFO {"first_seen_at":"1644177403218","ua.browser.family":"Firefox","ua.name":"Firefox","ua.os.family":"Linux","ua.browser.ver":"96.0","uri.method":"GET","ua":"96.0","ua.os.ver":"UNKNOWN","uri.path":"/1.0/sync/1.5","uid":"uuuuuuuu","metrics_uid":"xxxxxxx"}
Feb 06 23:13:39.447 INFO {"ua.os.ver":"UNKNOWN","ua.os.family":"Linux","uri.method":"GET","ua.name":"Firefox","ua.browser.family":"Firefox","uri.path":"/1.5/13/info/collections","ua":"96.0.3","ua.browser.ver":"96.0.3"}
Feb 06 23:13:39.453 INFO {"ua.os.ver":"UNKNOWN","ua":"96.0.3","ua.browser.ver":"96.0.3","uri.method":"GET","uri.path":"/1.5/13/info/configuration","ua.os.family":"Linux","ua.name":"Firefox","ua.browser.family":"Firefox"}
Feb 06 23:13:39.463 INFO {"ua.name":"Firefox","uri.method":"GET","ua.browser.ver":"96.0.3","uri.path":"/1.5/13/storage/meta/global","ua.os.ver":"UNKNOWN","ua.os.family":"Linux","ua":"96.0.3","ua.browser.family":"Firefox"}
If i connect with "identity.sync.tokenserver.uri=http://locahost/fxs/1.0/sync/1.5" with a modified node db entry "node=http://localhost/fxs" and over Apache as a proxy, i get the 401 error on the request to "/1.5/13/info/collections". This should mean that Apache is forwarding the request correctly to syncstorage_rs.
Here is the log of syncstorage_rs:
RUST_LOG="trace" /usr/bin/firefox-syncstorage --config=/etc/firefox-syncstorage.toml
Feb 06 23:18:35.409 INFO Starting 32 workers
Feb 06 23:18:35.418 INFO Starting "actix-web-service-127.0.0.1:8000" service on 127.0.0.1:8000
Feb 06 23:18:35.419 INFO Server running on http://127.0.0.1:8000 (mysql) No quota
Feb 06 23:18:46.206 INFO {"ua.os.ver":"UNKNOWN","ua.browser.family":"Firefox","first_seen_at":"1644177403218","ua.name":"Firefox","uri.method":"GET","uri.path":"/1.0/sync/1.5","uid":"uuuuuuuu","metrics_uid":"xxxxxxx","ua.browser.ver":"96.0","ua":"96.0","ua.os.family":"Linux"}
Feb 06 23:18:46.222 INFO {"ua.os.ver":"UNKNOWN","ua.browser.ver":"96.0.3","uri.path":"/1.5/13/info/collections","uri.method":"GET","ua.name":"Firefox","ua.os.family":"Linux","ua.browser.family":"Firefox","ua":"96.0.3"}
Feb 06 23:18:46.987 INFO {"uri.method":"GET","uri.path":"/1.0/sync/1.5","metrics_uid":"xxxxxxx","ua.browser.family":"Firefox","ua.name":"Firefox","ua.os.family":"Linux","uid":"uuuuuuuu","first_seen_at":"1644177403218","ua":"96.0","ua.browser.ver":"96.0","ua.os.ver":"UNKNOWN"}
The logs tell me that Apache strips the "/fxs" part from the path correctly and forwards the requests to syncstorage_rs.
To narrow it down more i removed the path "/fxs" from the apache config and tried to connect with "identity.sync.tokenserver.uri=http://locahost/1.0/sync/1.5" and node db entry of "node=http://localhost" over Apache and it worked.
Here are the syncstorage_rs logs:
RUST_LOG="trace" /usr/bin/firefox-syncstorage --config=/etc/firefox-syncstorage.toml
Feb 06 23:08:11.706 INFO Starting 32 workers
Feb 06 23:08:11.715 INFO Starting "actix-web-service-127.0.0.1:8000" service on 127.0.0.1:8000
Feb 06 23:08:11.715 INFO Server running on http://127.0.0.1:8000 (mysql) No quota
Feb 06 23:08:26.287 INFO {"ua.browser.family":"Firefox","ua.os.family":"Linux","ua":"96.0","first_seen_at":"1644177403218","uid":"uuuuuuuu","ua.name":"Firefox","uri.path":"/1.0/sync/1.5","ua.os.ver":"UNKNOWN","uri.method":"GET","metrics_uid":"xxxxxxx","ua.browser.ver":"96.0"}
Feb 06 23:08:26.308 INFO {"ua.name":"Firefox","ua.browser.ver":"96.0.3","ua.browser.family":"Firefox","ua.os.ver":"UNKNOWN","ua.os.family":"Linux","uri.method":"GET","ua":"96.0.3","uri.path":"/1.5/13/info/collections"}
Feb 06 23:08:26.317 INFO {"ua.browser.family":"Firefox","ua.os.ver":"UNKNOWN","uri.path":"/1.5/13/info/configuration","ua.name":"Firefox","ua":"96.0.3","ua.browser.ver":"96.0.3","uri.method":"GET","ua.os.family":"Linux"}
Feb 06 23:08:26.329 INFO {"ua.browser.family":"Firefox","ua.os.ver":"UNKNOWN","uri.path":"/1.5/13/storage/meta/global","ua.name":"Firefox","ua.os.family":"Linux","uri.method":"GET","ua":"96.0.3","ua.browser.ver":"96.0.3"}
At that point i thought that somehow the hostname and/or url is used in auth so i added a host entry "test.tld" to point to my lan ip (192.168.x.x) instead of localhost (127.0.0.1). I ran the same test this time with "identity.sync.tokenserver.uri=http://test.tld/1.0/sync/1.5" and node db entry of "node=http://test.tld" and it still worked.
Here is the log:
RUST_LOG="trace" /usr/bin/firefox-syncstorage --config=/etc/firefox-syncstorage.toml
Feb 07 00:23:35.153 INFO Starting 32 workers
Feb 07 00:23:35.161 INFO Starting "actix-web-service-127.0.0.1:8000" service on 127.0.0.1:8000
Feb 07 00:23:35.161 INFO Server running on http://127.0.0.1:8000 (mysql) No quota
Feb 07 00:23:44.184 INFO {"ua.os.ver":"UNKNOWN","ua":"96.0","uid":"uuuuuuuu","uri.path":"/1.0/sync/1.5","ua.name":"Firefox","ua.os.family":"Linux","ua.browser.ver":"96.0","metrics_uid":"xxxxxxx","uri.method":"GET","first_seen_at":"1644177403218","ua.browser.family":"Firefox"}
Feb 07 00:23:44.213 INFO {"uri.path":"/1.5/13/info/collections","ua.browser.family":"Firefox","ua.browser.ver":"96.0.3","ua.name":"Firefox","uri.method":"GET","ua.os.family":"Linux","ua":"96.0.3","ua.os.ver":"UNKNOWN"}
Feb 07 00:23:44.219 INFO {"ua.os.family":"Linux","ua.browser.ver":"96.0.3","uri.path":"/1.5/13/info/configuration","uri.method":"GET","ua.os.ver":"UNKNOWN","ua":"96.0.3","ua.browser.family":"Firefox","ua.name":"Firefox"}
Feb 07 00:23:44.231 INFO {"ua.browser.family":"Firefox","ua.os.family":"Linux","uri.method":"GET","ua.name":"Firefox","ua.browser.ver":"96.0.3","uri.path":"/1.5/13/storage/meta/global","ua":"96.0.3","ua.os.ver":"UNKNOWN"}
It seems the hostname and Apache as proxy are not the problem. Only if i add a path like "/fxs" ("identity.sync.tokenserver.uri=http://locahost/fxs" with node db entry "node=http://localhost/fxs") i get the auth error.
I started to capture the traffic with wireshark to check the requests and the auth data. The value of the Authorization header on a request to "/1.0/sync/1.5" is always exactly the same in every constellation (Authorization: Bearer xxx). The Authorization header on a request to "/1.5/13/info/collection" is different in every constellation, but that is expected because the hawk-id includes the node-url (every other field inside the ID is identical except some binary part at the end). I did not see anything else, and I don't know enough about hawk to judge if the auth strings are correct :(
Out of options i tried other combinations:
"identity.sync.tokenserver.uri=http://locahost/fxs/1.0/sync/1.5" with "node=http://localhost:8000" -> works
"identity.sync.tokenserver.uri=http://localhost:8000/1.0/sync/1.5" with "node=http://localhost/fxs" -> fails with 401
"identity.sync.tokenserver.uri=http://localhost:8000/1.0/sync/1.5" with "node=http://localhost" -> works
The tokenserver url inside firefox apparently does not matter. This further shows that Apache as proxy is not a problem, only using a node url like "http://localhost/fxs" is somehow causing the auth to fail. I could not find in the code if and where the node url gets used in the auth.
Maybe this is related to #671 ?
I don't know enough to further narrow down where exactly the error comes from but i need the added "/fxs" path in my setup so i hope someone else is able to find the problem or point me in the right direction where to look or what i'm doing wrong.
TL;DR:
When i add a node into the database that has a added path like "http://localhost/fxs" auth seems to fail and the server returns 401. Without the added path everything works. Even changing different hostname (like test.tld) or port does not produce an error. It seems somehow the url is used in auth but not the hostname?
Long version with steps i tried to find and diagnose the error:
Since #1189 is fixed i was able to run a working local instance of syncstorage_rs. I then tried to deploy syncstorage_rs on a dedicated server with public ip and for added security behind Apache2.4 as proxy. The connection with the tokenserver part (/1.0/sync/1.5) works without problems but every time firefox tries to connect to the syncstorage part (/1.5/{uid}/*) the server returns 401 as status. I suspected that apache as a proxy is somehow causing issues.
After investigating further i could replicate the error in a local setup with the following configs:
firefox-syncstorage.toml:
apache vhost:
When i directly connect to syncserver_rs with Firefox ("identity.sync.tokenserver.uri=http://locahost:8000/1.0/sync/1.5") and add a node into the tokenserver_rs database with "node=http://localhost:8000" everyting works as expected.
Here is the relevant part of the log of syncstorage_rs (everything after the "/1.5/13/storage/meta/global" request is removed to shorten the log to be readable):
If i connect with "identity.sync.tokenserver.uri=http://locahost/fxs/1.0/sync/1.5" with a modified node db entry "node=http://localhost/fxs" and over Apache as a proxy, i get the 401 error on the request to "/1.5/13/info/collections". This should mean that Apache is forwarding the request correctly to syncstorage_rs.
Here is the log of syncstorage_rs:
RUST_LOG="trace" /usr/bin/firefox-syncstorage --config=/etc/firefox-syncstorage.toml
The logs tell me that Apache strips the "/fxs" part from the path correctly and forwards the requests to syncstorage_rs.
To narrow it down more i removed the path "/fxs" from the apache config and tried to connect with "identity.sync.tokenserver.uri=http://locahost/1.0/sync/1.5" and node db entry of "node=http://localhost" over Apache and it worked.
Here are the syncstorage_rs logs:
RUST_LOG="trace" /usr/bin/firefox-syncstorage --config=/etc/firefox-syncstorage.toml
At that point i thought that somehow the hostname and/or url is used in auth so i added a host entry "test.tld" to point to my lan ip (192.168.x.x) instead of localhost (127.0.0.1). I ran the same test this time with "identity.sync.tokenserver.uri=http://test.tld/1.0/sync/1.5" and node db entry of "node=http://test.tld" and it still worked.
Here is the log:
RUST_LOG="trace" /usr/bin/firefox-syncstorage --config=/etc/firefox-syncstorage.toml
It seems the hostname and Apache as proxy are not the problem. Only if i add a path like "/fxs" ("identity.sync.tokenserver.uri=http://locahost/fxs" with node db entry "node=http://localhost/fxs") i get the auth error.
I started to capture the traffic with wireshark to check the requests and the auth data. The value of the Authorization header on a request to "/1.0/sync/1.5" is always exactly the same in every constellation (Authorization: Bearer xxx). The Authorization header on a request to "/1.5/13/info/collection" is different in every constellation, but that is expected because the hawk-id includes the node-url (every other field inside the ID is identical except some binary part at the end). I did not see anything else, and I don't know enough about hawk to judge if the auth strings are correct :(
Out of options i tried other combinations:
The tokenserver url inside firefox apparently does not matter. This further shows that Apache as proxy is not a problem, only using a node url like "http://localhost/fxs" is somehow causing the auth to fail. I could not find in the code if and where the node url gets used in the auth.
Maybe this is related to #671 ?
I don't know enough to further narrow down where exactly the error comes from but i need the added "/fxs" path in my setup so i hope someone else is able to find the problem or point me in the right direction where to look or what i'm doing wrong.