Adding our bandwidth file's tor_version. The spec claims that it's a version
1.5.0 attribute...
https://gitweb.torproject.org/torspec.git/commit/?id=7d8b4bc
However, our most recent bandwidth files have this attribute yet claim to be
1.4.0. Maybe our scanners added the attribute without bumping their version?
Working under that assumption and reached out to juga.
According to Mike and Karsten, OnionPerf and Shadow require the ability to
reset circuit timeouts when they change tor's guards. Adding this as an
argument to drop_guards().
https://github.com/torproject/stem/issues/73
Caught thanks to George...
11:42 <+asn> atagar: hello
11:42 <+asn> atagar: seems like the stem integration tests are failing travis:
https://travis-ci.org/github/torproject/tor/jobs/720852838
11:43 <+asn> atagar: is there a ticket for this?
11:44 <+asn> atagar: if u want i can open one!
23:56 <+atagar> asn: 'seems like the stem integration tests are failing
travis' => I just compiled tor's master branch (commit d4f3cfe) and our
integ tests pass. The problem looks to be that tor is testing its commit
7915b651, which predates https://gitlab.torproject.org/tpo/core/tor/-/issues/40005.
Stem added its test after tor changed its response
(https://gitweb.torproject.org/stem.git/commit/?id=0dba06fd).
23:56 <+atagar> I'd be happy to merge a patch if you'd care to generalize
the assertion.
Oops, I iterated over a set to deduplicate if our filesystem encoding is utf-8
or latin-1. However, there's no harm in decoding twice and doing so
inconsistently broke our unit test because set order is non-deterministic
(so we sometimes decoded our unicode path as latin-1).
======================================================================
FAIL: test_unicode_cookie
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3.7/unittest/mock.py", line 1204, in patched
return func(*args, **keywargs)
File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/unit/response/protocolinfo.py", line 160, in test_unicode_cookie
self.assertEqual(EXPECTED_UNICODE_PATH, control_message.cookie_path)
AssertionError: '/home/user/文档/tor-browser_en-US/Browser/TorBrowser/D[23 chars]okie' != '/home/user/æ\x96\x87æ¡£/tor-browser_en-US/Browser/To[33 chars]okie'
- /home/user/文档/tor-browser_en-US/Browser/TorBrowser/Data/Tor/control_auth_cookie
? ^^
+ /home/user/文档/tor-browser_en-US/Browser/TorBrowser/Data/Tor/control_auth_cookie
? ^^^^^^
When a PROTOCOLINFO's authentication cookie path mismatches our filesystem
encoding falling back to a couple common encodings (unicode and latin-1).
https://github.com/torproject/stem/issues/57
Oops! I thought that GitLab only had tickets for components that migrated, but
on reflection everything else is available under the 'legacy/trac' component.
Thanks to Roger for pointing this out!
Trac is read-only and will be discontinued at some point. I moved our open
tickets to GitHub but we'll lose closed tickets when it's gone. Definitely a
loss, but not too bad - our commit history archives their most useful
information.
Originally we added roles based on a tutorial from...
https://trac.torproject.org/projects/tor/ticket/8671
This was a lot more verbose than it needs to be. Trimming these to the simplest
functions I can come up with.
Oops, while adjusting the indentation we broke this table for Sphinx...
/home/atagar/Desktop/stem/stem/response/__init__.py:docstring of stem.response.convert:15: WARNING: Malformed table.
Text in column margin in table line 10.
Recently tor documented that MAPADDRESS can contain a mixture of successful and
failed responses...
https://gitweb.torproject.org/torspec.git/commit/?id=1417af05de247d9f858863849d16a7185577d369
Such a response didn't break us, our map_address() method simply ignored the
failures and returned a dictionary with whatever mappings succeeded.
However, we should provide our caller with failures as well for completeness.
This breaks backward compatibility in a couple ways...
1. Our map_address() method now returns a MapAddressResponse rather
than a dictionary.
2. Renamed MapAddressResponse's "entries" attribute to "mappings".
Personally I'd prefer if MAPADDRESS was all-or-nothing (that is to say, if any
entry is malformed then reject the request). That would produce a simpler API
for our callers. But I don't control that so this is the method response we'll
need to supply.
Integration tests should undo any configuration changes they make to avoid
unintentionally impacting later tests. Initially it was unclear to me how to
unassign MAPADDRESS, but I filed the following to clarify this...
https://gitlab.torproject.org/tpo/core/tor/-/issues/40104
Nick provided an alternate test rather than adjusting the existing one...
https://trac.torproject.org/projects/tor/ticket/25611
I'm not entirely clear what the issue was but that's an old issue so simply
dropping the obsolete test.
Tor uses a similar response code scheme as HTTP (2xx = ok, 4xx = error,
5xx = fatal). Most successful responses are 250, but better to accept
all 2xx codes.
pkill's '-x' flag picks a process via an exact match rather than a regex.
Caught thanks to uokf...
17:16 < uokf> I'd like to make a suggestion to change https://stem.torproject.org/faq.html#how-do-i-reload-my-torrc
17:17 < uokf> it says pkill -sighup tor
17:17 < uokf> but it should say pkill -x -sighup tor
17:17 < uokf> pkill sends the signal to all matching pids
17:18 <@arma> so if you have a process called 'extractor' then it'll hit that one too?
17:18 <@arma> that seems like a good bug. can you open a ticket for stem? i think atagar likes his tickets in github but i'm not sure.
Tor's present commit (67fc69c) isn't providing a bootstrap message with
progress above 0% so dropping that requirement. Also fixing...
Traceback (most recent call last):
File "./tor-prompt", line 8, in <module>
stem.interpreter.main()
File "/home/atagar/Desktop/stem/stem/interpreter/__init__.py", line 109, in main
password_prompt = True,
File "/home/atagar/Desktop/stem/stem/connection.py", line 285, in connect
connection = asyncio.run_coroutine_threadsafe(connect_async(control_port, control_socket, password, password_prompt, chroot_path, controller), loop).result()
File "/home/atagar/Python-3.7.0/Lib/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/home/atagar/Python-3.7.0/Lib/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/home/atagar/Desktop/stem/stem/connection.py", line 363, in connect_async
raise ValueError("'%s' isn't a valid port" % control_port[1])
ValueError: 'None' isn't a valid port
Our get_circuit() method is documented as taking an integer circuit id, but our
CircuitEvent class uses string id attributes. As a result calling with an int
argument would always fail with 'Tor currently does not have a circuit with the
id of x' error.
Caught thanks to Joel.
Thanks to Juan for the catch. On big-endian systems such as CentOS our unit
tests failed because we don't mock our IS_LITTLE_ENDIAN constant (so our
assertions are based on being little-endian).
By mocking the constant as 'False' we fail in the same way that Juan
reports...
https://github.com/torproject/stem/issues/71
Oops, adjusting this assertion in commit 435b980c broke our tests for prior tor
versions. Caught thanks to asn.
12:42 <+asn> AssertionError: Lists differ: [('0.0.0.0', 1113), ('::', 1113)]
!= [('0.0.0.0', 1113)]
...
20:06 <+atagar> asn, nickm: Thanks for pointing out the stem test assertion
error. I'd like to approach this via a conditional. What was the tor
version where the address behavior changed?
21:38 <+nickm> atagar: somewhere in 0.4.5.x
21:38 <+nickm> I believe that ">= 0.4.5.0" will do fine
21:42 <+nickm> (since 0.4.5.1 isn't out yet)
21:48 <+atagar> perfect, thanks
Mig5's parser was a fine proof of concept but stem parses everything within the
spec. Our list_hidden_service_auth() method now returns either a credential or
dictionary of credentials based on if we're requesting a single service or
everything.
These method names were based on the controller commands which is fine, but we
have some conventions of our own. Renaming these methods for a couple
reasons...
* For consitency Stem still calls these 'hidden services', and will continue
to do so until...
https://trac.torproject.org/projects/tor/ticket/25918
* We prefix getter methods like this with 'list_'.
This test failed when running with a control socket (the RUN_SOCKET test
target) because our constructor no longer implicity connects. Fixing this
tests' "am I using a control port" check.
The test failure message was...
======================================================================
FAIL: test_authenticate_general_example
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/atagar/Desktop/stem/stem/socket.py", line 456, in _open_connection
return await asyncio.open_connection(self.address, self.port)
File "/home/atagar/Python-3.7.0/Lib/asyncio/streams.py", line 77, in open_connection
lambda: protocol, host, port, **kwds)
File "/home/atagar/Python-3.7.0/Lib/asyncio/base_events.py", line 943, in create_connection
raise exceptions[0]
File "/home/atagar/Python-3.7.0/Lib/asyncio/base_events.py", line 930, in create_connection
await self.sock_connect(sock, address)
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 1111)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/atagar/Desktop/stem/stem/connection.py", line 583, in authenticate
protocolinfo_response = await get_protocolinfo(controller)
File "/home/atagar/Desktop/stem/stem/connection.py", line 1077, in get_protocolinfo
await controller.connect()
File "/home/atagar/Desktop/stem/stem/socket.py", line 182, in connect
self._reader, self._writer = await self._open_connection()
File "/home/atagar/Desktop/stem/stem/socket.py", line 458, in _open_connection
raise stem.SocketError(exc)
stem.SocketError: [Errno 111] Connect call failed ('127.0.0.1', 1111)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/atagar/Desktop/stem/test/integ/connection/authentication.py", line 146, in test_authenticate_general_example
await stem.connection.authenticate(control_socket, chroot_path = runner.get_chroot())
File "/home/atagar/Desktop/stem/stem/connection.py", line 587, in authenticate
raise AuthenticationFailure('socket connection failed (%s)' % exc)
stem.connection.AuthenticationFailure: socket connection failed ([Errno 111] Connect call failed ('127.0.0.1', 1111))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/atagar/Desktop/stem/test/require.py", line 75, in wrapped
return func(self, *args, **kwargs)
File "/home/atagar/Desktop/stem/stem/util/test_tools.py", line 701, in wrapper
result = loop.run_until_complete(func(*args, **kwargs))
File "/home/atagar/Python-3.7.0/Lib/asyncio/base_events.py", line 568, in run_until_complete
return future.result()
File "/home/atagar/Desktop/stem/test/integ/connection/authentication.py", line 160, in test_authenticate_general_example
self.fail()
AssertionError: None
----------------------------------------------------------------------
Unfortunately we can't differentiate socket disconnections from errors except
by its message. Asyncio sockets use a different message so revising our check.
This fixes the following RUN_SOCKET test...
======================================================================
ERROR: test_send_disconnected
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/atagar/Desktop/stem/stem/socket.py", line 536, in _write_to_socket
await writer.drain()
File "/home/atagar/Python-3.7.0/Lib/asyncio/streams.py", line 348, in drain
await self._protocol._drain_helper()
File "/home/atagar/Python-3.7.0/Lib/asyncio/streams.py", line 202, in _drain_helper
raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/atagar/Desktop/stem/test/require.py", line 75, in wrapped
return func(self, *args, **kwargs)
File "/home/atagar/Desktop/stem/stem/util/test_tools.py", line 701, in wrapper
result = loop.run_until_complete(func(*args, **kwargs))
File "/home/atagar/Python-3.7.0/Lib/asyncio/base_events.py", line 568, in run_until_complete
return future.result()
File "/home/atagar/Desktop/stem/test/integ/socket/control_socket.py", line 118, in test_send_disconnected
await control_socket.send('blarg')
File "/home/atagar/Desktop/stem/stem/socket.py", line 413, in send
await self._send(message, send_message)
File "/home/atagar/Desktop/stem/stem/socket.py", line 238, in _send
await handler(self._writer, message)
File "/home/atagar/Desktop/stem/stem/socket.py", line 525, in send_message
await _write_to_socket(writer, message)
File "/home/atagar/Desktop/stem/stem/socket.py", line 547, in _write_to_socket
raise stem.SocketError(exc)
stem.SocketError: Connection lost
I spent a few hours investigating the root cause but no luck. When closing a
unix socket that has been terminated by the other end our closed_wait() method
raises a BrokenPipeError. In the following test this causes us to fail to
reconnect the socket (because reconnection first closes us).
This only happens with a ControlSocket (ie. our RUN_SOCKET test target).
======================================================================
ERROR: test_pre_disconnected_query
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/atagar/Desktop/stem/test/require.py", line 75, in wrapped
return func(self, *args, **kwargs)
File "/home/atagar/Desktop/stem/stem/util/test_tools.py", line 701, in wrapper
result = loop.run_until_complete(func(*args, **kwargs))
File "/home/atagar/Python-3.7.0/Lib/asyncio/base_events.py", line 568, in run_until_complete
return future.result()
File "/home/atagar/Desktop/stem/test/integ/response/protocolinfo.py", line 122, in test_pre_disconnected_query
self.assert_matches_test_config(protocolinfo_response)
File "/home/atagar/Desktop/stem/stem/socket.py", line 293, in __aexit__
await self.close()
File "/home/atagar/Desktop/stem/stem/socket.py", line 203, in close
await self._close_wo_send_lock()
File "/home/atagar/Desktop/stem/stem/socket.py", line 215, in _close_wo_send_lock
await self._writer.wait_closed()
File "/home/atagar/Python-3.7.0/Lib/asyncio/streams.py", line 323, in wait_closed
await self._protocol._closed
File "/home/atagar/Desktop/stem/test/integ/response/protocolinfo.py", line 121, in test_pre_disconnected_query
protocolinfo_response = await stem.connection.get_protocolinfo(control_socket)
File "/home/atagar/Desktop/stem/stem/connection.py", line 1077, in get_protocolinfo
await controller.connect()
File "/home/atagar/Desktop/stem/stem/socket.py", line 181, in connect
await self._close_wo_send_lock()
File "/home/atagar/Desktop/stem/stem/socket.py", line 215, in _close_wo_send_lock
await self._writer.wait_closed()
File "/home/atagar/Python-3.7.0/Lib/asyncio/streams.py", line 323, in wait_closed
await self._protocol._closed
File "/home/atagar/Python-3.7.0/Lib/asyncio/selector_events.py", line 868, in write
n = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
Static methods such as from_port() and from_socket_file() cannot invoke
asynchronous methods. Fundimentally this is the same problem as our ainit -
when a loop is transtively running us we cannot join any futures we create.
Luckily in this case we can simply sidestep the headache. from_port() and
from_socket_file() are designed for 'with' statements so we can simply move the
act of connecting into our context management (which is already asynchronous).
I encountered this problem when I ran the following...
import asyncio
from stem.control import Controller
async def print_version_async():
async with Controller.from_port() as controller:
await controller.authenticate()
print('[with asyncio] tor is version %s' % await controller.get_version())
def print_version_sync():
with Controller.from_port() as controller:
controller.authenticate()
print('[without asyncio] tor is version %s' % controller.get_version())
print_version_sync()
asyncio.run(print_version_async())
Before:
% python3.7 demo.py
[without asyncio] tor is version 0.4.5.0-alpha-dev (git-9d922b8eaae54242)
/home/atagar/Desktop/stem/stem/control.py:1059: RuntimeWarning: coroutine 'BaseController.connect' was never awaited
controller.connect()
[with asyncio] tor is version 0.4.5.0-alpha-dev (git-9d922b8eaae54242)
After:
% python3.7 demo.py
[without asyncio] tor is version 0.4.5.0-alpha-dev (git-9d922b8eaae54242)
[with asyncio] tor is version 0.4.5.0-alpha-dev (git-9d922b8eaae54242)
Oops, we only exercise this line when using authentication (which isn't the
default). Our tests now pass when using the RUN_ALL target.
======================================================================
ERROR: test_authenticate_general_cookie
----------------------------------------------------------------------
Traceback (most recent call last):
File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/require.py", line 75, in wrapped
return func(self, *args, **kwargs)
File "/srv/jenkins-workspace/workspace/stem-tor-ci/stem/util/test_tools.py", line 701, in wrapper
result = loop.run_until_complete(func(*args, **kwargs))
File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
return future.result()
File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/integ/connection/authentication.py", line 221, in test_authenticate_general_cookie
if method in protocolinfo_response.auth_methods:
AttributeError: 'coroutine' object has no attribute 'auth_methods'
This class has grown sophisticated enough that it deserves its own module.
Also, I'd like to discuss this with the wider python community and this will
make it easier to cite.
Our asyncio branch was a large overhaul, and though the tests seemed to pass
locally it introduced several regressions...
* Python 3.6 support broke due to usage of asyncio.get_running_loop().
* Interpreter broke. This test was skipped locally because I can't run
python's readline module without segfaulting.
* Our ONLINE target had multiple failures. We don't run this target often so
some of the regressions predated this branch. Fixing the ONLINE target is
the significant bulk of these fixes.
Our integration tests are failing with...
======================================================================
FAIL: test_running_command
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/atagar/Desktop/stem/test/require.py", line 75, in wrapped
return func(self, *args, **kwargs)
File "/home/atagar/Desktop/stem/test/integ/interpreter.py", line 35, in test_running_command
self.assertEqual(expected, _run_prompt('--run', 'GETINFO config-file'))
AssertionError: Lists differ: ['250-config-file=/home/atagar/Desktop/stem/test/data/torrc', '250 OK'] != []
First list contains 2 additional elements.
First extra element 0:
'250-config-file=/home/atagar/Desktop/stem/test/data/torrc'
- ['250-config-file=/home/atagar/Desktop/stem/test/data/torrc', '250 OK']
+ []
Our actual reason surfaces in tor-prompt's stderr...
% python3.7 tor-prompt --run 'GETINFO config-file' --interface 9051
/home/atagar/Desktop/stem/stem/interpreter/__init__.py:115: RuntimeWarning: coroutine 'BaseController.__aenter__' was never awaited
with controller:
/home/atagar/Desktop/stem/stem/interpreter/commands.py:366: RuntimeWarning: coroutine 'BaseController.msg' was never awaited
output = format(str(self._controller.msg(command).raw_content()).strip(), *STANDARD_OUTPUT)
/home/atagar/Desktop/stem/stem/interpreter/__init__.py:182: RuntimeWarning: coroutine 'BaseController.__aexit__' was never awaited
break
The problem is that stem.connection.connect() returns asynchronous controllers,
whereas callers such as the interpreter require the class to be synchronous.
This workaround is pretty gross hackery but in the long run I expect to
completely replace the module prior to Stem 2.x.
For reasons I don't grok python 3.7's readline module segfaults whenever I use
it, so I've been unable to run our tor-prompt tests.
We only need readline for an interactive interpreter so narrowing it to that
scope so I can once again run its other tests.
Oops, just ran the integ tests prior to pushing. Type change broke a unit test.
======================================================================
FAIL: test_get_ports
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/atagar/Python-3.7.0/Lib/unittest/mock.py", line 1191, in patched
return func(*args, **keywargs)
File "/home/atagar/Desktop/stem/test/unit/control/controller.py", line 231, in test_get_ports
self.assertEqual([9050], self.controller.get_ports(Listener.CONTROL))
AssertionError: [9050] != {9050}
----------------------------------------------------------------------
Tor recently changed its ORPort behavior so it provides both an IPv4 and IPv6
endpoint by default...
https://github.com/torproject/stem/issues/70
Adjusting our tests. Our get_ports() method now provides a set rather than a
list so we don't return duplicate values.
Our Query instances now must be manually closed. This resolves the following
when running our ONLINE target...
Threads lingering after test run:
<_MainThread(MainThread, started 139802875361024)>
<Thread(Query asyncio, started daemon 139802728457984)>
<Thread(Query asyncio, started daemon 139802586375936)>
<Thread(Query asyncio, started daemon 139802594768640)>
<Thread(Query asyncio, started daemon 139802544412416)>
<Thread(Query asyncio, started daemon 139801990788864)>
<Thread(Query asyncio, started daemon 139801982396160)>
<Thread(Query asyncio, started daemon 139801974003456)>
