Does anyone use this? Written in response to a request from Karsten when I
first began stem I've never heard of someone actually using it. To simplify,
lets drop it.
Huh. Not sure why I added these to subclasses rather than their common parent.
Maybe there's a reason that will make me regret this, but certainly seems to
work.
Our data directory has up to two microdescriptor files: cached-microdescs and
cached-microdescs.new.
If the former is unavailable but the later is present we should use it...
https://trac.torproject.org/projects/tor/ticket/28508
Maybe more important, when looking into this I realized that our attempt to get
tor's data directory stacktraces if not explicitly present in the torrc...
>>> list(controller.get_microdescriptors())
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/atagar/Desktop/stem/stem/control.py", line 490, in wrapped
for val in func(self, *args, **kwargs):
File "/home/atagar/Desktop/stem/stem/control.py", line 1791, in get_microdescriptors
if not os.path.exists(data_directory):
File "/usr/lib/python2.7/genericpath.py", line 26, in exists
os.stat(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found
Changed it so we'll instead provide a generic exception saying we were unable
to determine the data directory. Fortunatley this whole thing is a fallback, so
eventually we'll be able to remove it.
Adding a parser for detached signatures, per irl's request...
https://trac.torproject.org/projects/tor/ticket/28495
You can use stem.descriptor.remote to download these, or parse with
DetachedSignature.from_str(). However, you cannot use parse_file()
until we have a @type annotation for these...
https://trac.torproject.org/projects/tor/ticket/28615
When downloaded these are only available for five minutes each hour making them
highly clunky to use, but irl suggested he might change that (hope so!). At
present during the window when they're available they can be fetched as
follows...
============================================================
Example script
============================================================
import stem.descriptor.remote
detached_sigs = stem.descriptor.remote.get_detached_signatures().run()[0]
for i, sig in enumerate(detached_sigs.signatures):
print('Signature %i is from %s' % (i + 1, sig.identity))
============================================================
When available (minutes 55-60 of the hour)
============================================================
% python demo.py
Signature 1 is from 0232AF901C31A04EE9848595AF9BB7620D4C5B2E
Signature 2 is from 14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4
Signature 3 is from 23D15D965BC35114467363C165C4F724B64B4F66
Signature 4 is from 27102BC123E7AF1D4741AE047E160C91ADC76B21
Signature 5 is from 49015F787433103580E3B66A1707A00E60F2D15B
Signature 6 is from D586D18309DED4CD6D57C18FDB97EFA96D330566
Signature 7 is from E8A9C45EDE6D711294FADF8E7951F4DE6CA56B58
Signature 8 is from ED03BB616EB2F60BEC80151114BB25CEF515B226
Signature 9 is from EFCBE720AB3A82B99F9E953CD5BF50F7EEFC7B97
============================================================
When unavailable (minutes 0-55 of the hour)
============================================================
% python demo.py
Traceback (most recent call last):
File "demo.py", line 3, in <module>
detached_sigs = stem.descriptor.remote.get_detached_signatures().run()[0]
File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 476, in run
return list(self._run(suppress))
File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 487, in _run
raise self.error
urllib2.HTTPError: HTTP Error 404: Not found
Nick encountered another error from this test...
======================================================================
ERROR: test_connections_by_ss
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/nickm/src/stem/test/integ/util/connection.py", line 50, in test_connections_by_ss
self.check_resolver(Resolver.SS)
File "/home/nickm/src/stem/test/require.py", line 58, in wrapped
return func(self, *args, **kwargs)
File "/home/nickm/src/stem/test/integ/util/connection.py", line 28, in check_resolver
connections = get_connections(resolver, process_pid = runner.get_pid())
File "/home/nickm/src/stem/stem/util/connection.py", line 300, in get_connections
raise IOError('No results found using: %s' % resolver_command)
IOError: No results found using: ss -nptu
----------------------------------------------------------------------
I'm unsure why this test is so flaky for him. Earlier I attempted to mitigate
this by catching OSErrors but on reflection what he was really getting were
IOErrors. Python *said* it was an OSError but that's because python3 has made
IOError an alias...
https://stackoverflow.com/questions/29347790/difference-between-ioerror-and-oserror
In Stem 2.x I should probably replace IOError throughout our codebase with
OSError.
Oops, my bad. Caught by teor on...
https://trac.torproject.org/projects/tor/ticket/28571
Our tests passed with python 2.7 (which is what I usually run), but python
3.x's filter() was changed to provide an iterable rather than a list so we
need to normalize.
Revamp digesting of Microdescriptors in the same way as server and extrainfo
descriptors...
https://trac.torproject.org/projects/tor/ticket/28398
This is definitely better *but* is backward incompatible with the class' old
'digest' attribute. Unfortunately I can't avoid this. The method needs to be
called digest() for consistency, and python cannot have name conflicts between
methods and attributes.
The old digest value was hex rather than base64 encoded which made it
relatively useless (it couldn't be used to fetch or validate microdescriptors,
the sole point of the damn thing) so fingers crossed that no one was using it.
I try very hard to provide backward compatibility in minor version bumps of
Stem but in this case I don't think we should be a slave to that here.
We deprecated get_microdescriptors() because tor hadn't implemented it on
DirPorts but now it has. Not only that but our existing method just works
(neat!). Undeprecating get_microdescriptors(), expanding its pydocs, adding
a test, and adding an alias to the base module to match other descriptor
types.
Router status entries don't have their own @type annotation, so our from_str()
method could not provide them. Implementing their own from_str() method so
things like RouterStatusEntryV3.from_str() will work.
Bah. Back when I added our 'digest' attribute to RouterStatusEntryMicroV3 I
tried to be consistent by making all our hashes hex. However, this was a
mistake. Uses of the microdescriptor digest expect base64 so deprecating the
'digest' attribute with another 'microdescriptor_digest' that's base64.
As requested by irl [1] adding a from_str() method to our Descriptor class,
similar to ControlMessage's from_str() [2].
This method makes it much simpler to parse descriptors from byte strings.
Unlike parse_file() this defaults to providing a single descriptor, with a
'multiple = True' flag that should be provided if your content has more than
one...
my_server_descriptor = RelayServerDescriptor.from_str(content)
consensus_relays = NetworkStatusDocumentV3.from_str(consensus_content, multiple = True)
[1] https://trac.torproject.org/projects/tor/ticket/28450
[2] https://stem.torproject.org/api/response.html#stem.response.ControlMessage.from_str
A warning message suggests wrapping bytes in BytesIO and then calling
parse_file(), but this is a simple step that could be included as a
convenience function.
In my particular use case, I'm loading from a file but I'd like to
perform the read asynchronously using asyncio.
https://trac.torproject.org/projects/tor/ticket/28450
When I first began Stem reading from cached descriptors in tor's data directory
was our primary mechanism to get descriptor data. This is no longer the case.
Mostly we use stem.descriptor.remote or the control port.
I included these annotation methods for completeness (ie. 'hey, there's data
here so lets expose it'). That said, I've never heard of someone actually
finding this to be useful, and not that it's extremely rare for people to even
*get* descriptors that have these annotations lets just drop it.
Brought to mind thanks to a discussion about annotations with irl...
https://trac.torproject.org/projects/tor/ticket/28503
Our digest methods rightfully raise a NotImplementedException if I bugger up
and add to the DigestEncoding enumeration without actually implementing it. But
if users provide other bad data for an encoding argument we should provide a
ValueError instead.
Ok, that was dumb. The whole point of my minor doc adjustment was to call
run()! Stupid me.
https://trac.torproject.org/projects/tor/ticket/28400
Ran this demo both successfully and with an exception hardcoded in run() to
ensure I didn't bugger it up again. :P
Iain made a great point that it's tougher to calculate descriptor digests than
it should be...
https://trac.torproject.org/projects/tor/ticket/28398
Digest type and encoding varies by their use. Mostly our spec sticks with
sha1/hex or sha256/base64, but sometimes it differs. For instance, the consensus
cites sha1/base64 server desciptor digests, whereas according to Karsten Tor
Metrics uses sha1/hex digests for filenames.
Presently server and extrainfo descriptors are the only classes with digest()
methods. Microdescriptors should (consensus votes cite microdescriptor digests)
but nobody has asked for those so we'll cross that bridge when we come to it.
This branch expands our digest() methods in the following ways...
* Add a hash_type argument so callers can specify sha1 or sha256
hashing.
* Add an encoding argument so callers can specify hex, base64, or
no encoding.
* In our digest documentation cite what references that descriptor
type's digest (ie. 'what the heck is this useful for?').
Our new digest type and encoding arguments make our _digest_for_content()
helper a poor fit. The only useful thing this helper does is narrow our
content to a specific range. As such adding a helper that does only that.
This doesn't yet change any of our _digest_for_content() callers. That's
next.
Digests are defined by a hash type and encoding tuple. I was using the first to
imply the second, but this doesn't always work. For example, the consensus
cites base64 encoded sha1 server descriptor digests but stem provides hex
encoded sha1s due to the following discussion with Karsten (subject: "Stem
Sphinx Documentation", 6/7/12).
>> - Why does digest() return the base64-encoded digest, not the
>> hex-formatted one? Network statuses are the only documents in Tor using
>> base64 (or rather, a variant of it without trailing ='s), so it's easier
>> to convert those to hex than to convert everything else to base64. Now,
>> if you switch to hex, you'll only have to decide between lower-case and
>> upper-case. I think Tor and metrics-lib use upper-case hex in most places.
>
> I went with base64 because I thought that this was only useful for
> comparing with the network status. What uses the hex encoded digest?
The hex-encoded server descriptor digest is used as file name in metrics
tarballs.
The (decoded) descriptor digest is used to verify the descriptor signature.
Other reasons for hex-encoding the digest() result is that the digest()
in extra-info descriptors should return the hex-encoded digest, too, or
you wouldn't be able to compare it to the extra-info-digest line in
server descriptors. Having both methods return a different encoding
would be confusing.
Oh, and router-digest lines in sanitized bridge descriptors also contain
the hex-encoded digest. You wouldn't want to convert that to base64
before writing it to the digest variable, nor would you want digest()
and digest to return differently encoded digests.
As such I'm going to leave both the hashing and encoding up to our callers
*and* cite all digest uses I know of in our digest method's pydoc.
When referencing digests tor now includes both sha1 and sha256 digests. As
such, beginning to expand our digest() methods to do the same...
https://trac.torproject.org/projects/tor/ticket/28398
I'm starting with extrainfo descriptors because their hashes are referenced by
our server descriptors, providing easy test data to see if we're doing this
right or not...
% curl http://128.31.0.39:9131/tor/server/fp/3BB34C63072D9D10E836EE42968713F7B9325F66 > /tmp/my_server_desc
% curl http://128.31.0.39:9131/tor/server/extra/3BB34C63072D9D10E836EE42968713F7B9325F66 > /tmp/my_extrainfo_desc
% grep extra /tmp/my_server_desc
extra-info-digest 5BEBC13FDA976050D3A0632EE6508FD1BF1D1750 FNzZZtYPlMjBeb78Wv0zS5DUIPGB3TrpJ3k79MZURMU
% python
>>> import stem.descriptor
>>> desc = next(stem.descriptor.parse_file('/tmp/my_extrainfo_desc', 'extra-info 1.0'))
# Good! The below shows that our sha1 digest matches what our server
# descriptor says it should be.
>>> desc.digest(stem.descriptor.DigestHashType.SHA1)
'5BEBC13FDA976050D3A0632EE6508FD1BF1D1750'
# Bad! This should *not* mismatch. >:(
>>> desc.digest(stem.descriptor.DigestHashType.SHA256)
'ciuNPeDfpiBQfowP7N1g7jPsuHR9fwceTTFyknNdyvY'
Unfortunately while I'm clearly doing something wrong, I'm puzzled about why we
mismatch. Our dir-spec's extra-info-digest description is pretty clear...
"sha256-digest" is a base64-encoded SHA256 digest of the extra-info
document, computed over the same data.
We're definitely hashing the same data (otherwise the sha1 wouldn't match).
This code also certainly seems to be doing exactly what the spec says (base64
encoding the sha256 digest). So... huh.
Gonna punt this over to irl who requested this to see if he can spot what I'm
doing wrong.
Oops. As mentioned by irl the surrounding text talks about using run() so our
demonstration should too. This example was correct (due to its 'block = True'
argument), but better to show what we say we show.
https://trac.torproject.org/projects/tor/ticket/28400
Interesting feature request from irl...
https://trac.torproject.org/projects/tor/ticket/28397
We can't knowledgeably provide a version number (those come from CollecTor,
for instance to specify which bridge scrubbing specification metrics used). But
we can certainly provide a valid annotation.
Oops! Iirc the V2Dir flag once was only granted to relays with a DirPort but
seems that's no longer the case. As a result our use_directory_mirrors() method
inserted endpoints with port values of 'None'...
https://trac.torproject.org/projects/tor/ticket/28393
Traceback (most recent call last):
File "bushel.py", line 21, in <module>
for desc in downloader.get_consensus().run():
...
File "/usr/lib/python3/dist-packages/stem/__init__.py", line 571, in __init__
raise ValueError("'%s' isn't a valid port" % port)
ValueError: 'None' isn't a valid port
We could use their ORPort to download descriptors instead, but Stem's ORPort
capabilities are still pretty immature so just gonna restrict ourselves to
DirPort mirrors for now instead.
Command thanks to Riastradh...
21:12 < Riastradh> atagar: I'm on tor-packagers. For a quick command, either
`pkg_add py37-stem' or `cd /usr/pkgsrc/net/py-stem && make install' might
work.
It wasn't visible on the live site, but when I rebuilt the site on my laptop
(which now runs Sphinx v1.6.5) the download page icons shrank to roughly 80x80.
This is because its haiku css added a new 'max-width: 100%' attribute to
images. Overriding that property to keep things looking how it was.
Dumb me. We should check if the tor process is running *after* the test, not
before. Otherwise tor segfaults that occure during our last test suite go
unreported (such as with tor commit 075acf905).
Initially I thought to myself 'meh, doing this for DocTor so no need for a
changelog entry' but as I thought about it more I realized this actually is
somewhat impactful for users since using Serge causes unrelability in
stem.descriptor.remote.
Like Bifroest before it, Serge is a bridge authority and frequently times out
when asked to serve descriptors. Avoiding it, both so descriptor downloading
works better and so I stop getting so many timeout notifications from DocTor.
On occasion our unit tests that emit then listen for controller events fail
with...
======================================================================
FAIL: test_events_get_received
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/mock/mock.py", line 1305, in patched
return func(*args, **keywargs)
File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/unit/control/controller.py", line 684, in test_events_get_received
self.bw_listener.assert_called_once_with(BW_EVENT)
File "/usr/lib/python2.7/dist-packages/mock/mock.py", line 947, in assert_called_once_with
raise AssertionError(msg)
AssertionError: Expected 'mock' to be called once. Called 0 times.
----------------------------------------------------------------------
The specific test varies, but it should always be one of our emit_event()
callers. This has caused nine jenkins test failures in the last three months
(so pretty noisy). Dave ran into this too...
https://trac.torproject.org/projects/tor/ticket/27173
I can't reliably (or even frequently) reproduce this locally but I think I know
what's going on...
* emit_event() adds an event to the queue via put(). This is a blocking call
so we can be assured the event's in there.
* emit_event() then calls set() on the event_notice condition so the
Controller's event thread will process the event.
* The controller's event thread calls get_nowait() on its event queue. This
*ususally* gets the event we just inserted, but since this is get_nowait()
rather than get() it's non-blocking, and may raise a queue.Empty instead.
* When queue.Empty gets thrown we once again wait on the event_notice
condition.
* emit_event() then calls close() on its Controller which joins and
terminates its threads. The close() method calls set() on event_noice to
give it a *second* opportunity to get the event.
* Once again, because we use get_nowait() there's a small chance we'll raise
queue.Empty without processing the event.
* The event thread then terminates with the enqueued event never being
processed.
* Finally, we get the assertion failure above.
This is actually a real bug that can cause Controller users to be highly
delayed or never receive an event. For instance...
* Tor emits an event.
* Our reader thread enqueues the event and calls set() on the condition to
notify the event thread.
* For the reasons mentioned above it fails to do so.
* Because we use wait() without a timeout our event thread then resumes
blocking, causing the event to not get another chance to be delivered until
a second event comes along.
This sucks, and is pretty easy to remediate. Rather than waiting indefinitely
we should instead have our event thread periodically check re-check its own
queue for undelivered messages (picked 50 ms as a decent sounding balance
between responsiveness and load). Our test's emit_event() in turn now blocks
until the event is delivered.
Needless to say this is a long song and dance so quite possible I'm in the
wrong on something but this seems, both in theory and practice thus far, to be
a good step forward.
Our flakiest integration tests on Jenkins tend to be our system tests...
======================================================================
FAIL: test_pid_by_name_ps_linux
----------------------------------------------------------------------
Traceback (most recent call last):
File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/require.py", line 58, in wrapped
return func(self, *args, **kwargs)
File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/require.py", line 58, in wrapped
return func(self, *args, **kwargs)
File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/require.py", line 58, in wrapped
return func(self, *args, **kwargs)
File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/integ/util/system.py", line 211, in test_pid_by_name_ps_linux
self.assertEqual(tor_pid, stem.util.system.pid_by_name(tor_cmd))
AssertionError: 31963 != None
----------------------------------------------------------------------
These tests skip themselves if multiple tor processes are running, *but* they
fail in this fashion if a tor process spawns *during* the test.
To reduce the chances of this the tests now check for additional tor processes
both before *and* after the test. In theory an error could still occure if a
tor process both starts *and* stops in the middle of a test but I'm unsure if
this will really happen in practice. Lets see. If these assertion errors
continue to show up I might simply move them under a special target.
Our test output notes our python version but not the intterpreter type. This
caused Pypy for instance to look like CPython 2.7. Intercluding the interpreter
in the version string if it isn't CPython.
Huh. With a 2018 date our datetime_to_unix() python2.6 workaround was off by an
hour, breaking one of the unit tests I added for stem.client's NETINFO cells.
Honestly not entirely sure what's up, but this is a far cleaner way of
converting datetimes to unix timestamps. Works too.
