packtpublishing / learning-concurrency-in-python Goto Github PK

for python3 use:
import urllib.request
import time
from bs4 import BeautifulSoup

linkArray = []
def getLinks():
req = urllib.request.urlopen('http://www.example.com')
soup = BeautifulSoup(req.read(),features="lxml")
for link in soup.findAll('a'):
linkArray.append(link.get('href'))
print(len(linkArray))

getLinks()

Wrong name used.
Should be myPublisher.

The code in Chapter 5 - lockedSet.py doesn't work propperly.

If you create a LockedSet object i.e.

z = LockedSet({1, 2, 3, 4})

and check if 2 is in z it will fail.
2 in z
returns
False
z.__contains__(2)
returns
None

The solution is to return the result of super(LockedSet, self).__contains__(elem)

def __contains__(self, elem):
     with self._lock:`
          return super(LockedSet, self).__contains__(elem)

I think this code is not working properly.
The result is very dependent on the number of threads you start.
The more threads you start, the more pages will be crawled.
I guess the problem is that the crawler threads finish due to empty queue and don't get back to work when there is new work in the queue.

Some results shown by the crawler when crawling https://tutorialedge.net

1 Thread: Total Number of Pages Visited 35
5 Threads: Total Number of Pages Visited 35
10 Threads: Total Number of Pages Visited 36
50 Threads: Total Number of Pages Visited 67
100 Threads: Total Number of Pages Visited 78

elem is used but not defined (in add and remove method).

As rx library has updated the code should be:
import rx

class temperatureObserver():

def on_next(self, x):
print("Temperature is: %s degrees centigrade" % x)
if (x > 6):
print("Warning: Temperate Is Exceeding Recommended Limit")
if (x == 9):
print("DataCenter is shutting down. Temperature is too high")
def on_error(self, e):
print("Error: %s" % e)

def on_completed(self):
print("All Temps Read")

xs = rx.from_iterable(range(10))
d = xs.subscribe(temperatureObserver())

Details here: https://stackoverflow.com/questions/70468810/problems-with-multithreading-queues-implementations

In main() function there are no consumer1 and consumer2 declared but used.

def main():
  integers = []
  condition = threading.Condition()

  # Our Publisher
  pub1 = Publisher(integers, condition)
  pub1.start()

  # Our Subscribers
  sub1 = Subscriber(integers, condition)
  sub2 = Subscriber(integers, condition)
  sub1.start()
  sub2.start()

  ## Joining our Threads
  pub1.join()
  consumer1.join()
  consumer2.join()

Will send PR for this.