dotnetage / crawling_the_web

《虫术:Python绝技》随书源码

Welcome to DotNetAge

##What's new? Updated: 2013/12/3

###Pluggable Module Framework Use to create an extension project for DotNetAge. It’s the best way to create or migrate existing ASP.NET MVC projects to DotNetAge in a short time. The following list is the features of the pluggable module framework: The way of creating and developing extension project is similar with asp.net mvc. Migrate existing ASP.NET MVC projects with few changes.

• The way of creating and developing extension project is similar with asp.net mvc.
• Migrate existing ASP.NET MVC projects with few changes.
• Support auto route mapping and custom routes.
• Support DotNetAge events.
• Support dashboard.
• Support tasks and schedules.
• Support Ioc.
• Support embedded widgets.
• Support embedded views.
• Support embedded resource files such as .js, .css,.png and etc.

###Embedded NoSQL Storage DotNetAge 3.1 add a small and powerful NoSQL database that allow developers

• Never need to create and maintain database.
• Never need to learn O-R Mapping libraries (Say good bye to EF).
• Never need to care about the data structures and working with objects directly (CURD).
• Implement the interface to support other NoSQL storage such as Window Azure, MongoDB , StsDB , RavenDB and more.

###New Web Event Mode DotNetAge 3.1 upgraded the Event Monitoring service to new web event mode. It’s easy to trigger and handle an event in MVC application or class library. The following list is the benefits of using the new Web Event Mode:

• Reduce coupling - Trigger event by name instead event object.
• Thread safe - The event handler run in the same thread of the web request.
• Easy to implement - Implement the ObserverBase class with few codes.
• Easy to deploy - Copy the library that contains observer implements to DotNetAge runtime folder then DotNetAge can auto detect and execute the observers when event raised.

###Task and Schedule DotNetAge 3.1 provides a multi-threaded and time trigger task mechanism. Using the Task and Schedule mechanism developed automatic data backup, automated report generation and reporting, and regular inventory detection, regular mail features such as unattended background service functions. The features of the Task and Schedule.

• Can be register in DotNetAge module.
• Support OneTime trigger and Recurring trigger.
• Build on Task Parallel Library (TPL).
• Provides schedule management tools.
• Thread safe.

###Others

• Logger - Provides a log viewer and a global logger object and extendable log interface to support other log frameworks.
• Business interfaces for contact conversion - Provides IOrganization, IPerson, IAddress interface to convert the Web, User and Profile for other application.
• Enhance Syndication feeds to support Medias.
• Enhance Dashboards Programmability.
• Email Queue - Prevent emails from being sent too frequently identified as spam.

More ...

##Licensing DotNetAge is under GPLV2 and Commercial dual-license terms.

###GPLV2 If you chose to use the software under the GPLv2 you have to release your own application under the same license. It could be used for educational, testing and evaluation purposes.

###Commercial The DotNetAge Commercial License allows customers to develop, use and distribute their applications under standard commercial terms. By purchasing commercial licenses, you do not longer have to release your application’s source code.

More...

##Resources

• DotNetAge Download Center
• DotNetAge Developer Network
• DotNetAge Support Center
• DotNetAge News Center

##Follow us

• Facebook
• Twitter
• Google+
• Github

##Contact us

• Contact: Ray
• Mobile: [phone]
• Facetime: [email protected]
• Email:

1. [email protected]
2. [email protected]

crawling_the_web's Issues

书中 代理池部分的代码错误

P366~367
5.2.3.2 代理池

// P366
self.redis.sadd('proxies',item_data,ex=ttl)

// P367
request.meta['proxy'] = self.redis.srandmember()

1. sadd 不支持 ex 参数
   去查了下，redis 不支持 给 set 的 item 设置 expire time
2. srandmember 没有带参数
   如果p366 是 用 item_val 做 key ，而不是 放到 set 里的话，可以设置超时时间。
   这样的话，如何去随机取出一个代理？

版本：
redis-py: 2.10.6