2025-05-09 06:57:03 [scrapy.utils.log] INFO: Scrapy 2.12.0 started (bot: spyingbee_crawler) 2025-05-09 06:57:03 [scrapy.utils.log] INFO: Versions: lxml 5.3.2.0, libxml2 2.12.10, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 24.11.0, Python 3.12.10 (main, Apr 9 2025, 00:26:06) [GCC 14.2.0], pyOpenSSL 25.0.0 (OpenSSL 3.3.2 3 Sep 2024), cryptography 43.0.3, Platform Linux-6.8.0-55-generic-x86_64-with 2025-05-09 06:57:04 [scrapy.addons] INFO: Enabled addons: [] 2025-05-09 06:57:04 [asyncio] DEBUG: Using selector: EpollSelector 2025-05-09 06:57:04 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor 2025-05-09 06:57:04 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop 2025-05-09 06:57:04 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor 2025-05-09 06:57:04 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop 2025-05-09 06:57:04 [scrapy.extensions.telnet] INFO: Telnet Password: df50879afb3716a0 2025-05-09 06:57:04 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.spiderstate.SpiderState'] 2025-05-09 06:57:04 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'spyingbee_crawler', 'DEPTH_LIMIT': 1, 'FEED_EXPORT_ENCODING': 'utf-8', 'FEED_EXPORT_INDENT': 2, 'JOBDIR': '/var/lib/scrapyd/jobs/dupefilter', 'LOG_FILE': '/var/lib/scrapyd/logs/myproject/spyingbee-changelog/cdfddd862ca211f0a4ee02420a000185.log', 'NEWSPIDER_MODULE': 'spyingbee_crawler.spiders', 'SPIDER_MODULES': ['spyingbee_crawler.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', 'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:134.0) ' 'Gecko/20100101 Firefox/134.0'} 2025-05-09 06:57:04 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2025-05-09 06:57:04 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.dupefilters.RFPDupeFilter', 'spyingbee_crawler.middlewares.SpyingbeeCrawlerSpiderMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2025-05-09 06:57:04 [scrapy.middleware] INFO: Enabled item pipelines: ['spyingbee_crawler.pipelines.StoreItemPipeline'] 2025-05-09 06:57:04 [scrapy.core.engine] INFO: Spider opened 2025-05-09 06:57:04 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2025-05-09 06:57:04 [spyingbee-changelog] INFO: Spider opened: spyingbee-changelog 2025-05-09 06:57:04 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2025-05-09 06:57:05 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2025-05-09 06:57:05 [spyingbee-changelog] INFO: found changelog page: https://senturopay.com/blog 2025-05-09 06:57:05 [openai._base_client] DEBUG: Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'json_data': {'messages': [{'role': 'system', 'content': "You are a URL classifier that determines if URLs likely to be web page articles. \n Respond with only a comma-separated list of numbers (e.g., '0,2,5') indicating the indices \n of URLs that are likely articles (0-based indexing)."}, {'role': 'user', 'content': 'Which of these URLs are likely to be articles?\n- https://senturopay.com/blog/maximizing-your-crypto-assets-a-guide-to-using-senturo-pay-s-virtual-and-physical-credit-cards\n- https://senturopay.com/blog/the-importance-of-securing-a-money-lender-license-in-hong-kong-\n- https://senturopay.com/blog/crypto-credit-cards-reshaping-the-financial-landscape\n- https://senturopay.com/blog/empowering-the-unbanked-tackling-payment-barriers-and-unlocking-the-potential-of-crypto-cards\n- https://senturopay.com/blog/eu-expands-travel-rules-what-does-it-mean-for-crypto-holders\n- https://senturopay.com/blog/mastering-crypto-20-key-terms-you-must-know\n- https://senturopay.com/blog?page=1\n- https://senturopay.com/blog?page=2'}], 'model': 'deepseek-chat', 'stream': False, 'temperature': 1.0}} 2025-05-09 06:57:05 [openai._base_client] DEBUG: Sending HTTP Request: POST https://api.deepseek.com/chat/completions 2025-05-09 06:57:05 [httpcore.connection] DEBUG: connect_tcp.started host='api.deepseek.com' port=443 local_address=None timeout=5.0 socket_options=None 2025-05-09 06:57:05 [httpcore.connection] DEBUG: connect_tcp.complete return_value= 2025-05-09 06:57:05 [httpcore.connection] DEBUG: start_tls.started ssl_context= server_hostname='api.deepseek.com' timeout=5.0 2025-05-09 06:57:05 [httpcore.connection] DEBUG: start_tls.complete return_value= 2025-05-09 06:57:05 [httpcore.http11] DEBUG: send_request_headers.started request= 2025-05-09 06:57:05 [httpcore.http11] DEBUG: send_request_headers.complete 2025-05-09 06:57:05 [httpcore.http11] DEBUG: send_request_body.started request= 2025-05-09 06:57:05 [httpcore.http11] DEBUG: send_request_body.complete 2025-05-09 06:57:05 [httpcore.http11] DEBUG: receive_response_headers.started request= 2025-05-09 06:57:06 [httpcore.http11] DEBUG: receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Date', b'Fri, 09 May 2025 06:57:06 GMT'), (b'Content-Type', b'application/json'), (b'Transfer-Encoding', b'chunked'), (b'Connection', b'keep-alive'), (b'vary', b'origin, access-control-request-method, access-control-request-headers'), (b'access-control-allow-credentials', b'true'), (b'x-ds-trace-id', b'c6c99a47dabbd3b43c2b8cfdcdbdd1a6'), (b'Strict-Transport-Security', b'max-age=31536000; includeSubDomains; preload'), (b'X-Content-Type-Options', b'nosniff'), (b'cf-cache-status', b'DYNAMIC'), (b'Set-Cookie', b'__cf_bm=jBFavEMPzoWHkj3hKGJz0kGI4Ml3W_obheevJIANkbU-1746773826-1.0.1.1-DqThxYjKa12XXNgOXdsvW0oiqVzGroYy64HR0aD1yZOEd2Z0kjBYzuBq0zLMnCcSh.A.Rznj3iCkXc47SrmS3_Godvur9XTK.BvJqleTiDA; path=/; expires=Fri, 09-May-25 07:27:06 GMT; domain=.deepseek.com; HttpOnly; Secure; SameSite=None'), (b'Server', b'cloudflare'), (b'CF-RAY', b'93cf4cfc1f0470fb-HEL'), (b'Content-Encoding', b'gzip')]) 2025-05-09 06:57:06 [httpx] INFO: HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK" 2025-05-09 06:57:06 [httpcore.http11] DEBUG: receive_response_body.started request= 2025-05-09 06:57:12 [httpcore.http11] DEBUG: receive_response_body.complete 2025-05-09 06:57:12 [httpcore.http11] DEBUG: response_closed.started 2025-05-09 06:57:12 [httpcore.http11] DEBUG: response_closed.complete 2025-05-09 06:57:12 [openai._base_client] DEBUG: HTTP Response: POST https://api.deepseek.com/chat/completions "200 OK" Headers({'date': 'Fri, 09 May 2025 06:57:06 GMT', 'content-type': 'application/json', 'transfer-encoding': 'chunked', 'connection': 'keep-alive', 'vary': 'origin, access-control-request-method, access-control-request-headers', 'access-control-allow-credentials': 'true', 'x-ds-trace-id': 'c6c99a47dabbd3b43c2b8cfdcdbdd1a6', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'x-content-type-options': 'nosniff', 'cf-cache-status': 'DYNAMIC', 'set-cookie': '__cf_bm=jBFavEMPzoWHkj3hKGJz0kGI4Ml3W_obheevJIANkbU-1746773826-1.0.1.1-DqThxYjKa12XXNgOXdsvW0oiqVzGroYy64HR0aD1yZOEd2Z0kjBYzuBq0zLMnCcSh.A.Rznj3iCkXc47SrmS3_Godvur9XTK.BvJqleTiDA; path=/; expires=Fri, 09-May-25 07:27:06 GMT; domain=.deepseek.com; HttpOnly; Secure; SameSite=None', 'server': 'cloudflare', 'cf-ray': '93cf4cfc1f0470fb-HEL', 'content-encoding': 'gzip'}) 2025-05-09 06:57:12 [openai._base_client] DEBUG: request_id: None 2025-05-09 06:57:12 [root] INFO: Found 6 relevant links with AI: [Link(url='https://senturopay.com/blog/maximizing-your-crypto-assets-a-guide-to-using-senturo-pay-s-virtual-and-physical-credit-cards', text='Feb 27 2025 • 1 min readMaximizing Your Crypto Assets: A Guide to Using SenturoPay’s Virtual and Physical Credit CardsWant to make the most out of your cryptocurrency? Learn how SenturoPay’s virtual and physical credit cards enable you to seamlessly integrate your digital assets into everyday spending, offering convenience, security, and global acceptance.', fragment='', nofollow=False), Link(url='https://senturopay.com/blog/the-importance-of-securing-a-money-lender-license-in-hong-kong-', text='Payments TransactionsFeb 26 2025 • 2 min readThe Importance of Securing a Money Lender License in Hong Kong', fragment='', nofollow=False), Link(url='https://senturopay.com/blog/crypto-credit-cards-reshaping-the-financial-landscape', text='Payments TransactionsFeb 28 2025 • 2 min readCrypto Credit Cards: Reshaping the Financial Landscape', fragment='', nofollow=False), Link(url='https://senturopay.com/blog/empowering-the-unbanked-tackling-payment-barriers-and-unlocking-the-potential-of-crypto-cards', text='Business EntrepreneurshipFeb 28 2025 • 3 min readEmpowering the Unbanked: Tackling Payment Barriers and Unlocking the Potential of Crypto Cards', fragment='', nofollow=False), Link(url='https://senturopay.com/blog/eu-expands-travel-rules-what-does-it-mean-for-crypto-holders', text='Business EntrepreneurshipFeb 28 2025 • 3 min readEU Expands Travel Rules – What does it Mean for Crypto Holders?', fragment='', nofollow=False), Link(url='https://senturopay.com/blog/mastering-crypto-20-key-terms-you-must-know', text='Business EntrepreneurshipFeb 28 2025 • 3 min readMastering Crypto: 20 Key Terms you Must Know', fragment='', nofollow=False)] 2025-05-09 06:57:12 [root] INFO: Article Link found by AI: https://senturopay.com/blog/maximizing-your-crypto-assets-a-guide-to-using-senturo-pay-s-virtual-and-physical-credit-cards 2025-05-09 06:57:12 [root] INFO: Article Link found by AI: https://senturopay.com/blog/the-importance-of-securing-a-money-lender-license-in-hong-kong- 2025-05-09 06:57:12 [root] INFO: Article Link found by AI: https://senturopay.com/blog/crypto-credit-cards-reshaping-the-financial-landscape 2025-05-09 06:57:12 [root] INFO: Article Link found by AI: https://senturopay.com/blog/empowering-the-unbanked-tackling-payment-barriers-and-unlocking-the-potential-of-crypto-cards 2025-05-09 06:57:12 [root] INFO: Article Link found by AI: https://senturopay.com/blog/eu-expands-travel-rules-what-does-it-mean-for-crypto-holders 2025-05-09 06:57:12 [root] INFO: Article Link found by AI: https://senturopay.com/blog/mastering-crypto-20-key-terms-you-must-know 2025-05-09 06:57:12 [spyingbee-changelog] DEBUG: Following link to: https://senturopay.com/blog/maximizing-your-crypto-assets-a-guide-to-using-senturo-pay-s-virtual-and-physical-credit-cards 2025-05-09 06:57:12 [scrapy.dupefilters] DEBUG: Filtered duplicate request: (referer: https://senturopay.com/blog) 2025-05-09 06:57:12 [spyingbee-changelog] DEBUG: Following link to: https://senturopay.com/blog/the-importance-of-securing-a-money-lender-license-in-hong-kong- 2025-05-09 06:57:12 [scrapy.dupefilters] DEBUG: Filtered duplicate request: (referer: https://senturopay.com/blog) 2025-05-09 06:57:12 [spyingbee-changelog] DEBUG: Following link to: https://senturopay.com/blog/crypto-credit-cards-reshaping-the-financial-landscape 2025-05-09 06:57:12 [scrapy.dupefilters] DEBUG: Filtered duplicate request: (referer: https://senturopay.com/blog) 2025-05-09 06:57:12 [spyingbee-changelog] DEBUG: Following link to: https://senturopay.com/blog/empowering-the-unbanked-tackling-payment-barriers-and-unlocking-the-potential-of-crypto-cards 2025-05-09 06:57:12 [scrapy.dupefilters] DEBUG: Filtered duplicate request: (referer: https://senturopay.com/blog) 2025-05-09 06:57:12 [spyingbee-changelog] DEBUG: Following link to: https://senturopay.com/blog/eu-expands-travel-rules-what-does-it-mean-for-crypto-holders 2025-05-09 06:57:12 [scrapy.dupefilters] DEBUG: Filtered duplicate request: (referer: https://senturopay.com/blog) 2025-05-09 06:57:12 [spyingbee-changelog] DEBUG: Following link to: https://senturopay.com/blog/mastering-crypto-20-key-terms-you-must-know 2025-05-09 06:57:12 [scrapy.dupefilters] DEBUG: Filtered duplicate request: (referer: https://senturopay.com/blog) 2025-05-09 06:57:12 [scrapy.core.engine] INFO: Closing spider (finished) 2025-05-09 06:57:12 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (0 items) in: file:///var/lib/scrapyd/items/myproject/spyingbee-changelog/cdfddd862ca211f0a4ee02420a000185.jl 2025-05-09 06:57:12 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 274, 'downloader/request_count': 1, 'downloader/request_method_count/GET': 1, 'downloader/response_bytes': 15050, 'downloader/response_count': 1, 'downloader/response_status_count/200': 1, 'dupefilter/filtered': 6, 'elapsed_time_seconds': 7.237203, 'feedexport/success_count/FileFeedStorage': 1, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2025, 5, 9, 6, 57, 12, 104418, tzinfo=datetime.timezone.utc), 'httpcompression/response_bytes': 90342, 'httpcompression/response_count': 1, 'items_per_minute': None, 'log_count/DEBUG': 36, 'log_count/INFO': 21, 'memusage/max': 128901120, 'memusage/startup': 128901120, 'request_depth_max': 1, 'response_received_count': 1, 'responses_per_minute': None, 'scheduler/dequeued': 1, 'scheduler/dequeued/disk': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/disk': 1, 'start_time': datetime.datetime(2025, 5, 9, 6, 57, 4, 867215, tzinfo=datetime.timezone.utc)} 2025-05-09 06:57:12 [scrapy.core.engine] INFO: Spider closed (finished)