- A+
所属分类：.NET技术
前言
在进行某些爬虫任务的时候，我们经常会遇到仅用Http协议难以攻破的情况，比如协议中带有加密参数，破解需要花费大量时间，那这时候就会用Selenium去模拟浏览器进行页面上的元素抓取
大多数情况下我们用Selenium只是爬取一下页面上可见的元素信息或者做一些模拟人工的操作，但页面可见元素的数据字段毕竟有限，有许多有用的字段隐藏在接口响应中的，但是要如何拿到接口响应内容呢？
在网上搜索Selenium如何获取Chrome中Network数据包响应结果，大多数的文章都是Python或者Java，C#的资源少之又少，虽然知道原理，但每个语言之间SDK代码实现相差很大，C#的SDK真的有点魔改，需要自己慢慢摸索
探索
通过寻找资料，大致就2种方案
方案1：通过Selenium指定本地一个代理去截取所有请求，类似于常见抓包工具的原理，但是C#是没有这种插件，也有可能是我没找到，比如Python和Java有一个叫Browsermob-Proxy的插件，可以和Selenium深度结合实现代理抓包。我利用FiddlerCore做了一个本地代理工具，但是并不好用，不能和Selenium进行深度绑定使用，会导致Selenium爬取过程和请求截取是异步进行的，强行用代码实现同步又很难受，达不到我想要的要求
方案2：Selenium通过chromedriver开启浏览器的性能日志功能，记录类型为Performance的日志，该功能在Selenium中叫做 PerformanceLoggingPreferences。网页加载完成后，可以通过Selenium拿到浏览器Performance Logs摘要信息，再利用Log中的RequestId调用Chrome CDP命令去浏览器端获取日志的完整内容。Selenium封装的CDP，本质上还是Http请求，只是带着驱动窗口的SessionId和Chrome的API做交互
方案1参考资料：
https://blog.csdn.net/qq_32502511/article/details/101536325 （Python + Browsermob-Proxy）
https://blog.csdn.net/fontcolor0/article/details/103297635/ （Java + Browsermob-Proxy）
https://www.cnblogs.com/airoot/articles/14888284.html (C# + FiddlerCore)
方案2参考资料：
https://chromedevtools.github.io/devtools-protocol/ （Chrome DevTools Protocol 介绍）
https://www.jianshu.com/p/615e3c0140a5 （Python + 开启PerfLoggingPref）
https://blog.csdn.net/weixin_49855251/article/details/112281901 (不同日志类型的介绍)
https://blog.csdn.net/bigcarp/article/details/115065730 （Java + 原生CDP协议获取日志内容）
上手
启用Logging：
C#伪代码示例：
首先需要安装最新的Nuget包：OpenQA.Selenium
var option = new ChromeOptions(); option.SetLoggingPreference("performance",OpenQA.Selenium.LogLevel.Info); //启用performance日志，等级为Info即可 option.PerformanceLoggingPreferences = new ChromiumPerformanceLoggingPreferences() { IsCollectingNetworkEvents = true //采集网络请求事件 }; using (ChromeDriver driver = new ChromeDriver(driverPath,option,TimeSpan.FromSeconds(5))) { driver.Navigate().GoToUrl("https://item.m.jd.com/product/10052422060501.html"); Thread.Sleep(3 * 1000); //等待页面加载完成 var logs = driver.Manage().Logs.GetLog("performance"); //获取所有performance日志 }
顺便放个Python代码做个对比：
caps = { 'browserName': 'chrome', 'loggingPrefs': { 'performance': 'Info', //启用performance日志，等级为Info即可 }, 'goog:chromeOptions': { 'perfLoggingPrefs': { 'enableNetwork': True, //采集网络请求事件 }, 'w3c': False, }, } driver = webdriver.Chrome(desired_capabilities=caps) //TODO 获取日志
//.....
分析Logging：
我们从返回的日志列表里随便挑一个看看原始内容：
{[2022-08-03T08:47:31Z] [Info] {"message":{"method":"Network.responseReceived","params":{"frameId":"78D9BC6F0CBE162DA6779F410AA1500C","hasExtraInfo":true,"loaderId":"33814E6BD702343CF0A2A38C976C772F","requestId":"33814E6BD702343CF0A2A38C976C772F","response":{"connectionId":242,"connectionReused":false,"encodedDataLength":546,"fromDiskCache":false,"fromPrefetchCache":false,"fromServiceWorker":false,"headers":{"access-control-allow-credentials":"true","access-control-allow-headers":"Origin, X-Requested-With, Content-Type, multipart/form-data, Accept, Authorization","access-control-allow-methods":"POST, GET, PATCH, DELETE, PUT, OPTIONS","access-control-allow-origin":"*","access-control-max-age":"3600","cache-control":"no-cache,no-store","content-encoding":"gzip","content-language":"zh-CN","content-type":"text/html;charset=UTF-8","date":"Wed, 03 Aug 2022 08:47:33 GMT","hit":"bj-9153118133147527","server":"jfe","strict-transport-security":"max-age=86400","vary":"Accept-Encoding"},"mimeType":"text/html","protocol":"h2","remoteIPAddress":"106.39.169.120","remotePort":443,"responseTime":1.659516451658764e+12,"securityDetails":{"certificateId":0,"certificateTransparencyCompliance":"compliant","cipher":"AES_256_GCM","issuer":"GlobalSign RSA OV SSL CA 2018","keyExchange":"","keyExchangeGroup":"X25519","protocol":"TLS 1.3","sanList":["*.jd.com","*.360buy.com","*.360buyimg.com","*.3.cn","*.7fresh.com","*.baitiao.com","*.chinabank.com.cn","*.e.jd.com","*.jd.co.th","*.jddglobal.com","*.jd.hk","*.jd.id","*.jdpay.com","*.jd.ru","*.jdworldwide.com","*.jdx.com","*.joybuy.com","*.joybuy.es","*.jr.jd.com","*.k.jd.com","*.m.jd.com","*.m.yhd.com","*.shop.jd.com","*.wangyin.com","*.yhd.com","*.yiyaojd.com","360buy.com","360buyimg.com","3.cn","7fresh.com","baitiao.com","chinabank.com.cn","jd.co.th","jddglobal.com","jd.hk","jd.id","jdpay.com","jd.ru","jdworldwide.com","jdx.com","joybuy.com","joybuy.es","wangyin.com","yhd.com","yiyaojd.com","jd.com"],"signedCertificateTimestampList":[{"hashAlgorithm":"SHA-256","logDescription":"Sectigo 'Mammoth' CT log","logId":"6F5376AC31F03119D89900A45115FF77151C11D902C10029068DB2089A37D913","origin":"Embedded in certificate","signatureAlgorithm":"ECDSA","signatureData":"3045022017A2AC492303F50786758D0B4B63EEB8D031850832031FC5A43139C0CDA5EB4F0221009063F884220327718857A6897B87ED5D9F785FFC97F23BD45C84975A11DE721E","status":"Verified","timestamp":1.634109209345e+12},{"hashAlgorithm":"SHA-256","logDescription":"Google 'Argon2022' log","logId":"2979BEF09E393921F056739F63A577E5BE577D9C600AF8F94D5D265C255DC784","origin":"Embedded in certificate","signatureAlgorithm":"ECDSA","signatureData":"3045022071D6D51A59CAA1764E478598AA8EE34B628C8F856B09CDB8382E090CF5D6D97D022100A8FE17E028BB6D439387721974ED3629B9CC44C90AC524B505B9B2375C75CBB9","status":"Verified","timestamp":1.634109210136e+12},{"hashAlgorithm":"SHA-256","logDescription":"Sectigo 'Sabre' CT log","logId":"5581D4C2169036014AEA0B9B573C53F0C0E43878702508172FA3AA1D0713D30C","origin":"Embedded in certificate","signatureAlgorithm":"ECDSA","signatureData":"3045022015D94080264A3FCA83C0F3DE2B85A703384BB678FEBBE5408B4FF7D30BD40900022100DFBAB7992EE99420724CA35A5C2C252298B750994B0EAA98ACB22992F97D545E","status":"Verified","timestamp":1.634109209393e+12}],"subjectName":"*.jd.com","validFrom":1634109205,"validTo":1668410005},"securityState":"secure","status":200,"statusText":"","timing":{"connectEnd":96.834,"connectStart":19.521,"dnsEnd":19.521,"dnsStart":0,"proxyEnd":-1,"proxyStart":-1,"pushEnd":0,"pushStart":0,"receiveHeadersEnd":217.707,"requestTime":2272960.91089,"sendEnd":97.25,"sendStart":97.03,"sslEnd":96.829,"sslStart":55.726,"workerFetchStart":-1,"workerReady":-1,"workerRespondWithSettled":-1,"workerStart":-1},"url":"https://item.m.jd.com/product/10052422060501.html"},"timestamp":2272961.1294,"type":"Document"}},"webview":"78D9BC6F0CBE162DA6779F410AA1500C"}}
我们格式化一下，可以发现日志里已经包含了请求的大部分描述信息，我们需要遍历过滤出method为Network.responseReceived的日志，当然你也可以再根据其他的一些参数过滤出你要的请求
每个日志都有一个requestId，我们再通过该值调CDP命令[Network.getResponseBody]换取具体内容
{ "message": { "method": "Network.responseReceived", "params": { "frameId": "78D9BC6F0CBE162DA6779F410AA1500C", "hasExtraInfo": true, "loaderId": "33814E6BD702343CF0A2A38C976C772F", "requestId": "33814E6BD702343CF0A2A38C976C772F", "response": { "connectionId": 242, "connectionReused": false, "encodedDataLength": 546, "fromDiskCache": false, "fromPrefetchCache": false, "fromServiceWorker": false, "headers": { "access-control-allow-credentials": "true", "access-control-allow-headers": "Origin, X-Requested-With, Content-Type, multipart/form-data, Accept, Authorization", "access-control-allow-methods": "POST, GET, PATCH, DELETE, PUT, OPTIONS", "access-control-allow-origin": "*", "access-control-max-age": "3600", "cache-control": "no-cache,no-store", "content-encoding": "gzip", "content-language": "zh-CN", "content-type": "text/html;charset=UTF-8", "date": "Wed, 03 Aug 2022 08:47:33 GMT", "hit": "bj-9153118133147527", "server": "jfe", "strict-transport-security": "max-age=86400", "vary": "Accept-Encoding" }, "mimeType": "text/html", "protocol": "h2", "remoteIPAddress": "106.39.169.120", "remotePort": 443, "responseTime": 1.659516451658764e+12, "securityDetails": { "certificateId": 0, "certificateTransparencyCompliance": "compliant", "cipher": "AES_256_GCM", "issuer": "GlobalSign RSA OV SSL CA 2018", "keyExchange": "", "keyExchangeGroup": "X25519", "protocol": "TLS 1.3", "sanList": ["*.jd.com", "*.360buy.com", "*.360buyimg.com", "*.3.cn", "*.7fresh.com", "*.baitiao.com", "*.chinabank.com.cn", "*.e.jd.com", "*.jd.co.th", "*.jddglobal.com", "*.jd.hk", "*.jd.id", "*.jdpay.com", "*.jd.ru", "*.jdworldwide.com", "*.jdx.com", "*.joybuy.com", "*.joybuy.es", "*.jr.jd.com", "*.k.jd.com", "*.m.jd.com", "*.m.yhd.com", "*.shop.jd.com", "*.wangyin.com", "*.yhd.com", "*.yiyaojd.com", "360buy.com", "360buyimg.com", "3.cn", "7fresh.com", "baitiao.com", "chinabank.com.cn", "jd.co.th", "jddglobal.com", "jd.hk", "jd.id", "jdpay.com", "jd.ru", "jdworldwide.com", "jdx.com", "joybuy.com", "joybuy.es", "wangyin.com", "yhd.com", "yiyaojd.com", "jd.com"], "signedCertificateTimestampList": [{ "hashAlgorithm": "SHA-256", "logDescription": "Sectigo 'Mammoth' CT log", "logId": "6F5376AC31F03119D89900A45115FF77151C11D902C10029068DB2089A37D913", "origin": "Embedded in certificate", "signatureAlgorithm": "ECDSA", "signatureData": "3045022017A2AC492303F50786758D0B4B63EEB8D031850832031FC5A43139C0CDA5EB4F0221009063F884220327718857A6897B87ED5D9F785FFC97F23BD45C84975A11DE721E", "status": "Verified", "timestamp": 1.634109209345e+12 }, { "hashAlgorithm": "SHA-256", "logDescription": "Google 'Argon2022' log", "logId": "2979BEF09E393921F056739F63A577E5BE577D9C600AF8F94D5D265C255DC784", "origin": "Embedded in certificate", "signatureAlgorithm": "ECDSA", "signatureData": "3045022071D6D51A59CAA1764E478598AA8EE34B628C8F856B09CDB8382E090CF5D6D97D022100A8FE17E028BB6D439387721974ED3629B9CC44C90AC524B505B9B2375C75CBB9", "status": "Verified", "timestamp": 1.634109210136e+12 }, { "hashAlgorithm": "SHA-256", "logDescription": "Sectigo 'Sabre' CT log", "logId": "5581D4C2169036014AEA0B9B573C53F0C0E43878702508172FA3AA1D0713D30C", "origin": "Embedded in certificate", "signatureAlgorithm": "ECDSA", "signatureData": "3045022015D94080264A3FCA83C0F3DE2B85A703384BB678FEBBE5408B4FF7D30BD40900022100DFBAB7992EE99420724CA35A5C2C252298B750994B0EAA98ACB22992F97D545E", "status": "Verified", "timestamp": 1.634109209393e+12 }], "subjectName": "*.jd.com", "validFrom": 1634109205, "validTo": 1668410005 }, "securityState": "secure", "status": 200, "statusText": "", "timing": { "connectEnd": 96.834, "connectStart": 19.521, "dnsEnd": 19.521, "dnsStart": 0, "proxyEnd": -1, "proxyStart": -1, "pushEnd": 0, "pushStart": 0, "receiveHeadersEnd": 217.707, "requestTime": 2272960.91089, "sendEnd": 97.25, "sendStart": 97.03, "sslEnd": 96.829, "sslStart": 55.726, "workerFetchStart": -1, "workerReady": -1, "workerRespondWithSettled": -1, "workerStart": -1 }, "url": "https://item.m.jd.com/product/10052422060501.html" }, "timestamp": 2272961.1294, "type": "Document" } }, "webview": "78D9BC6F0CBE162DA6779F410AA1500C" }
完整代码：
var option = new ChromeOptions(); option.SetLoggingPreference("performance",OpenQA.Selenium.LogLevel.Info); //启用performance日志，等级为Info即可 option.PerformanceLoggingPreferences = new ChromiumPerformanceLoggingPreferences() { IsCollectingNetworkEvents = true //采集网络请求事件 }; using (ChromeDriver driver = new ChromeDriver(driverPath,option,TimeSpan.FromSeconds(5))) { driver.Navigate().GoToUrl("https://item.m.jd.com/product/10052422060501.html"); Thread.Sleep(3 * 1000); //等待页面加载完成 var logs = driver.Manage().Logs.GetLog("performance").Where(o => o.Message.Contains(""Network.responseReceived""))；//获取所有performance日志，并过滤出所有类型为Network.responseReceived的日志 foreach (var log in logs) { //日志找不到就会抛出异常，必须要捕获异常 try { var json = JObject.Parse(log.Message); var url = json["message"]["params"]["response"]["url"].ToString(); //请求url，可通过url过滤出你要的请求 var requestId = json["message"]["params"]["requestId"].ToString(); //利用RequestId做为参数，执行CDP命令获取日志详细内容。 踩坑警告：返回的是一个字典，需要转换为Dictionary<string,object> var response = driver.ExecuteCdpCommand("Network.getResponseBody",new Dictionary<string,object>() {{ "requestId",requestId }}) as Dictionary<string,object>; if (response.TryGetValue("body",out object? bodyObj) && bodyObj != null) { string body = bodyObj.ToString(); Console.WriteLine($"输出Body内容：{body}"); } } catch(Exception ex){ //记录错误日志 } } }
封装
为了以后方便复用，我封装了一个类
其中过滤条件的入参，可以根据实际情况自行修改，如果不需要对body内容进行过滤可以去掉这个参数，这样就不用每次等待拿到结果再过滤，性能会好很多
/// <summary> /// Selenium网络请求日志帮助类 /// </summary> public class NetworkLoggingHelper { static readonly Logger logger = NlogProvider.GetLogger(); /// <summary> /// 开启网络请求日志 /// </summary> /// <param name="option"></param> public static void OpenNetworkPerformanceLogging(ref ChromeOptions option) { option.SetLoggingPreference("performance",OpenQA.Selenium.LogLevel.Info); option.PerformanceLoggingPreferences = new ChromiumPerformanceLoggingPreferences() { IsCollectingNetworkEvents = true }; } /// <summary> /// 获取网络请求数据 /// </summary> /// <param name="driver"></param> /// <param name="filter">过滤条件 入参1：请求的url | 入参2：请求的mimeType | 入参3：请求的body</param> /// <returns></returns> public static Dictionary<string,string> GetNetworkApiDatas(ChromeDriver driver,Func<string,string,string,bool> filter) { Dictionary<string,string> datas = new Dictionary<string,string>(); try { var logs = driver.Manage().Logs.GetLog("performance")?.Where(o => o.Message.Contains(""Network.responseReceived"")); foreach (var log in logs) { try { var json = JObject.Parse(log.Message); if (json["message"]["params"] == null || json["message"]["params"]["response"] == null) { continue; } var url = json["message"]["params"]["response"]["url"].ToString(); var mimeType = json["message"]["params"]["response"]["mimeType"].ToString(); var requestId = json["message"]["params"]["requestId"].ToString(); var response = driver.ExecuteCdpCommand("Network.getResponseBody",new Dictionary<string,object>() { { "requestId",requestId } }) as Dictionary<string,object>; if (response != null && response.Count > 0) { //是否base64编码 var isBase64Encode = false; if (response.TryGetValue("base64Encoded",out object? base64Encoded) && base64Encoded != null) { isBase64Encode = (bool)base64Encoded; } //获取响应内容 string body = string.Empty; if (response.TryGetValue("body",out object? bodyObj) && bodyObj != null) { body = bodyObj.ToString(); if (isBase64Encode) { body = body.DecodeBase64(Encoding.UTF8); } } //根据条件过滤，如果不需要body内容参与过滤条件判断，这个if语句可以移到获取response的上面，性能会好很多 if (filter.Invoke(url,mimeType,body)) { datas.Add(url,body); } } } catch (Exception ex) { logger.Error(ex.Message); } } } catch (Exception ex) { logger.Error($"获取日志失败：{ex.Message}"); } return datas; } }
调用示例：
var option = new ChromeOptions(); NetworkLoggingHelper.OpenNetworkPerformanceLogging(ref option); //开启日志 using (ChromeDriver driver = new ChromeDriver(driverPath,option,TimeSpan.FromSeconds(5))) { driver.Navigate().GoToUrl("https://item.m.jd.com/product/10052422060501.html"); Thread.Sleep(3 * 1000); //等待页面加载完成 var datas = NetworkLoggingHelper.GetNetworkApiDatas(driver,(url,mimeType,body) => { return url.Contains("//item.m.jd.com/product/") || mimeType.Contains("application/json") || body.Contains("windows.itemInfo"); }); Console.WriteLine(datas.Count); }
Selenium工作原理
叙述一下selenium工作的过程
1.selenium client(python等语言编写的自动化测试脚本)初始化一个service服务，通过Webdriver启动浏览器驱动程序chromedriver.exe
2.通过RemoteWebDriver向浏览器驱动程序发送HTTP请求，浏览器驱动程序解析请求，打开浏览器，并获得sessionid，如果再次对浏览器操作需携带此id
3.打开浏览器，绑定特定的端口，把启动后的浏览器作为webdriver的remote server
3.打开浏览器后，所有的selenium的操作(访问地址，查找元素等)均通过RemoteConnection链接到remote server，然后使用execute方法调用_request方法通过urlib3向remote server发送请求
4.浏览器通过请求的内容执行对应动作
5.浏览器再把执行的动作结果通过浏览器驱动程序返回给测试脚本
6. webdriver.是 w3c 的标准协议。提供一组接口,用于发现和操作web文档中的DOM元素。
webdriver.是一系列的API.它给测试代码提供了定位和操作 wEB元素的能力。不同的开发语言有相应的wsbdriver.
↑↑↑↑ 该部分内容来自：https://www.cnblogs.com/xrxc/p/14776895.html
作者：Harry
原文出处：https://www.cnblogs.com/simendancer/articles/16546199.html
有些文本描述和图片源自网络，如有侵犯请私信告知
