https://xgugeng.github.io/zh/tags/%E8%8B%A6%E9%80%BC%E8%BF%90%E7%BB%B4/ Recent content in 苦逼运维 on XGUGENG Hugo zh-CN Mon, 07 Jan 2019 14:11:18 +0800 https://xgugeng.github.io/zh/post/redis-anomalous-latency-because-of-page-fault/ Mon, 07 Jan 2019 14:11:18 +0800 https://xgugeng.github.io/zh/post/redis-anomalous-latency-because-of-page-fault/ <p>继续每天的知心姐姐时间，这次客户邮件说，CPU 使用率飙升，同时 GET/LLEN 等命令，都要执行几十毫秒。看了一下慢日志，我去：</p> <pre tabindex="0"><code>Id:900909 When:(1/1/2019 6:16:15 PM) Duration:41564us Command:(Llen PRIVATE ARG) Id:900908 When:(1/1/2019 6:16:14 PM) Duration:17041us Command:(get PRIVATE ARG) Id:900907 When:(1/1/2019 6:16:14 PM) Duration:51032us Command:(get PRIVATE ARG) Id:900906 When:(1/1/2019 6:16:12 PM) Duration:49292us Command:(get PRIVATE ARG) Id:900905 When:(1/1/2019 6:16:11 PM) Duration:56329us Command:(Llen PRIVATE ARG) Id:900904 When:(1/1/2019 6:16:10 PM) Duration:10562us Command:(get PRIVATE ARG) Id:900903 When:(1/1/2019 6:16:10 PM) Duration:35680us Command:(get PRIVATE ARG) Id:900902 When:(1/1/2019 6:16:10 PM) Duration:16367us Command:(get PRIVATE ARG) Id:900901 When:(1/1/2019 6:16:10 PM) Duration:29851us Command:(Llen PRIVATE ARG) Id:900900 When:(1/1/2019 6:16:08 PM) Duration:29050us Command:(get PRIVATE ARG) </code></pre> https://xgugeng.github.io/zh/post/100-server-load-because-of-redis-sync/ Wed, 28 Nov 2018 14:25:25 +0800 https://xgugeng.github.io/zh/post/100-server-load-because-of-redis-sync/ <p>今天一早，就有客户来报说机器负载 100%。从日志中看到的现象是，Slave 向 Master 请求部分同步，Master 却要求 Slave 做个全套：全量同步，因为 Slave 发过来的 replication ID 太老了。Slave 说好吧，整个全的，然后消费成功。接着，Slave 还想来个部分同步，上述的过程连续发生 3 次。Master 有点强买强卖的意思了，这样一来两边的 CPU 不干了，这不是加班嘛！二者之间的网络带宽也是突增，从下图可以看出：</p> <p><img src="https://xgugeng.github.io/images/redis-network-bandwidth-spiking.png" alt=""></p>