Fine tune performance-开源项目-CSDN问答

来自 : CSDN技术社区 发布时间：2021-03-24

私信 访问主页

weixin_39622901 2021-01-02 07:25 首页 开源项目 Fine tune performance Scrolling can be sluggish on files which have a very large number of lines. So far I have seen scrolling stutter on a file with 800k lines which was opened on a server share. In theory this can be solved by adjusting the scrolling sample rate, but in practice it is hard to predict when back pressures are formed. The idea solution would be for a task to run and if a queue has formed disregard all but the end of the queued job.Some times there is a minor latency when opening a file.Investigate whether the read algorithms can be improved

该提问来源于开源项目 RolandPheasant/TailBlazer

点赞 写回答 收藏 复制链接分享 删除 再等等 结题 再想想 22条回答

私信 访问主页

weixin_39622901 2月前

My suspicion was correct. I generated a file with lines which are greater than 10kB each and Tail Blazer ground to a halt

Now I know what the issue is I will work out a solution

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

The only time I have seen performance suffer is when there are very long lines in the log file. Is this the case for you? Is so then I am aware of the issue.

Is it possible for you to upload a sample file here so I can open it on my machine, or is the content sensitive?

点赞 评论 复制链接分享

私信 访问主页

weixin_39907220 2月前

I ve been testing version 0.5 and there are two major issues in the way it handles tailing my log files:- CPU usage is maxing out on a moderately updated log file ( 1 MB, sane line lengths)- Sometimes the UI seems to hang on a new line, becoming jittery and resuming normal operation after about half a second. This also happens when scrolling through files without updates happening.

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

version 0.3 should be much much better. Please try and let me know how it goes.

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

: 1: Thanks for checking. Someone in work just showed me a folder full of 20 Gigabyte sized log files so handling files that size has become my new challenge.

点赞 评论 复制链接分享

私信 访问主页

weixin_39907220 2月前

Performance has improved! I ll be testing some more tomorrow.

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

I have done lots of work on performance and found a few issues clearly slowing things down. The latest release (see v0.2.2) should be a lot quicker. I would appreciate if you take a look.

At the weekend I will be doing a release which according to my tests should be highly performing even for gigabyte sized files.

If these don t fix the performance for you than perhaps it could be a graphics issue

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

Again thanks for the input

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

Blimey! I will add a load of more logging to help get to the bottom of this.

点赞 评论 复制链接分享

私信 访问主页

weixin_39907220 2月前

Alright, but just to make sure: my log files are locally accessed, not over a network share.

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

I have raise issue #30 to deal this this one and will be opening issues specifically for different performance problems. I will let you know when that is done

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

Thanks for that. I suspect the underlying problem is network latency coupled with too fast a poll interval. I am looking in work at an active 1.3 M line log file (100 MB in size) over a network share and my cpu is doing almost nothing

So to address the issue I will:

a) slow down the poll rateb) rather than polling on a schedule, poll and allow for inactivity before polling again

I suspect in your instance the time it takes to check for changes and subsequently load the changes is longer than the polling interval leading to an ever growing queue of updates. I will add a load of logging soon.

点赞 评论 复制链接分享

私信 访问主页

weixin_39907220 2月前

Decoding seems to be fixed, great work! Tailblazer seems very CPU bound at the moment, it barely responds when other applications are taxing my CPU at the same time, even for small log files.

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

the files should decode correctly now.

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

I think I have worked out the issue for both problems. Should be able to get a fix out in a day or 2.

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

I just spotted that . Seems to be the way I have calculated the line lengths when indexing the file. I am on the case. Is it possible to attach as smallish extract here so I can test the indexing on your file type.

Edit: I will fix the encoding / indexing then I will address the scroll issue you have - my suspicion is the scroll issue is my sample rate is actually too fast which leads to a queue of updates hitting the screen at once.

点赞 评论 复制链接分享

私信 访问主页

weixin_39907220 2月前

I tried the latest release but it isn t able to read my log files correctly. Lines are garbled and distorted, sometimes beyond recognition. They are fairly standard WildFly log files.

So I m not really able to test your changes. However, it does seem like the UI updates every 500ms making it a somewhat slow experience compared to other tail tools like Baretail.

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

Its fast now check out this Fast Scrolling

For super large files probably more work required but for now I am going to put my feet up

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

and I were also discussing some more sophisticated indexing options and there is some overlap with your suggestion.

You can be assured that fine tuning performance is my current priorityThanks

点赞 评论 复制链接分享

私信 访问主页

weixin_39907220 2月前

You could reduce memory pressure by keeping a sparse index, e.g. indexing every 100th line instead of every line and using seeks for all lines in between. Doesn t have to be a constant, you can vary it by distance for example (index every line within 1000 lines of the current position, and decrease the resolution the further you go).

Alternatively, some tail tools (such as BareTail) give you the option to just keep the last X lines of the file in memory, discarding everything that falls off.

点赞 评论 复制链接分享

私信 访问主页

weixin_39622901 2月前

I have just been experimenting with indexing all the lines and have been able to achieve blisteringly fast scrolling. When the file is indexed, stream.seek method is used to load the current page.

The largest file I have tested has 1.3 million lines and it scrolls very well. Of course the indexing adds to memory pressures but the experiment has shown me I am on the right track.

点赞 评论 复制链接分享

私信 访问主页

weixin_39907220 2月前

1 on the scrolling. It seems that the UI is updated every 500ms or so, while other tools update instantaneously.

Also, when heavy file update activity occurs, the application seems to hog 100% of one CPU core.

点赞 评论 复制链接分享 提交 再想想 采纳 为你推荐 通过Golang SDK将大文件发送到Amazon S3时如何使用较少的内存？ memory 1个回答如何在Golang中实现可变字节编码算法 algorithm 1个回答图片文件夹的更短路径-Golang it技术互联网问答IT行业问题计算机技术编程语言问答 1个回答了解AppEngine的Go运行时上的“运行时mcycles”和“ cpu_ms”记帐 it技术互联网问答IT行业问题计算机技术编程语言问答 1个回答无效的内存地址或nil指针取消引用 it技术互联网问答IT行业问题计算机技术编程语言问答 2个回答在Golang中解析日期和时间的最佳方法 timeparsingperformancedate 2个回答Cgo：抱歉，未实现：未在64位模式下编译 cgo 2个回答将Web应用程序部署为独立可执行文件 nsisdeploymentphpgwt 4个回答如何在PHP中将日期时间划分为单独的日期 datetimephpdate 2个回答以程序方式编写一个相当大的，以数据库为中心的PHP应用程序而没有MVC是否合理？ oopphp 2个回答使用PHP从文本文件中打印给定数量的行 user-inputphpfile-io 1个回答PHP PDO_ODBC配置：错误：C编译器无法创建可执行文件 cdb2pdophpc++ 1个回答如何加快通过Amazon SES发送邮件的速度？ curlphp 1个回答如何删除网址[关闭] stringphp 3个回答cakePHP表现 mysqlphpperformance 2个回答无法构建pthreads：对\'zend_hash_index_insert\'的未定义引用 pthreadsphpcompilation 1个回答Facebook通知的cURL循环 curlfacebookphp 1个回答Json从python到php 5.3 jsonpythonphp 1个回答wordpress使用主题背景图不显示 html5phpchrome 1个回答更好的条带化方法php正则表达式 regexhtmlphp 4个回答 点击登录 提问题 欢迎建议意见 . 如何写高质量提问和回答？ 采纳榜7天 被采纳次数 ProfSnail 25 幻灰龙 19 qq_34124780 154 coagenth 115 规则边缘 116 GoCityPass新加坡曼谷通票 117 cpp_learner 98 Jack_Yang(数据分析及可视化) 99 我不喜欢这个世界 810 天际的海浪 6 加急问题 500 500 基于tensorflow的模型做预测时，cpu占用率过高，如何降低程序的cpu占用率？ 100 unity 设置Animation的culling type 导致游戏崩溃 100 java实现下面两个算法 100 ECDSA加密算法中，签名和验证都是需要时间的，那其时间复杂度怎么用椭圆曲线参数如阶或模p来表示呢？ 75 数据库只有myd文件没有frm,myi文件，怎么打开呢 51 利用Adams联合matlab仿真求运动学反解问题 50 labview使用datasocket与c/c#程序的通信 50 python引用xlwings出问题，这个报错什么意思 ，怎么解决，谢谢大佬 50 如何用C语言实现凯撒密码对文本文件(.txt)的读取穷举暴力破解？ 50 已知地图GPS坐标点A和坐标点B，获取A点往B点直线走一定距离的坐标点经纬度？