跳至主要内容

tomcat进程消失(被杀)问题

缘由

最近，一个机器上的tomcat进程莫名消失了，为了暂时解决问题，设置提升了内存设置。但是，之后过了没多久tomcat又消失了。为了排查问题，我仔细分析了机器上的log。
一开始去查看运行目录是否有dump文件，发现并没有，看似并不是OutOfMemoryError的问题, 也没有JVM crash之后会生成的hs_err_pidxxx.log, JVM也没有出错。
再去查看tomcat的日志，发现也没有异常，突然就结束了，看起来也并不是tomcat崩溃了。
不是JVM崩溃，也不是tomcat崩溃，这个让我们很困惑。通过搜索，得知可能是被os杀掉了。
通过demsg命令，得到如下的输出:

Java进程被杀

google之后，发现有可能是linux的奇怪特性。

分析

memory overcommit

linux有一个比较奇怪的特性，叫memory overcommit。详细内容可以参考这里。
主要表现就是malloc永远会返回一个合法的指针。举个例子，假设我们的机器有4G内存，有两个JVM，每个JVM都说我要4G，linux仍然会成功的分配给他们。

这个特性对于大多数程序来说没有影响，因为一般程序不会占满自己申请的内存，他们可能只用申请的30%内存，于是相安无事。

JVM and GC

但是对于JVM，尤其是长时间运行的JVM，memory overcommit会有很大影响。
对于大多数GC算法来说，只有在堆分配不出内存的时候才会进行GC活动，释放到达不了的object。
回到上面的例子，假设两个JVM长时间运行，内存都逼近1.5G，对于JVM来说，自己才使用了1.5G内存，可能新生代还很空，没有必要进行GC。但是linux不一样，他知道实际的内存(3G+内核+其他软件，比如vim)可能已经没有多少了，于是启动oom-killer杀死某一个进程（根据运行时间，内存占用等决定），而长时间运行，占用大量内存的Java进程很容易被选择。

总结

综上，如果tomcat真的是每次被内核杀死，要做的不是提高Xmx，Xms，反而是降低。
不过即使不是每次都被内核杀死，我也建议在beta机器上降低内存设置，试着让tomcat长时间的跑，看会不会OutOfMemoryError，并且看看那时JVM产生的dump，更加利于定位内存问题。

另外：

IBM Heap Analyzer其实挺坑爹的（附连接）：

他的root object根本不只是gc root，还包含所有可以被gc但是还没有被gc的object。所以他显示的leak suspect根本大多是可回收的对象，容易造成误解。

附

类似的问题

Written with StackEdit.

标签：

评论

发表评论

此博客中的热门博文

Spring Boot: Customize Environment

Spring Boot: Customize Environment Environment variable is a very commonly used feature in daily programming: used in init script used in startup configuration used by logging etc In Spring Boot, all environment variables are a part of properties in Spring context and managed by Environment abstraction. Because Spring Boot can handle the parse of configuration files, when we want to implement a project which uses yml file as a separate config file, we choose the Spring Boot. The following is the problems we met when we implementing the parse of yml file and it is recorded for future reader. Bind to Class Property values can be injected directly into your beans using the @Value annotation, accessed via Spring’s Environment abstraction or bound to structured objects via @ConfigurationProperties. As the document says, there exists three ways to access properties in *.properties or *.yml : @Value : access single value Environment : can access multi...
5 条评论
阅读全文

Elasticsearch: Join and SubQuery

Elasticsearch: Join and SubQuery Tony was bothered by the recent change of search engine requirement: they want the functionality of SQL-like join in Elasticsearch! “They are crazy! How can they think like that. Didn’t they understand that Elasticsearch is kind-of NoSQL 1 in which every index should be independent and self-contained? In this way, every index can work independently and scale as they like without considering other indexes, so the performance can boost. Following this design principle, Elasticsearch has little related supports.” Tony thought, after listening their requirements. Leader notice tony’s unwillingness and said, “Maybe it is hard to do, but the requirement is reasonable. We need to search person by his friends, didn’t we? What’s more, the harder to implement, the more you can learn from it, right?” Tony thought leader’s word does make sense so he set out to do the related implementations Application-Side Join “The first implementation ...
发表评论
阅读全文

Learn Spring Expression Language

When reading the source code of some Spring based projects, we can see some code like following: @Value( "${env}" ) private int value ; and like following: @Autowired public void configure (MovieFinder movieFinder, @ Value ("#{ systemProperties[ 'user.region' ] } ") String defaultLocale) { this.movieFinder = movieFinder; this.defaultLocale = defaultLocale; } In this way, we can inject values from different sources very conveniently, and this is the features of Spring EL. What is Spring EL? How to use this handy feature to assist our developments? Today, we are going to learn some basics of Spring EL. Features The full name of Spring EL is Spring Expression Language, which exists in form of Java string and evaluated by Spring. It supports many syntax, from simple property access to complex safe navigation – method invocation when object is not null. And the following is the feature list from Spring EL document : ...
发表评论
阅读全文