[vCSA 7.0] KB 85475 の Workaround 実施手順

vCSA 7.0 U3c 未満で /storage/log の容量枯渇の一因となる以下 KB 85475 の
Workaround 実施手順を以下に記載します。

・vCenter Server /storage/log filling up due to localhost_access.log and catalina.log in sso and lookupsvc log directories (85475)
  https://kb.vmware.com/s/article/85475

検証環境

vCenter Server Appliance 7.0 U2d

KB 85475 の Workaround 実施前の状態

まずは KB 85475 の Workaround を実施する前の状態を記録しておくため、
対象の各ディレクトリの状態を記録しておきます。

・/var/log/vmware/sso/tomcat

root@test [ /var/log/vmware/eam/web ]# ls /var/log/vmware/sso/tomcat
catalina.2022-08-28.log   localhost_access.2022-09-07.log  localhost_access.2022-09-29.log  localhost_access.2022-10-13.log  localhost_access.2022-10-26.log
catalina.2022-09-07.log   localhost_access.2022-09-08.log  localhost_access.2022-09-30.log  localhost_access.2022-10-14.log  localhost_access.2022-10-27.log
catalina.2022-09-16.log   localhost_access.2022-09-09.log  localhost_access.2022-10-01.log  localhost_access.2022-10-15.log  localhost_access.2022-10-28.log
catalina.2022-09-25.log   localhost_access.2022-09-10.log  localhost_access.2022-10-02.log  localhost_access.2022-10-16.log  localhost_access.2022-10-29.log
catalina.2022-10-07.log   localhost_access.2022-09-11.log  localhost_access.2022-10-03.log  localhost_access.2022-10-17.log  localhost_access.2022-10-30.log
catalina.2022-11-06.log   localhost_access.2022-09-12.log  localhost_access.2022-10-04.log  localhost_access.2022-10-18.log  localhost_access.2022-10-31.log
host-manager.log          localhost_access.2022-09-13.log  localhost_access.2022-10-05.log  localhost_access.2022-10-19.log  localhost_access.2022-11-01.log
localhost.2022-08-28.log  localhost_access.2022-09-14.log  localhost_access.2022-10-07.log  localhost_access.2022-10-20.log  localhost_access.2022-11-02.log
localhost.2022-09-07.log  localhost_access.2022-09-15.log  localhost_access.2022-10-08.log  localhost_access.2022-10-21.log  localhost_access.2022-11-03.log
localhost.2022-09-16.log  localhost_access.2022-09-25.log  localhost_access.2022-10-09.log  localhost_access.2022-10-22.log  localhost_access.2022-11-04.log
localhost.2022-09-25.log  localhost_access.2022-09-26.log  localhost_access.2022-10-10.log  localhost_access.2022-10-23.log  localhost_access.2022-11-05.log
localhost.2022-10-07.log  localhost_access.2022-09-27.log  localhost_access.2022-10-11.log  localhost_access.2022-10-24.log  localhost_access.log
localhost.2022-11-06.log  localhost_access.2022-09-28.log  localhost_access.2022-10-12.log  localhost_access.2022-10-25.log  manager.log

・/var/log/vmware/eam/web

root@test [ /var/log/vmware/eam/web ]# ls /var/log/vmware/eam/web
catalina-2022-08-28.log  catalina-2022-10-10.log  catalina-2022-11-01.log          localhost_access.2022-09-27.log  localhost_access.2022-10-20.log
catalina-2022-09-07.log  catalina-2022-10-11.log  catalina-2022-11-02.log          localhost_access.2022-09-28.log  localhost_access.2022-10-21.log
catalina-2022-09-11.log  catalina-2022-10-12.log  catalina-2022-11-03.log          localhost_access.2022-09-29.log  localhost_access.2022-10-22.log
catalina-2022-09-12.log  catalina-2022-10-13.log  catalina-2022-11-04.log          localhost_access.2022-09-30.log  localhost_access.2022-10-23.log
catalina-2022-09-13.log  catalina-2022-10-14.log  catalina-2022-11-05.log          localhost_access.2022-10-01.log  localhost_access.2022-10-24.log
catalina-2022-09-14.log  catalina-2022-10-15.log  catalina.log                     localhost_access.2022-10-02.log  localhost_access.2022-10-25.log
catalina-2022-09-15.log  catalina-2022-10-16.log  localhost-2022-08-28.log         localhost_access.2022-10-03.log  localhost_access.2022-10-26.log
catalina-2022-09-16.log  catalina-2022-10-17.log  localhost-2022-09-07.log         localhost_access.2022-10-04.log  localhost_access.2022-10-27.log
catalina-2022-09-25.log  catalina-2022-10-18.log  localhost-2022-09-16.log         localhost_access.2022-10-05.log  localhost_access.2022-10-28.log
catalina-2022-09-26.log  catalina-2022-10-19.log  localhost-2022-09-25.log         localhost_access.2022-10-07.log  localhost_access.2022-10-29.log
catalina-2022-09-27.log  catalina-2022-10-20.log  localhost-2022-10-07.log         localhost_access.2022-10-08.log  localhost_access.2022-10-30.log
catalina-2022-09-28.log  catalina-2022-10-21.log  localhost_access.2022-09-07.log  localhost_access.2022-10-09.log  localhost_access.2022-10-31.log
catalina-2022-09-29.log  catalina-2022-10-22.log  localhost_access.2022-09-08.log  localhost_access.2022-10-10.log  localhost_access.2022-11-01.log
catalina-2022-09-30.log  catalina-2022-10-23.log  localhost_access.2022-09-09.log  localhost_access.2022-10-11.log  localhost_access.2022-11-02.log
catalina-2022-10-01.log  catalina-2022-10-24.log  localhost_access.2022-09-10.log  localhost_access.2022-10-12.log  localhost_access.2022-11-03.log
catalina-2022-10-02.log  catalina-2022-10-25.log  localhost_access.2022-09-11.log  localhost_access.2022-10-13.log  localhost_access.2022-11-04.log
catalina-2022-10-03.log  catalina-2022-10-26.log  localhost_access.2022-09-12.log  localhost_access.2022-10-14.log  localhost_access.2022-11-05.log
catalina-2022-10-04.log  catalina-2022-10-27.log  localhost_access.2022-09-13.log  localhost_access.2022-10-15.log  localhost_access.log
catalina-2022-10-05.log  catalina-2022-10-28.log  localhost_access.2022-09-14.log  localhost_access.2022-10-16.log  localhost.log
catalina-2022-10-07.log  catalina-2022-10-29.log  localhost_access.2022-09-15.log  localhost_access.2022-10-17.log
catalina-2022-10-08.log  catalina-2022-10-30.log  localhost_access.2022-09-25.log  localhost_access.2022-10-18.log
catalina-2022-10-09.log  catalina-2022-10-31.log  localhost_access.2022-09-26.log  localhost_access.2022-10-19.log

・/var/log/vmware/lookupsvc/tomcat

root@test [ /var/log/vmware/eam/web ]# ls /var/log/vmware/lookupsvc/tomcat
catalina.2022-08-28.log          localhost_access.2022-09-09.log  localhost_access.2022-10-08.log  localhost_access.2022-10-27.log
catalina.2022-09-07.log          localhost_access.2022-09-10.log  localhost_access.2022-10-09.log  localhost_access.2022-10-28.log
catalina.2022-09-16.log          localhost_access.2022-09-11.log  localhost_access.2022-10-10.log  localhost_access.2022-10-29.log
catalina.2022-09-25.log          localhost_access.2022-09-12.log  localhost_access.2022-10-11.log  localhost_access.2022-10-30.log
catalina.2022-10-07.log          localhost_access.2022-09-13.log  localhost_access.2022-10-12.log  localhost_access.2022-10-31.log
catalina.2022-11-06.log          localhost_access.2022-09-14.log  localhost_access.2022-10-13.log  localhost_access.2022-11-01.log
host-manager.2022-08-28.log      localhost_access.2022-09-15.log  localhost_access.2022-10-14.log  localhost_access.2022-11-02.log
host-manager.2022-09-07.log      localhost_access.2022-09-25.log  localhost_access.2022-10-15.log  localhost_access.2022-11-03.log
host-manager.2022-09-25.log      localhost_access.2022-09-26.log  localhost_access.2022-10-16.log  localhost_access.2022-11-04.log
host-manager.2022-10-07.log      localhost_access.2022-09-27.log  localhost_access.2022-10-17.log  localhost_access.2022-11-05.log
host-manager.2022-11-06.log      localhost_access.2022-09-28.log  localhost_access.2022-10-18.log  localhost_access.log
localhost.2022-08-28.log         localhost_access.2022-09-29.log  localhost_access.2022-10-19.log  manager.2022-08-28.log
localhost.2022-09-07.log         localhost_access.2022-09-30.log  localhost_access.2022-10-20.log  manager.2022-09-07.log
localhost.2022-09-16.log         localhost_access.2022-10-01.log  localhost_access.2022-10-21.log  manager.2022-09-25.log
localhost.2022-09-25.log         localhost_access.2022-10-02.log  localhost_access.2022-10-22.log  manager.2022-10-07.log
localhost.2022-10-07.log         localhost_access.2022-10-03.log  localhost_access.2022-10-23.log  manager.2022-11-06.log
localhost.2022-11-06.log         localhost_access.2022-10-04.log  localhost_access.2022-10-24.log
localhost_access.2022-09-07.log  localhost_access.2022-10-05.log  localhost_access.2022-10-25.log
localhost_access.2022-09-08.log  localhost_access.2022-10-07.log  localhost_access.2022-10-26.log

・/var/log/vmware/lookupsvc

root@test [ /var/log/vmware/eam/web ]# ls /var/log/vmware/lookupsvc
logs                           lookupsvc_stream.log-1.stderr  lookupsvc_stream.log-4.stderr  vmware-lookupsvc-gc.log.0          vmware-lookupsvc-gc.log.5
lookupserver-default.log       lookupsvc_stream.log-1.stdout  lookupsvc_stream.log-4.stdout  vmware-lookupsvc-gc.log.0.current  vmware-lookupsvc-gc.log.6
lookupServer.log               lookupsvc_stream.log-2.stderr  lookupsvc_stream.log.stderr    vmware-lookupsvc-gc.log.1          vmware-lookupsvc-gc.log.7
lookupsvc-init.log             lookupsvc_stream.log-2.stdout  lookupsvc_stream.log.stdout    vmware-lookupsvc-gc.log.2          vmware-lookupsvc-gc.log.8
lookupsvc_stream.log-0.stderr  lookupsvc_stream.log-3.stderr  tomcat                         vmware-lookupsvc-gc.log.3.current  vmware-lookupsvc-gc.log.9
lookupsvc_stream.log-0.stdout  lookupsvc_stream.log-3.stdout  vmware-lookupservice-perf.log  vmware-lookupsvc-gc.log.4

KB 85475 の Workaround 実施手順

まずは KB 85475 の Attachments から [configs] , [tomcat_conpress] を
ダウンロードし、ファイルを解凍してください。

次に、WinSCP 等の SCP クライアントで対象の vCenter Server へ
接続してください。

※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※
WinSCP で vCenter Server に接続する場合、予めデフォルトシェルを bash に変更
する必要があります。

そのため、以下 KB 2107727 の手順 1 〜 5 までを実施した後に WinSCP での
接続を実施ください。

・WinSCP を使用してファイルを vCenter Server Appliance にアップロードすると
  エラーが発生する (2107727)
  https://kb.vmware.com/s/article/2107727?lang=ja
※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※

次に、KB 85475 の Attachments からダウンロードした [configs.zip] を
解凍した後、[configs] フォルダ内の以下の各種ファイルの名前を変更し、
対象のディレクトリに存在する既存のファイルと置き換えます。
※ファイル冒頭に記載されている “LOOKSVC_”, “SSO_” は削除する必要
があります。

[LOOKSVC_logging.properties]
---> /usr/lib/vmware-lookupsvc/conf/logging.properties

[LOOKSVC_server.xml]
--->/usr/lib/vmware-lookupsvc/conf/server.xml

[SSO_logging.properties]
---> /usr/lib/vmware-sso/vmware-sts/conf/logging.properties

[SSO_server.xml]
---> /usr/lib/vmware-sso/vmware-sts/conf/server.xml

名前を修正した [LOOKSVC_logging.properties], [LOOKSVC_server.xml]
を /usr/lib/vmware-lookupsvc/conf ディレクトリ配下にドラッグ&ドロップ。
(上書きコピーしてください。)

名前を修正した [SSO_logging.properties], [SSO_server.xml] を
/usr/lib/vmware-sso/vmware-sts/conf ディレクトリ配下にドラッグ
&ドロップ。(上書きコピーしてください。)

各種ファイルの置き換えが完了したら以下の各コマンドでサービスの
再起動を実施してください。
※手順に誤りがあるとサービスの起動に失敗する可能性があるので
注意ください。

service-control --stop vmware-stsd && service-control --start vmware-stsd
service-control --stop lookupsvc && service-control --start lookupsvc

次に KB 85475 の Attachments からダウンロードした
[tomcat_compress.zip] ファイルを解凍し、[tomcat_compress]
フォルダ内の [tomcat_compress.cron] を /etc/cron.d ディレクトリ
配下にコピーします。

※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※
KB からダウンロードした tomcat_compress.cron ファイルの
改行コードは CRLF になっています。

改行コードが CRLF のまま、/etc/cron.d ディレクトリ配下に配置
すると tomcat_compress.cron が動作しないという事例も過去に
見たことがあるため、改行コードを LF に変換することをおすすめ
します。

改行コードを CRLF から LF に変えるには、/etc/cron.d ディレクトリ
配下に tomcat_compress.cron ファイルを配置後、以下コマンドを
実行することで可能です。
※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※※

# sed -i -e 's/\r$//' tomcat_compress.cron

以上で KB 85475 の Workaround の手順は完了です。

KB 85475 の Workaround 実施直後の変化

Wordaround 実施前の状態を冒頭に記載しましたが、以下に Workaround
実施後の状態を記録していきます。

・/var/log/vmware/sso/tomcat

root@test [ ~ ]# ls /var/log/vmware/sso/tomcat
catalina.2022-11-06.log   localhost_access.2022-10-27.log  localhost_access.2022-10-30.log  localhost_access.2022-11-02.log  localhost_access.2022-11-05.log
host-manager.log          localhost_access.2022-10-28.log  localhost_access.2022-10-31.log  localhost_access.2022-11-03.log  localhost_access.log
localhost.2022-11-06.log  localhost_access.2022-10-29.log  localhost_access.2022-11-01.log  localhost_access.2022-11-04.log  manager.log

・/var/log/vmware/eam/web

root@test [ ~ ]# ls /var/log/vmware/eam/web
catalina-2022-08-28.log  catalina-2022-10-10.log  catalina-2022-11-01.log          localhost_access.2022-09-27.log  localhost_access.2022-10-20.log
catalina-2022-09-07.log  catalina-2022-10-11.log  catalina-2022-11-02.log          localhost_access.2022-09-28.log  localhost_access.2022-10-21.log
catalina-2022-09-11.log  catalina-2022-10-12.log  catalina-2022-11-03.log          localhost_access.2022-09-29.log  localhost_access.2022-10-22.log
catalina-2022-09-12.log  catalina-2022-10-13.log  catalina-2022-11-04.log          localhost_access.2022-09-30.log  localhost_access.2022-10-23.log
catalina-2022-09-13.log  catalina-2022-10-14.log  catalina-2022-11-05.log          localhost_access.2022-10-01.log  localhost_access.2022-10-24.log
catalina-2022-09-14.log  catalina-2022-10-15.log  catalina.log                     localhost_access.2022-10-02.log  localhost_access.2022-10-25.log
catalina-2022-09-15.log  catalina-2022-10-16.log  localhost-2022-08-28.log         localhost_access.2022-10-03.log  localhost_access.2022-10-26.log
catalina-2022-09-16.log  catalina-2022-10-17.log  localhost-2022-09-07.log         localhost_access.2022-10-04.log  localhost_access.2022-10-27.log
catalina-2022-09-25.log  catalina-2022-10-18.log  localhost-2022-09-16.log         localhost_access.2022-10-05.log  localhost_access.2022-10-28.log
catalina-2022-09-26.log  catalina-2022-10-19.log  localhost-2022-09-25.log         localhost_access.2022-10-07.log  localhost_access.2022-10-29.log
catalina-2022-09-27.log  catalina-2022-10-20.log  localhost-2022-10-07.log         localhost_access.2022-10-08.log  localhost_access.2022-10-30.log
catalina-2022-09-28.log  catalina-2022-10-21.log  localhost_access.2022-09-07.log  localhost_access.2022-10-09.log  localhost_access.2022-10-31.log
catalina-2022-09-29.log  catalina-2022-10-22.log  localhost_access.2022-09-08.log  localhost_access.2022-10-10.log  localhost_access.2022-11-01.log
catalina-2022-09-30.log  catalina-2022-10-23.log  localhost_access.2022-09-09.log  localhost_access.2022-10-11.log  localhost_access.2022-11-02.log
catalina-2022-10-01.log  catalina-2022-10-24.log  localhost_access.2022-09-10.log  localhost_access.2022-10-12.log  localhost_access.2022-11-03.log
catalina-2022-10-02.log  catalina-2022-10-25.log  localhost_access.2022-09-11.log  localhost_access.2022-10-13.log  localhost_access.2022-11-04.log
catalina-2022-10-03.log  catalina-2022-10-26.log  localhost_access.2022-09-12.log  localhost_access.2022-10-14.log  localhost_access.2022-11-05.log
catalina-2022-10-04.log  catalina-2022-10-27.log  localhost_access.2022-09-13.log  localhost_access.2022-10-15.log  localhost_access.log
catalina-2022-10-05.log  catalina-2022-10-28.log  localhost_access.2022-09-14.log  localhost_access.2022-10-16.log  localhost.log
catalina-2022-10-07.log  catalina-2022-10-29.log  localhost_access.2022-09-15.log  localhost_access.2022-10-17.log
catalina-2022-10-08.log  catalina-2022-10-30.log  localhost_access.2022-09-25.log  localhost_access.2022-10-18.log
catalina-2022-10-09.log  catalina-2022-10-31.log  localhost_access.2022-09-26.log  localhost_access.2022-10-19.log

・/var/log/vmware/lookupsvc/tomcat

root@test [ ~ ]# ls /var/log/vmware/lookupsvc/tomcat
catalina.2022-11-06.log          localhost_access.2022-10-28.log  localhost_access.2022-11-01.log  localhost_access.2022-11-05.log
host-manager.2022-11-06.log      localhost_access.2022-10-29.log  localhost_access.2022-11-02.log  localhost_access.log
localhost.2022-11-06.log         localhost_access.2022-10-30.log  localhost_access.2022-11-03.log  manager.2022-11-06.log
localhost_access.2022-10-27.log  localhost_access.2022-10-31.log  localhost_access.2022-11-04.log

　・/var/log/vmware/lookupsvc

root@test [ ~ ]# ls /var/log/vmware/lookupsvc
logs                           lookupsvc_stream.log-1.stderr  lookupsvc_stream.log-4.stderr  vmware-lookupsvc-gc.log.0          vmware-lookupsvc-gc.log.5
lookupserver-default.log       lookupsvc_stream.log-1.stdout  lookupsvc_stream.log-4.stdout  vmware-lookupsvc-gc.log.0.current  vmware-lookupsvc-gc.log.6
lookupServer.log               lookupsvc_stream.log-2.stderr  lookupsvc_stream.log.stderr    vmware-lookupsvc-gc.log.1          vmware-lookupsvc-gc.log.7
lookupsvc-init.log             lookupsvc_stream.log-2.stdout  lookupsvc_stream.log.stdout    vmware-lookupsvc-gc.log.2          vmware-lookupsvc-gc.log.8
lookupsvc_stream.log-0.stderr  lookupsvc_stream.log-3.stderr  tomcat                         vmware-lookupsvc-gc.log.3.current  vmware-lookupsvc-gc.log.9
lookupsvc_stream.log-0.stdout  lookupsvc_stream.log-3.stdout  vmware-lookupservice-perf.log  vmware-lookupsvc-gc.log.4

結果は以上となります。

/var/log/vmware/eam/web の catalina.log は変化が見られません
でしたが、他のディレクトに関しては Workaround 実施直後に自動的
にファイルが削除されました。

なお、/var/log/vmware/eam/web の localhost_access*.log は別途
KB 85249 の手順を実施する必要があり、こちらに関しては以下で記事
にしています。

[vCSA 7.0] KB 85249 の Workaround 実施手順
vCenter Server 7.0 において localhost*.log で /storage/log が圧迫されて しまう事象に関する KB 85249 の Workaround 実施手順について記載しました。
ss-engineer.com
2022.12.11

vCSA 7.0 U3c 以上のバージョンへアップデートすることでも、
上記 Workaround を実施した場合と同様に各ディレクトリの
ファイルが削除されることを確認しています。

tomcat_compress.cron について

tomcat_compress.cron は以下のような動作をしています。

・ 1 日に一度古い localhost_*.log 及び catalina.* を圧縮します。

・ １ 週間に一度古い localhost_*.log 及び catalina.* を削除します。

上記が正常に動作しているか確認する場合は
/var/log/vmware/messages ログから確認することが可能です。

以下 Daily ログ

2022-11-05T01:11:01.865973+00:00 test CROND[46549]: (root) CMD ( find /var/log/vmware/ -mindepth 2 -mtime +7 -type f -name localhost_* |xargs gzip > /dev/null  2>&1)
2022-11-05T01:15:01.092450+00:00 test CROND[48314]: (root) CMD ( find /var/log/vmware/ -mindepth 2 -mtime +7 -type f -name catalina.* |xargs gzip > /dev/null  2>&1)

以下 Weekly ログ

2022-11-06T02:22:01.749514+00:00 test CROND[51032]: (root) CMD ( find /var/log/vmware/ -mindepth 2 -mtime +14 -type f -name localhost_*.gz -delete)
2022-11-06T02:25:01.464425+00:00 test CROND[52382]: (root) CMD ( find /var/log/vmware/ -mindepth 2 -mtime +14 -type f -name catalina.*.gz -delete)

まとめ

/storage/log の容量が枯渇していた場合、vCSA の動作も不定となり
ますので、定期的に確認することをおすすめします。

なお、今回記事にした KB 85475 の Workaround に関しては比較的
時間もかからずに実施できるかと思いますので、vCSA 7.0 U3c への
アップデートが即座に厳しい場合には、実施いただくことを推奨します。

また、/storage/log 配下の各種ファイルを手動で削除したい場合には、
以下のような KB も公開されていますので、参考にしてください。

How to clear space on VCSA /storage/log partition (83070)
https://kb.vmware.com/s/article/83070?lang=en_us

参考情報

[vCSA 7.0] catalina*.log , localhost_access_log* により、/storage/log の容量が枯渇する
vCSA 7.0 を使用している環境において catalina*.log , localhost_access_log* により /storage/log の容量が圧迫されてしまう事象について解説します。
ss-engineer.com
2022.10.10

[vCSA 7.0] KB 85249 の Workaround 実施手順
vCenter Server 7.0 において localhost*.log で /storage/log が圧迫されて しまう事象に関する KB 85249 の Workaround 実施手順について記載しました。
ss-engineer.com
2022.12.11
タイトルとURLをコピーしました