Fix the failure of the test_cleanse.py - #6462
Conversation
|
Thanks, @pwalczysko. No immediate objections from my side. |
| assert not os.path.isfile(orig_file_path_and_name) | ||
| assert not os.path.exists(logfile_path) | ||
| # The log file itself must be removed. | ||
| # On NFS, if another process still |
There was a problem hiding this comment.
Do you have any idea on what keeps a reference to the file? In a standard import workflow, I would expect all file handles to be released on completion. Is this another example of latency due to the migration of the infrastructure (i.e. would a delay also allow the test to pass.
There was a problem hiding this comment.
Do you have any idea on what keeps a reference to the file?
Blitz, I know that. More precisely dont know, did not go deeper.
(i.e. would a delay also allow the test to pass.
Delay did not work. I did not try polling though, just a plain wait. It was longer than 30 sec afai can remember and the process still did not release the file.
There was a problem hiding this comment.
Do you have any idea on what keeps a reference to the file?
new investigation. Definitely a java Blitz process. It is not actively doing something. The leak is not worsening over time. Sometimes the .nfs file gets deleted after a minutes of wait. Sometimes not.
The cmds below shows the file descriptors currently open by process 1114099.
ls -l /proc/1114099/fd | grep .nfs0000008384567c850000160d
l-wx------. 1 omero omero 64 Aug 7 19:22 244 -> /home/omero/workspace/OMERO-test-integration/data/ManagedRepository/c1d4022e-2bcb-47af-a99f-7bb37c1826cf_4813/2026-08/07/.nfs0000008384567c850000160d
...
ps -fp 1114099
UID PID PPID C STIME TTY TIME CMD
omero 1114099 1114059 20 14:03 ? 01:11:19 java -Xmx9600m --add-opens java.base/java.lang=ALL-UNNAMED --add-exports java.naming/com.sun.jndi.ldap=ALL-UNNAMED -Djava.awt.headless=tru
(.venv3) bash-5.1$
jcmd 1114099 VM.command_line | grep blitz. # has a very long output below, which shows it is blitz
java_command: ome.services.blitz.Entry --Ice.Config=/home/omero/workspace/OMERO-test-integration/src/dist/var/master/servers/Blitz-0/config/config
java_class_path (initial): lib/server/proj4j.jar:lib/server/calcite-linq4j.jar:lib/server/minio.jar:lib/server/commons-lang3.jar:lib/server/ini4j.jar:lib/server/logback-core.jar:lib/server/spring-expression.jar:lib/server/httpcl.....
ls /proc/1114099/fd | wc -l
272
(.venv3) bash-5.1$ lsof -p 1114099 | grep -c nfs. # at the time of investigation, Blitz had 19 open files (not supposed to do anything though
19
lsof -p 1114099 | grep nfs | awk '{print $9}' | sort | uniq -c
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452795f000015d0
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527962000015ca
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452796f000015d9
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527975000015c8
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527977000015ce
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527993000015d1
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452799b000015cc
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452799d000015d3
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279a0000015d7
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279a2000015d6
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c0000015d2
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c2000015cd
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c5000015c9
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c6000015cf
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c7000015d4
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c8000015cb
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279ca000015d8
1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279cb000015d5
1 /home/omero/workspace/OMERO-test-integration/data/ManagedRepository/c1d4022e-2bcb-47af-a99f-7bb37c1826cf_4813/2026-08/07/.nfs0000008384567c850000160d
I am afraid I am not getting forward. I am told that we need to enable the enable Java file descriptor tracing. I should restart the java process with -Djava.security.debug=access but we are talking here about Blitz in OMERO server... no idea how to do that. Will leave it there, this is, I can continue after AL if please told how to configure the server with such option.
|
As discussed with @sbesson , the jury is out on this one. It stably passes with the fix in this PR, but not optimal and possibly underresearched. Will leave open for now. |
What this PR does
The failing test is https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/11/testReport/junit/OmeroPy.test.integration.clitest.test_cleanse/TestCleanseFullAdmin/testCleanseNonsenseName/
This PR is removing a line in the
test_cleanse.py.The assertion in that line which checks the existence of a path to a logfile which should have been deleted is not possible on NFS, because:
As the Blitz process is still holding the logfile open, on NFS this file cannot be deleted by NFS although the
cleansescript deletes it - it leaves a hidden file behind which the NFS renames to...nfs.Testing this PR
Check that the Jenkins build testintegration job is green.
Related reading
See #6461
cc @joshmoore @jburel @sbesson