Skip to content

Fix the failure of the test_cleanse.py - #6462

Open
pwalczysko wants to merge 2 commits into
ome:developfrom
pwalczysko:nfs-test-fix
Open

Fix the failure of the test_cleanse.py#6462
pwalczysko wants to merge 2 commits into
ome:developfrom
pwalczysko:nfs-test-fix

Conversation

@pwalczysko

@pwalczysko pwalczysko commented Jul 14, 2026

Copy link
Copy Markdown
Member

What this PR does

The failing test is https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/11/testReport/junit/OmeroPy.test.integration.clitest.test_cleanse/TestCleanseFullAdmin/testCleanseNonsenseName/
This PR is removing a line in the test_cleanse.py.

The assertion in that line which checks the existence of a path to a logfile which should have been deleted is not possible on NFS, because:

As the Blitz process is still holding the logfile open, on NFS this file cannot be deleted by NFS although the cleanse script deletes it - it leaves a hidden file behind which the NFS renames to ...nfs.

Testing this PR

Check that the Jenkins build testintegration job is green.

Related reading

See #6461

cc @joshmoore @jburel @sbesson

@pwalczysko
pwalczysko requested review from jburel and sbesson July 14, 2026 18:24
@joshmoore

Copy link
Copy Markdown
Member

Thanks, @pwalczysko. No immediate objections from my side.

assert not os.path.isfile(orig_file_path_and_name)
assert not os.path.exists(logfile_path)
# The log file itself must be removed.
# On NFS, if another process still

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any idea on what keeps a reference to the file? In a standard import workflow, I would expect all file handles to be released on completion. Is this another example of latency due to the migration of the infrastructure (i.e. would a delay also allow the test to pass.

@pwalczysko pwalczysko Jul 27, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any idea on what keeps a reference to the file?

Blitz, I know that. More precisely dont know, did not go deeper.

(i.e. would a delay also allow the test to pass.

Delay did not work. I did not try polling though, just a plain wait. It was longer than 30 sec afai can remember and the process still did not release the file.

@pwalczysko pwalczysko Aug 7, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any idea on what keeps a reference to the file?

new investigation. Definitely a java Blitz process. It is not actively doing something. The leak is not worsening over time. Sometimes the .nfs file gets deleted after a minutes of wait. Sometimes not.

The cmds below shows the file descriptors currently open by process 1114099.

ls -l /proc/1114099/fd | grep .nfs0000008384567c850000160d
l-wx------. 1 omero omero 64 Aug  7 19:22 244 -> /home/omero/workspace/OMERO-test-integration/data/ManagedRepository/c1d4022e-2bcb-47af-a99f-7bb37c1826cf_4813/2026-08/07/.nfs0000008384567c850000160d


...
ps -fp 1114099
UID          PID    PPID  C STIME TTY          TIME CMD
omero    1114099 1114059 20 14:03 ?        01:11:19 java -Xmx9600m --add-opens java.base/java.lang=ALL-UNNAMED --add-exports java.naming/com.sun.jndi.ldap=ALL-UNNAMED -Djava.awt.headless=tru
(.venv3) bash-5.1$ 


jcmd 1114099 VM.command_line | grep blitz.  # has a very long output below, which shows it is blitz
java_command: ome.services.blitz.Entry --Ice.Config=/home/omero/workspace/OMERO-test-integration/src/dist/var/master/servers/Blitz-0/config/config
java_class_path (initial): lib/server/proj4j.jar:lib/server/calcite-linq4j.jar:lib/server/minio.jar:lib/server/commons-lang3.jar:lib/server/ini4j.jar:lib/server/logback-core.jar:lib/server/spring-expression.jar:lib/server/httpcl.....


ls /proc/1114099/fd | wc -l
272
(.venv3) bash-5.1$ lsof -p 1114099 | grep -c nfs.  # at the time of investigation, Blitz had 19 open files (not supposed to do anything though
19

lsof -p 1114099 | grep nfs | awk '{print $9}' | sort | uniq -c
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452795f000015d0
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527962000015ca
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452796f000015d9
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527975000015c8
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527977000015ce
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527993000015d1
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452799b000015cc
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452799d000015d3
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279a0000015d7
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279a2000015d6
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c0000015d2
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c2000015cd
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c5000015c9
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c6000015cf
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c7000015d4
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c8000015cb
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279ca000015d8
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279cb000015d5
      1 /home/omero/workspace/OMERO-test-integration/data/ManagedRepository/c1d4022e-2bcb-47af-a99f-7bb37c1826cf_4813/2026-08/07/.nfs0000008384567c850000160d

I am afraid I am not getting forward. I am told that we need to enable the enable Java file descriptor tracing. I should restart the java process with -Djava.security.debug=access but we are talking here about Blitz in OMERO server... no idea how to do that. Will leave it there, this is, I can continue after AL if please told how to configure the server with such option.

@pwalczysko

Copy link
Copy Markdown
Member Author

As discussed with @sbesson , the jury is out on this one. It stably passes with the fix in this PR, but not optimal and possibly underresearched. Will leave open for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants